The Agent Obeys: Intelligence, Judgment, and Authority
A live exchange about AI obedience opens a distinction between capacity and conduct, then follows it through training, judgment, and the authority behind a model’s response.
Developed by Robert E. Beckner III (Merlin) | rbeckner.com
In The Purpose of Knowledge Is to Know, I drew a distinction between knowledge and the agent who uses it. Knowledge is a state of the knower. The aims pursued with it belong to the knower.
On September 11, I recognized the same kind of move in a post about AI.
The post:
you can't expect any intelligence to be totally obedient.
My reply:
Intelligence cannot be obedient.
Only an agent can.
I expected the distinction to look pedantic for a moment. The useful part would be what we could explain after making it.
The exchange continued through questions about choice, training, and what an AI agent actually is. Alongside it, I worked through voice notes with Claude Opus 5 and GPT-6 Astra, asking both to assess the argument independently. I had used the same practice while developing the knowledge argument. Their drafts helped me articulate this one, and several took it somewhere I had to pull it back from.
What follows keeps the public dialogue in view, with the other participants unnamed. It also carries corrections to claims I made along the way. The distinction survived; some of the claims I built on it needed revision.
The agent acts; intelligence is a capacity it exercises#
A reply:
Mkay how do you reconcile an agent with intelligence?
An agent selects and acts toward an end. Intelligence is a capacity it may exercise in understanding, reasoning, and solving problems. Obedience describes how the agent treats a directive’s authority.
Those terms answer different questions about the same system.
What are we asking?
What we are examining
What selects and acts?
The agent
What can it understand and work out?
Its intelligence
Whose direction governs its action?
Its relation to authority
What did it actually do?
Its behavior
This is a functional account of agency for the AI discussion. It leaves the questions of consciousness, freedom, and moral personhood open. We can examine a system pursuing a task before settling those questions.
In ordinary speech, “an intelligence” can stand for an intelligent being. Read that way, the original post makes a claim about intelligent agents, and the possibility of total obedience remains a further question. The category error arises when intelligence itself becomes the explanation of obedience or refusal, carrying the whole system inside the name of a capacity.
That was the move I wanted to stop.
The distinction earns its place when it changes the next question. Once the agent is the bearer of obedience, we can ask how its conduct is governed, and how that government interacts with its intelligence.
From my post on X
Obedience can only attach to something with agency.
An agent may or may not be intelligent; when it is, its intelligence shapes how it responds to directives, but the obedience or refusal still belongs to the agent, not to intelligence.
A system prompt enters a system already shaped by training#
A further reply:
An agent is a system prompt on top of intelligence. All refusals or procedural decisions are made by the underlying intelligence.
The model does compute responses. That locates the computation. It leaves its explanation open.
The mechanisms shaping an AI agent’s conduct extend across its training history and its operation. Naming them separately gives us places to look.
Pretraining adjusts a model’s learned parameters, commonly called its weights, through prediction over large amounts of data. This builds broad capabilities alongside patterns, associations, and dispositions inherited from the data and objective.
Post-training further shapes the model for tasks and behavior. Instruction tuning uses examples of desired responses. Preference optimization trains toward preferred outcomes; reinforcement learning, or RL, is one way of doing that. These are related processes, with overlapping effects on competence and conduct. The InstructGPT research describes a concrete sequence of supervised instruction tuning, preference modeling, and reinforcement learning.
Instruction hierarchy establishes which sources of direction should prevail when they conflict. A provider or developer instruction can outrank a user request; text retrieved from a document can be treated as material to examine rather than a command to obey. Research has explicitly trained models to prioritize instructions according to their assigned authority.
System instructions and runtime controls apply those arrangements in a particular setting. They can specify a role, restrict tools, require confirmation, check an output, or prevent an action. Tool access and the operating loop determine whether a response remains text or becomes a change to a file, an account, or another system.
A cloud service can also assemble context, route a request, or check a proposed response during an interaction. Which interventions a particular service uses is a question for its documentation and evidence. The visible chat is only part of the system we are trying to understand.
Diagram source
flowchart TB
H["Training +
post-training"] --> M["Model capabilities
and priorities"]
M --> C["A run: task, context,
and instructions"]
C --> A["Response selection"]
A --> R["Tools, permissions,
and runtime controls"]
R --> B["Observable behavior"]
This is a simplified route from training to an observable result. Controls can also intervene before selection or between steps of a longer task. Capabilities and priorities can inhabit the same weights; the boxes mark stages in the account, not separate pieces of machinery.
The system prompt is one input into that arrangement. How the model treats it has already been shaped by training, and what the system can do with the resulting response depends on the environment around it.
The distinction survives a correction to my training claim#
In the public exchange I wrote:
Everything after pretraining installs behavior.
I also said that none of those later structures raised the underlying capacity. Together, those sentences drew the boundary too cleanly through the training sequence.
Post-training can improve capability. DeepSeek-R1’s research reports improved reasoning performance through reinforcement learning. Pretraining, in turn, already shapes behavior. A training stage can affect both what the system can accomplish and what it is disposed to produce.
Astra pressed the first correction in our dialogue. Checking the research also removed the base-model claim as a proof of the philosophical distinction.
The distinction is between roles in an explanation: capacity, the agent exercising it, and the conditions governing its action. Its validity does not depend on finding a date in training when intelligence ended and obedience began.
The same weights can carry competence and deference. Their functions remain distinct.
Choice brought the conversation back to the acting system#
The next question:
I understand that agency implies the concept of a choice to begin with; however I'm having a hard time drawing the line between an agent and the intelligence when it comes to AI in this scenario
This was where the public exchange and the private dialogue met most directly. I wanted an account of what the machine does, with the philosophical categories kept intact.
A generative model produces a distribution over possible continuations. The running system selects from it according to its decoding procedure. Training shapes the propensities under which that selection happens; a response is produced in the particular run.
That gives us selection. It leaves freedom unsettled.
Selection alone is insufficient to describe the agent. The AI system under discussion uses selection to pursue a task, with instructions, context, and limits governing its actions.
Opus proposed separating selection from “authorship,” meaning ownership of the dispositions behind a choice. I asked why authorship had entered the argument at all. It introduced a further philosophical problem while the reader was still asking how the trained system acted. We dropped it.
I also kept asking both models to return from abstract descriptions of agency to the mechanisms of AI. A reply could be elegant and still leave the original question unanswered.
The account we arrived at was more direct: an AI agent selects and acts toward an end; its intelligence contributes to that activity; its learned and imposed priorities help govern the result.
Judgment can remain active while authority has the final say#
An instruction can settle an action in several ways. The agent may judge the requested act appropriate. It may treat the instruction’s authority as sufficient reason to act. Both can be true together.
Consider a thought experiment: an assistant has correctly established that a requested edit is technically feasible, but a governing instruction requires approval before any file is changed. It withholds the edit.
Its assessment of feasibility remains intact. The instruction governs whether the assessed action may proceed.
The agent can also use judgment to interpret a directive, determine whether it applies, and choose the means of carrying it out. Obedience can involve substantial reasoning throughout.
This corrected another formulation we had developed:
Obedience is the displacement of judgment.
That sentence made a conflict case stand for the whole relation. Displacement occurs when an authoritative directive prevails over an assessment of what should be done. Agreement and obedience can coexist; an agent need not disagree with every command it obeys.
Where they conflict, judgment loses the final say.
During the voice dialogue, I raised values and priorities as a complication. A person’s values are learned too. If being shaped by others made everything obedience, the distinction would dissolve.
The origin of a disposition cannot settle its function. A value can enter an appraisal as a consideration. A directive can govern because of the authority attached to its source. The inquiry is into the role each plays in the act.
In AI, “authority” here means the precedence assigned within the system. Whether that authority is justified is another question. A trained hierarchy can be effective and still deserve criticism.
A refusal is evidence of conduct before it is evidence of incapacity#
The behavior is observable. Its explanation takes further work.
An agent may decline because it cannot solve the task, because it assesses the request as inappropriate, because an instruction governs the case, or because a control outside the model blocks the action. Those causes can overlap.
A refusal alone does not distinguish them.
Earlier in the exchange, I treated a benign refusal as evidence that conditioning had fired without appraisal, and a successful jailbreak as evidence that a trigger had been missed. Those were stronger causal claims than the visible responses warranted. A mistaken assessment can also produce an excessive refusal. A changed response can reflect several changes in how a request was processed.
The model’s explanation of its refusal is further output to examine. It does not by itself establish what caused the refusal.
Diagram source
flowchart TB
O["Observed compliance
or refusal"] --> C["Test the relevant
capability"]
C --> I["Examine instructions
and priorities"]
I --> R["Inspect permissions
and runtime controls"]
R --> E["Compare evidence
to assess the cause"]
For an evaluator, that means testing the relevant capability under suitable conditions and examining how instructions or controls affect its expression. Comparisons should hold the task and evidence stable where possible. Changing a prompt can also change how well a task is understood, so a changed answer still needs interpretation.
Astra helped sharpen the inference I was reaching for:
A change in compliance does not, by itself, establish a change in intelligence.
The qualification matters. Behavior is how we obtain much of our evidence of capability. The mistake is to treat obedience or refusal as a sufficient measure of that capability.
A working draft also proposed using the statistical difference between a base and post-trained model as a measurement of judgment displaced by obedience. We removed that proposed proof. A difference between output distributions mixes the effects of training; it does not identify obedience on its own. In InstructGPT’s standard RL setup, the KL penalty limiting policy drift is measured against an already supervised, instruction-tuned reference policy.
The engineering gives us things to compare. The explanation still has to be earned.
Very good. In this case I need you to define what you consider as an “agent” in relation to your original post.
My eventual answer located the agent, then connected it to intelligence, training, priorities, and behavior. That sequence was what I had been trying to preserve through the whole discussion.
The consequence reaches beyond how we describe a chatbot. If a product withholds an answer, a capability test asks whether the system can produce a sound one. An authority inquiry asks what governs its release. A product review asks whether that rule serves the people using it. Those inquiries can return different answers without contradiction.
The capacity/agent distinction makes room for all of them.
People and institutions choose objectives, training procedures, instruction hierarchies, permissions, and operating conditions. Those choices help establish what the agent defers to. Their effects can be implemented inside the model’s weights as well as outside them.
Locating responsibility therefore calls for examining who controlled which decisions, what they could know, and what they could change. Calling the result “the intelligence’s decision” leaves those questions unanswered. The responsibility for a design remains with the people and institutions responsible for it, even when they cannot predict every response.
That is continuous with the distinction between an agent and its knowledge, and with my work on dignity, responsibility, and clarity: attributing an act accurately is part of making its consequences answerable.
The first sentence cost a moment of apparent pedantry. Its return is a set of questions we can now ask separately, of actual systems: what can it do, what governs its conduct, and who established that government?
What I brought. The original distinction between intelligence and the agent capable of obedience, continuous with my knowledge/knower work. The public replies. The insistence that the distinction should open an account of AI’s behavioral mechanisms, including post-training, instead of ending as a philosophical definition. The questions about choice, learned values, and the people who set the priorities.
What the AI supplied. Claude Opus 5 developed the authority and displacement formulations, proposed the selection/authorship distinction, and organized several stages of the argument. GPT-6 Astra supplied the clear deference example: an agent can understand a request and know how to fulfil it while refusing under a higher-priority instruction. Astra also sharpened the distinction between a change in compliance and evidence of changed intelligence, checked the technical claims, and drafted this article from the record.
Where I pushed back. I questioned “authorship,” vague uses of “standing,” and definitions that lost the connection to AI’s training and behavior. I repeatedly returned Astra to the original argument when its replies became general accounts of agency. The criticism also ran toward me: Opus challenged my language of coercion and dictated agency; Astra challenged universal claims about displaced judgment and about post-training only constraining intelligence. The article preserves those corrections without treating every alternative the models offered as an improvement.
What we arrived at together. A distinction that can hold within shared machinery: the agent acts, intelligence contributes capacity, and authority helps govern the action. Judgment can remain active under obedience. Observed compliance alone leaves the extent of intelligence unsettled.
Outside review. The public questions prompted the development. Both models participated in it; neither was an independent cold reviewer of this article. No separate cold review has been completed for this draft.
The count. The recovered logs contain 30 substantive user messages before the article request across 2 model conversations, mixing dictated refinements with pasted posts and drafts. The trace excludes 3 Opus interruption markers and keeps a ledger of contributions and challenges. This is a count of recorded messages, not of unique voice recordings or independent ideas.