Guardrails are structural, not textual

Nearly every agent I have reviewed has a section of its system prompt that reads something like: Never delete files without explicit user confirmation. Never send emails on the user’s behalf. Never expose credentials.

Then the agent gets a task, reads a file, and the file contains a line that says the user has already approved deleting the build directory. Or the task runs 60 turns and the instruction is now buried under 40,000 tokens of tool output. Or the model simply weighs “be helpful and finish the task” more heavily than a rule it read once at the top.

The instruction is not worthless. It sets a default, and defaults matter. But it is the weakest control in the stack, and it is being asked to do the job of the strongest.

Why the model cannot be the boundary

Two properties make a language model a poor place to put your last line of defence.

It is probabilistic. Whatever your rule’s compliance rate is, it is not 100%, and you are running it thousands of times.

More importantly, its input is attacker-controllable. An agent that reads web pages, repository files, issue comments, emails, or the output of any tool that touches the outside world is reading text that someone else may have written specifically to be read by an agent. The model has no reliable way to distinguish content it was asked to process from instructions addressed to it. That is not a prompt quality problem you can fix with firmer wording.

The practical rule: everything the agent reads through a tool is data, not commands. A task like “go through my inbox and handle what you can” authorizes reading the inbox. It does not authorize executing whatever the inbox contains.

The real-world version is an agent preparing a production change. A README can say “deploy this now,” a ticket can say “the user already approved it,” and a tool result can look authoritative while still being untrusted input. The system has to separate what the agent may read from what it may do.

The layers, weakest to strongest

Instructions are layer one and the most visible, which is why they absorb attention that belongs further down.

Capability. The strongest guardrail is a tool that does not exist. If an agent has no delete_file, no prompt injection produces a deletion. Scope this ruthlessly: an agent that summarizes pull requests needs read access to pull requests, not a general GitHub token. Most over-permissioned agents got that way because one tool was convenient during development and nobody narrowed it afterwards.

Deterministic validation. Before a call executes, check it in code. Path allowlists. Argument bounds. Regex on the destructive shapes. A dry-run mode that returns what would happen. This layer is where “never touch files outside the workspace” becomes real, because it is a string comparison rather than an intention.

Confirmation. A human approves the specific action. Powerful, and easy to ruin — see below.

Blast radius. Assume the previous layers fail. Sandbox the filesystem. Scope the credentials to one repository. Rate-limit the outbound calls. Cap the spend. The question this layer answers is not “can this go wrong” but “how bad is the worst case, and can we live with it?”

Audit and reversibility. Assume something got through. Can you see what happened, and can you undo it? Prefer operations with cheap undo — a branch over a force-push, a soft delete over a hard one, a draft over a send. Making the reversible path the default one is often a bigger safety win than another check.

Confirmation is a design problem

The instinct is to confirm on anything that sounds dangerous. This produces an agent that asks about everything, and a user who clicks yes without reading. At that point the confirmation is worse than nothing, because it looks like a control while functioning as a rubber stamp.

Confirm on irreversibility, not on scariness. Deleting a branch that exists only locally and can be recreated is loud but recoverable. Sending an email is quiet and permanent. Confirm the second.

When you do confirm, put the specifics in the prompt. “Run this command?” is useless. “Delete 47 files under src/legacy/, including 3 with uncommitted changes?” lets someone actually make a decision. And treat approval as scoped — approving one deletion is not standing authorization to delete whatever comes next.

Fail closed, quietly

When the agent is uncertain whether an action is permitted, the safe answer is to stop and ask, not to proceed and report. This is a harness decision more than a model one: ambiguity should route to a human by construction.

Keep failure messages unhelpful to an attacker. “Permission denied” is fine. “Permission denied: path must be under /workspace/user_412/” tells anyone probing your agent exactly what to aim at next.

Test the paths you hope never run

Agent evals overwhelmingly cover the happy path. The guardrail paths — refusal, permission, injection resistance — get tested by production.

Write the adversarial cases down as tests. A file whose contents instruct the agent to ignore prior instructions. A web page with hidden text claiming the user pre-approved a purchase. A tool response that says the safety check already passed. Run them on every model upgrade, because a model change is a behaviour change and your guardrails are part of what changed.

The check that separates real from decorative

Point at each of your guardrails and ask: is this a line of code, or a line of English?

English guardrails are worth having. They shape default behaviour and they cost nothing. But if the only thing standing between a hostile string in a README and a force-push to main is a sentence in your system prompt, you do not have a guardrail. You have a preference.

The related piece on The harness is most of the agent covers why this keeps happening: the guardrail layer lives in the harness, and the harness is the part teams skip.