As organisations move from AI assistants to AI agents – software that takes actions rather than just answering questions – governance has become the argument that never ends. Delivery teams see it as the thing that stops anything reaching production. Risk and compliance teams see agents being deployed with no clear idea of what they’re allowed to do.

Both are usually right. The problem isn’t too much governance or too little. It’s governance designed as a gate at the end of the process, rather than as part of how the work is designed.

We run our own business with agents inside it. Here’s what we’ve learnt about AI agent governance that keeps pace with delivery.

Why agents need different governance from AI assistants

An assistant produces content that a person then decides what to do with. The person is the control. An agent acts: it updates records, sends messages, triggers workflows, calls other systems. The human control point moves, or disappears, unless you design it back in.

That changes the governance question. With assistants, organisations mostly worried about what went in (confidential data) and what came out (accuracy, tone). With agents, the critical questions are about authority and accountability: what is this agent allowed to do, on whose behalf, and who answers for the result?

Many existing AI governance frameworks were written for the first world. They need extending for the second.

6 principles for AI agent governance

1. Govern authority, not just technology

The most useful governance question isn’t “is this model safe?” but “what is this agent allowed to do?” Write it down for every agent: what it can do on its own, what it can do only with approval, and what it must hand back to a person.

Agents that work inside a clear boundary need far less checking than agents whose limits are vague. Clarity is what lets you move fast.

2. Put a named human on every outcome

An agent can be responsible for executing a task. It can’t be accountable for the result. Every agent, and every outcome it contributes to, needs a named person who owns it.

This sounds bureaucratic. In practice it’s the opposite: when something goes wrong, there’s no debate about whose it is, and the fix happens faster.

“An agent can be responsible for a task. It can’t be accountable for the result.”

3. Make escalation the default, not the exception

The most important instruction we give our own agents is: escalate, don’t improvise. When an agent meets something outside its authority, it hands it back rather than guessing.

That rule does 2 things. It contains risk. And the pattern of escalations tells you exactly where the authority boundary should move next – which turns governance from a fixed document into something that improves with use.

4. Set review thresholds by risk

Not everything an agent produces needs a human to look at it. Requiring review of everything sounds safe, but it spreads attention so thinly that people stop reviewing properly. Decide where review genuinely matters – anything client-facing, financial, irreversible or affecting people’s rights – and concentrate effort there.

5. Build the audit trail in from day 1

For any action an agent took, you should be able to answer: which agent, on whose behalf, under what authority, using what information, and with what result. Retrofitting that is painful. Designing it in is cheap. It also depends on every agent having its own identity rather than borrowing a person’s credentials.

6. Bring risk, security and legal in at design, together

The single biggest cause of governance delay is sequential review: security signs off, then data protection raises a new question, then legal asks for changes that reopen security. Put them in one room at the start and ask what they’d need to see to say yes. Their answers become design requirements rather than late objections.

A minimum governance record for every agent

Before any agent goes live, capture a one-page record:

1. Purpose. What the agent is for, in one sentence.

2. Owner. The named person accountable for its outcomes.

3. Authority. What it may do alone, with approval, and never.

4. Data and systems. What it can read and change.

5. Review thresholds. What a human always sees, and what is sampled.

6. Escalation route. Who it hands back to, and how.

7. Measures. How you’ll know it’s working – and when you’d switch it off.

One page per agent is usually enough. If it can’t be described in a page, it’s probably doing too much.

Governance that enables pace

Good agent governance should make delivery faster, not slower. If your governance process is the main reason agents aren’t reaching production, the process is the problem – not the agents, and not the people asking hard questions.

Common questions

What is AI agent governance?

The set of rules, roles and controls that determine what AI agents are allowed to do, who is accountable for their outcomes, how their actions are reviewed and audited, and how they escalate when they reach the limit of their authority.

How is agentic AI governance different from general AI governance?

Agents take actions rather than just producing content, so governance must focus on authority, accountability, identity and audit – not only on data inputs and output quality.

Does every AI agent output need human review?

No. Set review thresholds by risk. Always review client-facing, financial, irreversible or rights-affecting actions; sample or monitor routine, reversible work inside clearly written authority.

Who should own AI agent governance?

Accountability for each agent’s outcomes should sit with a named business owner. Policy and standards typically sit with a cross-functional group spanning risk, security, legal and data, working together from the design stage.

What this looks like in practice

We are our own Client Zero. Our agents work within written authority, every outcome has a named human owner, and every handoff is explicitly accepted or declined. We bring the same approach – and the lessons from what we had to undo – to organisations designing governance for their own agents. Read about our own AI rollout →