AI Agent Governance: A 90-Day Playbook for CISOs

AI agent governance is the set of rules, checks and ownership that keeps autonomous agents safe and auditable. It is not a document. It is a working control loop. Two things get you most of the way there. Name an owner for every agent. Add runtime checks that catch bad actions before they land. Get those right and you have a fleet you can explain to an auditor.

What is AI agent governance, and how is it different from AI governance?

Governing a model and governing an agent are different jobs.

A model answers a question and stops. An agent does not stop. It holds state, makes a plan, and acts across many steps. It might call five tools, touch three systems, and make a decision nobody sees until it is done.

That is the whole problem in one line. Agents are path dependent. What they do at step four depends on what happened at step one. A prompt filter cannot catch that. Neither can an access list someone set up six months ago.

Think of it as the difference between checking an ID at the door and watching what the person does once they are inside. Old-style AI governance checks the ID. Agent governance watches the whole visit.

That is why the old controls fall short:

• Prompt-only filters catch bad instructions. They do not catch bad action sequences.
• Static identity and access management (IAM, the system that decides who can access what) assumes a fixed set of permissions. Agents need permissions that shift with context.
• Design-time reviews check the system before it launches. They miss what only shows up once the agent is live and chaining actions together.

Runtime enforcement closes that gap. It checks each step as it happens, not just the plan on paper. Microsoft's cloud governance guidance is blunt about it. Do not build a second governance model for agents. Extend what you already run for cloud and identity, then add the runtime layer on top.

Why does agent governance matter now for security and risk teams?

Because agents are spreading faster than most security teams can track them.

Teams spin up agents to handle support tickets, draft contracts, or move data between systems. Nobody logs it centrally. That is shadow deployment, and it creates blind spots exactly where you need sight. Njin's write-up on shadow AI inside a business covers how quickly this spreads once one team proves it works.

The harms are not hypothetical. An agent with broad tool access can leak customer data. One with delegated credentials can act without approval. One left unwatched for months can drift from its original policy and nobody notices.

Statistic callout: OWASP's agentic application risk guidance identifies unauthorised tool use and excessive agency as some of the top risks for agentic systems, comparable to serious attack categories security teams commonly manage.

Regulators are watching. NIST's AI Risk Management Framework and ISO 42001 both push firms toward auditable, risk-mapped AI processes. If your fleet cannot produce a clean audit trail, an auditor will find that gap before you do.

The cost of doing nothing builds quietly. Every unregistered agent and every unwatched tool call adds to a risk balance you pay down later, usually at the worst moment.

What are the main risks and attack surfaces from AI agents?

Agents add features. They also add new ways for things to go wrong.

Five risks matter most:

1. Loss of execution control. When agents call other agents, a small error at step one can become a large one by step ten.
2. Unauthorised tool calls. An agent with access to a payment API or a database can act well outside its job if nothing checks each call.
3. Data exfiltration and PII handling. Agents that summarise, forward or store customer data can leak personal information without a line of malicious code.
4. Privilege escalation via delegated identities. An agent inheriting a service account's permissions ends up with far more access than its task needs.
5. Accountability diffusion and policy drift. When five teams share one agent and nobody owns it, policy rots quietly. Nobody notices until something breaks.

Picture an agent built to answer customer emails. It gets access to a CRM to look up order history. Six months later somebody bolts on a refund tool to save time. Nobody updates the risk review. That agent can now move money, and its original approval never covered that.

Pro Tip: Treat every new tool connection like a new hire. Ask what it can touch, why, and who is watching it. If you would not hand a new employee that access without a manager signing off, do not hand it to an agent either.

Our piece on AI agent security breaks down how these risks stack in practice.

The five-part governance framework every program needs

Good governance is not one big control. It is five smaller ones working together, like a kitchen with separate stations for prep, cooking, plating and cleanup. Miss a station and the whole service slows down.

1. Identity and authorisation Every agent gets its own identity. Never a shared login. Keep a central agent registry: the one list of every agent, what it does, and who owns it. Microsoft treats this as the starting point for everything else.

2. Guardrails and constraints Set hard limits on what each agent can do. Use allow lists rather than deny lists. You cannot predict every bad action in advance. You can define the good ones.

3. Tool governance Vet every third-party tool before an agent touches it. Limit integrations to what the task needs. Our guide on connecting agents across business operations covers how to map those connections without creating a mess.

4. Runtime enforcement This is the layer that checks actions as they happen. A policy engine sits between the agent's decision and its execution, and catches problems before they land.

5. Human oversight Set approval checkpoints for high-risk actions. Someone with real authority signs off before the agent moves money, deletes data, or contacts a customer.

These five map onto identity tools, policy platforms and monitoring systems you probably already own. The work is connecting them, not replacing them.

How do you actually implement AI agent governance?

Here is the rollout order that works. Six steps.

1. Set your risk appetite and name a sponsor. Decide in writing what level of agent risk the business will accept. Give one senior person the job of owning that call.
2. Find and register every agent you have. Most firms run more agents than anyone realises. Build a registry with the agent name, owner, purpose, tools it can reach, and data it touches.
3. Classify agents by impact and write policy templates. A customer-facing agent that can issue refunds needs tighter rules than one drafting meeting notes. Match the policy weight to the blast radius.
4. Map identity and enforce least privilege. Every agent gets its own identity and only the permissions its task needs. Nothing extra just in case.
5. Add a runtime policy engine. This layer intercepts a proposed action before it fires, checks it against policy, then lets it through, blocks it, or flags it for a human.
6. Test with canaries and red teams, then repeat. Run controlled tests on new agents before full rollout. Set service-level objectives for how often violations should occur, then measure against them.

Pro Tip: If you have zero tooling right now, start with a spreadsheet. A basic registry beats no registry, and Microsoft's own field guidance backs manual tracking as a real first step. The goal is not a perfect system on day one. It is knowing which agent did what and who is accountable for it.

Our article on AI policy development walks through drafting playbooks that stay usable, not just legally tidy. If you want an outside sanity check once the framework is live, this AI audit checklist gives teams a practical run-through they can finish without hiring a consultant.

What does a runtime policy engine actually do?

This is the layer most governance programs skip. It is also the one that matters most.

An agent's actions form an execution path: the full sequence of decisions and tool calls from start to finish. A policy that only checks the opening prompt or the final output misses everything in between. Research on runtime governance argues that many compliance conditions depend on that full path, not just identity or intent.

Checking every step against the entire history sounds impossible at scale. It is not, if you compress it. The practical approach keeps a compact governance state vector, a short summary of everything relevant so far, and evaluates each new action against that summary instead of the full history.

"A Policy Engine formalises this by evaluating and intervening on proposed actions at each step, rather than relying solely on upfront checks." That is the shift agent governance asks for. Judgment happens continuously, not once.

The policy engine has four jobs. Evaluate the proposed action. Decide whether it is allowed. Intervene if it is not. Log everything in a tamper-evident record that nobody can quietly edit later. That record is what an auditor wants to see.

Intervention type	What happens	When it's used
Block	Action is stopped entirely	Clear policy violation
Steer	Action is modified before execution	Partial violation, fixable in flight
Human approval	Action is paused for sign-off	High-risk or ambiguous case
Terminate	Agent session is ended	Repeated or severe violations

There is a real trade-off. Tighter enforcement caps risk and also slows the agent down or blocks legitimate work. Microsoft's agent governance toolkit shows one way to balance it, using fail-closed defaults so utility does not come at the cost of control.

Who owns what: roles and human controls

Governance fails fast when nobody is clearly on the hook. Fix that first.

• Agent owner. One named person per agent, responsible for its behaviour, its tools, and its risk classification.
• Central governance owner. One role holding the whole fleet's policy, standards and registry. Usually a CISO or a named AI risk lead.
• Decision rights for high-risk actions. Write down in advance who can approve refunds, data deletion, or external communication. Do not decide it mid-incident.
• Kill switch and SLAs. Your incident response team needs a way to stop any agent instantly, with a clear time target for how fast that happens.
• Operator training. Anyone running or supervising agents needs a written playbook, not tribal knowledge held by one person who might leave.
• Vendor accountability. If a third-party tool is wired into your agent, get written terms on what happens when that tool fails.

Global ethics guidance from UNESCO makes the same point in broader terms. Without named human accountability, responsibility for AI decisions disappears into the system. Naming owners is not paperwork. It is the difference between a fixable incident and a mystery.

How do you measure and audit AI agent governance?

You cannot govern what you do not measure. Build these signals in from day one.

• Fleet-level risk SLOs. Set an expected-violation budget, how many policy breaches you will tolerate per month, and track actual numbers against it.
• Tamper-evident decision records. Every proposed action and every intervention, logged in a format nobody can rewrite later.
• Standards alignment checks. Review against OWASP's agentic risk list and NIST AI RMF controls on a schedule, so drift surfaces before an external audit finds it.
• Red-team cadence. Book adversarial testing on a fixed calendar, not just after an incident.

Statistic callout: Runtime evaluation research shows that keeping expected violations inside a fleet-level budget depends on a decision function that actively trades off utility against risk, rather than a static one-time approval.

Audit is not an annual event for agents the way it might be for a model release. It is continuous, because the fleet keeps changing shape. New agents get added. Old ones get new tools. Your measurement cadence has to match that pace or it falls permanently behind.

How we build agent governance with clients

We start with ownership, not tech. A perfect policy engine means nothing if nobody is named to run it.

Every engagement opens with a registry and one accountable owner per agent. That is the unglamorous bit nobody wants to do first. It is also the bit that prevents the disasters.

Then we do the part most governance programs never reach. We encode the rules into the system that runs the agents.

Our 90-day program builds an AI Operating System with Claude Code: a coordinated set of AI employees that carry the founder's IP, their decision rules, and their approval thresholds in the same place the work happens. Governance stops being a document somebody reads once a year. It becomes the allow list, the escalation rule and the audit log that the agent runs on. Change the rule and every agent inherits it on the next run.

That is why we build with Claude Code rather than wiring generic automation tools together. The rules and the work live in one system, so the audit trail is a by-product of running, not a separate reporting job. Our piece on custom AI delivery systems built with Claude Code shows how that gets assembled.

The pitfall we see most often: teams buy monitoring tools before they have decided who is allowed to approve a high-risk action. Fix ownership first. The tooling gets much easier after that.

James Killick

Want a governance system that works, not one that looks good on paper?

Most consultants hand you a framework and walk away. We build the system with you.

The AI Orchestrators runs a 90-day program that turns your governance plan into a working setup: a live agent registry, clear ownership for every agent, and runtime checks built into your own AI Operating System rather than a generic template. You get hands-on prototyping from week one, not a report you file and forget.

If you are running agents across content, support, delivery or ops and cannot yet name who owns each one, that is the gap we close first. Check our AI Orchestration Glossary if some of these terms are new to your team, then look at our generative AI consulting program to see how the 90 days runs.

Next step: book a strategy call and bring your agent list, even if it is a messy spreadsheet. We will help you turn it into something an auditor will not flinch at.

Sources

• Governance and security across the organisation, Microsoft Azure
• Runtime Governance for AI Agents: Policies on Paths, arXiv
• NIST AI Risk Management Framework
• OWASP agentic AI security and governance resources
• ISO 42001