What is runtime governance for AI agents?

Runtime governance evaluates every AI agent action at execution time and returns one of five verdicts before the action ships. Not after-the-fact monitoring. Not a static filter.

The question none of the existing layers answer

What governs an AI agent while it’s executing real work?

Enterprise AI has made enormous progress. Organizations now have identity systems for AI agents, observability platforms that capture execution traces, and guardrails that evaluate prompts and model responses. Each solves an important problem.

Yet one critical question remains unanswered: Should this action be allowed to execute, right now?

Most production failures are not caused by missing logs or incomplete traces. They happen when an agent performs an action it was never authorized to perform. By the time the failure appears in telemetry, the action has already occurred. The cost has already landed. The audit trail records a violation, not a prevention.

Identity establishes who an agent represents. Observability explains what happened. Guardrails evaluate the text of prompts and responses. Runtime governance answers a different question, at a different moment: at execution time, on the action itself.

The definition

Runtime governance is the discipline of evaluating every action an AI agent is about to take, at execution time and before dispatch, and returning one of five verdicts:

The five verdicts are canonical. They cover the full space of outcomes for a single agent action:

• Allow. Lets the action ship as-is.
• Redact. Lets it ship with sensitive fields removed.
• Throttle. Queues it under a rate limit.
• Escalate. Requires a human to sign off before dispatch.
• Block. Prevents the action entirely and returns an explanation to the calling agent.

Every verdict is issued in under 50 milliseconds at p99, tied to a delegated principal, and written to a tamper-evident audit trail. The enforcement decision itself is sub-25 milliseconds; the 50ms envelope covers evidence-write and the network round-trip so the whole gate stays inside the agent's request budget.

Nothing about the pipeline is ex-post; the verdict has to arrive in time to change the outcome.

The four responsibilities

The five verdicts are what runtime governance produces. The layer itself has to do four things — and any framework that skips one produces a false sense of safety.

Three failures that only runtime governance catches

The category is defined by the failure modes it exists to prevent. Three of them recur across the regulated deployments we’ve observed, and none of them are visible in time to an identity, observability, or guardrail layer.

A delegated workflow exceeds its authority

A support agent hands an escalation to a refund agent, which chains to a payment agent that issues a $50,000 credit. Each agent, in isolation, had permission for the step it took. The composition did not. Static IAM cannot see the composition because it evaluates identity at session start, not authority at the moment of action. Observability sees the refund after it clears. Runtime governance sees the delegation chain widening and refuses the last hop.

A tool call reaches a sensitive system without the required approval

An analytics agent, generating a report, calls a tool that touches a production database with PHI in it. The prompt was safe. The output was safe. The action was not. Guardrails inspect text, not tools; observability records the query after execution. Runtime governance evaluates the tool call before it dispatches and returns Escalate or Block.

An autonomous process acts on behalf of the wrong principal

A cron-triggered agent, meant to run under a service account, is spawned in a context where it inherits a customer’s session. Its actions are now attributed to that customer, without their knowledge or consent. Identity says the token is valid. Observability records the actions as coming from a legitimate identity. Runtime governance verifies the delegation chain at every step and refuses when the principal binding is broken.

These are not edge cases. They are the recurring shape of agent incidents in production. In each one, the pre-runtime layers cannot see the failure and the post-runtime layers see it too late.

What runtime governance is not

The category is easier to see once you separate it from three adjacent things it is regularly confused with.

It is not observability

Observability records what happened. Runtime governance decides what happens.

Traces, spans, and evaluations are all telemetry; they are read after the agent acted. Runtime governance is the perimeter check that runs before.

Observability answers "what did this agent do yesterday?" Runtime governance answers "should this agent be allowed to do the thing it is about to do, right now?"

It is not guardrails on prompts

Prompt and output guardrails, the category occupied by tools like Lakera and NeMo Guardrails, filter language at the LLM boundary. They catch jailbreaks, prompt injection, and unsafe generations.

But the moment an agent turns a generation into an action, a tool call, an API request, a file write, the guardrail is out of scope. Runtime governance evaluates the action, not the text.

It is not static IAM

Identity and access management defines what a principal is permitted to do in principle. It is evaluated at authorization time, typically once per session.

Runtime governance is evaluated at every action, with context that IAM cannot see:

• What the agent has already done in this session
• What the delegation chain looks like
• Whether an attack pattern is forming across multiple actions

IAM says "this identity can call this API." Runtime governance says "this specific call, right now, from this delegation chain, in this context: allow, redact, throttle, escalate, or block."

The three layers of AI governance

Responsible AI assesses the model before deployment. Observability reports what agents did after. Runtime governance decides, at execution time, what they’re allowed to do, and signs the record.

The AI governance stack has three layers, each with its own question, verbs, buyer, and vendor category. Runtime governance sits in the middle. It is the thinnest layer of the three, and the only one that acts in the request path.

The layers are complementary, but the signal flow is not symmetrical.

Identity → Veldt: Principal identity and delegation context inform every runtime verdict.

Veldt → Responsible AI: Signed decisions and verifiable evidence support audit, assurance, and compliance programs.

Veldt → Observability: Runtime verdicts and evidence become high-value operational events. Veldt also builds behavioral history directly from the actions it governs, enabling detection at the decision point rather than relying solely on post-hoc trace reconstruction.

Runtime governance sits at the action boundary, where context becomes an enforceable decision and evidence.

The category label is Runtime Governance for AI Agents. The budget line is typically AI agent security (same CISO, same buying center), but the discipline itself is runtime governance. Not Responsible AI, not observability. Those are different budget lines, different buyers, and different questions.

Delegated authority: the concept that makes it enforceable

Runtime governance is not just a decision engine. It is a decision engine tied to a delegation chain that can be reasoned about. Without that, verdicts have nothing to check against.

An autonomous actor never acts with its own authority. It acts on behalf of a principal, and the concept of "principal" is broader than most teams start with. A principal can be a human, an AI agent, a service account, an upstream agent that is itself acting on behalf of someone else, or a non-human identity attached to a machine (a drone, a robot, an industrial controller, or an autonomous vehicle). Every one of them holds authority delegated from an accountable party.

Every action, at every hop, must trace back through the delegation chain to an original human accountable party. Runtime governance verifies the chain at every action, not once per session.

If delegated authority expands rather than narrows as work moves across an agent chain, you’ve created an authority amplification problem—a privilege escalation that no one explicitly granted.

KYA and KYP: the two frames of the same problem

The formal literature names two adjacent frames on this problem.

Both are introduced in the same source: Quadri (2026), "KYA: A Framework-Agnostic Trust Layer for Autonomous Systems with Verifiable Provenance and Hierarchical Policy Composition" (arXiv:2605.25376). KYA is the framing; KYP is the schema-level unification introduced in section 5.6 of the same paper.

KYA (Know Your Agent) is the frame that starts from AI agents specifically: what is this agent authorized to do, who does it act for, and what evidence records what it did? The KYA paper formalizes the five architectural primitives (identity, authority, evidence, runtime, and attack chain) that make agent governance enforceable rather than aspirational.

KYP (Know Your Principal) is the generalization. Section 5.6 of the KYA paper defines KYP as a unified principal-trust schema: one relational table with a discriminator column that carries every principal category through a single scoring substrate (humans, AI agents, service accounts, non-human machine identities, drones, robots, industrial controllers, and autonomous vehicles). Same delegation chain, same only-tighten rule, same signed evidence, applied to whichever principal holds authority at the moment of action.

Runtime governance is the enforcement discipline that both frames converge on. KYA answers "what makes an agent governable"; KYP answers "what makes a principal governable, whatever it is". Both flow into a single trust ledger that runtime governance evaluates against.

Delegated authority is what turns a runtime verdict from "does this agent’s identity have this permission?" (a static IAM question) into "does the current delegation chain, considering every hop that led here, still authorize this action?" (the runtime governance question).

This is why static IAM alone is insufficient: it can approve the identity at session start but cannot re-evaluate the chain of authority that the agent is now operating under, four hops later, in a context IAM never saw.

Why it emerged now

The category did not exist five years ago, because it did not need to.

Traditional AI systems answered: they returned text, a classification, or a prediction. A model that answers can be reviewed after the fact without much cost, because the answer is a text artifact and its consequences propagate slowly through humans.

Agents act. Answers don't.

AI agents act. They call APIs, move money, write to shared drives, spawn sub-agents, send messages to customers, and take actions that cannot be undone.

When a system acts autonomously and irreversibly, "we'll review it tomorrow" stops being an acceptable operating model. The correct place to intervene is before the action ships, not after.

That is the shift that produced runtime governance. It is not a new tool category invented for marketing; it is the direct consequence of moving from "AI that answers" to "AI that acts".

Regulators recognized the shift too. The EU AI Act, NIST AI RMF, and adjacent frameworks increasingly place obligations at the action boundary, not only at the model boundary.

How runtime governance works

A runtime governance layer sits between the agent and the outside world. When the agent decides to take an action, whether a tool call, an API request, or a side effect, the intended action is sent to the runtime as a proposal.

The runtime evaluates it against several signals and returns a verdict.

The five signals

In the Veldt implementation:

1. The delegation chain. Does this agent hold a delegation that authorizes this action, from a human principal, under only-tighten composition?
2. The policy. Does the action fall within the scope the operator has configured for this class of agent, workflow, and context?
3. The behavioral history. Does the action fit the pattern of this agent, or does it look like the beginning of a multi-step attack chain?
4. The data classification. Does the action involve sensitive fields that must be redacted or escalated per regulation?
5. The rate and blast-radius envelope. Has this workflow already used its budget of high-risk actions? Should this be throttled?

All five signals feed into a single verdict returned in under 50 milliseconds at p99. The verdict, the reasoning, and the full context are written to a signed audit trail.

The agent sees the verdict and either proceeds, retries with a narrower scope (which is itself evaluated by the same pipeline, so a retry cannot escape governance), or escalates to a human.

The full specification of this pipeline, including the five architectural primitives, the only-tighten composition rule, and the tamper-evident evidence chain, is described formally in the KYA paper.

Containing compromise: the revoke family

When a compromise is detected — an agent behaving anomalously, a delegation chain showing signs of hijack, a tool being called in a pattern that should not happen — the runtime must be able to revoke authority faster than the compromise can execute more actions.

Revoke is not one operation. It is a small family, each with a distinct blast radius:

Two more capabilities the response layer needs to expose:

• Blast-radius preview. Before a destructive operation, the operator should see exactly which principals will be affected. A "lineage revoke" that surprises the operator with 40 revoked agents is an incident, not a containment.
• Sub-second propagation. Compromise moves fast. If the revoke takes minutes to reach every runtime, the compromised agent has minutes to keep acting under the old authority.

Grounded in the literature

Runtime governance is a formalized discipline. If a vendor is claiming it and their claims do not trace back to a paper, they are extrapolating from adjacent categories.

Veldt is built on:

• Quadri (2026), KYA: A Framework-Agnostic Trust Layer for Autonomous Systems with Verifiable Provenance and Hierarchical Policy Composition. arXiv:2605.25376. Introduces the five architectural primitives, only-tighten composition, the tamper-evident evidence chain, and the KYP (Know Your Principal) unified trust schema (section 5.6) that generalizes agent governance to every principal type.

Every substantive claim on this page maps to the paper. Where a claim does not, it is called out as opinion or industry practice, not as formal result.

Where it fits with adjacent disciplines

Runtime governance replaces none of the adjacent disciplines. A production AI agent stack that is safe to run typically includes all of the following, at different points in the request lifecycle:

• Identity / IAM at session start: who is this agent, and what is it in principle allowed to do?
• Prompt and output guardrails at the LLM boundary: is this generation safe as text?
• Runtime governance at the action boundary: allow, redact, throttle, escalate, or block this specific action?
• Observability after the fact: what did the agent actually do, and how did it perform?

The first three are enforcement disciplines. The fourth is a telemetry discipline.

Teams that skip runtime governance end up with observability of things they could not stop. That is not the operating model that regulated industries can defend.

What buyers should look for

Use this as the floor when evaluating an AI runtime governance platform, whether you are buying it or building it. A platform that ticks every line item is doing runtime governance. A platform missing any one of them is doing something adjacent — observability, prompt guardrails, or IAM — but not this discipline.

Products that answer "no" or "kind of" to any of these are not doing runtime governance in the sense the term is now used. They are doing observability, prompt guardrails, or IAM. Those are useful, but different disciplines.