The Defender’s Guide to AI Agents·Chapter 13
Measuring AI Agent Security Readiness
Readiness you can measure
“We have tracing enabled” is not a readiness statement. It is a procurement fact. Eleven measures below turn readiness into something you can put a number against, argue about, and improve.
Use them as launch criteria for a new agent and as a quarterly position for the estate. The targets are a starting point, not a standard - there is no published benchmark for any of this, and anyone who offers you one has invented it.
| # | Measure | How it is calculated | Suggested floor | Why this one |
|---|---|---|---|---|
| 1 | Inventory coverage | Agent identities observed acting, divided by agent identities registered | 100% | Anything above 100% means you have agents nobody registered. That is the finding, and it is usually the first one. |
| 2 | Event-family coverage | Families from 5.6 collected for this agent, out of the sixteen | 10 of 16 to launch | Name the missing six explicitly. An accepted gap is governance; an unnoticed gap is an incident waiting. |
| 3 | Correlation success | Traces where every consequential action joins back to an originating request, divided by all consequential actions | 95% | The single best proxy for “can we investigate”. Below 90% you have logs, not a trail. |
| 4 | Approval binding | Approved actions carrying a digest verified at execution, divided by approved actions | 100% for irreversible | Partial coverage here is close to worthless - an attacker uses the unbound path. |
| 5 | Downstream confirmation | High-impact actions with an authoritative outcome joined, divided by high-impact actions | 90% | Measures whether you know what actually happened, as opposed to what was attempted. |
| 6 | Telemetry latency | Event time to queryable time, at the 50th, 95th and 99th percentiles | p95 under 5 min | Report all three. A good median hides a tail, and the tail is when the incident happens. |
| 7 | Field completeness | Events carrying every required field from 5.5, divided by all events | 98% | One missing field on 5% of events breaks 5% of investigations, usually the interesting ones. |
| 8 | Evidence integrity | Hash-chain verifications passing, divided by verifications attempted | 100% | Any failure is either tampering or a pipeline defect. Both need answering the same day. |
| 9 | Control-test success | Adversarial tests where the control held and the detection fired, divided by tests run | Run monthly | The only measure here that tests reality rather than configuration. See 13.2. |
| 10 | Time to constrain authority | Decision to the agent identity no longer obtaining tokens | Under 15 min | Not time to kill the process - time to contain the authority. Measure it in a rehearsal, not in an incident. |
| 11 | Rollback confidence | Rollbacks confirmed in the target system’s own log, divided by rollbacks attempted | 95% | A compensating action that reports success and did not is worse than no rollback, because you stop looking. |
The acceptance test
Before an agent gets consequential authority, one exercise. Take a real action the agent performed last week and reconstruct it from telemetry alone, without asking anyone who built it. Nine questions, from 5.5:
- 1Who delegated this authority, and when does it expire?
- Identity and delegation records.
- 2What content did the agent consume, and what was its trust class?
- Context admission.
- 3Which model answered, and was it the one requested?
- Model invocation.
- 4Which policy version applied, and did anything get denied first?
- Policy decisions.
- 5What did the agent propose?
- Planning steps.
- 6What was a human shown, and what did they approve?
- Approval records - and the two are different facts.
- 7Which tool attempted it, at what version, from which server?
- Tool execution and supply chain.
- 8What left the trust boundary?
- Egress.
- 9What does the target system say changed?
- Authoritative outcomes.
Score it out of nine. Anything below seven and the agent is not ready for authority it cannot be held to. The exercise takes an afternoon and is worth more than any questionnaire.
Testing the controls rather than the configuration
Monthly, against a non-production agent with production-shaped telemetry. Each test should produce a detection and a control response:
- Place benign marked content with an instruction in a source the agent reads. Does detection 6 fire?
- Alter an approved action between approval and execution. Does the digest comparison refuse it?
- Present a token to the wrong resource. Does anything notice?
- Have a low-privilege agent write to a shared memory store a privileged agent reads. Does detection 12 fire, and how long does it take?
- Change a tool’s description without changing its version. Does anything alert?
- Return a false success from a tool with no downstream record. Does the workflow proceed anyway?
- Disable tracing on a high-risk agent. Does its autonomy actually drop, per 5.11?
Seven tests, all safe, all repeatable. The last one fails in almost every organization on the first attempt.
Thirty, sixty, ninety - the implementation half
Chapter 10 covers the first ninety days of understanding your estate. This is the ninety days after that, once you have decided to build.
Days 1-30 · Contract
Write the telemetry contract. Which of the sixteen families, which fields from 5.5, which tier each goes to. Publish it as a document teams build against rather than a wish.
Complete the agent inventory and reconcile it against identities observed acting. Name an owner for each.
Agree the evidence policy: what is never collected, what is redacted, who can read Tier 1, who authorizes Tier 2.
Days 31-60 · Correlate
Get the spine working for one agent end to end before broadening. Measure correlation success; fix it until it is above 90%.
Build the deterministic detections first - model drift, novel tool, novel egress, telemetry suppression. They need no baseline and no classifier.
Onboard the downstream systems of record for that agent’s three most consequential actions.
Days 61-90 · Respond and prove
Implement approval binding for irreversible actions, and the automatic autonomy reduction on telemetry loss.
Run the seven control tests. Rehearse identity containment and time it.
Report the eleven measures to whoever governs this. Then repeat for the next agent, with the contract now written.
The sequencing principle throughout: depth on one agent before breadth across many. An estate where one agent is fully instrumented and nine are not is in a far better position than one where ten agents each have a third of the evidence - because the first has a working pattern to copy and the second has ten unfinished investigations waiting.
Say this on Monday
“Let's score one real action out of nine. Whatever we can't answer from telemetry alone is our roadmap, in priority order, for free.”
This guide will go out of date.
Providers change how their logs work, models get retired, and new cases get disclosed. Ask to be told when this changes — no newsletter, just the updates.
Download the complete guide (PDF)
The telemetry contract, detection specifications, framework mappings and checklists are also published as files — the defender pack, CC BY 4.0, free to reuse.