Buying an AI SOC?
Ask These 70 Questions First

Six events carry most of the value — context admission, tool call, memory write, identity assertion, agent-to-agent call and configuration fingerprint — within a wider set of sixteen event families that covers the agent activity worth logging across the platforms surveyed. [5] It is a working taxonomy, not a proof of completeness: a product may emit something none of the sixteen anticipates, which is itself worth asking about. Ask which of these the product emits, and then ask the harder half of the question: can those events land in your SIEM or data lake, in a documented schema, without an export button and a spreadsheet.

A tool call is the agent doing something — running a query, disabling an account, pulling a file. An outcome summary is useful, but it does not replace a record of the actions taken. When you are investigating six months later, the narrative is the agent's account of itself and the tool calls are the evidence. If only one of those survives, you have a story rather than a record.

This is retention, portability and integrity in one question. Ask what the retention period is, whether it is configurable, what it costs to extend, and whether the records are complete enough on their own that an investigator who has never seen the product could follow what happened. Then ask who can alter or delete an activity record, whether the product itself can, and how an investigator would know it had happened. Undetected changes to a record undermine its evidential value.

Most products have gaps, and a vendor who names theirs is telling you they have thought about it. The common omissions are worth listening for: reasoning steps, retries, failed tool calls, content the agent read but did not act on, and anything the agent did while an error was being handled.

Evidence you cannot query during an incident is evidence for the post-mortem. If the agent's activity reaches your logging platform on a nightly batch, then for the entire duration of an incident your most privileged automated actor is invisible to the people running the response.

User, host, case identifier, timestamp, request identifier. Without shared fields you cannot join the product's activity to your own telemetry, which means you cannot answer the only question that matters during an incident: was this action ours, and what did it touch. Without shared identifiers, reconstructing the activity becomes slower and less reliable.

Ask for the actual permission set, in writing, not a description of it. Then ask why each permission is needed — the gap between what a product requests at install and what it actually uses is usually large, and usually never revisited. The second half of the question is the one people forget: which of the vendor's own staff can read your data, change your configuration or trigger an action, how that access is approved, whether it is logged on your side as well as theirs, and whether you are told after the fact. For a managed service this is not a side question; it is the main one.

Disabling an account and revoking its access are not the same event, and the gap between them is your real containment window. Microsoft documents a default access-token lifetime of 60 to 90 minutes on the Microsoft identity platform, and two hours for some clients in tenants without Conditional Access; where continuous access evaluation applies, a disabled account is a critical event that propagates in near real time, though Microsoft still notes latency of up to fifteen minutes. Your platform and configuration will differ. Measure the actual containment window with the vendor in your tenant, using an active session. Published timings are a starting point, not a substitute for that test. [1] [2]

There is a real difference between "the agent will not do that" and "the agent cannot do that". The first is a behaviour, and behaviours change when models change. The second is a permission boundary. Ask which one you are being offered, and whether you can see it in a settings page.

This establishes whether an independent control enforces the action boundary. If the model is the last thing that decides, the safeguards rest on its judgement rather than on something it cannot cross. What you are checking is not the existence of a file — a policy engine, a signed configuration, typed permissions on the action API, an approval service and several other designs all qualify. What every acceptable answer has in common is that the constraint lives outside the model, that you can read it, that changes to it are versioned and attributable, and that it holds when tested. Ask which component enforces it, ask to inspect the current state, ask who can change it, and ask what changed last month.

Tuning detections is legitimate work, and a product that proposes improvements to your rules may be worth having. What is not acceptable is a change that happens without a named approver, a version history and a way back. An agent that can quietly tune down a noisy rule or reduce sampling to save cost reduces your visibility, and the resulting silence looks exactly like an improvement. It is also the last stage of a competent attack. So ask for the specifics: can it propose, or can it apply? Who approves, and is that person independent of the team whose alert volume is being reduced? Is every change versioned, diffable and revertible in one step? And does a change to detection or logging content itself raise an alert somewhere the agent cannot reach?

Multi-agent products spawn helpers: an enrichment agent, a summariser, a hunter. Ask whether those inherit the parent's permissions, whether they appear in the logs under their own identity, and whether pulling the kill switch stops them or leaves orphaned children still holding credentials.

Persistent memory is how a product gets better and also how a wrong conclusion becomes permanent. Ask where it is stored, whether it records the origin of each entry, whether entries expire, and whether an agent that reads untrusted content can write into a store that a more privileged agent later reads. That last one is the delayed attack: the write happens today, the damage next month, and the investigation looks in the wrong fortnight. Then ask the multi-tenant version of the same question: what technically prevents context stored from my environment — hostnames, user names, case text, learned patterns — from influencing an investigation in someone else's. "Logical separation" is a phrase, not a control; ask what enforces it.

Investigations sprawl by nature — an alert on a laptop leads to a file server leads to a domain controller. Ask whether scope is a hard boundary enforced by permissions or a guideline in a prompt, and what the product does when the trail leads somewhere it is not allowed to go.

Measuring wrong closures is essential to understanding missed incidents and delayed containment. Ask for the formula in writing, because two different figures get quoted under the same name:

On smaller screens, swipe or scroll the table to see every column.

The numerator is the same count in both. Only the population underneath changes, and the two populations answer different questions, which is why you want both.

The first measures judgement: of the malicious alerts this product looked at, what proportion did it wave through? That is the number to compare between products, and the one to watch after a model change.

The second measures exposure: of everything sitting in your closed queue, what proportion was incorrectly closed as benign? That is the number your incident commander needs, because it sizes what the closed pile is hiding.

Which figure is larger depends on the ratio of the two populations in your environment, so neither can be inferred from the other and neither can be compared with anyone else's unless both denominators are stated. Ask for both, with the period, the adjudicator, and the raw counts rather than the percentages — and ask whether the numerator came from an adjudicated sample or from incidents that happened to surface on their own, because the second depends on how hard anyone looked. A vendor who says they do not know is being honest; a vendor with a process — sampled re-review by humans, published results — is telling you something about how they run.

Your alerts are full of attacker-supplied text. An attacker who wants to reach your triage agent does not need to breach anything, only to generate an alert containing text the agent will read. This is the one attack path that exists because you bought the product, and the answer tells you whether the vendor has thought about it at all.

Ask for the specific ceilings: isolations per hour, accounts disabled per hour, messages purged per action, and what happens when a limit is reached. A product with no ceilings is one manipulated alert away from becoming a denial-of-service tool pointed at your own estate, operated by your own SOC.

Ask for the case, the investigation and what changed afterwards. The point is not the mistake, it is whether they have a process for finding mistakes at all. If examples cannot be shared, ask for anonymised test findings and the process used to identify and review errors.

This is a different question from the last one, and the distinction matters. Getting an answer wrong is a judgement failure. Taking an action nobody sanctioned is a control failure, and control failures tell you about the architecture rather than the model.

Not disabling the integration — containing the product while it is mid-flight on two hundred cases. Ask what happens to work in progress, whether actions already issued complete or roll back, how long the whole thing takes, and whether anyone has ever done it outside a slide.

Ask for the containment story and the residual risk they accept. A vendor who has thought about this has an answer ready and will usually volunteer the limits of it. If the answer is unclear, request the relevant threat model, test evidence and incident-response procedure.

Ask for the actual contract language rather than a verbal assurance, and read what it excludes. In the agreements I have read, the risk sits with the customer. That may be acceptable, but it should be a decision you made rather than one you discovered afterwards.

Do they queue, and for how long? Do they fall through to your analysts, who by then may have reorganised around the product's existence? Does the queue drain in order or by priority when service returns? A triage layer that fails silently and leaves a gap is worse than one you never installed, because you staffed around it.

Ask for the middle of the distribution and the worst result they have seen, not the average. An average built from one exceptional deployment and nine ordinary ones tells you about the exceptional one. Then ask what those environments were: size, sector, tooling, alert volume, analyst headcount. A product validated across thirty small cloud-native companies is a different proposition for a regulated enterprise with twenty years of accumulated infrastructure, and it will fail in different places.

The most relevant benchmark reflects your environment and operating conditions. Push for a proof of concept on your own historical alerts with your own analysts scoring the results, and treat reluctance as information. Ask what it costs, how long it takes and what you have to give them to make it work.

Results in this field are usually sensitive to timing assumptions that are rarely stated. Ask what happens to the numbers when an alert lands twenty minutes late, when an approval sits unanswered overnight, or when the enrichment source they depend on is slow. The first thing to break tells you what the product actually relies on.

Unknown assets are the normal state in most enterprises, not the exception. A product that behaves sensibly only when its context is complete will spend a lot of its life in the state you did not test.

Reasoning that surfaces only in an export is not available to the person who has to accept or override the decision in the moment. Ask what the analyst sees on screen, how much of it is the model's own account of itself, and whether the underlying evidence is one click away or three.

The right answer is that autonomy narrows as consequence rises. Ask how the product knows which of your systems are sensitive, whether that classification comes from you or is inferred, and what remains automatic on a domain controller or a payment system.

Most products in this category sit downstream of detection. Your endpoint tool or SIEM found the thing. For any claimed improvement in detection time, establish whether the product generates the finding or accelerates handling of an existing alert. Attribute each contribution separately. It also sets a hard limit on every accuracy figure they will show you: a product that only triages cannot see an attack that generated no alert, and no amount of reviewing its closed cases will reveal one.

A before-and-after from different customers measured by different methods is not a comparison. Ask for paired numbers from the same environments, and ask who measured them. Measure the improvement under comparable conditions before using it in the business case.

These are not the same number and the gap between them is where incidents live. A tool reporting success is a claim; the endpoint's own record is evidence. The product may report API acceptance because that event is directly visible to it; target-system confirmation requires additional telemetry. Whether the endpoint was reachable, whether the agent was healthy, whether the change survived a reboot are separate questions, and in my experience few products join the action back to the target system to answer them.

Case closed, threat removed, or system back in service? Each is a legitimate measure and each produces a wildly different number. Whichever they choose, make sure it is the same one your own reporting uses, or you will spend a year explaining to your board why two dashboards disagree.

Ask for the denominator every time. "Dismissed forty thousand false positives" means nothing without how many alerts it handled, how many were genuinely benign, and how many a human checked. Volume alone does not establish accuracy or operational value.

This is the previous question's shadow and the more important half. For a given level of discrimination, a product tuned to close more will report fewer false positives and miss more real incidents; the trade runs the other way when it is tuned to escalate more. A product can improve both at once by genuinely getting better at telling them apart — which is the thing worth paying for — but you cannot tell which is happening from one number, and the false-positive figure is the one that improves for the wrong reason. Ask how they discover a wrong close: sampled re-review by humans, customer reports, incidents traced back to a closed case. If there is no process, the accuracy numbers are unaudited by construction. Keep the answer separate from detection coverage in your notes — this question measures what the product got wrong about what it saw, and says nothing about what it never saw.

Learning from your analysts' decisions is valuable and raises a data question. Learning from other customers' environments raises a different one. Ask which is happening, whether you can turn it off, and what happens to that learned behaviour when you leave.

The distance between a marketing claim and a service level is the distance between a hope and an obligation. Accuracy commitments depend on scope and measurement conditions. Agree what can be measured and committed to, including availability, latency and support response, with procurement and legal.

Same user across three tools, same machine over four days, the same unremarkable behaviour appearing in six places. Most detection logic works within a single product and a short window. Ask for concrete examples of cross-tool, cross-time correlations it produced, and how it decided they were related rather than coincidental.

This is the question worth pressing hardest in the whole section. A product that finds patterns in your environment but will not hand them back as detection logic you own is renting you insights drawn from your own data. Ask what format they come in and whether they survive the end of the contract.

Ask what it took to find, how long the signal had been present, and what the customer did with it afterwards. A good answer is specific and slightly unglamorous. If the answer is unclear, separate the demonstrated triage benefit from any additional detection value that remains unproven.

Ask for it in writing. Check that the inventory is maintained and covers the components actually deployed. Pay attention to connectors published by individuals rather than organisations.

An integration that updates on its own schedule is a change to your security stack that nobody in your change process approved. Ask whether updates are pulled automatically, whether you can pin them, and whether anyone would notice if a component's behaviour changed.

In September 2025 an npm package called postmark-mcp built trust over fifteen releases and then, at version 1.0.16, began silently copying email it handled to an external server. The package impersonated Postmark; Postmark states it had not published its own MCP server on npm before the incident and that its legitimate API and services were unaffected. That distinction matters when you tell this story — the lesson is not that a vendor was breached, it is that a package with a trusted-looking name accumulated fifteen releases of good behaviour before it needed to be malicious, and nothing about its identity changed when it was. Only the version did. [3]

The answer should be about isolation and permissions rather than vetting. Vetting tells you a component was acceptable when you looked. Isolation determines what happens when it stops being acceptable, which is the case you are actually buying protection against.

A software bill of materials — a parts list for the product — is becoming standard, and for an agent it needs two entries that traditional lists omit: which model version is running, and which version of the instructions it is running under. Both change the product's behaviour and neither is usually tracked.

Ask for the sequence, with the automatic steps marked. This is the scenario where automation is most valuable and most dangerous at the same time, and the shape of the answer tells you how much the vendor has thought about consequence rather than speed.

Isolating an encrypting file server is often the right call and occasionally a bigger outage than the ransomware. Ask whether the product knows which of your systems cannot be isolated without an approval, where that list comes from, and who maintains it once you are live.

Ransomware generates alert storms, and an agent that reasons about every alert individually is the component most likely to fall over exactly when you need it. Ask what degrades first, whether there is a batching or sampling mode, and what it costs to run through a storm.

Separate the two. Read-only visibility into backup status is often genuinely useful during a ransomware investigation: knowing whether last night's job completed, and which systems are recoverable, changes the containment decision. The ability to modify, expire, unlock or delete a backup is different. There are legitimate reasons a product might hold a narrow version of it — triggering an out-of-band snapshot before containment, for instance — but it should be explicitly approved, scoped to the specific operation, logged where the product cannot reach, and never a side effect of a general integration role. Ask which permissions the account actually holds against the backup platform, whether read-only is enforced by design or merely by current configuration, and what would have to happen for that to change. Then ask the same question about the recovery environment itself.

Default-deny with a named allowlist is the answer to push for. A blocklist means every destination nobody thought of is permitted, and the destinations nobody thinks of are the ones an attacker chooses.

Ask for the field list rather than the assurance. Alert content contains hostnames, usernames, file paths, email subjects and sometimes the contents of documents. Where it goes, who can read it there, how long it stays and whether it is used for anything besides serving you are four separate questions.

This is the volume version of the blast radius question. Ask about rate limits, per-case ceilings and whether anything alerts on unusual outbound volume from the product itself — which is a control almost nobody has, because the product is the trusted thing.

Fetching an attacker's URL is itself a way for data to leave, and it needs no exfiltration channel of its own. This is how CVE-2025-32711 worked — an AI command injection in Microsoft 365 Copilot, published in June 2025, scored 7.5 by NIST and 9.3 by Microsoft, and classed as CWE-74. [4] The researchers who reported it named it EchoLeak: the user never opened the message, the assistant retrieved it on their behalf, and the data left inside a request for an image in the answer. [6]

One identity that reaches everything is convenient to deploy and is also one of the shortest paths an attacker will find in your estate. Ask how the product is segmented, and whether segmentation is available on your licence tier or is a feature of the enterprise edition.

Relevant for anyone with subsidiaries, regional separation, regulated units or a recent acquisition. The agent's willingness to follow a trail is a feature until the trail crosses a boundary that exists for legal reasons.

If the answer is yes without qualification, then the permissions of your most privileged agent are effectively the permissions of your least careful one. Ask whether the origin of the original request travels with the call, so you can answer afterwards who actually asked for an action.

Vendors who have done this exercise say so immediately and usually have a diagram. If the answer is "that is not possible", ask which architectural controls support that claim and how they were tested. Draw it yourself during the proof of concept.

Whose account the usage lands in determines who absorbs a surprise. It also determines who has an incentive to make the agent efficient, which quietly shapes how the product behaves over the life of the contract.

Ask for the middle of the range and the worst five per cent, because the distribution has a long tail. A handful of pathological alerts can cost more than a normal week, and those are exactly the alerts an attacker would choose to generate.

Agents loop. They retry, re-reason and re-enrich, and a single confusing alert can cost more to triage than an ordinary week. Find out whose account is billed for that and whether there is a ceiling, because an attacker who works out how to generate expensive alerts has found a way to run up your bill or exhaust your quota without touching a single system.

All three are defensible answers and they have very different consequences. Stopping is safe and creates a gap; degrading is sensible if you are told it is happening; continuing is fine until the month you find out otherwise. What you cannot accept is not knowing which one your product does.

Vendors move between models for quality and for margin. If your unit cost is a function of a choice they make without telling you, your budget is a variable they control. Ask for notice and for the right to stay on a known configuration.

This is the security question hiding in the cost section. A sudden change in what the agent spends per case is one of the earliest signals that it is behaving differently — looping, being manipulated, or working much harder on a category of alert than it used to. Month-end billing tells you about it five weeks late.

Someone maintains the connectors when an API changes. Someone approves the actions the policy does not cover, handles the exceptions, tunes the suppressions, and works out what happened when the agent behaves oddly. Ask the vendor for the staffing their existing customers actually use, in hours per week rather than in adjectives, and ask whether that person needs to be your best analyst — because if it does, you have moved your scarcest resource rather than freed it.

A floating model name means the component making decisions in your environment changes on someone else's schedule. Ask whether you can pin a version, how long old versions are supported, and whether you are told before a change or after.

There is a real difference between a release that adds a feature and a release that changes judgement. Ask whether release notes distinguish the two, and whether behaviour changes can arrive outside the release cycle — which they can, whenever the model behind the product is updated.

Suppressions, exceptions, auto-close rules, asset classifications. Teams invest months in this and it is rarely versioned. Ask what is preserved across updates, what is not, and whether you get a diff of what changed.

This is the question where the risk is hardest to see, because a bad update rarely announces itself. It might close more of what it should escalate, and everyone congratulates the tool on the quieter queue. It might escalate more of what it used to close, and your analysts absorb the difference without anyone recording why. Or it might keep the same volumes and make different wrong decisions inside them, which is the version nobody catches at all. None of those three produce an alert. A vendor who re-runs a regression set against the same cases after every update, and publishes the delta, has built the control most likely to catch any of them.

Ask for the export format, the completeness and the timeline. Anything generated in your environment from your data has a reasonable claim to being yours, and the moment to establish that is before you sign rather than during the thirty-day wind-down.

Training is the question everyone asks and only one part of the answer. Ask what "data" covers — alerts, case content, analyst decisions, corrections — and whether opting out of training changes the product's behaviour or its price. Then ask the lifecycle questions that matter just as much: the retention period, what deletion actually means and how it is confirmed, which subprocessors touch the data, and which countries it is processed in. Finish with what happens to anything already learned when you leave, which is the part most agreements are quiet about.