Any patient who has compared a clinic’s brochure against its outcomes data knows the drill: the brochure promises, the data remembers. Buying security tooling asks for the same discipline, because AIOps for Security — the young, unstandardised label for security work that machine learning and LLMs assist — sells itself almost entirely through claim sheets: glossy one-pagers where every capability is dressed in its best adjective and nothing carries a footnote. So this piece runs the comparison properly: each common claim on one side, the published measurement on the other, and a verdict at the end that a small team can act on without a procurement department. The voice here is human on purpose; these comparisons are ultimately about trust, and trust is decided by people reading two documents side by side and asking which one updates when the facts change.
Round One: The Accuracy Claim Meets the AIOps for Security Bench
The claim sheet says: near-perfect detection, validated. The bench says: be careful what “validated” carried with it. The TESSERACT research line (2019–2022) took headline F1 scores of roughly 0.99 — numbers that sound like the end of the discussion — and showed they could be manufactured by leakage in how the test data was assembled, spatial and temporal bias doing the work the model was credited for. Around the same time, the Arp et al. audit of thirty leading security-ML papers reported leakage-adjacent problems at scale: sampling bias appeared in nine out of every ten papers, data snooping in close to three-quarters. The bench’s guidance for reading any accuracy figure follows directly: a number brushing 1.0 is not reassurance but a flag, and the honest question is always what the test set contained and who assembled it. There is nothing exotic about the failure either — a test set that trains on yesterday’s incidents and evaluates on last week’s will flatter any model, and nobody has to intend it. Brochures, meet outcomes data.
Round Two: “Agents Resolve Cases End-to-End”
The claim sheet says: autonomous agents handle security work from alert to resolution. The bench says: agents complete a fraction of independent scenarios. ITBench, presented at ICML 2025, recorded agents finishing 25.2% of its compliance-and-security tasks. CyberSOCEval — Meta and CrowdStrike’s 2025 benchmark — spread between 23% and 34% when agents tackled malware analysis, and between 43% and 53% on reasoning through threat intel; blind guessing, for scale, manages 0.63% and 1.7%. Read those pairs honestly: the agents are far better than chance and nowhere near autonomous, and the gap between “demonstrates fluency” and “completes the job” is the whole market in one paragraph. It helps to say what the benchmarks actually measure — multi-step scenarios where an agent must gather evidence, decide, and carry an action through, exactly the chain a claim sheet compresses into the word “resolves”. A claim sheet that quotes no benchmark has chosen not to stand where it can be measured, which is itself information.
Round Three: “The Market Has Spoken”
The claim sheet says: analysts and analysts’ analysts agree this is the future. The bench says: the market’s own scorekeepers are urging caution. Gartner shelved the AI-SOC-agent category on the inflated-expectations peak of the 2026 cycle — a placement that does not say the technology fails, but does say the promises are running ahead of delivery — and popularised the term “AI washing” for capability claims polished past their proofs. There is even older bench data for the sceptic: Sommer and Paxson wrote in 2010 that the security industry had largely kept machine learning out of production, for the plain reason that errors cost too much. Sixteen years of progress later, the errors still cost what they cost; what changed is how confidently they are marketed. Score this round for the scorekeepers — when the industry’s own referees reach for caution flags, the brochure’s “analysts agree” line needs a footnote.
Round Four: “Your Data Is Handled Carefully”
The claim sheet says: enterprise-grade data protection. The bench says: read the defaults, because the defaults are the policy. Security Copilot leaves customer-data sharing enabled unless an E5 or E7 licence changes the setting — a licence tier quietly deciding a governance posture. Gemini’s work inside Security Ops happens on global endpoints; prompts stay out of training, which answers the training question while the geography question stays open. Multi-scanner convenience has its own fine print: VirusTotal notes that submitted files can reach its premium customers as well. None of these facts is hidden; all of them live where buyers rarely look, in defaults and terms rather than feature lists. The pattern repeats across the industry: what a system does by default, where processing physically happens, and who else can see submissions — three questions no claim sheet answers unprompted. Clinics comparing treatment options already know this habit — reading the risks column with the same attention as the results column, exactly as good guides to treatment options, risks and results teach — and the same habit transfers here without translation.
Round Five: “Autonomy Included”
The claim sheet says: closes cases by itself, saves headcount. The bench says: autonomy is priced in evidence, and the price list is public. Recommendations may begin once a system has studied around 200 graded alerts with at least 30 genuine ones. Silent closing waits for a miss bound of 2% or better across 150-plus genuine cases, planted decoys, a scored blind holdout, and a reopen control that puts a case back in one click. Reversible actions require wrong-action rates under 1% across 300-plus verdicts on cases that genuinely warranted action, plus rollback that has actually been rehearsed. Irreversible actions are never granted, regardless of evidence. The market’s 2026 behaviour illustrates both ends of the price list: Microsoft’s phishing agent starts closing mail in its very first minute — its documentation, checked in October 2026, contains no observe-only mode — while Google’s triage-and-investigation agent holds to recommend-only at roughly a minute per case — and Microsoft’s own Sentinel Fusion engine hands correlation duties to Defender XDR on a hard deadline of 31 March 2027. Products churn; the price list does not. Notice, too, what the two vendors’ contrasting designs imply: one asks your team for an observe stage it skipped, the other builds the human into the loop by construction — and a buyer can tell which philosophy a product follows with a single question about what happens on day one.
The Verdict on AIOps for Security Claims
Score it round by round and the pattern is consistent: claims run ahead of the bench everywhere except on narrow, measured jobs — sorting a queue, drafting a summary, recommending a disposition — where the technology genuinely helps today. The winning move for a small team is to become its own bench, and that is a shorter project than most people expect:
- Export your recent dispositions and label a blind sample — two people, one tie-break rule, the answers frozen before any machine sees them.
- Score any candidate tool against that sample, and insist on per-category results, because blended figures once masked a 42.3% failure rate behind a 9.1% headline.
- Grant recommend-only while evidence accumulates; autonomous closing waits for a proven miss bound, not for a quarterly webinar.
- Plant canaries before any autonomy — fifty a quarter, withdrawn at three closures — since random spot-checks of roughly 1,845 closures can miss everything and still bound the miss rate near 38%.
- Keep the published decision rubric on file: above 14 of 27 justifies shadow mode; from 9 through 14, a pilot with an end date; anything less, a decline.
Teams that want the full dossier behind every figure above will find it in the Ops Atlas comparison-ready briefing on AIOps for Security. And when the next claim sheet arrives — it will — handle it the way careful patients handle safety guidelines for cosmetic procedures: check who wrote it, what evidence sits behind each line, and what happens when something goes wrong. The brochure is a starting position, never the result. The bench always remembers, and now you can make it remember for you.