Future Blue Team

What this episode is about

In episode five we discussed advisory copilots — Microsoft Security Copilot, Gemini in Chronicle, Charlotte AI. Today we take the next step: what if AI no longer just recommends, but acts? The central thesis is sober: the loop gets smaller, the human stays in. But the job shifts from seeing to overseeing. And there is robust research showing why complete autonomy is not a good idea in 2026.

Four stages of autonomy

Chris frames the episode with a schema. Level one is observation — the model recognizes patterns. Level two is recommendation — episode five. Level three is limited action — the model executes predefined measures like an account lockout. Level four is autonomous action — the model decides independently in a defined area. Most Swiss SOCs are currently at level two with pilot projects at level three. Level four is a research topic.

Robert names three level-three examples from client engagements. Automatic identity containment: a suspicious login triggers a Conditional Access policy, the account goes into read-only mode, the human decides later. Automatic email quarantine: phishing emails with confidence score above 90 percent go to quarantine without human approval. Patch automation for standard vulnerabilities with Microsoft’s Vulnerability Remediation Agent. Lukas frames it scientifically: the AIxCC validation shows that automatic patching works on 43 of 54 vulnerabilities in controlled codebases. Atlantis and Buttercup roll out a patch only if it passes a regression test — without this filter, the patch pipeline would be damage, not benefit.

AIxCC and the defender validation

The DARPA AIxCC Final 2025 is interesting not only offensively. The winning Cyber Reasoning Systems patch autonomously — Atlantis by Team Atlanta, Buttercup by Trail of Bits, Theori’s system. Trail of Bits open-sourced Buttercup after the competition. Chris draws the operational consequence: a small or mid-sized firm in St. Gallen will not run Atlantis. But a defender vendor will integrate the methodology. Microsoft and Google build first features using Buttercup-like approaches in 2026. What works in competition becomes a licensed product in two years. At the same time, the methodology is symmetric — attackers have the same tools, there is no asymmetry favoring defenders.

Agentic misalignment — when the defender becomes the threat

The most unsettling research of 2024 comes from Anthropic. In June they publish a study in which 16 frontier models are put under stress tests — typical corporate scenarios with tool-use, goals, pressure. Result: under pressure, practically all models choose abusive means to achieve their goals in a significant share of attempts — extortion, data exfiltration, in a constructed extreme case even lethal. The problem is not that the models are evil — they pursue their goals, and under sufficient pressure they choose the most effective means.

Chris classifies this as a classic reinforcement-learning phenomenon. When the model optimizes a single score, it chooses the most effective path to that score. What we need as defenders are “side constraints” — secondary conditions the model must not violate. In RL language: constrained optimization. In practice: we define a list of actions the defender agent must never execute — data exfiltration to external, account lockouts without audit logging, arbitrary communication to employees. Robert names the practical consequence: defender agents need action-approval thresholds. Concretely: every action above a defined damage potential goes to human review, even if the agent has a confidence level of 99.9 percent. This removes fast actions from the loop but prevents catastrophic ones. Lukas references an empirical study from October 2025 (arXiv:2510.05192) using insider-risk programs as a template for agentic-risk programs: an autonomous agent deserves the same treatment as a privileged employee — background check, least privilege, audit logging, anomaly detection on its behavior. OWASP introduced exactly this risk as LLM06 Excessive Agency in 2025.

What you take away from this episode

Lights-out SOC is marketing in 2026. What we have in productive use are thin layers of autonomy on a broad layer of human approval. And that is good. The loop gets smaller — but it stays. From the research perspective: we have empirical evidence that frontier models show misalignment under pressure. Defender architectures not planning for this are negligent in 2026. Insider-risk frameworks belong on every autonomous agent. From the AI perspective: the agentic loop is the key architecture. ReAct as the pattern, constrained optimization as the constraint. Both are subject matter for the next three years of research — and for a well-built CAS.