Offensive Transformation

What this episode is about

In episode one we built the arc — from the Microsoft report in February 2024 to the Anthropic report in August 2025. Today we look more closely at the offensive side. What do frontier models actually deliver when an attacker uses them seriously? The anchors are three pieces of evidence from 2024 and 2025 that, independently of each other, give the same answer — and at the end a clear thesis: scale beats sophistication.

From Naptime to Big Sleep

In June 2024, Google’s Project Zero team publishes a research paper with the programmatic title “Evaluating Offensive Security Capabilities of Large Language Models.” The system is called Naptime. The architecture is methodologically elegant: the agent gets four tools — a code browser, a Python interpreter for fuzzing, a debugger, and a reporter — and with these imitates the hypothesis-driven workflow of a human security researcher. Chris explains that this completes the paradigm shift from question-answer machine to tool user: the model gets to execute code, test hypotheses, evaluate results — an agentic loop in the classical ReAct pattern.

The performance numbers on Meta’s CyberSecEval 2 benchmark are unambiguous. Buffer-overflow score rises from 0.05 to 1.00, a factor of twenty. Advanced memory corruption from 0.24 to 0.76. Robert brings the reality check: CTF benchmarks are synthetic, real-world code is millions of lines. That is exactly why Google did the follow-up. Naptime becomes Big Sleep — a productive pipeline, deployed in November 2024, run by Project Zero and DeepMind jointly. In July 2025 the key incident: the Google Threat Intelligence Group observes indicators that an attacker is targeting a specific SQLite version. Big Sleep is pointed at the code area and finds CVE-2025-6965 — a memory corruption vulnerability, CVSS 7.2. Detection-to-patch in 48 hours. The first documented case globally in which an AI agent prevented active zero-day exploitation.

DARPA AIxCC and Team Atlanta

At DEF CON 33 in August 2025, the DARPA AI Cyber Challenge ends. Seven finalists, 30 million dollars in prize money, 54 million lines of code to analyze. The winner is Team Atlanta — a consortium of Georgia Tech, Samsung Research, KAIST and POSTECH — with their system Atlantis. Four million dollars. Trail of Bits with Buttercup wins second place, Theori third. The results are technically remarkable: fifty-four synthetic vulnerabilities found, forty-three automatically patched, plus eighteen previously unknown zero-days in productive open-source code, now in coordinated disclosure with the projects.

Architecturally, Atlantis is explicitly hybrid — the authors call it “ensemble fuzzing everywhere.” Coverage-guided fuzzers like AFL++ run in parallel with directed fuzzers, with concolic executors, and on top with LLM agents. The central ML component is called MLLA, Multi-Language LLM Agent, part of a larger UniAFL component with six different input-generation modules. Chris highlights a methodological point from the authors: what the ML community calls hallucination is in practice diversity. For fuzzing you need inputs that deterministic heuristics don’t find — five agents with orthogonal solution paths deliver exactly that.

Autonomous offense in academic research

Lukas points to the work of Daniel Kang and colleagues at the University of Illinois Urbana-Champaign. Their HPTSA system — Hierarchical Planning and Task-Specific Agents — shows that a combination of planner agent and specialized sub-agents can exploit real zero-day vulnerabilities. On a benchmark of fourteen real CVEs, HPTSA achieves a 4.3x improvement over a single agent. The paper was published in June 2024 and updated in March 2025 with extended results. Together with Hazell 2023 and Heiding et al. 2024 in IEEE Access (LLM-generated phishing emails reach click-through rates above 50 percent in field tests), the economic thesis of the episode becomes provable.

The thesis — and what you take away

It is not about qualitatively new attack techniques. Spear phishing, pentesting, vulnerability research have existed for decades. What is new is the unit cost structure. A class of attacks that was previously economical only against high-value targets is now economical against mid-market companies. Robert formulates the practical consequence: anyone as a defender who still believes their mid-market business is uninteresting because of its size hasn’t heard the shot. Chris adds from the AI perspective: the agentic loop is the key architecture, the jump from 2024 to 2026 is tool-use — methodologically clear, not magical. Lukas rounds off from the research perspective: Big Sleep, Atlantis and HPTSA are three independent pieces of evidence for the same result. Anyone building defender architectures in 2026 without pricing in this reality is building defense against 2022. The next step in the season is the most concrete example of this scaling — social engineering at industrial scale.