AI for Social Engineering

What this episode is about

This episode makes the thesis from episode two — scale beats sophistication — concrete on the attack type everyone knows from daily life: social engineering. Phishing, business email compromise, voice phishing, pig butchering. The question is no longer whether AI makes phishing better. That is answered. The question is what the new volume and quality class means — and how to defend. The cold open features a 30-second audio demo of a fictitious AI-generated phishing call our editorial team produced in under two minutes.

The economics of the dark-web LLM

Lukas walks through the history. WormGPT appears in July 2023 on a Russian-language underground forum — a fine-tuned open-source model, presumably GPT-J based, with safety layers removed, subscription from sixty euros per month. FraudGPT follows, presumably from the same actor. Both are withdrawn at the end of 2023 under law enforcement pressure. Robert highlights this was not the main channel — more a marketing stunt. The actual main channel since 2024 is different: in October 2024 and February 2025, Cato Networks documents two new variants on BreachForums, both named WormGPT. What has changed is the build plan: they are no longer self-trained models, but wrappers around commercial frontier models. One variant bypasses safety layers in xAI’s Grok via manipulated system prompts; the other uses an unfiltered Mixtral variant. The business model is no longer “build an evil LLM” — it’s “operate a good LLM the wrong way.”

The numbers are clear

The IBM Cost of a Data Breach Report 2025 provides the robust values. Global average damage per breach: 4.44 million dollars. Phishing overtakes stolen credentials as the most common initial vector — 16 percent of all breaches, damage volume 4.8 million dollars. The killer number: the time to write a convincing phishing email has dropped from 16 hours to 5 minutes. Factor of two hundred. Lukas frames the defender side, also from IBM: organizations with extensive AI-defense use save 1.9 million per breach and shorten the lifecycle by 80 days. Shadow AI — uncontrolled AI use without IT approval — costs an additional 670,000 dollars per breach. 97 percent of organizations with AI security incidents had inadequate access controls.

The FBI IC3 report 2024, published in April 2025, complements the picture. Total cybercrime damage in the USA: 16.6 billion dollars. Business email compromise alone: 21,442 reports, 2.8 billion damages. Cumulated since 2015, BEC losses have risen by over a thousand percent — 17.1 billion over the past decade. Robert confirms from in-house telemetry: BEC is now the single threat class that most frequently leads to real financial damage in Swiss engagements — before ransomware, before data exfiltration.

Conversational phishing and the Swiss case

Heiding and Schneier show in IEEE Access 2024 that LLM-generated phishing emails achieve over 50 percent click-through — significantly above human baselines. And the same LLM class detects these emails only in a fraction of cases. Chris calls this asymmetric detection gap the generator-detector gap — it does not close on its own. Lukas points to a study published in December 2025 by Roy, Naragam and Nilizadeh: in controlled weekly studies of pig-butchering scenarios, LLM agents reach 46 percent compliance with critical requests — transfer, identity data, investment confirmation. Human operators: 18 percent. More than double. Chainalysis quantifies the volume: 9.9 billion dollars to pig-butchering wallets in 2024, projection of 12.4 billion for 2025, number of deposits up 210 percent.

Robert describes anonymously a Swiss critical-infrastructure case from the energy sector: a treasurer receives an email from the supposed CFO referencing a real acquisition negotiation announced in the financial press two days earlier. 480,000 francs to be transferred to a new supplier account. Saved by the four-eyes principle — the second person asks, calls, the CFO knows nothing about it.

What you take away from this episode

Three measures, in this order: out-of-band verification for everything above a defined threshold — callback to a known number from the supplier directory, not from mail or chat. Code words between management and treasury, between family members for voice-cloning scenarios. And consolidated identity telemetry — XDR instead of isolated EDR islands, sequence analysis across mail, identity provider and endpoint. The central message: linguistic anomaly as a detection feature is yesterday. Identity and behavior are today. In the next episode we deepen this at the most emotional topic of the series — deepfakes: voice, face, trust.