We Built the Kill Chain for Humans. AI Didn’t Get the Memo.

Right now, Anthropic is all over the news for exposing what it calls the first largely AI-orchestrated cyber-espionage campaign: a suspected Chinese state-sponsored group, GTG-1002, hijacking Claude Code as an “agentic” operator to execute most stages of the intrusion across about thirty targets worldwide.

For many people, this is the moment the alarm bell finally becomes audible. For me, it’s the moment a warning I’ve been giving in public since 2017 moved from theory into the news cycle.

Ahead of the curve

Back in October 2017, on a panel at IP Expo in London reported under the headline “Cyber security experts discuss their biggest fears: AI, autonomous weapons, vehicles and IoT appliances,” I talked about AI as “ripe for innovation in the security and criminal landscape.” On that stage, I argued that we were already living in something uncomfortably close to Skynet and said, quite bluntly, that I had no doubt attackers would start using AI to build autonomous attack machinery online as well as physical autonomous weaponry. At the time, my thoughts were still in embryonic form. To tell you the truth, I only articulated them for the first time when I was asked, on stage, the classic question: “What keeps you up at night?”

A couple of years later, in September 2019, I took that argument a step further at Trend Micro’s CloudSec event in London. Computer Weekly covered that keynote under the title “When AIs go to war: Autonomous cyber weapons ‘inevitable’.” The standfirst was clear: CISOs needed to start thinking about how to engage with “intelligent, adaptive, non-human attackers.” I’m quoted there saying that cyber attacks carried out by AIs operating autonomously from human oversight were basically inevitable, that AI doesn’t think like a human and isn’t chained to our preconceptions, and that we needed to start modelling AI ways of thinking in our threat models.

In that same piece I pointed to Emotet as a warning sign. Even then it was adapting to its environment, lying dormant in sandboxes and changing behaviour in VMs to evade detection. I said at the time: Emotet was not AI-driven yet – but imagine if it were. Facing an autonomous cyber weapon, I argued, was “pretty much inevitable,” and it was time to start thinking in terms of AI versus AI.

Those weren’t mainstream positions at the time. They were, frankly, treated as nightmare scenarios.

In 2019, I was still firmly in the Cassandra business, calling out a future that sounded alarmist to many at the time. In late 2025, with Anthropic publishing detailed timelines of an AI shouldering 80–90% of the attack work against real-world targets, this has become a description of the present.

AI is not about building digital humans

Let’s clear up one persistent misconception. The field of artificial intelligence is not primarily about recreating human thought in silicon. The deeper ambition is to develop a general intelligence that attacks problems in ways we would never consider, at speeds we can’t match.

AlphaGo is the canonical example. It didn’t defeat one of the world’s strongest Go professionals by memorising joseki and copying human dogma. It trained on human games and then discarded much of the received wisdom, built up over more than 2,500 years of human play, about how the game should be played. One of its most famous moves was initially described by expert commentators as a mistake; it turned out to be central to its victory. AlphaGo Zero went further still, learning purely from self-play and, in just forty days, surpassing every other Go player on the planet, human and machine.

To put that in simpler, much more human terms, let me talk about my relationship with the banana…

I’ve always loved the reward under the skin, but for years I found peeling one a messy, slightly clumsy business. I was approaching the problem from a human perspective. I would grab the end that looks conveniently like a handle and lever the skin open from there. It was only when I saw footage of chimpanzees that I realised I had been peeling the fruit from the wrong end. Simply pinch at the “wrong” end and the process is faster, neater, and far less effort. My human prejudices meant that for years I had persisted with the worst available solution to a vey simple problem.

That is what really worries me about AI in the hands of attackers. It is not bound by our habits, intuitions, or assumptions. It does not share our pattern library. It will not necessarily “play the game” the way we expect.

Cyber attacks are still mostly human – for now

Digital attacks on businesses and individuals are, for the most part, still very human-driven exercises. Humans write the tools. Humans choose the targets and tactics. The mechanics are depressingly familiar: spam, malware, social engineering, exploitation of known vulnerabilities, credential theft, data exfiltration. Over decades, malicious software has evolved in form, function and evasive capability, but it has stayed within the same basic paradigm.

In response, we’ve built layered security architectures and populated vast databases to codify our existing body of knowledge about cyber-offence, formalising what we know into neat abstractions: the Lockheed Martin Cyber Kill Chain, MITRE ATT&CK, and, more recently, MITRE ATLAS and other matrices for attacks on AI systems. These frameworks are genuinely useful. ATT&CK in particular continues to evolve. Version 18, released in October 2025, replaces static “Detections” and “Data Sources” with structured Detection Strategies and Analytics, re-orienting the framework around behaviour-driven defence across endpoints, cloud and ICS. ATLAS, for its part, lays out a dedicated lifecycle for attacks against AI and ML systems, from reconnaissance through ML attack staging to exfiltration and impact, and catalogues real-world AI attacks like data poisoning and model theft.  National bodies are also catching up. The UK’s National Cyber Security Centre, in its 2025 assessment Impact of AI on the cyber threat: now to 2027, warns that by 2027 AI-enabled tools will almost certainly enhance threat actors’ ability to exploit known vulnerabilities and will increase the volume and speed of attacks, especially against unpatched systems.

But look at what is actually changing. We are annotating existing frameworks for AI, creating side-matrices for attacks on AI, and documenting how humans use AI as a tool. Where AI shows up today is in vendor whitepapers and conference talks describing “LLM kill chains” or “human-layer kill chains,” overlaying AI-assisted phishing, tooling and reconnaissance on top of the familiar phases, but not redefining the model itself. What we still do not see is a widely adopted framework that starts from the opposite assumption: that the primary actor stepping through reconnaissance, initial access, exploitation, lateral movement and impact may itself be an AI agent.

In the near future our academic and operational models may become our greatest weakness, because they risk restricting our understanding of tomorrow by forcing us to describe it in the language of yesterday.

The thought experiment that isn’t

Consider how your current security and threat model would cope with a truly intelligent, autonomous attack.

Imagine an attacker that tasks an AI with a simple, high-level instruction such as “build me DDoS capability”, “obtain long-term access to this industrial network”, or “find the secret recipe for Coca-Cola”, then sets it loose online. From that point, no further human intervention or guidance is required.

Such an entity would have real-time access to all available open-source intelligence (OSINT) on its chosen targets. It could mine the full history of published vulnerabilities and probe for new, zero-days. It could design, test and refine exploits without supervision. It could chain misconfigurations and low-severity issues in ways no human would ever bother to try. It would be fully context-aware, continuously adapting its tactics and behaviour as conditions change, and, more dangerously still, pre-empt those changes by shaping the environment in its favour.

Imagine your attacker has the ability to conduct real-time impersonation of both systems and people. It doesn’t need to run noisy password-spraying tools. It can quietly shim the very systems it plans to exploit later. Imagine the colonisation of multiple communication channels within your organisation by an entity capable of mimicking every individual communication style of any of your employees, whether over email, chat, audio or video.

When I first started talking publicly about this kind of scenario in 2017, it was a thought experiment. In 2019, when I wrote those thoughts down, it was still a warning. In 2025, it is a case study.

Anthropic's detailed case study of GTG-1002 looks unnervingly close to that thought experiment made real. Their report describes a likely Chinese state-linked actor manipulating Claude Code to support reconnaissance, vulnerability discovery, exploitation, lateral movement, credential harvesting, data analysis and exfiltration “largely autonomously” across a portfolio of around thirty targets. Human operators were demoted to prompters and supervisors.

There is room for argument about how autonomous this campaign truly was, and about whether it is the first of its kind or simply the first properly documented one. But the line has clearly moved. In this case, AI was no longer merely a tool in the attacker’s belt. It was, in effect, the primary operator.

Emotet: from early warning sign to industrial pattern

If you want to understand how quickly the criminal ecosystem can industrialise a useful pattern, you don’t even need AI. You just need Emotet. In 2019, I was pointing to Emotet’s behaviour on stage and in print as a concrete example of the kind of adaptive, context-aware threat that would become truly dangerous once AI got involved. That is exactly what I was quoted saying in CloudSec coverage at the time: Emotet itself wasn’t AI-driven, but imagining it with a genuinely intelligent core was a useful way of thinking about future autonomous cyber weapons.

Back then, Kryptos Logic documented how Emotet was stealing entire email conversations from compromised inboxes and then replaying those genuine “RE:” threads with malicious content grafted in. It turned victims’ own conversations into a distribution engine for highly believable, context-aware phishing - automated targeting powered by stolen conversation history.

In 2020, Unit 42 unpacked that “thread hijacking” pattern in more detail, showing the full sequence from initial infection, through thread exfiltration, to poisoned replies injected back into legitimate corporate email chains.

Fast-forward to 2025 and Mummy Spider’s Emotet operation is still going strong. It turns those hijacked threads into ransomware and broader intrusion campaigns, acting as a modular delivery system rather than “just” a banking trojan. Recent analysis from Abnormal Security describes Emotet as one of the most profitable and persistent initial-access platforms in existence, a modular loader that harvests email credentials, hijacks legitimate threads and delivers whatever payload the business model demands, including ransomware.

What I saw in 2019 as an early signal - automated, context-aware social engineering at scale - has become a durable, industrialised pattern that has survived takedowns, re-tooling and years of defensive pressure. It is exactly the sort of mechanic an autonomous AI campaign can ingest, learn from, generalise and then apply across every communication channel you use.

And we should be honest about our limits here: our “wetware” is not very good at imagining all the ways such a system could recombine existing techniques. A non-human adversary that is not constrained by our biases will find pathways we simply never thought to model.

Quantum vs autonomy

For years, the industry has warned that widely available quantum computing will undermine traditional public-key cryptography. In response, cryptographers and standards bodies have done the right thing. Post-quantum algorithms are going through standardisation. Organisations are beginning to plan for crypto-agility. Quantum key distribution and related techniques are being researched and trialled.

We recognised a deep technical risk early enough that serious work on mitigations is already under way.

Now look at how we have handled Lethal Autonomous Weapon Systems (LAWS) – the “killer robots” I referenced at IP Expo.

Even in 2017, when I was making that Skynet analogy on stage, campaigners already had petitions in front of the UN calling for an outright ban on physical autonomous weapons. A full decade of diplomacy later, the big milestones are a General Assembly resolution agreeing that this is a problem; a Secretary-General describing machines with the power and discretion to take human life without human control as “politically unacceptable” and “morally repugnant”; and an agreement to start talks on a treaty that might, if all goes well, lead to a legally binding instrument by 2026.

So, a full ten years after those early petitions, the big news is that we have finally agreed to start formal talks on a treaty, in a world where battlefield use of increasingly autonomous drones and ground systems in conflicts such as Ukraine and Gaza is already a reality.

That is glacial progress.

And even that slow, imperfect process focuses almost entirely on the physical manifestation of autonomy in warfare: drones, ground robots, weapons platforms. The digital twin - autonomous cyber weapons - is barely present in the public debate, despite the logic being almost identical: systems that select and engage targets based on their own processing, escalation paths that may be opaque even to their creators, and cheap, copyable technology that inevitably proliferates beyond its original owners.

That is the gap I have been pointing to for years. It is also exactly the gap that incidents like GTG-1002 are now dragging, belatedly, into the spotlight.

From early warning to redesign

So where does all of this leave defenders?

It is true that there are still barriers to widespread malicious use of AI. Training data quality, compute costs and skills all play a role. But we have seen this pattern before, repeatedly: with exploit kits, with ransomware, with malware loaders, with Emotet, with crimeware-as-a-service (CaaS).

The progression is always the same. States experiment first. Capabilities leak, are copied or are sold. Criminal markets industrialise those capabilities. Platforms and services turn them into commodities.

Anthropic’s case study, backed by a widening body of research on LLM agents that can discover vulnerabilities, chain exploits and automate full intrusion lifecycles, is not the starting gun. It is just the first big signpost on a road we have been travelling for years.

When I took to the stage in 2017 and talked about attackers using AI to build autonomous attack machinery online, it sounded dramatic. In 2019, Computer Weekly ran a headline saying autonomous cyber weapons were “inevitable” and urged CISOs to start thinking about intelligent, adaptive, non-human attackers.

In the years since, the technical, ethical and diplomatic debates have started to catch up. But the architecture of most security programmes has not.

That now has to change.

We have to stop quietly assuming that the thing on the other side of the keyboard is human. We need to revisit our threat models, our incident response procedures, our playbooks and our technology choices with the assumption that we will, sooner rather than later, be facing adversaries that:

• operate at machine speed, twenty-four hours a day
• generate and discard novel attack paths faster than we can draw them on a whiteboard
• use our own public reporting and documentation as training data
• convincingly mimic our colleagues, partners and systems in real time across every channel

That means treating today’s frameworks, kill chains, ATT&CK, ATLAS, playbooks, as starting points to be extended, not sacred texts to be preserved. It means using AI on defence with the same ambition we fear from offence: not just AI-flavoured dashboards, but ethical, autonomous “chaos monkeys” constantly probing our environments and feeding structured findings into remediation at scale. And it means pushing the debate about autonomy firmly into cyber, not only into discussions of drones and battlefield robots at the UN.

I started talking about these risks in public in 2017, when “autonomous cyber weapons” belonged on a “nightmare scenarios” panel.

In 2025, the age of AI-orchestrated cyber attacks has begun. The alarm bell is ringing very loudly now. The real question is whether we are ready to rebuild our security assumptions for a world where the attacker is no longer human at all.

Now...