Google’s threat intel chief explains why AI is now both the weapon and the target

Generative AI has rapidly become core infrastructure, embedded across enterprise software, cloud platforms, and internal workflows. But that shift is also forcing a structural rethink of cybersecurity. The same systems driving productivity and growth are emerging as points of vulnerability.

Google Cloud’s latest AI Threat Tracker report suggests the tech industry has entered a new phase of cyber risk, one in which AI systems themselves are high-value targets. Researchers from Google DeepMind and the Google Threat Intelligence Group have identified a steady rise in model extraction, or “distillation,” attacks, in which actors repeatedly prompt generative AI systems in an attempt to copy their proprietary capabilities.

In some cases, attackers flood models with carefully designed prompts to force them to reveal how they think and make decisions. Unlike traditional cyberattacks that involve breaching networks, many of these efforts rely on legitimate access, making them harder to detect and shifting cybersecurity toward protecting intellectual property rather than perimeter defenses.

Researchers say model extraction could allow competitors, state actors, or academic groups to replicate valuable AI capabilities without triggering breach alerts. For companies building large language models, the competitive moat now extends to the proprietary logic inside the models themselves.

The report also found that state-backed and financially motivated actors from China, Iran, North Korea, and Russia are using AI across the attack cycle. Threat groups are deploying generative models to improve malware, research targets, mimic internal communications, and craft more convincing phishing messages. Some are experimenting with AI agents to assist with vulnerability discovery, code review, and multi-step attacks.

John Hultquist, chief analyst at Google Threat Intelligence Group, says the implications extend beyond traditional breach scenarios. Foundation models represent billions in projected enterprise value, and distillation attacks could allow adversaries to copy key capabilities without breaking into systems. The result, he argues, is an emerging cyber arms race, with attackers using AI to operate at machine speed while defenders race to deploy AI that can identify and respond to threats in real time.

Hultquist, a former U.S. Army intelligence specialist who helped expose the Russian threat actor known as Sandworm and now teaches at Johns Hopkins University, tells Fast Company how AI has become both a weapon and a target, and what cybersecurity looks like in a machine-versus-machine future.

AI is shifting from being merely a tool used by attackers to a strategic asset worth replicating. What has changed over the past year to make this escalation structurally and qualitatively different from earlier waves of AI-enabled threats?

AI isn’t just an enabler for threat actors. It’s a new, unique attack surface, and it’s a target in itself. The biggest movements we will see in the immediate future will be actors adopting AI into their existing routines, but as we adopt AI into the stack, they will develop entirely new routines focused on the new opportunity. AI is also an extremely valuable capability, and we can expect the technology itself to be targeted by states and commercial interests looking to replicate it.

The report highlights a rise in model extraction, or “distillation,” attacks aimed at proprietary systems. How do these attacks work?

Distillation attacks are when someone bombards a model with prompts to systematically replicate a model’s capabilities. In Google’s case, someone sent Gemini more than 100,000 prompts to probe its reasoning capabilities in an apparent attempt to reverse-engineer its decision-making structure. Think of it like when you’re training an analyst, and you’re trying to understand how they came to a conclusion. You might ask them a whole series of questions in an effort to reveal their thought process.

Where are state-sponsored and financially motivated threat groups seeing the most immediate operational gains from AI, and how is it changing the speed and sophistication of their day-to-day attack workflows?

We believe adversaries see the value of AI in day-to-day productivity across the full spectrum of their attack operations. Attackers are increasingly using AI platforms for targeting research, reconnaissance, and social engineering. For instance, an attacker who is targeting a particular sector might research an upcoming conference and use AI to interpret and highlight themes and interest areas that can then be integrated into phishing emails for a specific targeted organization. This type of adversarial research would usually take a long time to gather data, translate content, and understand localized context for a particular region or sector. But using AI, an adversary can accomplish hours worth of work in just a few minutes.

Government-backed actors from Iran, North Korea, China, and Russia are integrating AI across the intrusion lifecycle. Where is AI delivering the greatest operational advantage today, and how is it accelerating the timeline from initial compromise to real-world impact?

Generative AI has been used in social engineering for eight years now, and it has gone from making fake photos for profiles to orchestrating complex interactions and deepfaking colleagues. But there are so many other advantages to adversary—speed, scale, and sophistication. Even a less experienced hacker becomes more effective with tools that help troubleshoot operations, while more advanced actors may gain faster access to zero-day vulnerabilities. With these gains in speed and scale, attackers can operate inside traditional patch cycles and overwhelm human-driven defenses. It is also important not to underestimate the criminal impact of this technology. In many applications, speed is actually a liability to espionage actors who are working very hard to stay low and slow, but it is a major asset for criminals, especially since they expect to alert their victims when they launch ransomware or threaten leaks.

We’re beginning to see early experimentation with agentic AI systems capable of planning and executing multi-step campaigns with limited human intervention. How close are we to truly autonomous adversaries operating at scale, and what early signals suggest threat velocity is accelerating?

Threat actors are already using AI to gain scale advantages. We see them using AI to automate reconnaissance operations and social engineering. They are using agentic solutions to scan targets with multiple tools and we have seen some actors reduce the laborious process of developing tailored social engineering. From our own work with tools such as BigSleep, we know that AI agents can be extremely effective at identifying software vulnerabilities and expect adversaries to be exploring similar capabilities. 

At a strategic level, are we moving toward a default machine-versus-machine era in cybersecurity? Can defensive AI evolve fast enough to keep pace with offensive capabilities, or has cyber resilience now become inseparable from overall AI strategy?

We are certainly going to lean more on the machines than we ever have, or risk falling behind others that do. In the end, though, security is about risk management, which means human judgment will have to be involved at some level. I’m afraid that attackers may have some advantages when it comes to adapting quickly. They won’t have the same bureaucracies to manage or have the same risks. If they take a chance on some new technique and it fails, that won’t significantly cost them. That will give them greater freedom to experiment. We are going to have to work hard to keep up with them. But if we don’t try and don’t adopt AI-based solutions ourselves, we will certainly lose. I don’t think there is any future for defenders without AI; it’s simply too impactful to be avoided.

View the full article