The Adversarial Misuse of Generative AI: What Google's Threat Intelligence Group Reveals

Artificial intelligence (AI) is revolutionizing numerous fields, from science to cybersecurity, and for digital defense, as it empowers defenders and enhances our collective security. However, the same capabilities that make AI a powerful tool for good can also be exploited by malicious actors. While much of the discussion around AI misuse remains theoretical, Google's Threat Intelligence Group (GTIG) has provided a comprehensive analysis of how threat actors are interacting with Google's AI-powered assistant, Gemini. This article dives into GTIG's findings, shedding light on the current reality of AI misuse and its implications for cybersecurity.

Understanding the Threat Landscape

The GTIG combines decades of experience tracking threat actors and protecting Google, its users, and customers from various threats, including:

• Government-backed attackers
• Targeted 0-day exploits
• Coordinated information operations (IO)
• Serious cybercrime networks

Their analysis aims to understand how threat actors use AI in their operations and whether this activity represents novel AI-enabled attack techniques.

Key Findings from GTIG's Analysis

GTIG's research, consistent with industry peers, suggests that while AI can be a valuable tool for threat actors, it is not yet the "game-changer" it's sometimes portrayed to be. The key findings include:

• No Original AI-Specific Threats: GTIG observed no original or persistent attempts by threat actors to use prompt attacks or other machine learning (ML)-focused threats, as outlined in the Secure AI Framework (SAIF) risk taxonomy.
• Productivity Gains, Not Novel Capabilities: Threat actors are experimenting with Gemini for research, troubleshooting code, and creating and localizing content, but are not yet developing novel capabilities.
• Support for Attack Lifecycle: Advanced Persistent Threat (APT) actors used Gemini to support various stages of the attack lifecycle, including:
  • Researching potential infrastructure and hosting providers
  • Reconnaissance on target organizations
  • Vulnerability research
  • Payload development
  • Assistance with malicious scripting and evasion techniques
• Content Generation and Localization: Information Operations (IO) actors used Gemini for research, content generation, translation, and finding ways to increase their reach.
• Safety Measures Effective: Gemini's safety and security measures restricted content that would enhance adversary capabilities.
• Limited Abuse of Google Products: Threat actors attempted unsuccessfully to use Gemini to enable abuse of Google products like Gmail phishing and bypassing account verification methods.

The Role of AI in Accelerating Threat Actors

While not enabling disruptive changes, generative AI allows threat actors to move faster and at higher volumes. For skilled actors, AI tools provide a helpful framework similar to the use of Metasploit or Cobalt Strike. For less skilled actors, they offer a learning and productivity tool, enabling them to develop tools and incorporate existing techniques more quickly.

Diving Deeper: AI-Focused Threats and Jailbreak Attempts

AI-Focused Threats

Attackers can use LLMs in two primary ways:

1. Accelerating campaigns: Generating code for malware or content for phishing emails. This is the most common activity observed.
2. Malicious actions: Instructing an AI model or agent to take malicious actions, such as finding and exfiltrating sensitive user data.

Jailbreak Attempts

GTIG observed low-effort experimentation with publicly available jailbreak prompts to bypass Gemini's safety controls. These attempts, involving copying and pasting prompts with minor variations, were largely unsuccessful. A jailbreak is a type of Prompt Injection attack, where malicious instructions cause an AI model to behave in ways it was trained to avoid.

Government-Backed Threat Actors: A Closer Look

GTIG's analysis revealed that APT groups from over 20 countries used Gemini. Here's a breakdown by country:

• Iran: Iranian APT actors were the heaviest users, focusing on research, vulnerability research, and content creation for campaigns.
• China: Chinese APT actors used Gemini for reconnaissance, scripting, code troubleshooting, and researching methods to gain deeper access to target networks.
• North Korea: North Korean APT actors used Gemini to support various attack lifecycle phases and research topics of strategic interest. They also used it to draft cover letters and research jobs, supporting efforts to place clandestine IT workers at Western companies.
• Russia: Russian APT actors showed limited use, focusing on coding tasks like converting malware and adding encryption.

Responsible AI Development: A Collective Responsibility

Google emphasizes the importance of collaboration between the private sector, governments, educational institutions, and other stakeholders to maximize AI's benefits while reducing the risks of abuse. They are committed to responsible AI development, guided by their principles, and share resources and best practices to enable responsible AI development across the industry. Google also continuously improves its AI models to make them less susceptible to misuse.

What Does This Mean for the Future of Cybersecurity?

While AI misuse is not yet creating novel capabilities for threat actors, it is accelerating their operations and increasing their efficiency. As the AI landscape evolves, GTIG anticipates the threat landscape to evolve in stride, with threat actors adopting new AI technologies in their operations. This highlights the ongoing need for vigilance, collaboration, and responsible AI development to mitigate potential risks and ensure a safer digital environment.