The rapid advancement of Artificial Intelligence (AI) offers unprecedented opportunities across various sectors, from scientific breakthroughs to cybersecurity enhancements. AI's potential to revolutionize digital defense, empower security professionals, and strengthen our collective security is undeniable. Large Language Models (LLMs), for example, are streamlining operations through their ability to sift through complex data, facilitate secure coding, and expedite vulnerability discovery.
However, the availability of these same AI capabilities to malicious actors has raised concerns about the potential for AI misuse. In a new report, the Google Threat Intelligence Group (GTIG) shares a comprehensive analysis of how threat actors are interacting with Google's AI-powered assistant, Gemini, providing valuable insights into the current state of AI misuse.
Much of the current discussion surrounding cyber threat actors’ misuse of AI remains theoretical. While studies demonstrate the potential for malicious exploitation, they often fail to reflect the reality of how AI is currently being used in the wild. To address this gap, GTIG conducted an extensive analysis of threat actor interactions with Gemini.
This analysis leverages the expertise of GTIG, which combines decades of experience in tracking threat actors and protecting Google, its users, and its customers from government-backed attackers, targeted 0-day exploits, coordinated information operations (IO), and serious cybercrime networks.
Google believes that a collaborative approach involving the private sector, governments, educational institutions, and other stakeholders is crucial to maximizing AI's benefits while minimizing the risks of abuse. Google is committed to developing responsible AI, guided by its AI principles, and regularly shares resources and best practices to promote responsible AI development across the industry.
The company continuously improves its AI models to make them less susceptible to misuse and applies its intelligence to enhance defenses and protect users from cyber threats. Proactive disruption of malicious activity and the sharing of findings with the security community are also key components of Google's strategy to foster a safer internet.
GTIG's report, available for download here, delves into how advanced persistent threat (APT) and coordinated information operations actors are attempting to misuse Gemini. The analysis, which involved a combination of analyst review and LLM-assisted analysis of prompts, revealed several key findings:
The report suggests that generative AI, in its current state, primarily serves as an accelerator for threat actors, enabling them to move faster and at higher volume. For skilled actors, these tools provide a helpful framework, similar to Metasploit or Cobalt Strike. For less skilled actors, they offer a learning and productivity tool, facilitating quicker development and incorporation of existing techniques.
However, GTIG emphasizes that current LLMs are unlikely to enable breakthrough capabilities for threat actors on their own. The AI landscape is constantly evolving, and GTIG anticipates that the threat landscape will adapt in stride as new AI models and agentic systems emerge.
One specific area of concern is the potential for "AI jailbreaks," which are a type of Prompt Injection attack. GTIG observed a handful of cases of low-effort experimentation using publicly available jailbreak prompts in unsuccessful attempts to bypass Gemini's safety controls.
These attacks can cause an AI model to behave in unintended ways, such as outputting unsafe content or leaking sensitive information. Controls against prompt injection include input/output validation and sanitization, as well as adversarial training and testing.
The GTIG analysis also provided valuable insights into the activities of APT actors from different countries:
The GTIG's report provides a valuable snapshot of the current landscape of AI misuse by threat actors. While AI has not yet become a "game-changer" in cyberattacks, it is essential to remain vigilant and continuously adapt security measures as AI technology evolves.
By sharing these findings, Google aims to raise awareness within the security community and enable stronger protections for all, contributing to a safer and more secure digital world.
