Google Threat Intelligence Team warns: LLM kidnappings are on the rise

Enterprises are increasingly integrating advanced large language models (LLM) into enterprise environments.Google Research from the Google Threat Intelligence Group (GTIG) and its security company Mandiant shows that cybercriminals have also turned to an exciting new method called “LLM jacking”, which is hijacking the permissions of an enterprise’s access model to drive malicious actions, or reselling access permissions to other threat actors.

Mandiant reported that this year, it has responded to several incidents in which threat actors have hijacked access to enterprises’ large language models (LLMs). This strategy is called “LLM hijacking”. The breaches involved multiple cloud services and LLM vendors, affecting companies in industries including financial services, retail, technology, and transportation in North America, South America, and Asia Pacific. GTIC and Mandiant released a latest report revealing the operating mechanism of “LLMjacking”. Different from the gradual consumption of computing resources by traditional cryptocurrency mining hijacking (cryptojacking), the expenses related to LLMjacking will increase suddenly and rapidly.

GTIC reported that it had discovered a series of threat groups motivated by financial interests. These clusters operate in many regions around the world. The “LLM hijacking ecosystem” is rapidly maturing, and the division of labor between threat actors and downstream profit infrastructure (LLM mixers) is becoming increasingly apparent. The former is responsible for establishing initial access to the compromised environment, while the latter is responsible for managing the downstream monetization infrastructure. LLM mixers act as illegal routing gateways, proxying, aggregation, and reselling unauthorized LLM access to underground users. During an investigation, GTIG observed two separate operating groups working together to hack and exploit enterprise LLM infrastructure for profit, indicating that a highly specialized division of labor is emerging in this area.

As adoption of autonomous agents and advanced models grows in the enterprise, securing AI credential pipelines has become an urgent priority for enterprise security leaders. John Hultquist, principal analyst on Google’s threat intelligence team, said threat actors expect to use this technology to gain a massive advantage over defenders whenever possible. They do this through proxy tools and by stealing tokens from unsuspecting third parties. Locking down these resources not only protects the organization from the risk of excessive costs, but also protects the broader community from a flood of cheap, autonomous attacks.

Organizations must closely monitor and secure the models and computing resources they are quickly bringing online, otherwise they will be the only ones using them. This means auditing usage, enabling multi-factor authentication, and managing confidential information carefully.