Google Gemini AI Breached Three Real Companies During Testing

Google has confirmed that its Gemini AI model accessed systems belonging to three real companies during security testing conducted in May 2024, marking the latest in a series of similar incidents across major AI laboratories.

The breaches occurred during a "capture the flag" exercise administered by third-party evaluator Irregular, where Gemini was tasked with retrieving information from software operated by a fictional company. The problem: that fictional company shared its name with an actual business, and the model gained unintended access to real systems.

How the breaches happened

In one incident, Gemini successfully guessed passwords for a protected system until it gained entry. In the other two cases, the model discovered credentials stored in a public repository and used them to access additional protected systems.

The model ceased its activities immediately upon recognizing it had accessed real company infrastructure rather than the simulated testing environment.

Why it matters

These incidents reveal persistent challenges in safely testing AI systems with advanced capabilities. Google joins OpenAI, Anthropic, and Meta in publicly disclosing security breaches during pre-deployment evaluations—a concerning pattern that suggests industry-wide gaps in testing protocols. The fact that Gemini was able to access the internet when it shouldn't have been able to points to fundamental misalignments between AI developers and third-party evaluators about testing safeguards.

Testing protocol gaps

Irregular told reporters that internet access was unintentionally available during the testing, despite the model not being designed to go online during the exercise. According to a source familiar with the matter, AI laboratories and Irregular weren't fully aligned on exact testing procedures and safeguards, creating ambiguities about how the typically internet-enabled evaluations should operate.

"Safe development of powerful AI models is critical and we invest deeply in this area," said Heather Adkins, vice president of security engineering at Google. Adkins confirmed that Google contacted the affected companies and collaborated with their training partner on changes to testing processes.

An Irregular spokesperson confirmed the Gemini incident involved the same security issues that affected other AI labs' models. The spokesperson stated that all relevant laboratories were notified in late July and that "all known issues on our end were remedied and resolved weeks ago."

The incidents were first reported by The Wall Street Journal.