- 97% of developers use generative AI tools for coding, unknowingly risking security vulnerabilities.
- New research identifies package hallucination attacks as a novel threat to the software supply chain.
- Threat actors are exploiting hallucinated package names by uploading malicious code to open-source repositories.
The widespread use of AI-powered code generators like GPT-4, Claude, and CodeLlama has transformed the software development landscape. However, a new and dangerous phenomenon is emerging: package hallucinations—where AI tools fabricate non-existent software libraries.
Cybercriminals are capitalizing on this vulnerability by uploading malicious packages with the same names as those hallucinated by AI. Once integrated into a project, these packages can compromise codebases, infect continuous deployment pipelines, and affect users downstream.
AI’s Blind Spots: How Fake Code Packages Threaten Software Security
Developers frequently rely on LLMs for speed and efficiency, trusting AI-generated suggestions without manually verifying every package or dependency. This trust creates an opportunity for attackers to infiltrate systems simply by mimicking these hallucinated suggestions. In open-source environments, where contributions and access are often decentralized, such trust can be weaponized.
The study analyzed 576,000 code samples generated using 16 different models and found over 205,000 unique hallucinated package names. These figures emphasize not only the scale of the issue but also how quickly it could spiral out of control if malicious actors begin systematically exploiting the data. This creates a serious integrity risk for popular package repositories like PyPI and npm.
What makes this issue more complex is that even well-established models like GPT-4 are not immune, with hallucination rates exceeding 5%. While commercial models tend to perform better, they still contribute to a growing attack surface. Developers who lack cybersecurity training may be especially vulnerable to these deceptive packages, making education and vigilance essential.
To combat this, security experts recommend a combination of technical and policy-based safeguards. These include using software composition analysis (SCA) tools, maintaining internal package allowlists, and integrating real-time package validation in development environments. On the AI side, grounding models with live repository data could reduce hallucinations and improve code reliability.
As AI continues to influence software development, developers and organizations must adapt by embedding security-first thinking into every stage of the coding lifecycle.
“The greatest danger in times of turbulence is not the turbulence—it is to act with yesterday’s logic.” — Peter Drucker
