✅ Report Title: AI-powered Threats Case Study #02: Vulnerabilities

The S2W Threat Intelligence Center, TALON, has published an in-depth report analyzing cases of vulnerability exploitation involving generative AI and corresponding response strategies. This report provides a comprehensive overview based on real-world incidents observed on the dark web, examining attempts to exploit vulnerabilities using AI and LLMs, responses from academia and industry, and attacks targeting LLM services themselves.

✅ Executive Summary:

1) Abuse Potential of LLMs and Dark Web Trends

As large language models (LLMs) become increasingly capable, actors on the dark web have begun using them as offensive tools. References to popular models such as ChatGPT, Claude, and DeepSeek are widespread on forums, where they are leveraged for exploit generation, malware development, and circumvention of security safeguards.

2) Key Threat Cases Involving LLMs on the Dark Web

2.1 Exploiting Vulnerabilities Using LLMs

- Case: On January 24, 2025, a user named “KuroCracks” posted on the Cracked forum with the title, “CVE-2024-10914 SCANNER - REMOTE CODE EXECUTION - GREAT FOR BOTNETS - OPEN SOURCE.”

- Details: The post shared a scanner for CVE-2024-10914 and described how it was developed using AI. Portions of Masscan-based automated scanner code were included.

- Analysis: Queries submitted to ChatGPT and similar LLMs, when configured to bypass default safety settings, demonstrated the potential to return responses resembling exploit code. Techniques to circumvent these safeguards are being actively shared on the dark web.

2.2 Distribution of LLM-Related Research

- Forums are increasingly populated with content discussing LLM fine-tuning, prompt engineering, and optimization techniques.

- Academic and industrial publications, including code repositories, are frequently cited, indicating growing interest in applying these insights to malicious use cases.

2.3 Attacks Targeting LLM Infrastructure

- Threat activity is evolving from using LLMs as tools to directly targeting the models and their APIs.

- In February 2025, a threat actor named “MTU1500Tunnel” posted on BreachForums offering to sell an exploit for the Gemini API, claiming it enabled balance manipulation and restriction bypasses (jailbreaking).

2.4 Circumventing Security Constraints

- LLMs are typically equipped with safety layers and security guardrails to prevent malicious use.

- Nonetheless, attempts to disable or bypass these protections have surged, particularly with open-source models.

- A notable example is WormGPT, an LLM fine-tuned for malicious purposes, promoted on cybercrime forums as a “no-restrictions” alternative for attackers.

3) Research Progress on LLM-Based Vulnerability Discovery and Patching

- The U.S. Defense Advanced Research Projects Agency (DARPA) actively supports research through the AI Cyber Challenge (AIxCC), which seeks to develop AI-driven solutions for automated vulnerability detection and remediation in core software.

- LLM development frameworks such as LangChain and the Model Context Protocol (MCP) now support tool integrations, creating more seamless environments for vulnerability analysis and attack automation.

- Beyond analysis, LLMs are also being explored for automated proof-of-concept (PoC) and exploit code generation.

4) Attacks Targeting LLM Services and Security Responses

- Mainstream LLM services such as ChatGPT, Gemini, Claude, and DeepSeek have faced consistent attacks, exposing both traditional cyber threats and novel LLM-specific vulnerabilities.

- Prompt injection is one of the most prominent of these, prompting providers to deploy additional security guardrails.

- Defending against these threats requires a multi-layered strategy, including technical safeguards, real-time threat monitoring, user awareness, and continuous intelligence updates to adapt to the evolving landscape.

✅ Recommendations and Mitigation Measures:

- LLMs offer significant benefits for cybersecurity, including improved efficiency in vulnerability detection and automated patching. However, the increasing abuse of LLMs on the dark web and the emergence of structural vulnerabilities within the models themselves present substantial risks to the security industry.

- LLM service providers must strengthen multi-layered defense systems against prompt injection and guardrail bypass attacks. At the same time, user awareness initiatives and up-to-date threat intelligence must be prioritized. Ethical deployment guidelines and community-led response frameworks should also be actively developed and implemented.

- To fully harness the potential of LLMs while minimizing associated risks, organizations must adopt proactive, rapid-response strategies to effectively mitigate threats arising from the use and misuse of generative AI.

🧑‍💻 Author: S2W TALON

👉 Contact us: https://s2w.inc/en/contact

*The full report is available upon request or with a subscription to the S2W platform.