AI IN CYBERCRIME: FROM WORMGPT TO MACHINE-WRITTEN PHISHING

Generative AI didn’t invent cybercrime; it democratized it. Since mid-2023, jailbroken “dark LLMs” have been sold openly to criminals, AI now writes the majority of phishing emails, and the models businesses deploy are themselves under attack. The through-line: AI doesn’t create fundamentally new attacks so much as let far more people run sophisticated ones, far faster.

Thank you for reading this post, don't forget to subscribe!

Crime-as-a-service

WormGPT appeared on a hacking forum in June 2023, built on the open-source GPT-J 6B model and fine-tuned on malware and phishing material, sold by subscription for €60–100 a month. FraudGPT followed weeks later on dark-web Telegram channels, advertising phishing kits, malicious code, and hacking tutorials. The original tools were disrupted, but the model held: new WormGPT variants riding on Grok and Mixtral were found on underground markets through early 2025.

The phishing explosion

AI removed the tells that used to give phishing away — bad grammar, clumsy phrasing — and added scale and personalization. The result: AI-written phishing now dominates inboxes, business-email-compromise incidents rose with AI assistance (the FBI’s IC3 noted a 37% jump in AI-assisted BEC), and voice phishing became the fastest-growing vector. Over 80% of cyberattacks now involve AI in some part of the kill chain.

“AI doesn’t create fundamentally new attacks so much as let far more people run sophisticated ones, far faster.”

Attacking the models themselves

As businesses bolt LLMs onto products, the models become a new attack surface. Prompt injection — slipping hidden instructions into content the model reads — tops OWASP’s LLM risk list, and research shows jailbreak success rates above 90% against systems without strong guardrails. Even state actors are in: in early 2024 OpenAI and Microsoft disrupted five state-affiliated groups using LLMs for reconnaissance and scripting.