WormGPT: The First Fraud-Focused LLM Goes on Sale

Quick Answer — In July 2023, researchers at Hoxhunt flagged WormGPT, a GPT-J-derived large language model marketed on criminal forums as “a blackhat’s ChatGPT”: no ethics filters, no refusal, purpose-built for BEC spam and phishing. The subscription bot (€500/month, per researcher screenshots) was the first widely-documented fraud-focused LLM — proof that generative AI’s cost curve had fallen far enough that offensive automation became a service industry, and the moment enterprise phishing-defense assumptions had to be rewritten from “spot the broken grammar” to “detect the fluent, personalized, machine-scaled lure”.

What happened

• The product: WormGPT was built on the open-source GPT-J 6B architecture (2021-vintage), trained — per its author’s own marketing — on malware-related data, with no alignment tuning whatsoever. Its selling points were the absence of refusals and “all illegal content” available on demand.
• The pitch: Forum threads showed the author demonstrating BEC email generation, phishing-kit copy, and malicious code snippets; pricing tiers ran from forum access to subscription API via Telegram bot (€500/month reported by Hoxhunt’s screenshots of the sales thread).
• The lifecycle: Hype peaked within weeks; by August 2023 the author claimed to be shutting down (a “commitment to a legal project”), amid speculation the exit was driven by law-enforcement heat, scammers scamming scammers, and the plain fact that jailbroken mainstream models often outperformed it.
• The market response: Security vendors (Abnormal, SlashNext, others) published detection analyses; “WormGPT” became shorthand for the whole criminal-LLM category that followed — FraudGPT, EvilGPT, WolfGPT and dozens of copycats.

Why a 6B model mattered

Timeline

Defensive lessons

• The grammar tell is dead. WormGPT-class tools exist precisely to fix the typo-ridden BEC email; detection must move to intent, context, and channel anomalies — never to prose quality.
• Open weights are a dual-use commodity. Any 2021-era open model could be repurposed; assume adversaries rent frontier-adjacent capability at commodity prices and plan controls for fluency, not brokenness.
• Criminal SaaS has marketing departments. The screenshots, tiers, and “shutting down” drama were textbook underground commerce behavior — threat-intel teams should track criminal products like commercial vendors (changelogs, pricing, SLAs).
• Hype ≠ capability. Independent tests found WormGPT often worse than jailbroken GPT-4-class models; risk communication should separate the category’s trajectory from any single product’s prowess.

The economics of a criminal foundation model

The uncomfortable arithmetic: GPT-J’s weights were free, fine-tuning cost a few hundred dollars of GPU time, a Telegram bot is infrastructure anyone can rent, and the addressable market — BEC losses ran in the billions per year per FBI IC3 tallies — meant even a tiny conversion rate on a €500/month subscription printed money. Nothing about WormGPT required nation-state resources; it required patience and a payment processor. That is the durable strategic fact: the marginal cost of fluent offensive automation collapsed to commodity levels in 2023, and every defensive budget line since has been catching up to that reality.

Why it still matters in 2026

WormGPT was the category-defining proof that offense would industrialize generative AI the moment weights and GPU hours became cheap. By 2026, the “fraud LLM” is no longer a curiosity: deepfake video calls, voice clones of executives, and agentic phishing kits are table stakes, and the defensive industry WormGPT spawned — LLM-lure detection, behavioral mail analysis, human-in-the-loop verification for money movement — is now core enterprise budget. The name shut down in a month; the market it demonstrated never did.

What was WormGPT actually built on?

GPT-J, a 6-billion-parameter open-source model from EleutherAI, dating to 2021 — deliberately chosen because weights were free, hosting was cheap, and there was no vendor ToS to violate. Everything dangerous about it was the application layer wrapped around an ordinary model.

Did it work?

As a product, marginally: demonstrated BEC emails were fluent but generic, and buyers on criminal forums complained about quality. As a signal, enormously: it drew the defensive industry’s attention and budgets to LLM-generated phishing years before agentic attacks made the threat mainstream-scale.

Is the category still around?

Yes — consolidated. Standalone “evil chatbots” faded; their capabilities merged into full attack platforms (phishing-kit builders with LLM copy generation, deepfake modules, and infrastructure rental bundled as one subscription). WormGPT’s shutdown marked the shift from novelty products to integrated criminal platforms, which is precisely the consolidation legitimate SaaS went through a decade earlier.

Why peg this to July 13, 2023?

That’s the date press coverage of the Hoxhunt research reached global scale — the moment “criminal LLM” became a board-briefing topic. The sales threads predate it by months, which is itself a lesson: criminal tooling adoption runs on forum time, and defenders only see it when it breaks into press time.

Part of the hmmnm.com security-timeline series — one event per month, 2021–2024, indexed here.

Detection and response playbook

WormGPT (2023) was the first commercially marketed criminal LLM: a 6-billion-parameter model, no safety tuning, sold by subscription for phishing and BEC generation. For defenders, it marked the moment offensive AI stopped requiring skill — and the detection playbook shifted from spotting bad grammar to spotting good.

1. Retire the grammar heuristic everywhere it survives. The historic tell of malicious email — broken language, odd phrasing — is gone. AI-generated phishing is fluent, context-aware, and personalized at scale from scraped data. Detection has to move to substance: link and attachment analysis, sender infrastructure reputation, and behavioral signals, because prose quality now signals nothing.
2. Assume personalization from public data. WormGPT pipelines ingested scraped context — role, company, recent news — to tailor messages. The defense-side equivalent is knowing your public footprint: what a synthetic adversary learns from your website, LinkedIn surface, and filings, and rehearsing users against lures built from exactly that material.
3. Watch for BEC pattern evolution. The fraud-focused use cases (invoice redirection, executive impersonation, vendor fraud) concentrate on payment-process seams. Controls that predate AI still stop AI-assisted versions: out-of-band verification for payment changes, dual approval, and supplier-onboarding validation. The model writes better emails; it still cannot answer a callback to a known number.
4. Track the criminal-model market, not just this product. WormGPT vanished (reputedly a scam exit), but successors — FraudGPT and peers — institutionalized the category. Threat-intelligence tracking of criminal AI tools is now standard: knowing which capabilities are rented tells you which fraud patterns will spike.
5. Red-team with the attackers tools. The most honest preparedness: generate realistic AI-phishing against your own users in controlled campaigns, measure click and report rates, and tune training on the failures. Organizations that test with 2023-grade lures are training for a threat that no longer exists.

Where this sits in the broader pattern

WormGPT opened the criminal-LLM era that this series tracks through its consequences: the deepfake executive frauds of the following years, AI-generated identity documents defeating KYC, and the voice-cloning incidents that turned a thirty-second sample into an authorization bypass. The original product was almost irrelevant — a modest model with the guardrails removed — but the demonstration was category-defining: capability without conscience, available by subscription.

The economics follow the same curve as every criminal tooling market: initial hype, quality collapse (WormGPT itself was reportedly an exit scam), and then quiet institutionalization — professional crews integrating LLM assistance into existing pipelines rather than buying standalone products. The mature threat is not a criminal ChatGPT; it is your existing adversaries operating with an editor that never sleeps.

The defensive bookends are covered elsewhere in this series: prompt injection for the model-security side, mfa bypass techniques for where AI-assisted social engineering converges with technical attacks. WormGPT itself was neither sophisticated nor durable — it was the announcement that fluency had become free, and every detection heuristic built on its absence needed to retire.

The bottom line. A 6B model with the guardrails stripped made one permanent change: malicious prose is now flawless and personal. Detection must live in infrastructure and process — verification callbacks, payment controls, sender authentication — because the words no longer confess. And the deeper lesson of the criminal-LLM market: assume your adversaries are early adopters, because the subscription price of fluency is zero and falling.

The red-team point deserves emphasis because it is the one control that improves with the threat: each generation of criminal language models makes test lures more realistic, which makes user training more honest, provided the tests actually use current-generation tools. A phishing simulation library last refreshed years ago measures resilience against a threat that no longer exists — and organizations that know this often find their measured click rate is a comfortable fiction.

Refresh the library this quarter — the threat already did.

Rate your people against this year threat, then repeat next year. That loop is the entire discipline.

Frequently Asked Questions

{“@context”:”https://schema.org”,”@type”:”FAQPage”,”mainEntity”:[{“@type”:”Question”,”name”:”What was WormGPT?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”The first commercially marketed criminal large language model, offered in 2023 by subscription: a roughly 6-billion-parameter model without safety guardrails, marketed for phishing, business-email-compromise, and fraud content generation.”}},{“@type”:”Question”,”name”:”Is WormGPT still available?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”No — the operator reportedly shut it down in an exit scam after selling subscriptions. But it legitimized the criminal-LLM category, and successors like FraudGPT and integrated AI-assisted tooling continued the capability.”}},{“@type”:”Question”,”name”:”How do you defend against AI-generated phishing?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Stop relying on grammar and phrasing tells; use email authentication (DMARC), link and attachment analysis, out-of-band verification for financial requests, and dual approval for payment changes — process controls the AI cannot answer for.”}}]}

Related Reading

• What Studying 33 Historic Cyberattacks Taught Me About Defending Systems Today
• Securing AI Agents in Production: A Practical Checklist After a Year of Real Deployments
• Passwordless in 2026: Why Passkeys Are Finally Killing the Password (and How Attackers Are Adapting)
• Authoritative reference: CISA Known Exploited Vulnerabilities

More on this topic

• KnowBe4 vs a Fake North Korean IT Worker: The AI-Era Insider Case Stud
• Windows Recall: The Privacy Debate Before Launch
• 0ktapus: The SMS Phishing Wave That Hit 130+ Companies

Published by Hmmnm

Hands-on cybersecurity tutorials, CVE breakdowns, and guided learning paths — written and lab-tested by the Hmmnm team.

Learning Paths · About Hmmnm · Editorial policy