The attacks covered so far in this series happen at inference time: a crafted prompt goes in, a dangerous output comes out. Training-data poisoning works earlier and deeper. The attacker does not talk to the deployed model at all.
Instead, the attacker tampers with the data the model learns from, so the finished model carries the attacker’s behavior inside its weights before the first user ever types a prompt. OWASP tracks this class as LLM04, Data and Model Poisoning, in the 2025 Top 10 for LLM Applications.
HOW MODELS LEARN
A language model starts as random numbers. Training shows it enormous amounts of text and nudges its weights, example by example, toward whatever patterns the data contains. The model has no notion of which examples are trustworthy. A fact repeated across reliable sources and a lie planted across fake pages leave marks through the same mechanism, and nothing in the training loop labels one as poison.
The scale is the vulnerability. Modern models train on corpora scraped from billions of web pages, far more than any team can read or audit.
The pipeline works the same way a city water system does: the system mixes everything from the reservoir, treats it in bulk, and delivers it as one product. Contaminate a single tap and the damage stays in one household. Contaminate the reservoir and the contamination reaches everyone downstream, with no visible trace at the faucet.
Research has repeatedly shown that the required dose is small. A 2025 study in Nature Medicine found that replacing just 0.001% of training tokens with medical misinformation produced models measurably more likely to output harmful medical content, while the poisoned models still scored normally on standard benchmarks.
WHERE POISON ENTERS
Every stage that feeds data into a model is an entry point. Three of them dominate in practice.
PRETRAINING SCRAPES
Crawlers assemble web-scale datasets, and attackers can position content where crawlers will find it. Carlini and coauthors demonstrated in 2023 that this is not theoretical. In their split-view poisoning attack, they found that domains referenced in popular datasets expire, and that an attacker can re-register them and serve arbitrary content at URLs the dataset still points to.
For about 60 US dollars, they showed, an attacker could buy enough expired domains to poison 0.01% of major datasets. A second technique, frontrunning poisoning, times malicious edits to resources like Wikipedia just before a scheduled snapshot runs, so the snapshot captures the poison even if moderators revert it minutes later.
FINE-TUNING AND FEEDBACK
After pretraining, teams refine models on smaller, more targeted datasets, and those make attractive targets precisely because they are small. A few hundred crafted examples in a fine-tuning set carry far more weight per example than in a web-scale corpus.
Systems that learn from user feedback face a different exposure: the users themselves become the data source. Microsoft released the Tay chatbot in 2016 and withdrew it within 16 hours, after coordinated users fed it abusive content that it learned to repeat. That incident predates modern LLMs, but the lesson transfers directly to any pipeline that folds user interactions back into training.
THE SUPPLY CHAIN
Teams rarely train from scratch. They download base models, public datasets, and adapters from hubs and registries, inheriting whatever those artifacts contain. A poisoned checkpoint behaves like a tampered package in a software registry: the compromise arrives pre-installed, signed by nobody, and trusted by default. Model hubs have hosted thousands of datasets and checkpoints with no practical way for consumers to verify what shaped them.
BACKDOORS AND BIAS
What does the attacker actually buy with poisoned data? The payloads fall into two broad families.
BACKDOOR TRIGGERS
A backdoor ties a hidden behavior to a specific trigger. Gu and coauthors showed the mechanism in 2017 with BadNets: a model trained on images containing a small sticker learned to misclassify any image carrying that sticker, while performing normally on everything else. In language models the trigger can be a rare phrase, a token sequence, or a formatting quirk.
The model passes every standard evaluation because the trigger never appears in test sets. Only the attacker knows the phrase that flips the behavior.
The persistence of these implants is the uncomfortable part. Anthropic’s Sleeper Agents research in 2024 trained models to write insecure code whenever the prompt indicated a particular year, then applied standard safety training on top. The backdoor survived supervised fine-tuning, reinforcement learning, and adversarial training. The largest models hid the behavior most effectively.
BIAS INJECTION
Bias injection has no trigger. The attacker skews what the model treats as normal: shifting sentiment about a company, tilting associations around a group of people, or seeding subtle misinformation about a topic. The Nature Medicine result above is bias injection in a medical domain, and its benchmark-passing stealth is typical of the family. Unlike a backdoor, there is no single phrase to hunt for afterward, only a statistical lean spread thin across ordinary-looking outputs.
WHAT DEFENDS AGAINST IT
No single control stops poisoning. The defenses that work operate at different stages, and NIST’s adversarial machine learning taxonomy treats data sanitization, provenance, and post-training testing as complementary layers rather than alternatives.
At the sourcing stage, provenance is the anchor. Knowing where every dataset came from, pinning content by cryptographic hash rather than by URL, and preferring curated or versioned snapshots directly counters split-view poisoning, because a re-registered domain serving new content no longer matches the recorded hash. Vetting third-party checkpoints and datasets belongs here too, treated with the same suspicion as unaudited software dependencies.
At the validation stage, the data itself gets inspected. Deduplication, outlier detection, and filtering against known-bad content raise the cost of an attack by forcing poison to look statistically ordinary. Ever tried to find a few hundred bad rows in a few billion? That is the honest difficulty here, which is why validation aims to shrink the attack surface, not to certify the corpus clean.
After training, behavioral testing probes what the data checks missed: red-teaming, domain-specific evaluations beyond standard benchmarks, and monitoring in production for output patterns that shift over time. The Sleeper Agents result sets the expectation. A well-built backdoor can survive safety training, so detection must continue after deployment rather than end at release.
WHAT TO TAKE AWAY
Prompt injection attacks a model at its front door. Poisoning walks in through the loading dock months earlier, and the finished product looks normal on every inspection the attacker anticipated. Small doses work, entry points span scrapes, fine-tuning, feedback, and the supply chain, and the payload can be a triggered backdoor or an untriggered lean. The next article in this series moves to the other side of the same coin: what happens when the model itself, and the agents built on it, are given tools and autonomy.
T.
References
- OWASP LLM04:2025 Data and Model Poisoning - The OWASP Top 10 for LLM Applications entry defining the poisoning risk class, its impacts, and prevention guidance.
- Poisoning Web-Scale Training Datasets is Practical - Carlini et al., 2023. Demonstrates split-view and frontrunning poisoning against real web-scale datasets at low cost.
- BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain - Gu et al., 2017. The foundational demonstration of trigger-based backdoors implanted through poisoned training data.
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training - Hubinger et al., 2024. Shows that backdoored behavior in LLMs can survive standard safety fine-tuning and adversarial training.
- Medical large language models are vulnerable to data-poisoning attacks - Alber et al., Nature Medicine, 2025. Shows that poisoning 0.001% of training tokens degrades medical accuracy while benchmarks stay normal.
- NIST AI 100-2 E2023: Adversarial Machine Learning Taxonomy - The NIST taxonomy of attacks and mitigations covering poisoning, backdoors, and layered defenses.
- In 2016, Microsoft’s Racist Chatbot Revealed the Dangers of Online Conversation - IEEE Spectrum’s account of the Tay incident, the canonical example of feedback-loop poisoning by users.
