What is 'data poisoning' in generative AI models?

Data poisoning is a cyberattack where threat actors manipulate or corrupt the training data used to develop artificial intelligence and machine learning models^[1]. Because neural networks, large language models, and deep learning models rely heavily on the quality and integrity of their training data, introducing incorrect or biased data points can subtly or drastically alter a model's behavior^[2].

Impact on Model Outputs

Data poisoning can cause machine learning models to misclassify inputs, which reduces overall efficacy and accuracy^[3]. In consumer-facing applications, this leads to inaccurate recommendations that erode user trust^[4]. Attackers can also target specific demographics to amplify existing biases, resulting in discriminatory or unfair outcomes in sensitive areas like hiring or facial recognition^[5]. Furthermore, these attacks can degrade general model robustness, introduce severe cybersecurity risks in critical industries such as healthcare and autonomous vehicles, and open the door to backdoor threats^[6].

Trust and Vulnerability

Because training data is often scraped from open sources, the internet, or third-party providers, maintaining strict data integrity is challenging^[7]. Attackers use various methods such as label flipping, data injection, backdoor attacks, and clean-label attacks—where malicious modifications remain difficult for traditional validation methods to detect^[8]. This undermines the sanctity and reliability of AI systems, making data validation, sanitization, and adversarial training essential defenses for secure deployment^[9][10].Would you also like to know how clean-label attacks and label flipping differ?AskAnswer complete. One follow-up option available.