Addressing AI-generated misinformation
Minimize the risks and consequences of flawed inference from AI models
AI-generated misinformation is on the rise — and it is a growing concern. According to researchers from Google and several fact-checking organizations, the amount of misleading content produced by AI models has increased dramatically since mid 2023.
Most of us are eager to use AI for enhancing efficiencies, driving innovation, augmenting cyber security, and more. But we need to be aware that AI also has the power to produce misinformation, which can have serious consequences for individuals, businesses, and society.
What kinds of misinformation can AI models produce? Why do they generate inaccurate information? And what should we do as security leaders to safeguard both organizational and public interests?
Some limitations of AI models
The problem of AI-generated misinformation in part stems from the current limitations of AI models. We hear a lot about AI inference, which is the ability for AI models to reason or draw conclusions from new, live data. As I’ve written previously, AI models can’t truly reason yet, and their ability to generate meaningful insights from live data is far from perfect.
What AI models can do very well is deliver fact-based answers. If you ask a large language model (LLM) to tell you the year that Columbus landed in the Americas, it should give you the correct answer — unless there was a problem with how the model was built or trained.
However, an AI model might also deliver an opinion as if it were a fact. So, if you ask whether Europeans had the right to settle the Americas, an AI model could provide an opinion without telling you it’s an opinion. In a business context, a company’s AI-based chatbot could provide customers with product information that is derived more from marketing content than from fact.
An AI model could also present you with multiple, competing opinions, which might be unhelpful in a different way. If you ask an LLM like ChatGPT about who is the greatest baseball player that ever lived, it will present multiple opinions, providing general arguments for each. It’s possible that none of these opinions or arguments are actually “wrong,” but the model might not adequately distinguish better points of view from worse ones.
What causes AI-generated misinformation?
Incorrect answers, opinions presented as facts, and multiple opinions presented as having equal value might all be considered forms of misinformation. This misinformation might be produced intentionally or unintentionally.
Intentional
AI models can be manipulated — or “poisoned” — to produce specific answers. For example, a cyber attacker could poison data used by a certain business to train an LLM. The attacker might want to embarrass the business and damage their reputation by rigging the LLM to produce bad answers in an AI chatbot or AI-based application.
Of course, model poisoning isn’t exclusively the work of cyber attackers. Political candidates or governments might also be intent on spreading disinformation by intentionally introducing data into AI models or using other manipulation techniques to disseminate false information.
Another scenario is that individuals or an organization could intentionally manipulate data or information from their model to reinforce their particular point of view. A television news network might portray a politician in a good light and intentionally exclude certain data in an election model to suggest the politician has a good chance of winning.
For example, the network could selectively choose data points that favor one politician, ignoring less favorable data. The network could construct narratives in a similar way, highlighting factual successes or positive policy proposals while ignoring any controversies or failures of a politician.
Unintentional
Misinformation can also be unintentional. An organization might unknowingly feed a model biased data, leading to distorted conclusions. Take the same television news network example: The network might inadvertently discount some information or add weight to other information without any malicious intent. The result would still be a model producing misleading predictions. In many cases, models deliver misinformation simply because they have insufficient data. You might not have collected enough data to provide accurate answers. In addition, the timing of data input into the model could also lead to problems if the sequence of information is important to decision-making.
Finding the source of misinformation — and determining whether it was the result of a purposeful act — can be very difficult, especially since AI models often exist in a closed, opaque system. With traditional machine learning (ML), you can see and understand the decisions behind the model. But with LLMs, which are built on neural networks, the decisioning logic is masked within hidden layers. LLMs summarize data and — from the user’s perspective — do some magic to provide answers. Users don’t have access to the decision-making process. As a result, they can’t easily find the source of potential misinformation.
Implications for AI misinformation
Individuals might make poorly informed decisions based on AI-generated results. For example, incorrect answers from an AI-based chatbot could lead to poor investment, healthcare, legal, or job decisions.
Corporate leaders could similarly make poor decisions based on AI-generated misinformation. AI models might emphasize the wrong types of cyber security threats, causing security leaders to adjust their strategies in the wrong ways or implement suboptimal solutions. Or more generally, leaders might trust AI-generated insights without sufficient human oversight. The AI models might not account for nuanced human factors or ethical considerations.
Even worse, they could completely miss a threat. Let’s say your team submits eight registry keys as part of a malware sample. Your model might wrongly determine that this is not malware but software. Why? Because no piece of malware that you previously submitted had more than eight registry keys. Consequently, your organization would be vulnerable to any malware that did not fit this previous template.
Misinformation doesn’t stop with a single individual or company — it could have widespread consequences. For example, false information spread about major publicly traded corporations could destabilize financial markets. Similarly, disinformation about a country could affect geopolitical relationships or foreign policy, with AI models potentially exacerbating these issues.
Adjusting our expectations
The issue is not only that AI models produce misinformation, it’s that we assume those AI-generated answers are always correct. In my interactions with security teams at other companies, I’ve noticed that some people believe that AI will provide them with the correct answers and simply fix their problems — and that’s troubling.
AI models aren’t absolute. They do not necessarily provide a single, definitive answer. Take mapping software, for example. If I ask a mapping app to chart a car route to a particular destination a few states away, it might provide multiple options. The software might not have enough data to calculate the fastest path. Or it might recommend a fast route that is unsafe.
The app might also suggest a few options that have distinct advantages and disadvantages. For example, it might present a direct — but boring — freeway route. And it might also suggest a more scenic route that takes a little longer. Which is better? There is no single “right” answer.
We also expect AI models to accurately anticipate what will happen in the future. Models can predict outcomes, but those predictions are based on past events and data. As new and more relevant data emerges, it has to be incorporated into existing models. Even then, the models don’t know for sure what will happen. They can’t intuit what will happen next with 100% accuracy.
Adjusting our expectations for AI is key. But of course, we still need to do all that we can to reduce the likelihood that they generate misinformation.
Combating misinformation
There are multiple strategies that can help us reduce the incorrect information generated by AI models. Taken together, these strategies will enable us to minimize inference errors — even as we temper our expectations. As governments enact new regulations for the use of AI (such as the EU’s AI Act and the FTC’s Operation AI Comply in the US), implementing strategies to combat misinformation will be essential for avoiding fines or other regulatory actions.
Collect more data: Generally, AI model developers with the most data will be in the best position to produce accurate responses. However, collecting the data is just the first step. Model and app builders will need to turn that data into features that can address specific issues.
Implement data quality and integrity measures: Misinformation can result from poor data governance. When AI models are built on incomplete, inaccurate, inconsistent, corrupted, or erroneous / outlier data, they deliver incorrect answers. We need to run regular checks and audits to verify data sources are legitimate and haven’t been tampered with.
Validate outputs: In addition to validating input data, we should filter what LLMs produce while monitoring usage patterns and prompts. Validating outputs enable us to spot misinformation — intentional or unintentional — before we use that misinformation for decision making. We could actually use AI models to validate the output (and input) of other AI models. Doing so would help us make sure that answers are in the range of what’s factually accurate.
Control access to open source models: Shadow AI — in particular, the unauthorized use of open source, public AI models — can amplify the impact of misinformation in an enterprise. Monitoring the use of these models within an enterprise can help protect sensitive information from being exposed and minimize the risk of employees making ill-informed decisions based on misinformation.
Tighten security for internal models: Locking down internal AI models with strict access controls, version tracking, encryption, and digital certificates can help prevent intentional and unintentional poisoning. We should keep a close eye on how models are being used and deployed to catch any tampering or theft attempts.
Prepare for misinformation attacks: All organizations should prepare for misinformation attacks. Just like preparing for a DDoS or ransomware attack, we should have a plan for detecting the attack, reporting events, containing the damage, and communicating the issue with customers, partners, and the public. Most of this planning should align with typical crisis communication strategies.
Use human judgement: We need to discern what’s real from what isn’t, and that is becoming more difficult with AI. “Deepfakes,” for example, are becoming much more realistic, and they will continue to improve over time. Meanwhile, individuals and organizations are producing numerous AI-generated fake articles, social media posts, and comments on posts. Unfortunately, the more we encounter an idea or an alleged fact, the more we will believe it’s authentic — even if it isn’t.
Human judgement will be critical for deciding whether we are receiving misinformation or disinformation. We don’t need to know the correct answer to a question (if there is only one correct answer), we just need to determine whether an answer is in a range of possibilities, given our own past experiences. It’s just like when someone exaggerates or tells a wild story — we have to determine what is truth and what is fiction. By creating an environment of critical thinking, transparency, and continuous learning, we can mitigate the risks posed by AI-generated misinformation.
The future of AI misinformation
There’s no doubt that AI will play an increasingly important role in our daily lives and work. As we prepare for the near and distant future with AI, we need to be aware of the potential for AI-generated misinformation and find ways to minimize its consequences.
The growth of artificial general intelligence (AGI) could help us better identify potential problems with models. This field, which attempts to produce more human-like intelligence, could open the closed system of AI models. We might be able to determine how a model arrived at a particular answer — and figure out whether it could be incorrect.
For now, however, some of the most important tools are our own powerful capabilities of human reasoning. We need to be skeptics of information. Just because we read something or watch a video, it doesn’t mean it’s true. The best way to combat misinformation is to use good old-fashioned common sense.
This article is part of a series on the latest trends and topics impacting today’s technology decision-makers.
Dive deeper into this topic.
Learn how to build, use, and secure generative AI at scale in the Ensuring safe AI practices guide for CISOs.
Author
Grant Bourzikas — @grantbourzikas
Chief Security Officer, Cloudflare
Key takeaways
After reading this article you will be able to understand:
What causes AI misinformation
Consequences of making decisions based on flawed inference from AI models
How to combat misinformation