Filling in the Blanks

The hidden assumptions we make about AI

In daily life, we all tend to explain to ourselves how things work. Let’s take the example of example of a car. Most people know that pressing the accelerator makes it go faster, and pushing the brake makes it stop. We might have some idea of what happens under the hood to make this happen, but only a trained mechanic will understand the engine system behind it. This is the concept of mental models: the ideas we have about ourselves, others, our environment, and the things we interact with.

The same idea applies to computer systems. The black box effect means that many AI systems make decisions in ways that are difficult or even impossible for humans to interpret. So how do we cope when interacting with AI systems we don’t fully understand?

A mental model represents the user’s – often simplified – understanding of how an AI system works. Mental models are usually not technically accurate. They are our own, internal representations of how a system works. For complex AI technologies, our mental models can get particularly incoherent and incomplete. Even technically experienced users tend to carry simplified understandings for complicated systems.

Our brains do not like uncertainty. When faced with gaps in our understanding, we tend to fill in these gaps ourselves. Where our knowledge ends, our brain fills in the blank spaces. We base these explanations on known patterns, knowledge and experience. The challenge is that we rarely notice when we have filled a gap, because once filled, it no longer feels like one. The result is that people may overestimate their own understanding of how an AI system works, creating a false sense of confidence that is difficult to detect and correct.

For example, a finance worker reviewing an AI-generated credit score might assume the system weighted income most heavily, because that is how their previous non-AI system worked. Or a recruiter using an AI screening tool might assume that it is looking for the same keywords that they were trained to look for themselves. These assumptions feel reasonable. In the case of deep learning algorithms, they are often wrong. Such algorithms are non-linear and follow patterns that may not align with human logic and reasoning.

It can become problematic when a user’s mental model and the actual workings of the system are too far apart. A flawed mental model increases the risk of wrong usage and mismatched expectations, which may lead to unexpected outputs that can reduce user trust in the system.

Let’s take the well-known example of viral social media posts where Large Language Models (LLMs) are asked to count how often the letter R appears in the word “strawberry”. LLMs often get these questions wrong, as they process language in tokens, not letters. A mental model where the LLM is expected to process language in the same way a person would, leads to expectations that the system was never designed to meet. The result is that trust in the system is reduced: “AI is stupid. AI doesn't work.”

It works the other way around as well. To some users, AI has become synonymous with LLMs. These users may extrapolate their understanding of how LLMs work to all AI technologies, even to technologies that have nothing to do with Natural Language Processing. As a result, the internal logic and decision-making processes of different AI technologies gets confused.

This problem can be made worse by how AI is marketed and communicated. Artificial Intelligence is an umbrella term that covers a huge range of technologies. The term refers to any system that mimics human intelligence and human decision-making, which can range from clustering and regression tasks to generative AI. These models work in fundamentally different ways, with very different logic and decision-making processes. Yet non-technical end users are rarely informed about the specific model they are interacting with. The applications are presented to users under the same label. AI-assisted. AI-powered. AI-enhanced. But when every system is marketed as “AI-assisted” regardless of the underlying technology, it becomes nearly impossible for users to build an accurate mental model for any of them.

For this reason, the call for organisations to invest in AI literacy and enablement is more urgent than ever. Knowledge gaps don’t stand empty. If your organisation doesn’t provide the accurate context, employees will build their own understanding from whatever is available: how the tool was marketed to them, social media, or their experience with previous systems. This could result in a false sense of confidence and understanding, and potentially inappropriate use of the system.

The logical response to all of this may seem straightforward: if users don't understand how AI systems work, we should explain it to them. But is it as easy as that?

Next week, we'll look at what happens when we try to open the black box and how users interact with explanations.

---

References

Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2023). Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Frontiers in Computer Science, 5, 1096257. https://doi.org/10.3389/fcomp.2023.1096257

Sanchez, T., Vereschak, O., & Deroy, O. (2026). Mental Models in Human-AI Interaction: Systematic Review of Empirical Methodologies and Guidelines. Proceedings of the 31st International Conference on Intelligent User Interfaces, 663–682. https://doi.org/10.1145/3742413.3789223