But receiving an explanation and understanding it are not the same thing. We can read one carefully and still not know what it actually tells us. Part of the difficulty is that there is no universal agreement in transparency frameworks on what a clear and meaningful explanation actually looks like. Therefore, the question is not just whether the system provides an explanation, but whether it works for the person reading it.

So what makes an explanation helpful, and how do explanations affect user trust in an AI system?

The relationship between trust and transparency is not as straightforward as it sounds. Having no explanation at all leaves users without the context they need to understand an output. Providing an explanation of the reasoning behind the decision, is known to increase trust in AI predictions. But the solution is not simply to provide as much information as possible. The positive effect of providing an explanation can actually be cancelled out by providing too much technical information.

The challenge lies in finding the right balance between technical accuracy and user-friendliness. Imagine a loan officer presented with a full chart of model weights, feature scores and confidence scores, explaining which factors led to the loan being approved or rejected. Technically, this is the most transparent type of explanation. But for a non-technical end user, it may leave them not knowing where to look or how to make sense of what they're seeing. That's cognitive overload: when explanations stop being helpful and start overwhelming. On the other hand, oversimplified explanations that leave out too much detail can give users a false sense of understanding that may reinforce flawed mental models about how the system works.

Getting this balance right is harder than it looks, because many factors influence how users perceive explanations. When an explanation does not match human expectations, judgement, or intuition, it can reduce trust in the system, even if the system is correct. This is a particular issue for deep learning models, whose decision-making processes may not align with our own mental models of how the system works. Explanations are also subject to confirmation bias: users are more likely to accept an explanation that fits what they already believed, and to challenge one that does not. This means that explanations can reinforce misunderstandings we may have about the system.

How users respond to explanations also depends on their background, the type of decision they are making, and how much they already understand about AI. A business manager with no AI experience and a data scientist reading the same output may walk away with very different levels of understanding.

So what can we do to help users better interpret AI outputs?

First, it is important that we make explanations accessible to non-technical end users and write them in plain English. Think of a supplier risk score: if the explanation behind it is written for a data scientist rather than a procurement team, the risk is that it gets ignored or misread. That said, accessible does not mean dumbed down. Explanations that oversimplify to the point of being misleading can do more harm than providing no explanation at all.

Where possible, we should consider using interactive explanations for complex models. Static information we may not fully understand can leave users with more questions than answers, and may actually reduce trust in the system. Research suggests that allowing users to ask questions about explanations improves understanding and acceptance. Being able to ask a follow-up question to a human expert or digital assistant helps users to better interpret the information. If this kind of interaction is not feasible, even a well-designed FAQ or tooltip can make a difference.

Finally, it is crucial that we invest in AI literacy. Currently, most enablement for end users is focused on how to use the system: how to prompt effectively, how to navigate the interface, etc. There is much less emphasis on understanding the overall logic of the system: why a chatbot responds the way it does, or what a model was designed to do at a conceptual level. This can cause misunderstandings about outputs. Users who have a high-level understanding of how the technology works, are better able to interpret what it tells them. This does not require technical training in coding or mathematics. It means giving employees enough context about how the system works and what it was designed to do, so they can make informed decisions about what to do with AI-generated answers.

The goal of this approach is to allow users to make informed decisions and establish an appropriate level of trust in AI predictions. When users understand what they are working with, they are more likely to apply the right AI tools in the right situations. It allows users to be confident in AI systems without sliding into blind trust. And that is what a responsible, durable adoption of business AI looks like in practice.

This article draws on research conducted for my thesis, which studied the effect of Explainable AI on user trust in high-risk DNN-based AI systems. If you would like more details, please contact me.

References

Bauer, K., Von Zahn, M., & Hinz, O. (2023). Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing. Information Systems Research, 34(4), 1582–1602. https://doi.org/10.1287/isre.2023.1199

Cheong, B. C. (2024). Transparency and accountability in AI systems: Safeguarding wellbeing in the age of algorithmic decision-making. Frontiers in Human Dynamics, 6, 1421273. https://doi.org/10.3389/fhumd.2024.1421273

Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2023). Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Frontiers in Computer Science, 5, 1096257. https://doi.org/10.3389/fcomp.2023.1096257

Kizilcec, R. F. (2016). How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2390–2395. https://doi.org/10.1145/2858036.2858402

Lundberg, S., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions (Version 2). arXiv. https://doi.org/10.48550/ARXIV.1705.07874

Pavlidis, G. (2024). Unlocking the black box: Analysing the EU artificial intelligence act’s framework for explainability in AI. Law, Innovation and Technology, 16(1), 293–308. https://doi.org/10.1080/17579961.2024.2313795

Ribeiro, M., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, 97–101. https://doi.org/10.18653/v1/N16-3020

Souza, G. H. C., De Araujo Wanderley, C., & Braga De Aguiar, A. (2025). The influence of artificial intelligence (AI) transparency on AI acceptance in managerial decision-making. Journal of Management Control. https://doi.org/10.1007/s00187-025-00396-7

Zhang, T., Yang, X. J., & Li, B. (2025). May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability. International Journal of Human–Computer Interaction, 41(9), 5623–5647. https://doi.org/10.1080/10447318.2024.2364986

Thank you for reading AI Ethics Made Practical! Subscribe for fee to receive new posts and support my work.

Subscribe now

Help to spread the message and share this article.

Share

AI Ethics Made Practical is free of charge. But if you found this article helpful and would like to support my work, please consider buying me a coffee!

Buy Me a Coffee

The same idea applies to computer systems. The black box effect means that many AI systems make decisions in ways that are difficult or even impossible for humans to interpret. So how do we cope when interacting with AI systems we don’t fully understand?

A mental model represents the user’s – often simplified – understanding of how an AI system works. Mental models are usually not technically accurate. They are our own, internal representations of how a system works. For complex AI technologies, our mental models can get particularly incoherent and incomplete. Even technically experienced users tend to carry simplified understandings for complicated systems.

Our brains do not like uncertainty. When faced with gaps in our understanding, we tend to fill in these gaps ourselves. Where our knowledge ends, our brain fills in the blank spaces. We base these explanations on known patterns, knowledge and experience. The challenge is that we rarely notice when we have filled a gap, because once filled, it no longer feels like one. The result is that people may overestimate their own understanding of how an AI system works, creating a false sense of confidence that is difficult to detect and correct.

For example, a finance worker reviewing an AI-generated credit score might assume the system weighted income most heavily, because that is how their previous non-AI system worked. Or a recruiter using an AI screening tool might assume that it is looking for the same keywords that they were trained to look for themselves. These assumptions feel reasonable. In the case of deep learning algorithms, they are often wrong. Such algorithms are non-linear and follow patterns that may not align with human logic and reasoning.

It can become problematic when a user’s mental model and the actual workings of the system are too far apart. A flawed mental model increases the risk of wrong usage and mismatched expectations, which may lead to unexpected outputs that can reduce user trust in the system.

Let’s take the well-known example of viral social media posts where Large Language Models (LLMs) are asked to count how often the letter R appears in the word “strawberry”. LLMs often get these questions wrong, as they process language in tokens, not letters. A mental model where the LLM is expected to process language in the same way a person would, leads to expectations that the system was never designed to meet. The result is that trust in the system is reduced: “AI is stupid. AI doesn't work.”

It works the other way around as well. To some users, AI has become synonymous with LLMs. These users may extrapolate their understanding of how LLMs work to all AI technologies, even to technologies that have nothing to do with Natural Language Processing. As a result, the internal logic and decision-making processes of different AI technologies gets confused.

This problem can be made worse by how AI is marketed and communicated. Artificial Intelligence is an umbrella term that covers a huge range of technologies. The term refers to any system that mimics human intelligence and human decision-making, which can range from clustering and regression tasks to generative AI. These models work in fundamentally different ways, with very different logic and decision-making processes. Yet non-technical end users are rarely informed about the specific model they are interacting with. The applications are presented to users under the same label. AI-assisted. AI-powered. AI-enhanced. But when every system is marketed as “AI-assisted” regardless of the underlying technology, it becomes nearly impossible for users to build an accurate mental model for any of them.

For this reason, the call for organisations to invest in AI literacy and enablement is more urgent than ever. Knowledge gaps don’t stand empty. If your organisation doesn’t provide the accurate context, employees will build their own understanding from whatever is available: how the tool was marketed to them, social media, or their experience with previous systems. This could result in a false sense of confidence and understanding, and potentially inappropriate use of the system.

The logical response to all of this may seem straightforward: if users don't understand how AI systems work, we should explain it to them. But is it as easy as that?

Next week, we'll look at what happens when we try to open the black box and how users interact with explanations.

References

Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2023). Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Frontiers in Computer Science, 5, 1096257. https://doi.org/10.3389/fcomp.2023.1096257

Sanchez, T., Vereschak, O., & Deroy, O. (2026). Mental Models in Human-AI Interaction: Systematic Review of Empirical Methodologies and Guidelines. Proceedings of the 31st International Conference on Intelligent User Interfaces, 663–682. https://doi.org/10.1145/3742413.3789223

Share

In my article about Rethinking Trust in Business AI, I made the case that an appropriate level of trust is the foundation of sustainable AI adoption, and that the three pillars to focus on are AI enablement, transparency, and explainability. The black box effect sits at the heart of all three.

Simply put, the black box effect means that the decision-making process of AI systems is not transparent. While the inputs (questions) and outputs (answers) of an AI model are known, the way the system comes to its decision is difficult to understand.

There is a linear relationship between model accuracy and complexity. Relatively simple AI models may be less accurate than complex models, but their decision-making process is more easily explained and understood.

However, the availability of big data means that organisations are now increasingly deploying complex models such as deep learning models. Let’s have a look at Deep Neural Networks (DNNs) to demonstrate this. They are among the most powerful and accurate AI models, but their outputs are also the most difficult to interpret, even for trained experts.

Let’s take a DNN-based customer churn prediction model as an example. It is designed to predict which customers are likely to cancel their subscription next month. This diagram shows a simplified DNN architecture.

The input layer is the first layer, containing the input variables: things like login frequency, last purchase date, support ticket history, and contract type. The output layer contains the prediction: a probability score indicating how likely a customer is to churn. The hidden layers sit in between, and this is where the "black box" effect comes into play. The training data does not teach the model how the hidden layers should behave or what values to take. Instead, the model independently learns which features are relevant to explain relationships in the data. In this example, the model might learn that a combination of declining login frequency and a recent support ticket is a stronger churn signal than either factor alone — but it won't tell you that explicitly. A DNN can have between one and hundreds of hidden layers, and the decisions of these models are the result of millions of interdependent calculations. There is no single parameter or rule that controls a decision.

This complexity is what makes transparency so difficult to achieve. We often rely on external interpretation algorithms, such as LIME and SHAP, to help identify how inputs were translated into specific outputs.

The black box issue is highly relevant in a business context, as humans remain legally responsible for the decisions they make. The problem here lies in accountability. If humans cannot justify and explain the reasons for what they are doing, how can they make an informed decision on what action to take?

To understand why this matters in practice, let’s look at an example in the Business Intelligence domain. A business analyst has used an AI-powered forecasting tool to prepare a presentation for their leadership. The model predicts a significant rise in revenue for the next quarter. When asked why the model shows this, the analyst cannot explain it clearly. Was it based on seasonal patterns? A sudden change in customer behaviour? An anomaly in the data? The analyst is left defending a recommendation they don’t fully understand.

Acting on AI recommendations without being able to justify the reasons also carries ethical, social and financial business risks. Consider a procurement manager using an AI tool to assess supplier risk. The system flags a supplier as high-risk, but the manager doesn’t understand exactly what factors drove that assessment. Acting on the recommendation blindly could damage a business relationship. Ignoring it could pose a financial risk to the business. Without understanding the reasoning, there’s no good basis for a decision either way.

While transparency legislation exists at international, local, and organisational level, the legal scope is limited and practical guidance is minimal. Transparency regulation in the EU AI Act is mostly targeted at high-risk AI use cases. You can read more about the risk classification here. This risk-based approach means that many of the most widely used AI technologies, such as chatbots and forecasting tools, fall outside legal explainability requirements.

These unregulated use cases are particularly important to be aware of. For example, Large Language Models (LLMs) are among the most popular AI technologies right now. Yet many users rely on their outputs without fully understanding how those tools generate their answers. As a result, we may need to look beyond legal transparency requirements, and take our own responsibility to educate users.

So what does this mean for your business? Have you considered how it impacts your organisation if an employee provides incorrect information to a peer or customer based on an AI tool’s output? Are there reputational or financial risks? How does it affect your employee’s confidence and development? And how could this impact AI adoption?

In my next article, we’ll explore why this issue is not so easily solved and how we interact with AI systems we don’t fully understand.

Share

My name is Lisa and I am an applied AI researcher with a passion for AI Ethics and business AI. My background is in ERP software and I have a B.Sc. in applied Artificial Intelligence. I write about human-computer interaction, AI adoption, and what a responsible use of business AI looks like in practice. My goal is to translate scientific research into concrete insights for anyone who is interested, no academic or technical background required.

If your organisation is deploying AI and you want to learn more about these topics, you’re in the right place.

Let’s consider a scenario that many organisations will recognise. You have deployed an AI-assisted tool to help employees make decisions in a routine task. One employee blindly trusts the system and approves every recommendation without question. The other employee mistrusts the system on principle and therefore completely ignores it. It becomes a tug-of-war between the AI enthusiasts and the AI sceptics. Both responses feel like a failure, and neither approach represents a sustainable form of adoption.

Many organisations frame the trust problem as one-directional. From an AI adoption perspective, it may seem tempting to try and maximise trust in AI systems. Not enough trust means a low user adoption rate, therefore increasing trust becomes a goal in itself. The dominant approach to this tends to be promotional in nature – showcasing the benefits of AI, building hype and excitement, and emphasizing the productivity and efficiency benefits. What gets less attention is addressing the genuine concerns around AI and building a realistic understanding of the tools.

The risk in maximising trust via promotional means is that it can easily slide into blind trust and overreliance on the tool. Employees with less domain knowledge, such as new-hires or those transitioning into new roles, are particularly vulnerable to this. They may be more dependent on AI assistance and might be less able to verify correctness of the outputs.

The answer to this problem lies in establishing a calibrated level of trust. This level of trust means that a user’s confidence in the AI system reflects the system’s actual capabilities and limitations. Calibrated trust is the foundation of sustainable AI adoption. Users who understand a tool’s limitations are more likely to use it appropriately over time and are less likely to abandon it after a single bad experience or incorrect output.

The goal of calibrated trust is to allow users to make informed decisions based on the system’s output. Employees who understand how a tool reaches its conclusions are better equipped to use it critically, and to trust it appropriately. This also means that, where appropriate, we should help employees to identify scenarios when not to follow the outputs of an AI tool.

Achieving this level of trust is easier said than done, but three main topics to focus on are AI enablement, transparency and explainability.

Let’s start with enablement. As more employees are getting into contact with different AI technologies, AI proficiency becomes an increasingly important skill in the workplace. Are the employees in your workplace given enough context about what the AI tool can and cannot do, before they are expected to use it?

Transparency refers to the nature of the AI technology that is being used. It refers to clarity and openness in how AI algorithms operate and make decisions. How well do users understand the technology they are using, and do they know how the tool reaches its conclusions?

Explainability or Explainable AI (XAI) focuses on specific outputs of AI systems, attempting to “explain” individual decisions. Are employees able to understand why the AI system generated an answer?

Each of these topics comes with its own complexity and challenges. For example, the relationship between trust and transparency is complex, with many factors affecting how explanations are perceived and understood. In addition, there are technical, practical and financial implications to consider when implementing XAI techniques. We will be exploring these challenges in more detail in the coming months. Specifically, we will talk about how to approach these topics ethically and responsibly.

Building trust in AI is not a marketing challenge, but a challenge in understanding. Organisations that invest in helping their employees understand the tools they work with, will prevent the extremes of mistrust and overreliance. The goal was never to reach the top of the hype cycle, but to reach the slope of enlightenment, eventually leading into long-term, sustainable adoption of business AI.

Subscribe now

Share