The Transparency Paradox

How AI explanations affect user trust

Explainable AI and interpretation methods, such as LIME and SHAP, are techniques used to reduce the black box effect and increase transparency and trust in AI systems. These explanations can take different forms and transparency levels. For example, a credit risk score may come with a list of contributing factors, or a flagged anomaly with a summary of what triggered it.

But receiving an explanation and understanding it are not the same thing. We can read one carefully and still not know what it actually tells us. Part of the difficulty is that there is no universal agreement in transparency frameworks on what a clear and meaningful explanation actually looks like. Therefore, the question is not just whether the system provides an explanation, but whether it works for the person reading it.

So what makes an explanation helpful, and how do explanations affect user trust in an AI system?

The relationship between trust and transparency is not as straightforward as it sounds. Having no explanation at all leaves users without the context they need to understand an output. Providing an explanation of the reasoning behind the decision, is known to increase trust in AI predictions. But the solution is not simply to provide as much information as possible. The positive effect of providing an explanation can actually be cancelled out by providing too much technical information.

The challenge lies in finding the right balance between technical accuracy and user-friendliness. Imagine a loan officer presented with a full chart of model weights, feature scores and confidence scores, explaining which factors led to the loan being approved or rejected. Technically, this is the most transparent type of explanation. But for a non-technical end user, it may leave them not knowing where to look or how to make sense of what they're seeing. That's cognitive overload: when explanations stop being helpful and start overwhelming. On the other hand, oversimplified explanations that leave out too much detail can give users a false sense of understanding that may reinforce flawed mental models about how the system works.

Getting this balance right is harder than it looks, because many factors influence how users perceive explanations. When an explanation does not match human expectations, judgement, or intuition, it can reduce trust in the system, even if the system is correct. This is a particular issue for deep learning models, whose decision-making processes may not align with our own mental models of how the system works. Explanations are also subject to confirmation bias: users are more likely to accept an explanation that fits what they already believed, and to challenge one that does not. This means that explanations can reinforce misunderstandings we may have about the system.

How users respond to explanations also depends on their background, the type of decision they are making, and how much they already understand about AI. A business manager with no AI experience and a data scientist reading the same output may walk away with very different levels of understanding.

So what can we do to help users better interpret AI outputs?

First, it is important that we make explanations accessible to non-technical end users and write them in plain English. Think of a supplier risk score: if the explanation behind it is written for a data scientist rather than a procurement team, the risk is that it gets ignored or misread. That said, accessible does not mean dumbed down. Explanations that oversimplify to the point of being misleading can do more harm than providing no explanation at all.

Where possible, we should consider using interactive explanations for complex models. Static information we may not fully understand can leave users with more questions than answers, and may actually reduce trust in the system. Research suggests that allowing users to ask questions about explanations improves understanding and acceptance. Being able to ask a follow-up question to a human expert or digital assistant helps users to better interpret the information. If this kind of interaction is not feasible, even a well-designed FAQ or tooltip can make a difference.

Finally, it is crucial that we invest in AI literacy. Currently, most enablement for end users is focused on how to use the system: how to prompt effectively, how to navigate the interface, etc. There is much less emphasis on understanding the overall logic of the system: why a chatbot responds the way it does, or what a model was designed to do at a conceptual level. This can cause misunderstandings about outputs. Users who have a high-level understanding of how the technology works, are better able to interpret what it tells them. This does not require technical training in coding or mathematics. It means giving employees enough context about how the system works and what it was designed to do, so they can make informed decisions about what to do with AI-generated answers.

The goal of this approach is to allow users to make informed decisions and establish an appropriate level of trust in AI predictions. When users understand what they are working with, they are more likely to apply the right AI tools in the right situations. It allows users to be confident in AI systems without sliding into blind trust. And that is what a responsible, durable adoption of business AI looks like in practice.

---

This article draws on research conducted for my thesis, which studied the effect of Explainable AI on user trust in high-risk DNN-based AI systems. If you would like more details, please contact me.

---

References

Bauer, K., Von Zahn, M., & Hinz, O. (2023). Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing. Information Systems Research, 34(4), 1582–1602. https://doi.org/10.1287/isre.2023.1199

Cheong, B. C. (2024). Transparency and accountability in AI systems: Safeguarding wellbeing in the age of algorithmic decision-making. Frontiers in Human Dynamics, 6, 1421273. https://doi.org/10.3389/fhumd.2024.1421273

Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2023). Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Frontiers in Computer Science, 5, 1096257. https://doi.org/10.3389/fcomp.2023.1096257

Kizilcec, R. F. (2016). How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2390–2395. https://doi.org/10.1145/2858036.2858402

Lundberg, S., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions (Version 2). arXiv. https://doi.org/10.48550/ARXIV.1705.07874

Pavlidis, G. (2024). Unlocking the black box: Analysing the EU artificial intelligence act’s framework for explainability in AI. Law, Innovation and Technology, 16(1), 293–308. https://doi.org/10.1080/17579961.2024.2313795

Ribeiro, M., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, 97–101. https://doi.org/10.18653/v1/N16-3020

Souza, G. H. C., De Araujo Wanderley, C., & Braga De Aguiar, A. (2025). The influence of artificial intelligence (AI) transparency on AI acceptance in managerial decision-making. Journal of Management Control. https://doi.org/10.1007/s00187-025-00396-7

Zhang, T., Yang, X. J., & Li, B. (2025). May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability. International Journal of Human–Computer Interaction, 41(9), 5623–5647. https://doi.org/10.1080/10447318.2024.2364986

---

Thank you for reading AI Ethics Made Practical! Subscribe for fee to receive new posts and support my work.

Help to spread the message and share this article.

AI Ethics Made Practical is free of charge. But if you found this article helpful and would like to support my work, please consider buying me a coffee!

---