When AI Can't Explain Itself

Understanding the black box problem and what it means for your organisation

A commonly used term in AI is the “black box” concept. But what does this actually mean and how does it affect business users working with AI?

In my article about Rethinking Trust in Business AI, I made the case that an appropriate level of trust is the foundation of sustainable AI adoption, and that the three pillars to focus on are AI enablement, transparency, and explainability. The black box effect sits at the heart of all three.

Simply put, the black box effect means that the decision-making process of AI systems is not transparent. While the inputs (questions) and outputs (answers) of an AI model are known, the way the system comes to its decision is difficult to understand.

There is a linear relationship between model accuracy and complexity. Relatively simple AI models may be less accurate than complex models, but their decision-making process is more easily explained and understood.

However, the availability of big data means that organisations are now increasingly deploying complex models such as deep learning models. Let’s have a look at Deep Neural Networks (DNNs) to demonstrate this. They are among the most powerful and accurate AI models, but their outputs are also the most difficult to interpret, even for trained experts.

Let’s take a DNN-based customer churn prediction model as an example. It is designed to predict which customers are likely to cancel their subscription next month. This diagram shows a simplified DNN architecture.

The input layer is the first layer, containing the input variables: things like login frequency, last purchase date, support ticket history, and contract type. The output layer contains the prediction: a probability score indicating how likely a customer is to churn. The hidden layers sit in between, and this is where the "black box" effect comes into play. The training data does not teach the model how the hidden layers should behave or what values to take. Instead, the model independently learns which features are relevant to explain relationships in the data. In this example, the model might learn that a combination of declining login frequency and a recent support ticket is a stronger churn signal than either factor alone — but it won't tell you that explicitly. A DNN can have between one and hundreds of hidden layers, and the decisions of these models are the result of millions of interdependent calculations. There is no single parameter or rule that controls a decision.

This complexity is what makes transparency so difficult to achieve. We often rely on external interpretation algorithms, such as LIME and SHAP, to help identify how inputs were translated into specific outputs.

The black box issue is highly relevant in a business context, as humans remain legally responsible for the decisions they make. The problem here lies in accountability. If humans cannot justify and explain the reasons for what they are doing, how can they make an informed decision on what action to take?

To understand why this matters in practice, let’s look at an example in the Business Intelligence domain. A business analyst has used an AI-powered forecasting tool to prepare a presentation for their leadership. The model predicts a significant rise in revenue for the next quarter. When asked why the model shows this, the analyst cannot explain it clearly. Was it based on seasonal patterns? A sudden change in customer behaviour? An anomaly in the data? The analyst is left defending a recommendation they don’t fully understand.

Acting on AI recommendations without being able to justify the reasons also carries ethical, social and financial business risks. Consider a procurement manager using an AI tool to assess supplier risk. The system flags a supplier as high-risk, but the manager doesn’t understand exactly what factors drove that assessment. Acting on the recommendation blindly could damage a business relationship. Ignoring it could pose a financial risk to the business. Without understanding the reasoning, there’s no good basis for a decision either way.

While transparency legislation exists at international, local, and organisational level, the legal scope is limited and practical guidance is minimal. Transparency regulation in the EU AI Act is mostly targeted at high-risk AI use cases. You can read more about the risk classification here. This risk-based approach means that many of the most widely used AI technologies, such as chatbots and forecasting tools, fall outside legal explainability requirements.

These unregulated use cases are particularly important to be aware of. For example, Large Language Models (LLMs) are among the most popular AI technologies right now. Yet many users rely on their outputs without fully understanding how those tools generate their answers. As a result, we may need to look beyond legal transparency requirements, and take our own responsibility to educate users.

So what does this mean for your business? Have you considered how it impacts your organisation if an employee provides incorrect information to a peer or customer based on an AI tool’s output? Are there reputational or financial risks? How does it affect your employee’s confidence and development? And how could this impact AI adoption?

In my next article, we’ll explore why this issue is not so easily solved and how we interact with AI systems we don’t fully understand.