Rethinking Trust in Business AI

Why calibrated trust is the foundation of sustainable AI adoption

With many Artificial Intelligence (AI) technologies currently at the top of their hype cycle, it’s no wonder that organisations are eager to adopt AI. However, research by McKinsey shows that businesses are still in the early phases of rolling out AI in their daily operations. After all, integrating AI into your business processes does not happen overnight. One of the common blockers faced by organisations is AI adoption and user trust.

Let’s consider a scenario that many organisations will recognise. You have deployed an AI-assisted tool to help employees make decisions in a routine task. One employee blindly trusts the system and approves every recommendation without question. The other employee mistrusts the system on principle and therefore completely ignores it. It becomes a tug-of-war between the AI enthusiasts and the AI sceptics. Both responses feel like a failure, and neither approach represents a sustainable form of adoption.

Many organisations frame the trust problem as one-directional. From an AI adoption perspective, it may seem tempting to try and maximise trust in AI systems. Not enough trust means a low user adoption rate, therefore increasing trust becomes a goal in itself. The dominant approach to this tends to be promotional in nature – showcasing the benefits of AI, building hype and excitement, and emphasizing the productivity and efficiency benefits. What gets less attention is addressing the genuine concerns around AI and building a realistic understanding of the tools.

The risk in maximising trust via promotional means is that it can easily slide into blind trust and overreliance on the tool. Employees with less domain knowledge, such as new-hires or those transitioning into new roles, are particularly vulnerable to this. They may be more dependent on AI assistance and might be less able to verify correctness of the outputs.

The answer to this problem lies in establishing a calibrated level of trust. This level of trust means that a user’s confidence in the AI system reflects the system’s actual capabilities and limitations. Calibrated trust is the foundation of sustainable AI adoption. Users who understand a tool’s limitations are more likely to use it appropriately over time and are less likely to abandon it after a single bad experience or incorrect output.

The goal of calibrated trust is to allow users to make informed decisions based on the system’s output. Employees who understand how a tool reaches its conclusions are better equipped to use it critically, and to trust it appropriately. This also means that, where appropriate, we should help employees to identify scenarios when not to follow the outputs of an AI tool.

Achieving this level of trust is easier said than done, but three main topics to focus on are AI enablement, transparency and explainability.

Let’s start with enablement. As more employees are getting into contact with different AI technologies, AI proficiency becomes an increasingly important skill in the workplace. Are the employees in your workplace given enough context about what the AI tool can and cannot do, before they are expected to use it?

Transparency refers to the nature of the AI technology that is being used. It refers to clarity and openness in how AI algorithms operate and make decisions. How well do users understand the technology they are using, and do they know how the tool reaches its conclusions?

Explainability or Explainable AI (XAI) focuses on specific outputs of AI systems, attempting to “explain” individual decisions. Are employees able to understand why the AI system generated an answer?

Each of these topics comes with its own complexity and challenges. For example, the relationship between trust and transparency is complex, with many factors affecting how explanations are perceived and understood. In addition, there are technical, practical and financial implications to consider when implementing XAI techniques. We will be exploring these challenges in more detail in the coming months. Specifically, we will talk about how to approach these topics ethically and responsibly.

Building trust in AI is not a marketing challenge, but a challenge in understanding. Organisations that invest in helping their employees understand the tools they work with, will prevent the extremes of mistrust and overreliance. The goal was never to reach the top of the hype cycle, but to reach the slope of enlightenment, eventually leading into long-term, sustainable adoption of business AI.