An essential requirement of intelligence is its explanation. When we question a child about why they did something, we are not only attempting to understand their justifications but also encouraging them to think back on their thought process. Similarly, when we ask an AI system for an explanation, we are simultaneously seeking to comprehend how it makes decisions and forcing it to consider why it took a particular action.
The requirement for explainability stems from the fact that many AI models are “black boxes,” which means that the models are built on such a complex architecture that it becomes very difficult to explain why the model is making such a decision. Furthermore, we are deploying such models in production every day and some of them are developed and deployed for high-risk environments such as medical or financial decision-making. Often, data scientists measure their success using parameters like accuracy. But is this enough for a model to be highly accurate to deploy in production? How representative of real-world data is the data used for training or testing the model? Also, in many situations, these models are deployed to aid human decision-making. How should the person who gets this model output understand and trust the model output?
This is where model explainability plays an important role. The goal of explainable AI is to make it easier for humans to understand how AI systems work, how they make decisions, and why they do so. This enhances the dependability and credibility of AI systems. Additionally, it helps locate and correct errors in AI systems and improve performance.
When Do you need Model Explanation?
Model explainability is an expensive process thus not all models and processes need an explanation. Model Explainability becomes important when consequences are high if the model output is not explained to the user. For example, with a recommendation system, if our model is recommending an item that we are not interested in, or a person as a friend whom you don’t know, you can simply ignore this. There are no consequences for this kind of mistake by the model and you don’t want to know why the engine is recommending this to you. On the other hand, if you have a model that is used to reject your loan application, you would be interested to know why this application is rejected. Similarly, in another case, when you are getting diagnosed for a specific disease by a model, you and your doctor want to know on what basis the model is taking a decision.
Model explainability also becomes important when you are not sure if the data used for model training and testing are representative of the real-world scenario. If this model is deployed in the production environment, it is always good to explain why the model is behaving in this manner. So that user is aware of the output and reason to take their own decision.
Along with it, there are some additional situations such as regulatory requirements and safety, where model explanation becomes a must for using them in the production environment.
The Benefits of Explainable AI
Explainable AI provides numerous advantages, particularly in the realm of business decision-making. By comprehending the process and reasoning behind machine learning algorithms’ predictions, businesses can make well-informed choices about how to attain their objectives.
The key benefits of Explainable AI are as follows: –
- Accountability and Transparency – It helps in understanding how an AI system makes decisions. It can also assist in spotting biases that may have snuck in during development. As a result, we can address these prejudices.
- Trust and Acceptance – End users are frequently hesitant to adopt and use tools that are powered by AI because they do not know how the AI system makes decisions. Lack of explanation promotes mistrust and skepticism, eventually preventing the widespread adoption of AI-powered solutions. If users can understand the reasoning behind an AI system’s output, they will be more confident in using it.
- Performance – Explaining the model also helps in debugging model behavior and in turn improves its performance.
How to achieve explainability
There are potentially two main methods to achieve model explainability.
- Use Inherently Explainable Models: Some models are inherently easy to explain for example Rule-based models, Regression models, or Shallow Tree models. Predictions from these models can be explained using features, coefficients, or simple if-else rules.
2. Post Hoc Explanation: Sometimes, models are too complex or when you are examining a third-party model which is a kind of black box for you, you will need some external mechanism to explain the model. This external mechanism is known as the Explainer algorithm. This explainer algorithm takes the black box model and generates a simple explainer model that can explain output using a simple equivalent algorithm. For example, a large neural network-based model can be explained by using an explainer algorithm that captures the most important features used by this large NN.
This explainer algorithm should have two main characteristics:
- Faithful: The explanation generated by this explainer algorithm should faithfully describe model behavior.
- Understandable: The explanation generated by this explainer should be understandable by the end user. This nature of understandability depends on who this user is. Is this the ML Expert, or domain expert? For example: If you send the values of coefficients to ML engineers or rules/most important features (Doctor) to the domain expert.
Types of Explanation
There are various types of model explanations, that can be broadly classified into two categories:
- Local Explanation: Local explanation attempt to explain how a model operates in a certain area of interest. They typically approximate the model around the instance the user wants to explain, in order to extract explanations that describe how the model operates when encountering such instances. Local explanations unearth biases in the neighborhood of the given instance. For example; sharing an explanation of an instance where the model rejected a loan application. Some of the popular tools for local explanations are LIME, SHAP, and Saliency Map.
- Global Explanation: Global explanation on the other hand attempts to explain the complete behavior of the overall model. They typically unearth bias on a large subgroup of the model domain. This type of explanation is suitable in the scenarios such as sharing model explanations with authorities so that they can take decisions to deploy in production. Some popular tools for global explanations are SP-LIME and TCAV.
There are several other ways you can use to make AI more explainable. These include
- Human-in-the-Loop: Human-in-the-Loop (HITL) is when a human is actively involved in the decision-making process of an AI system. A human might look at an AI system’s decision and then explain it to other humans.
- Open Source: Another great way to make AI more explainable is to make sure that your AI models are open source. This way, other people can check your algorithms and make sure that your models are legitimate.
- Visualization: You can also make AI more explainable by visualizing data so that humans can understand it more easily. This is helpful for both humans and AI to better understand data and make decisions based on it.
Conclusion:
Making AI more explainable is crucial for the adoption of AI. It is essential to help humans understand how AI systems work and make decisions. With the help of these tools, we can make AI more understandable to humans and thus reduce the concerns of potential misuse and abuse of AI systems.
Share this Post