Model Trade-Offs
Interpretability trade-offs
Interpretability is a feature of model transparency. Interpretability is the degree to which a human can understand the cause of a decision. This might sound a lot like explainability, but there is a distinction difference.
Interpretability
Interpretability is the access into a system so that a human can interpret the model’s output based on the weights and features. For example, if a business wants high model transparency and wants to understand exactly why and how the model is generating predictions, they need to observe the inner mechanics of the AI/ML method.
Explainability
Explainability is how to take an ML model and explain the behavior in human terms. With complex models (for example, black boxes), you cannot fully understand how and why the inner mechanics impact the prediction. However, through model agnostic methods (for example, partial dependence plots, SHAP dependence plots, or surrogate models) you can discover meaning between input data attributions and model outputs. With that understanding, you can explain the nature and behavior of the AI/ML model.
Interpretability example
An economist might want to build a multi-variate regression model to predict an inflation rate. They can view the estimated parameters of the model’s variables to measure the expected output given different data examples. In this case, full transparency is given, and the economist can answer the exact why and how of the model’s behavior.
Explainability example
A news media outlet uses a neural network to assign categories to different articles. The news outlet cannot interpret the model in depth. However, they can use a model agnostic approach to evaluate the input article data compared to the model predictions. With this approach, they find that the model is assigning the sports category to business articles that mention sport organizations. Although the news outlet did not use model interpretability, they were still able to derive an explainable answer to reveal the model’s behavior.
If a business wants high model transparency and wants to understand exactly why and how the model is generating predictions, then they need a model that offers interpretability. However, high interpretability typically comes at the cost of performance, as seen in the diagram.
If a company wants to achieve high performance but still wants to have a general understanding of the model behavior, model explainability starts to play a larger role.
When starting a new AI/ML project, you need to consider whether interpretability is required. Model explainability can be used in any AI/ML use case, but if detailed transparency is required, then your AI/ML method selection becomes limited.
Safety and transparency trade-offs
Model safety
Model safety is the ability of an AI system to avoid causing harm in its interactions with the world. This includes avoiding social harm, such as bias in decision-making algorithms, and avoiding privacy and security vulnerability exposures. Model safety is important for ensuring that AI systems are used in ways that benefit society and do not cause harm to individuals or groups.
Model safety and model transparency trade-offs
With model safety focusing on protecting information, and model transparency focusing on exposing information, you can understand that there is a delicate balance needed between them.
Accuracy
Complex models like large neural networks tend to be more accurate but less interpretable than simpler linear models, which are more transparent.
Privacy
Privacy-preserving techniques like differential privacy can improve safety but make models harder to inspect. This can make models less transparent.
Safety
Constraining or filtering model outputs for safety can reduce transparency into the original model reasoning.
Security
Highly secured air-gapped train models (models that are trained on networks that are private and do not have access to external data) might be less open to external auditing.
Model controllability
Model control
A controllable model is one where you can influence the model's predictions and behavior by changing aspects of the training data. Higher controllability provides more transparency into the model and allows correcting undesired biases and outputs.
Model controllability is measured by how much control you have over the model by changing the input data. Models that are more controllable are easier to steer towards desired behaviors. This is important for fairness because you want to be able to understand and control bias in the model. Controllability of a model is also important for transparency and debugging in a model.
Controllability depends on the model architecture. Linear models tend to be more controllable than complex neural models. You can test for controllability by evaluating if manipulating the data, such as adding or removing examples, causes expected changes in the model's outputs and predictions. Controllability can be improved through data augmentation techniques and by adding constraints to the model training process.
In the next lesson, you will learn about the principles of human-centered design for explainable AI.