eXplainable AI

eXplainable AI

1. Taxonomomy of interpretability

   • Pre-model
     • Good understanding of data.
       • Meaningful intuitive features
       • EDA techniques
       • Visualizations
   • In-model
     • How the model works?
     • Constraints imposed on the complexity of the ML model
   • Post-model
     • What else does the model tell us?
     • Applying methods that analyze the model after training.
   • Intrinsic
   • Post-hoc
   • Model-specific
     • Limited to specific model classes.
       • Each method is based on some specific model's internals (e.g., the weights in a linear model)
     • interpretable models
       • Linear regression
       • Decision trees
       • RuleFit
       • Logistic Regression
       • Naive Bayes
   • Model-agnostic
     • Applied to any ML model.
       • Advantage: Completely independent of the original model class.
     • Cannot have access to model's inner workings (e.g., weights, structural information)
       • Disadvantage: They do not leverage intrinsic properties of specific types of models.
     • Model-agnostic explanation methods
       • Partial Dependence Plot
       • Individual Condition Expectation
       • Accumulated Local Effects Plot
       • Feature Interaction
       • Feature Importance
       • Local Surrogate Model
       • Shapley Values (or SHAP)
       • BreakDown
       • Anchors
       • Counterfactual Explanations
       • Prototypes and Criticism (or MMD-Critic)
       • Influence Functions
     • Analyzing pairs of feature input and output.
   • Results of explanation methods
     • Feature summary
       • Summary statistics for each feature (e.g., feature importance).
       • They can (and maybe must) be visualized.
     • Model internals
     • Data point
       • Example-based methods. Suitable for images and text, not for tabular data.
     • Surrogate intrinsically interpretable model
       • Approximation for black-box models to provide insights

2. Scope of interpretability

   • Algorithm transparency
     • How does the algorithm create the model?
     • It doesn't require knowledge of the data or the learned model.
   • Global model interpretability
     • Holistic
       • Global model output
       • Knowledge of the algorithm
         • Difficult to achieve in practice.
       • Data
       • How does the trained model make predictions?
       • Comprehend the entire model at once; understand the distribution output based on the features.
     • Modular
       • How do parts of the model affect the predictions?
         • Linear models: weights
         • Decision trees: splits (features and cut-off values) and leaf node predictions
   • Local model interpretability
     • Single prediction
       • Approximate a small region of interest in a black-box model using a simpler interpretable model.
     • Group of predictions
       • Global methods
         • Apply to a group of predictions as it was the whole dataset.
       • Local methods
         • Apply on each prediction individually and aggregate afterwards.

3. Properties of explanation

   • Accuracy
     • The predictive accuracy of the explanation regarding unseen data.
   • Fidelity
     • How well an explanation approximates the prediction of the black-box model.
   • Consistency
     • Two different models, trained on the same task, output the same predictions.
   • Stability
     • How similar are the explanations for similar instances (using the same model).
   • Comprehensibility
     • How well humans understand the explanations.
   • Certainity
     • Model's confidence on the correctness of the prediction
   • Importance (of features or parts of explanation)
     • Is it clear which of the conditions of the rule were the most important?
   • Novelty
     • Locate the data instance to be explained in the distribution of the training data. If it's far away, the explanation might be useless.
   • Representativeness
     • How many instances are covered by the explanation.

4. Interpretability evaluation

   • Application-level (end-task)
     • End-user experiments
     • Baseline: How good a human would explain the same decision.
     • End-user is domain expert
   • Human-level (simple task)
     • End-user is layperson
   • Functional-level (proxy-task)
     • No human experiments required
     • Some formal definition of explainability serves as proxy to assess the quality of an explanation
       • Depth of decision tree
       • Model sparsity
       • Uncertainty
     • Preferable: The class of model being used has already been evaluated by someone else in a human-level evaluation.

5. Interpretability goals

   • Accuracy
   • Understandability
   • Efficiency
     • Needed time for a user to grasp the explanation

6. Human-friendly explanations

   • Contrastiveness
     • Why P rather than Q?
       • P is the target event
       • Q is the counterfactual contrast case that did not occur.
   • Selectivity
   • Social
   • Focus on the abnormal
   • Truthful
   • Consistent with prior beliefs
   • General and probable

7. Interpretability importance

   • Satisfy human curiosity
   • Scientific findings
   • Find meaning
   • Regulation requirements
     • GDPR
     • ISO/IEC 27001
   • Social acceptance and trust
   • Safety
     • Models can be tested, audited, and debugged
   • Acquire new knowledge

8. Properties of explanation methods

   • Expressive power
   • Translucency
   • Portability
   • Algorithmic complexity