Interpret Model (SHAP)

---

1. Objective

The objective of this workflow is to open the "Black Box". Stakeholders do not trust models they do not understand. SHAP (SHapley Additive exPlanations) provides a game-theoretic approach to explain the output of any machine learning model by distributing the "payout" (prediction) among the "players" (features) fairly.

---

2. Context & Scope

In Scope

This workflow covers selecting the right Explainer (Tree, Kernel, Deep), calculating SHAP values, visualizing Global Importance (Summary Plot), and Local Importance (Force Plot for a single user).

Assumption: You have a trained model (Scikit-Learn, XGBoost, CatBoost, Keras).

Out of Scope

• Causal Inference: SHAP explains correlations the model found, not necessarily real-world causality.
• Fairness Metrics: Assessing bias requires specific fairness metrics (Equalized Odds), though SHAP helps diagnose it.

---

3. When to Use / When Not to Use

✅ Use This Workflow When

• A loan applicant asks "Why was I rejected?" (Regulatory requirement).
• You need to debug if the model is learning "garbage" correlations (e.g., predicting "Wolf" because of snow in background).
• You want to measure interaction effects between features.

❌ Do NOT Use This Workflow When

• The model is a simple Linear Regression (Just look at the Coefficients).
• You have millions of rows and strict time constraints (SHAP is slow; use FastTreeSHAP or subsample).

---

4. Inputs (Required/Optional)

Required Inputs

Input	Description	Format	Example
MODEL	The prediction object.	Object	XGBClassifier
DATASET	Evaluation data.	Dataframe	X_test

Optional Inputs

Input	Description	Default	Condition
SAMPLE_SIZE	Rows to explain.	100	KernelExplainer is slow.

---

5. Outputs (Artifacts)

Artifact	Format	Destination	Quality Criteria
Summary Plot	Image	Report	Shows feature impact + direction.
Force Plot	HTML/Image	Dashboard	Explains individual prediction.

---

6. Operating Modes

⚡ Fast Mode

Timebox: 15 minutes Scope: TreeExplainer. Details: Calculating SHAP values for a Tree Model (XGB/RF) on a small subsample. Generating a Summary Bar Chart.

🎯 Standard Mode (Default)

Timebox: 2 hours Scope: Deep Dive. Details: Using Explainer to auto-select method. Generating Dependence Plots to see non-linear relationships. Exporting values to CSV for business analysis.

🔬 Deep Mode

Timebox: 1 day Scope: Model Agnostic / Deep. Details: Using KernelExplainer or DeepExplainer for Neural Networks. Analyzing Interaction Values (2nd order effects) to see how Feature A changes Feature B's impact.

---

7. Constraints & Guardrails

Technical Constraints

• Speed: KernelExplainer is exponential time. Do NOT run it on 1M rows. Sample 100-500 background samples.
• Additivity: The sum of SHAP values + Base Value MUST equal the Model Output. If not, the explanation is broken.

Security & Privacy

CAUTION

Privacy Leakage Inverting the SHAP values can sometimes reconstruct the training data (Model Inversion Attack). Be careful sharing detailed explanations publicaly for sensitive models.

Compliance

• GDPR: "Right to Explanation". This workflow validates that you can generate human-readable reasons for automated decisions.

---

8. Procedure

Phase 1: Initialize Explainer

Objective: Pick the tool.

Identify Model Type.

• Tree-based (XGB, LightGBM, RF): Use shap.TreeExplainer(model). Fast (C++ optimized).
• Deep Learning (TF/Torch): Use shap.DeepExplainer or shap.GradientExplainer.
• Anything else (SVM, KNN): Use shap.KernelExplainer(model.predict, background_data). Slow.

Pass the background_data (e.g., X_train.sample(100)). This serves as the "Baseline" against which impact is measured.

Verify: Explainer object created without error.

Phase 2: Calculation

Objective: Number crunching.

Calculate values: shap_values = explainer.shap_values(X_test). Note: For classification, this returns a list (one per class). Select the class of interest (e.g., Class 1 = Fraud). Check Additivity: explainer.expected_value + sum(shap_values) ≈ model.predict(X_test).

Verify: The shape of shap_values matches X_test.

Phase 3: Visualization

Objective: Tell the story.

Global: shap.summary_plot(shap_values, X_test).

• Dots: Each dot is a row.
• Color: Red = High Feature Value, Blue = Low.
• X-Axis: Impact on prediction. Reading: "High Income (Red) pushes prediction to the right (Positive Impact)."

Local: shap.force_plot(base_value, shap_values[0], X_test.iloc[0]).

• Explains one specific user. "Why was John rejected?"
• Red bars push higher, Blue bars push lower.

Dependence: shap.dependence_plot("Age", shap_values, X_test).

• Shows non-linear relationships (e.g., Age risk U-curve).

Verify: Plots are readable. Insights align with domain intuition (or reveal bugs).

---

9. Technical Considerations

Correlation vs Causation: SHAP says "Age is important". It does not say "Getting older causes rejection" (though likely true). It reflects the model's logic, which might be flawed.

Training Data Mismatch: If you explain a Sample that is "Out of Distribution" (OOD) compared to background data, SHAP values become unreliable.

Cohorts: You can slice SHAP values. "What drives predictions for Women vs Men?" Split the shap_values array and plot separately.

---

10. Quality Gates (Definition of Done)

Checklist

• [ ] Correct Explainer selected.
• [ ] Background data provided (for Kernel/Deep).
• [ ] Additivity check passed.
• [ ] Summary plot generated.

Validation

Criterion	Method	Threshold
Accuracy	Additivity	Error < 1e-5
Performance	Runtime	< 1 hour (use sampling if needed)

---

11. Failure Modes & Recovery

Failure Mode	Symptoms	Recovery Action
Kernel Freeze	Cell runs forever.	Reduce nsamples or background_data size. Switch to TreeExplainer if possible.
Shape Mismatch	"Dimension mismatch".	Classification models return list of arrays. Index shap_values[1] for the positive class.
Unintuitive Signs	"Income" has negative impact?	Check feature encoding (Did -1 mean missing?). Or Model found a real weird correlation.

---

12. Copy-Paste Prompt

TIP

One-Click Agent Invocation Copy the prompt below, replace placeholders, and paste into your agent.

Role: Act as a Senior Data Scientist specializing in Explainable AI (XAI).
Task: Execute the Interpret Model (SHAP) workflow.

## Objective & Scope
- **Goal**: Explain model predictions (Global and Local) to stakeholders using SHAP.
- **Scope**: Explainer Selection, SHAP Value Calculation, and Visualization.

## Inputs
- [ ] MODEL: Trained Model Object (e.g., XGBClassifier).
- [ ] DATASET: Test DataFrame (X_test).
- [ ] SAMPLE_SIZE: Rows to explain (default 100).

## Output Artifacts
- [ ] Summary Plot (Global Importance)
- [ ] Force Plot (Local Explanation)
- [ ] Analysis Notebook

## Execution Steps
1. **Setup**
   - Initialize appropriate Explainer (TreeExplainer for trees,
     KernelExplainer for others). Pass background data if needed.
2. **Compute**
   - Calculate SHAP values for the dataset. Verify additivity (Base + Shap == Output).
3. **Visualize**
   - Generate Summary Plot to show feature impact direction. Generate Force Plot for outliers.

## Quality Gates
- [ ] Explainer initialized correctly.
- [ ] Additivity check passed (Error < 1e-5).
- [ ] Plots are readable and interpreted.
- [ ] Sensitive data (PII) protected in outputs.

## Failure Handling
- If blocked, output a "Clarification Brief" detailing missing info or blockers.

## Constraints
- **Resource**: Sample data for KernelExplainer (slow).
- **Compliance**: Do not expose PII in explanations.

## Command
Now execute this workflow step-by-step.

---

Appendix: Change Log

Version	Date	Author	Changes
1.0.0	2026-01-14	AI Engineering Team	Initial release