| Myth | Reality |
|---|---|
| Any explainability technique satisfies GDPR’s “right to explanation.” | GDPR Articles 13 to 15 require “meaningful information about the logic involved,” a standard that raw SHAP values or attention maps do not meet without translation into human-actionable terms. |
| SHAP and LIME are interchangeable for compliance purposes. | SHAP has stronger theoretical guarantees and is more defensible under scrutiny; LIME’s locally approximate, sampling-based explanations are more likely to be challenged for inconsistency between runs. |
| Attention weights in a transformer model count as an explanation. | Research has repeatedly shown attention weights do not reliably correspond to the features actually driving a model’s output, making them weak evidence in a legal or audit context. |
| If a technique is useful for internal debugging, it is also good enough for regulators. | Internal debugging tools optimize for engineer intuition; regulatory acceptance requires stability, reproducibility, and a direct link to actionable information for the affected individual. |
Two Different Jobs Wearing the Same Name
“Explainable AI” gets used to describe two fundamentally different activities that happen to share tooling. The first is internal: a data scientist wants to understand why a model is behaving a certain way, to debug it, improve it, or catch a spurious correlation before deployment. The second is external and adversarial in nature: a regulator, auditor, or affected individual wants to know why a specific decision was made about them, and that explanation may need to hold up in a legal proceeding or a regulatory review.
Most of the explainability literature, and most of the tooling built around it, was designed for the first job. SHAP, LIME, counterfactual explanation libraries, and attention visualization all originated as ways to help the people who built a model understand its behavior. They were not designed with an eye toward whether their output would satisfy a specific legal standard, would survive being challenged by opposing counsel, or would remain stable if regenerated a year later on a slightly retrained model. That gap, between debugging usefulness and regulatory defensibility, is where most compliance programs run into trouble, and it is the gap this article is about.
What Regulators Actually Require, Not What Is Merely Nice to Have
Two legal regimes anchor most of the current explainability obligations facing organizations operating in or serving the EU: GDPR and the EU AI Act. Neither actually names SHAP or LIME. Both establish a standard the explanation has to meet, and it is the organization’s job to pick a technique that clears that bar.
GDPR Article 22 restricts certain solely automated decisions that produce legal or similarly significant effects on a person. It is read together with the transparency obligations in Articles 13 through 15, which require that a data subject be given “meaningful information about the logic involved” in automated decision-making, along with the significance and envisaged consequences of that processing. The phrase “meaningful information about the logic involved” is doing the real work here: it is not a demand for the literal mathematics of the model, and it is not satisfied by a vague statement that “the algorithm considered many factors.” It sits between those two extremes, requiring something specific enough to be actionable by the person it affects.
The EU AI Act’s Article 13 transparency obligations, which apply to high-risk AI systems from August 2026, take a more prescriptive stance. Providers of high-risk systems must ensure the system is accompanied by instructions for use containing information that is relevant, accessible, and comprehensible to deployers, sufficient for the deployer to interpret the system’s output and use it appropriately. This is a stricter and more operational standard than GDPR’s “meaningful information,” because it is aimed at professional deployers who need to actually act on the explanation, not just be informed by it.
| Legal standard | What it actually requires | What it does not require |
|---|---|---|
| GDPR Article 22 (with Articles 13-15) | Meaningful information about the logic involved, plus significance and consequences of automated decisions | Full mathematical disclosure of model weights or architecture |
| EU AI Act Article 13 | Instructions for use that are relevant, accessible, and comprehensible, enabling deployers to interpret and act on outputs | A specific named technique; the Act is method-agnostic as long as the standard is met |
| General due-process expectation | An explanation specific enough for the affected person to contest the decision | Any explanation that is technically accurate but practically unusable by a layperson |
Note what is absent from both regimes: neither specifies a required algorithm. This is precisely why picking the right technique matters so much. The law sets a standard, not a checklist, which means an organization can satisfy it with a well-implemented counterfactual explanation or fail it with a poorly presented SHAP output. The technique is a means, and regulators evaluate the end result against the standard, not the brand name of the method used to get there.
How the Major Techniques Hold Up Under Scrutiny
SHAP: theoretically grounded, but only as good as its presentation
SHAP, short for SHapley Additive exPlanations, computes feature attribution using a game-theoretic approach borrowed from cooperative game theory, distributing a prediction’s deviation from the baseline across the input features in a mathematically consistent way. It is the most rigorous and most widely adopted feature attribution method precisely because it comes with theoretical guarantees, most notably consistency and local accuracy, that weaker approximation methods lack.
Those guarantees make raw SHAP output more defensible in a regulatory review than most alternatives, because an auditor can verify the attribution mathematically rather than taking the vendor’s word for it. But defensibility depends on how the output is used. A SHAP value expressed as a raw log-odds contribution is not “meaningful information about the logic involved” to the person it was used against; it becomes meaningful only once it is translated into a plain statement tied to that person’s actual inputs, something like “your application was scored lower primarily because of a short credit history and a high existing debt ratio.” The mathematical rigor of SHAP is necessary for legal defensibility but not sufficient on its own; the translation step is what regulators are actually evaluating.
LIME: fast and intuitive, but its instability is a real liability
LIME, Local Interpretable Model-agnostic Explanations, approximates a complex model’s local behavior around a single prediction by fitting a simple, interpretable model, typically linear, to perturbed samples near that data point. It produces simpler, faster, and often more intuitive explanations than SHAP for a single case.
The tradeoff is stability. Because LIME relies on random sampling of the neighborhood around a prediction, running it twice on the same input can produce meaningfully different explanations, and the quality of the local approximation depends heavily on how the neighborhood is defined. In a regulatory or legal context, an explanation that changes when regenerated is a serious weakness. If a rejected loan applicant challenges a decision and the bank’s own tool produces a different rationale the second time it is run, that inconsistency undermines the credibility of the entire explanation process. LIME remains useful for rapid internal exploration; it is a weaker choice as the explanation of record for a contestable, high-stakes decision.
Counterfactual explanations: the closest fit to the legal standard
Counterfactual explanations answer a different question than attribution methods: not “which features mattered,” but “what would need to change to get a different outcome.” An optimization process finds the smallest change to a person’s inputs that would flip the model’s decision, and the result is stated directly in terms the affected person can act on: increase income by a stated amount, reduce an existing balance, wait a certain number of months.
This format maps almost directly onto the “meaningful information about the logic involved” standard in GDPR, arguably better than a SHAP waterfall chart, because it does not require the recipient to understand attribution scores, log-odds, or feature weighting at all. It hands the person a concrete, actionable path. Counterfactual explanations also sidestep a common legal trap: because they describe a hypothetical alternative input rather than claiming to fully expose the model’s internal reasoning, they are less exposed to the objection that the organization has revealed only a partial or misleading account of “why.” For regulatory purposes, counterfactuals are frequently the strongest single technique available, though they work best paired with a feature attribution method that shows the full picture, not just one alternate path.
Attention visualization: popular, but weak as legal evidence
Attention visualization, common with transformer-based models, displays which input tokens or regions received the highest attention weights when the model produced a given output. It is visually compelling and widely used in research papers and internal tooling.
Its weakness as regulatory evidence is well documented: attention weights do not reliably correspond to the features that actually drove a model’s decision. A token can receive high attention weight without being causally important to the output, and a model can be trained to produce a particular output while distributing attention in ways that look plausible but are not faithful to the underlying computation. Presenting an attention map to an auditor as “the reason” for a decision is a weak defense, because the technique itself has known faithfulness problems that a technically informed challenger can raise. Attention visualization remains a legitimate research and debugging tool; it is a poor choice as the primary evidence in a compliance file.
Plain feature importance: useful context, insufficient alone
Global feature importance rankings, whether from a tree-based model’s built-in importance scores or a permutation importance test, tell you which features matter most across the whole population on average. They are useful for governance documentation and model cards, and regulators do want to see this global picture as context. But a global ranking cannot answer the specific question a GDPR data subject or an EU AI Act deployer is actually asking, which is why did this decision happen to this person. Global importance is necessary supporting evidence; it is not, by itself, an individual explanation.
| Technique | Regulatory standing | Best used for |
|---|---|---|
| SHAP (with plain-language translation) | Strong, defensible when output is translated into actionable language | Individual decision explanations in credit, insurance, hiring |
| Counterfactual explanations | Strong, closest natural fit to “meaningful information about the logic” | Consumer-facing adverse decisions where the person can act on the explanation |
| LIME | Moderate, weakened by run-to-run instability | Rapid internal exploration, not the explanation of record for contested decisions |
| Global feature importance | Supporting, not sufficient alone for an individual decision | Model card documentation and governance context |
| Attention visualization | Weak as standalone evidence, known faithfulness problems | Internal research and qualitative model debugging |
Where each technique sits on the internal-to-regulatory spectrum
Counterfactual explanations and translated SHAP output sit closest to the regulatory end of the spectrum; attention visualization and raw feature importance sit closer to the internal-debugging end, useful but insufficient alone as compliance evidence.
Building a Defensible Explanation, Not Just a Technically Correct One
The gap between a technically correct explanation and a legally defensible one comes down to a handful of practical decisions that have nothing to do with the underlying math.
- Translate, do not just compute. A SHAP value or a counterfactual delta is not an explanation until it is rendered in language the affected person can actually use, tied to their specific inputs rather than generic feature names.
- Choose stability over speed for high-stakes, contestable decisions. LIME’s convenience is not worth its run-to-run variability when the explanation may need to be reproduced identically months later in a dispute.
- Document the method choice itself. Regulators and auditors increasingly ask not just for the explanation but for why that technique was chosen for that use case, so the rationale needs to be part of the compliance record, not an implementation detail buried in code.
- Pair individual and global views. An individual counterfactual or SHAP explanation answers “why me”; a global feature importance summary answers “is this fair across the population.” Auditors expect both, not one or the other.
- Test explanation stability over model retraining. If a model is retrained quarterly, confirm the explanation method still produces coherent, comparable output, since a discontinuity here undermines the credibility of the whole process.
Common mistake
Treating whichever explainability library the data science team already had installed as the compliance answer. SHAP and LIME were built to help engineers understand models, not to satisfy a specific legal standard, and dropping raw output from either into a compliance file without translating it into person-specific, actionable language is a routine reason explanations get rejected on review.
What worked
Organizations that pair a counterfactual explanation with a translated, plain-language SHAP summary for the same decision tend to satisfy both the “why” and the “what would change it” questions a regulator or affected individual is likely to ask, without requiring the recipient to understand any underlying mathematics.
- FaithfulnessWhether an explanation actually reflects the computation that produced the output, as opposed to merely looking plausible, which is the specific weakness that undermines attention visualization as legal evidence.
- Explanation stabilityWhether regenerating the same explanation produces a consistent result, a property LIME lacks due to its random sampling and that auditors specifically probe for.
- ActionabilityWhether the explanation tells the affected person something they can actually do differently, which is why counterfactual explanations map so closely to regulatory expectations.
- Method justification recordA documented rationale for why a specific explainability technique was selected for a specific use case, increasingly requested alongside the explanation output itself.
- Global-to-local pairingCombining a population-level feature importance view with an individual explanation, since auditors expect evidence of both overall fairness and case-specific reasoning.
Glossary
- SHAP
- SHapley Additive exPlanations, a feature attribution method based on cooperative game theory that distributes a prediction’s deviation from baseline across input features with mathematical consistency guarantees.
- LIME
- Local Interpretable Model-agnostic Explanations, a method that approximates a model’s local behavior around a single prediction using a simpler surrogate model fit to perturbed samples.
- Counterfactual explanation
- An explanation stating the smallest change to a person’s inputs that would have produced a different model outcome, framed as an actionable path rather than an attribution score.
- Faithfulness
- The degree to which an explanation accurately reflects the computation that actually produced a model’s output, as opposed to merely appearing plausible.
- Meaningful information about the logic involved
- The GDPR standard, drawn from Articles 13 through 15 alongside Article 22, requiring that automated decision explanations be specific and useful rather than vague or purely technical.
- Attention weights
- Values in a transformer model indicating how much a given output token attended to each input token, widely visualized but not reliably indicative of causal importance.
Key Takeaways
- Most explainability techniques were built to help engineers debug models, not to satisfy a specific legal standard, and the two goals require different design choices.
- Neither GDPR nor the EU AI Act names a required technique; both set a standard the explanation must meet, leaving the method choice to the organization.
- Counterfactual explanations tend to map most directly onto GDPR’s “meaningful information about the logic involved” standard because they hand the affected person an actionable path.
- SHAP is more theoretically defensible than LIME, but raw SHAP output still requires translation into plain, person-specific language to count as a real explanation.
- LIME’s run-to-run instability is a genuine liability for high-stakes, contestable decisions, even though it remains useful for fast internal exploration.
- Attention visualization is popular but has documented faithfulness problems, making it weak as standalone evidence in an audit or legal proceeding.
- Auditors increasingly expect both an individual explanation and a global feature importance view, plus a documented rationale for why a given technique was chosen.
FAQs
Do regulators require a specific explainability technique like SHAP or LIME?
No. Neither GDPR nor the EU AI Act names a required algorithm. Both set a standard, meaningful information about the logic involved under GDPR, and accessible, comprehensible instructions for use under the EU AI Act, leaving organizations free to choose whichever technique actually meets that standard for a given use case.
Why are counterfactual explanations considered strong for regulatory purposes?
Counterfactual explanations state the smallest input change that would produce a different outcome, giving the affected person a concrete, actionable path rather than an abstract attribution score. This format closely matches the GDPR standard of meaningful, useful information without requiring the recipient to understand any underlying mathematics.
Is SHAP more legally defensible than LIME?
Generally yes. SHAP carries theoretical consistency guarantees that make its output verifiable and reproducible, while LIME’s reliance on random local sampling means the same input can produce different explanations on separate runs, which is a real weakness if an explanation must be reproduced identically during a dispute.
Why is attention visualization considered weak as legal evidence?
Research has repeatedly shown that attention weights in transformer models do not reliably correspond to the features actually driving a model’s output. A token can receive high attention without being causally important, so presenting an attention map as “the reason” for a decision is vulnerable to a well-informed technical challenge.
What does GDPR Article 22 actually require regarding explanations?
Article 22 restricts certain solely automated decisions with legal or similarly significant effects and, read alongside Articles 13 through 15, requires that the affected person receive meaningful information about the logic involved plus the significance and envisaged consequences of the processing, not the full mathematical detail of the model.
What does the EU AI Act’s Article 13 require for high-risk systems?
Article 13 requires providers of high-risk AI systems to accompany the system with instructions for use containing information that is relevant, accessible, and comprehensible to deployers, sufficient for them to interpret the system’s output and use it appropriately, applying from August 2026.
Can a global feature importance ranking satisfy an individual’s right to an explanation?
Not on its own. A global ranking shows which features matter most across the whole population, which is useful governance context, but it does not answer why a specific decision was made about a specific person, which is what an individual explanation, such as a counterfactual or a translated SHAP output, is required to address.
How should an organization choose between these techniques for a given use case?
Match the technique to the stakes and the audience. For high-stakes, contestable individual decisions, pair a counterfactual explanation with a translated, plain-language attribution summary; reserve faster, less stable methods like raw LIME output or attention maps for internal debugging rather than the compliance record.
References
- Lundberg, S. M., and Lee, S. I. “A Unified Approach to Interpreting Model Predictions.” Advances in Neural Information Processing Systems, 30 (2017).
- Ribeiro, M. T., Singh, S., and Guestrin, C. “Why Should I Trust You? Explaining the Predictions of Any Classifier.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016).
- European Union. “Article 22: Automated Individual Decision-Making, Including Profiling.” General Data Protection Regulation.
- European Union. “Article 13: Transparency and Provision of Information to Deployers.” EU Artificial Intelligence Act.
- Molnar, C. “Interpretable Machine Learning: A Guide for Making Black Box Models Explainable.”
- Jain, S., and Wallace, B. C. “Attention Is Not Explanation.” Proceedings of NAACL-HLT (2019).
For related coverage on this site, see how these individual techniques feed into broader documentation practices in model cards and system cards, how they connect to systematic fairness testing in ML fairness auditing tooling, and how jurisdictions differ on explanation requirements in global AI ethics standards comparison. Readers building business-facing interfaces around these same methods should also see explainable ML dashboards for business users, which covers the presentation-layer, non-legal side of this topic. Evaluation of whether an explanation technique performs reliably ties into LLM evaluation frameworks, and organizations building compliance workflows around these techniques may also find automated compliance relevant.
