Causal Interventions and Sparse Features Outperform Attention Maps in Reasoning LLMs

Large language models light up with attention heatmaps, but the glow is often misleading. Across dense Transformers, Mixture-of-Experts (MoE) models, and retrieval/tool-use systems, raw attention weights routinely fail basic checks of faithfulness, completeness, and stability in reasoning tasks. When attention looks most persuasive, it is often because it tracks where information flowed, not how decisive computations were performed. The real action lives elsewhere: in MLP/residual pathways, routing policies, and sparse, reusable features that survive paraphrase and decoding changes.

This matters now because models are increasingly evaluated on multi-step, compositional reasoning benchmarks such as GSM8K, MATH, BBH, MMLU, ARC, and DROP. In these settings, inspecting attention alone gives a partial—and frequently wrong—story of why a model reached an answer. This article details the mechanisms underlying that gap, explains where attention-based explanations still add value, and lays out what replaces them: causal interventions, feature-level analyses, and carefully validated attributions that can withstand counterfactual tests. Readers will learn where and why attention explanations break, what interventional and sparse-feature methods recover, and how to evaluate reasoning claims in dense, MoE, and RAG/tool-use systems with confidence.

Architecture/Implementation Details

Limits of raw attention: non-uniqueness, manipulability, and failed faithfulness/stability

• Non-uniqueness: Multiple distinct attention configurations can yield the same output. That undermines any claim that observed weights uniquely explain a prediction.
• Manipulability: Attention can be perturbed without changing outputs, producing attractive but unfaithful “explanations.”
• Missing mediation: Even path-aggregated methods like attention rollout/flow visualize influence but miss decisive computations mediated in non-attention pathways.
• Stability failures: Attention patterns swing under paraphrase, adversarial edits, and decoding changes, breaking consistency requirements for explanations.

Net effect: Raw attention functions best as a visibility mechanism for routing, not a faithful account of computation.

Dense decoder-only Transformers: MLP/residual pathways as key–value memories; induction heads as a validated exception

Mechanistic audits repeatedly localize factual associations and compositional transformations to the MLP/residual stack, not the attention matrices. Feed-forward layers act as key–value memories, retrieving and transforming latent features that ultimately decide predictions. This holds up under targeted knowledge editing, which reliably changes outputs by modifying non-attention parameters, and under activation patching and causal scrubbing, which identify decisive computations outside attention.

• Validated exception: Induction heads implementing copying/next-token induction are a prominent, replicable attention-mediated circuit. Here, head-level ablations and patching demonstrate causal necessity; attention is genuinely explanatory because the computation is mechanistically understood and attention-mediated.
• Reasoning benchmarks: On GSM8K, MATH, BBH, MMLU, ARC, and DROP, reasoning relies on distributed features across many layers. Attention weights fail to recover the actual internal steps producing correct answers and degrade under paraphrase and decoding changes. Specific metrics are unavailable, but consistency of these findings across tasks is emphasized.

Implication: Treat attention in dense models as a component of routing and tracing token-to-token interactions, not as the primary locus of reasoning.

Mixture-of-Experts Transformers: routers and expert MLPs dominate causal pathways omitted by attention maps

MoE architectures introduce per-token routing to specialized experts (most often MLPs). The router’s logits and the selected expert computations add decision points that self-attention weights do not expose.

• Dominant causal pathway: Router decisions and expert MLP activations frequently determine outcomes. Attention maps, even when aggregated across heads and layers, omit this control flow.
• Increased opacity: Head roles become less informative because crucial determinants move to the routing plane. Effective interpretability requires examining the routing distributions and intervening on expert internals.

Takeaway: In MoE models, attention-only explanations are even less complete than in dense models because they ignore the most consequential choices.

Retrieval and tool-use systems: cross-attention as provenance, not proof of reliance

In retrieval-augmented generation (RAG) and RETRO, cross-attention to specific passages provides credible provenance—which sources were consulted. That visibility aids auditing, but it does not validate whether the model used the content correctly in reasoning. Hallucinations and misattributions can persist despite attention to relevant passages.

• Stronger test: Leave-one-document-out retrieval and controlled context removal demonstrate actual reliance by observing performance changes when purportedly critical documents are withheld.
• Tool-augmented agents: Attention over tool tokens and outputs reflects surface reading, not decision policies. Faithful explanations require tracing routing decisions, function selection, and execution results through causal audits and ablations.

Bottom line: Use cross-attention for source attribution; use interventions to establish reasoning over retrieved content and tool choices.

Comparative performance: gradients versus attention; activation patching, causal scrubbing, and knowledge editing as strongest evidence

• Gradient-based attributions (Integrated Gradients, Layer-wise Relevance Propagation) satisfy useful axioms such as completeness and frequently align better with causal influence than raw attention, especially when path-aware. They remain sensitive to baselines and can capture correlations without causation unless validated.
• Causal methods—activation/attention patching, causal scrubbing, and targeted knowledge editing—provide the strongest evidence of faithfulness. These techniques enable necessity/sufficiency tests and circuit localization that generalize across inputs better than attention weights.
• Representation-level approaches: Sparse autoencoders (SAEs) and probing uncover sparse, interpretable features that recur across layers and models. These features are more stable under paraphrase and decoding variation and provide a truer substrate for explaining reasoning than raw attention patterns.
• Model-generated chain-of-thought (CoT): Helpful for performance and readability, but frequently post hoc and unfaithful to internal computation; never accept as explanation without triangulation via interventions.

Benchmark-driven findings: distributed computation and instability under paraphrase/decoding

Across GSM8K, MATH, BBH, MMLU, ARC, and DROP:

• Attention-only methods miss multi-hop, algebraic, and factual transformations that decide final answers.
• Attention can highlight plausible tokens or spans while failing faithfulness under intervention-based audits.
• Occasional attention-mediated sub-steps (e.g., copying) appear, but end-to-end correctness depends on interactions in MLP/residual pathways and distributed features.
• Quantitative breakdowns are model- and setup-dependent; specific metrics are unavailable.

Scaling effects and superposition: why attention roles degrade with size and long context

As model size and context length grow:

• Superposition increases: Features overlap within neurons and heads, making head roles less clean and attention patterns less stable.
• Redundancy in head configurations blunts head-importance signals; sparse/linear attention variants do not consistently improve faithfulness at the weight level.
• Long-context scenarios diffuse attention over many tokens; visualization (e.g., attention flow) can help but remains incomplete without interventions.
• Decoding parameters alter attention distributions and token paths, further eroding stability. Domain/language shifts change head specialization, limiting cross-task transfer of attention-based explanations.

Conclusion: Scale and long context amplify the weaknesses of attention-as-explanation while strengthening the case for feature-level analyses and causal tests.

Comparison Tables

Explanatory methods for reasoning LLMs

Method family	Causal faithfulness	Completeness	Stability/robustness	Cross-model/task transfer	When most effective
Raw attention weights	Low; can be manipulated without output change	No	Low; sensitive to paraphrase/decoding	Poor	Quick plausibility checks; early layers; small models
Head importance/pruning	Mixed; redundancy obscures causality	No	Moderate; task-dependent	Limited	Identifying dispensable heads; coarse specialization
Attention rollout/flow	Better than raw maps but incomplete	Partial at best	Moderate; still brittle without interventions	Limited	Long-range influence visualization; paired with causal tests
Attention masking/mediation	Higher when pre-registered and causal	Partial	Moderate to high (experiment-dependent)	Moderate	Testing specific attention circuits (e.g., induction heads)
Gradients/IG/LRP	Moderate to high with careful design	Yes (IG)	Moderate; baseline-sensitive	Moderate	Token-/layer-wise attribution; validated with interventions
Causal tracing/patching/editing	High; strongest evidence	N/A (interventional)	High (with controlled designs)	Moderate to high (circuit-level)	Mechanistic localization; counterfactual testing
Representation features (probes/SAEs)	Moderate; improves with interventions	N/A	Moderate to high (feature-dependent)	Moderate to high (feature-level)	Discovering stable features; complements patching
Model-generated CoT	Low (often post hoc)	No	Variable	Poor	Human-facing rationales; not explanations

What attention shows—and misses—by architecture/setting

Architecture/setting	What attention reveals	What attention misses	Additional components needed
Dense Transformers	Circuits for induction/copying; some entity tracking	MLP/residual-mediated computations; distributed features	Activation patching, mediation, feature analyses
MoE LLMs	Token-to-token routing via self-attention	Router decisions; expert computations	Router logit audits; per-expert interventions
RAG/RETRO	Which passages were consulted (provenance)	Whether evidence was used correctly; reasoning over content	Leave-one-out retrieval and context ablations; causal tracing
Tool-augmented agents	Surface attention to tool tokens	Policy for tool selection; execution reliance	Causal audits of tool routing and outputs

Best Practices

A disciplined evaluation protocol turns interpretability from glossy pictures into testable science 🔬

• Start with mechanistic hypotheses:

• Specify candidate heads, paths, or features believed to mediate a computation (e.g., an induction head or a sparse feature representing carry in arithmetic).

• Pre-register expectations where possible to avoid hindsight bias.

• Triangulate explanations:

• Compute multiple signals: raw attention, attention flow, gradients/IG/LRP, and candidate feature activations from SAEs or probes.

• Use each as a generator of hypotheses, not as proof.

• Run causal tests:

• Head/path masking and attention editing to test attention-mediated claims.

• Activation patching across layers to identify decisive locations and features.

• Causal scrubbing to replace hypothesized variables with counterfactuals and check whether predictions follow.

• Evaluate on reasoning benchmarks with robustness checks:

• Use GSM8K, MATH, BBH, MMLU, ARC, and DROP as primary arenas.

• Stress stability with paraphrases, adversarial/counterfactual edits, and varied decoding settings.

• Track performance and qualitative behavior under targeted interventions; specific metrics may be unavailable but should be recorded when possible.

• For MoE models:

• Log and analyze router logits and expert selections alongside attention.

• Perform per-expert interventions to validate causal roles.

• For RAG and tool-use systems:

• Treat cross-attention as provenance, not reliance.

• Use leave-one-document-out retrieval and structured context ablations to verify dependence on specific sources.

• For tools, audit routing and execution results; ablate tool outputs to confirm necessity.

• Prefer feature-level substrates:

• Use SAEs or targeted probes to surface sparse, interpretable features that recur across layers/models.

• Validate feature causality with activation patching and localized edits.

• Handle CoT carefully:

• Collect CoT for human comprehension and performance gains.

• Do not equate CoT with the model’s internal computation without supporting causal tests.

• Document controls:

• Record model size, attention head configurations, router visibility (MoE), retrieval set composition, decoding hyperparameters, CoT usage, and domain/language so results are interpretable and transferable.

• Report limitations:

• Be explicit when metrics are unavailable or when evidence is specific to tasks, architectures, or setups.

Conclusion

Attention maps changed how practitioners visualize neural models, but they are not up to the job of explaining reasoning in today’s LLMs. Decisive computations typically unfold in MLP/residual pathways and routing policies, and the signals that best recover those computations come from causal interventions and feature-level analyses, optionally supported by carefully designed gradients. Attention retains value in narrow, mechanistically specified settings—induction heads and cross-attention provenance—but fails as a general-purpose explanation of reasoning. The path forward blends hypothesis-driven experiments with interventional audits and sparse features that stand up to paraphrase, decoding variation, and architectural shifts.

Key takeaways:

• Attention is visibility, not computation: treat it as routing evidence unless validated causally.
• MLP/residual pathways and MoE routing/expert choices are the dominant causal loci.
• Causal methods (activation patching, causal scrubbing, knowledge editing) provide the strongest proof of explanation fidelity.
• Sparse features from SAEs and probing offer a more stable explanatory substrate than head-level weights.
• Cross-attention in RAG is good for provenance; reliance requires leave-one-out and ablation tests.

Actionable next steps:

• Build evaluation harnesses that automate activation patching, mediation, and leave-one-out tests across benchmarks.
• Incorporate router/expert logging in MoE interpretability pipelines.
• Train and deploy SAEs to furnish candidate features; prioritize features that transfer across tasks.
• Treat CoT as a user interface feature, not an explanation, unless causally validated.

Looking ahead, scaling will continue to magnify superposition and distribute computation. Explanations that center on causal interventions and sparse, mechanistic features will travel best across architectures and tasks, while attention maps will remain useful—but only in the narrow lanes where the computation itself is known to be attention-mediated.

Sources & References