Are we interpreting the model of interest?

The model we interpret should be the same computational artifact that exhibits the behavior of interest.

Part 0: Getting attention weights can make a model behave differently

A while ago, I was analyzing the attention patterns of a vision-language model when I noticed something strange: its accuracy on my benchmark dropped from about 70% to 20% after I enabled output_attentions=True. I hadn't changed anything else other than that one argument for output_attentions. How could observing the model change its behavior so much?

One warning message caught my eye:

`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.

So, requesting attention weights changed the attention implementation. The model I was trying to inspect was no longer being executed in the same way as the model whose behavior interested me.

Motivated by the huge performance drop, I wanted to understand whether switching attention implementations really could change model outputs, how much floating-point precision mattered, and whether there was a way to recover attention weights without switching away from the original computation.

That also leads to a broader question for interpretability: how do we make sure that we are studying the computational artifact that actually exhibited the behavior of interest?

Part I: Same weights can produce different behavioral outputs under different attention implementations

Attention implementations are supposed to compute the same underlying operation. But mathematically equivalent doesn't always mean numerically identical: fused kernels may group and round floating-point operations differently. The practical question is whether these numerical differences can lead to different observed model behavior (i.e. generated text). In the meantime, small rounding errors that do not lead to a difference in behavior would be fine and should not be a big problem.

I compared three attention backends while holding everything else fixed:

• Eager: Hugging Face attn_implementation="eager", which explicitly materializes attention weights.
• SDPA: attn_implementation="sdpa", using PyTorch's regular scaled-dot-product-attention dispatch.
• FA2: attn_implementation="flash_attention_2", using the external FlashAttention 2 package.

All continuations below use greedy decoding to make sure the process is deterministic. If two runs generate different token sequences, the difference cannot be attributed to random token sampling, and must be from the attention implementation.

The same input results in different continuations

We begin with Qwen3-VL-4B-Instruct in BF16, using the prompt "Describe the image." and a COCO image of a bakery counter.

The identical input image for all three BF16 generations: COCO val2017 image 226903. Prompt: “Describe the image.”

The full greedy generations are shown below. Yellow highlights mark spans differing from both other backends under pairwise word/punctuation alignment.

All three executions describe roughly the same scene, but the details and wording diverge.

To see whether this was an isolated example, I ran the same prompt on 20 randomly sampled COCO images. The counts below shows whether the there exists any generated different token among the output sequences between each pair of backends:

15-17 out of 20 generations result in different output sequences across the three attention backends. While this does not necessarily mean they convey very different semantic information, it is already quite bothersome if we want to study some low-level/token-level mechanisms or predictions -- just by using another attention backend, the mechanism of interest may no longer be used.

from urllib.request import urlopen

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "Qwen/Qwen3-VL-4B-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
vlm = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype=torch.bfloat16, attn_implementation="eager",
).to("cuda").eval()
image = Image.open(urlopen(
    "https://s3.amazonaws.com/images.cocodataset.org/val2017/000000226903.jpg"
)).convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Describe the image."},
]}]
inputs = processor.apply_chat_template(messages, tokenize=True,
    add_generation_prompt=True, return_dict=True, return_tensors="pt").to("cuda")


def compare_backends(model, inputs, decoder, backends):
    logits, texts = {}, {}
    for backend in backends:
        model.set_attn_implementation(backend)
        with torch.inference_mode():
            logits[backend] = model(**inputs, use_cache=False).logits[0, -1].cpu()
            tokens = model.generate(**inputs, do_sample=False, num_beams=1,
                                    max_new_tokens=128)
        generated = tokens[0, inputs["input_ids"].shape[1]:]
        texts[backend] = decoder.decode(generated, skip_special_tokens=True)
    return logits, texts


backends = ["eager", "sdpa", "flash_attention_2"]
logits, texts = compare_backends(vlm, inputs, processor, backends)
for backend, text in texts.items():
    print(backend, text)

The differences are visible in next-token logits

Different continuations can amplify small numerical changes over many generation steps. To take a closer look at the upstream cause, here I compare next-token logits on the identical image-and-prompt prefix.

For every input, I calculate the root-mean-squared difference between each pair of backends' logits, then average this value over the 20 inputs:

So the runs already have measurably different next-token scores on the same prefix, even before we compare their later continuations.

import torch


def pairwise_logit_differences(logits_by_input):
    # One [backend, vocabulary] tensor per identical input prefix.
    z = torch.stack(logits_by_input).double()  # [input, backend, vocabulary]
    difference = z[:, :, None, :] - z[:, None, :, :]
    rmse = difference.square().mean(-1).sqrt().mean(0)
    max_abs = difference.abs().amax(-1).mean(0)
    return rmse, max_abs


# Opening example only. For the 20-input average, pass 20 such tensors.
logits_by_input = [torch.stack([logits[b] for b in backends])]
rmse, max_abs = pairwise_logit_differences(logits_by_input)
print(rmse)
print(max_abs)

Precision makes a substantial difference

If the choice of attention backend matters because of numerical computation, what happens when we change the floating-point precision?

I repeat the Qwen3-VL comparison on the same 20 inputs, holding precision fixed within each backend comparison:

The implementations agree much more closely at higher precision on these inputs. In FP32 especially, eager and SDPA generated identical token sequences across all 20 inputs, with a near zero logit RMSE difference. This pattern suggests that float number precision also plays an important role in determining model's behavior, as suggested in prior work (e.g. Qi et al., 2025).

However, since many models (such as the Qwen models) are trained in BF16 with flash attention and suggests inference to use the same setup, even if there is a better consistency across attention backends at FP32, switching to FP32 would cause a difference in behavior that is undesirable for carrying out interpretability analysis (see appendix for related comparisons).

import torch
from transformers import AutoModelForImageTextToText

# Reuse inputs, processor and compare_backends from the first example.
for dtype in (torch.float16, torch.float32):
    methods = ["eager", "sdpa"]
    if dtype == torch.float16:
        methods.append("flash_attention_2")  # FA2 does not support FP32.
    precision_model = AutoModelForImageTextToText.from_pretrained(
        model_id, dtype=dtype, attn_implementation="eager",
    ).to("cuda").eval()
    z, text = compare_backends(precision_model, inputs, processor, methods)
    print(dtype, text)
    print(pairwise_logit_differences([torch.stack([z[b] for b in methods])])[0])
    del precision_model

This isn't limited to vision-language models

Could this be a quirk of image inputs or Qwen3-VL? To check, I repeat the BF16 comparison on Qwen3-4B (a separately trained text-only model) and GPT-2 (the OG decoder-only transformer). Both models received the same 20 unfinished prose prefixes as literal text, with no chat template or special tokens added.

The following example shows the complete generations for both models, with differences highlighted.

Example: greedy continuations

Prefix: Fifty years later, the small coastal town had become

Both models use BF16 and greedy decoding. Yellow highlights mark spans that disagree with both other backends within that model.

Qwen3-4B

GPT-2

Across the full 20-prefix sample:

Differences across generated outputs are found in the tested text-only models too and it is not exclusive to VLMs.

Mean pairwise next-token logit RMSE, averaged over 20 inputs per checkpoint. These are within-model backend comparisons, not comparisons between different checkpoints.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# One prefix is shown; use continuation20.json for the 20-prefix average.
prefix = "Fifty years later, the small coastal town had become"
for checkpoint in ("Qwen/Qwen3-4B", "openai-community/gpt2"):
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    lm = AutoModelForCausalLM.from_pretrained(
        checkpoint, dtype=torch.bfloat16, attn_implementation="eager",
    ).to("cuda").eval()
    text_inputs = tokenizer(prefix, add_special_tokens=False,
                            return_tensors="pt").to("cuda")
    z, text = compare_backends(lm, text_inputs, tokenizer, backends)
    print(checkpoint, text)
    print(pairwise_logit_differences([torch.stack([z[b] for b in backends])])[0])
    del lm

The interpretability problem remains: if requesting attention weights causes a switch from SDPA or FA2 to eager, we may end up measuring internal patterns from a run that doesn't even generate the same sequence as the original. What then should we do to (1) access faithful intermediate computational variables and (2) not change the model's native computation graph?

Part II: Recovering attention from fused kernels

Part I exposed a practical problem: the attention implementation that produces the behavior we want to understand may not expose its attention weights. With optimized backends such as SDPA and FlashAttention, the full attention matrix is often never materialized as an accessible tensor. Asking for it through output_attentions=True can instead switch the model to eager attention—and, as we've seen, switching implementations can change the generated continuation.

Can we recover the attention weights of the original backend without switching to a different implementation?

This is in principle doable. We don't need the kernel to return its attention matrix. We can patch the values supplied to it and observe its output. By choosing the values carefully, we can leverage the existing blackbox attention kernel to reconstruct the attention weights. This idea was suggested in this PyTorch discussion. I'll walk through the intuition, explain how to use it for long sequences, and then test how accurately it recovers attention from real backends.

1. The trick: reconstructing attention weights with identity value matrices

The final step of a single attention head is:

Here, \(A\) is the attention matrix we want, \(V\) contains the value vectors, and \(Z\) is the attention output. An optimized kernel may give us \(Z\) without ever exposing \(A\).

Consider the following observation: if we set \(V\) to an identity matrix, the output becomes \(A\) itself, via \(AI=A\).

That gives us a way to reconstruct attention weights using the original kernel. We leave all other variables unchanged, and replace only the value vectors.

In practice, the value dimension is usually much smaller than the sequence length, so we can't fit the entire identity matrix into a single attention call. We can use partial identities: each call reveals a block of columns, and several calls recover the complete matrix.

Reading attention from its output. (A) Replacing the value vectors with a partial identity reveals the corresponding columns of the attention matrix. (B) Shifting the identity entries across successive passes recovers the remaining columns, which are concatenated.

For example, if a sequence has six key positions but each value vector has only two dimensions, each identity-value pass reveals two columns of attention. Three passes recover all six columns. More generally, a sequence with \(T_k\) key positions and value dimension \(d_v\) takes \(\lceil T_k/d_v\rceil\) passes.

The key part is that this doesn't require the fused kernel to produce a full attention matrix internally. We reconstruct it from the kernel's responses to carefully chosen values.

import torch

for length, head_dim in [(2, 4), (6, 2)]:
    # Toy attention weights: the fused kernel is represented here by A @ V.
    A = torch.softmax(
        torch.randn(length, length, dtype=torch.float64), dim=-1
    )
    blocks = []

    for offset in range(0, length, head_dim):
        width = min(head_dim, length - offset)
        V = torch.zeros(length, head_dim, dtype=A.dtype)
        V[offset:offset + width, :width] = torch.eye(
            width, dtype=A.dtype
        )
        Z = A @ V
        blocks.append(Z[:, :width])

    recovered = torch.cat(blocks, dim=-1)
    torch.testing.assert_close(recovered, A, atol=0, rtol=0)
    print((length, head_dim), recovered.shape)

2. Applying the trick to a real attention implementation

The mathematical idea is simple. To use it with a real transformer, we need access to the attention operation at the point where its values enter and its output leaves. We need to be able to replace intermediate value tensors and read intermediate outputs.

Using NNsight, we can do the following for a selected attention head on HF open-sourced models:

1. Run the model with the desired setup, so that the queries and keys correspond to the computation we're studying.
2. Repeat the attention computation with successive partial-identity value tensors, keeping the queries, keys, mask, and other attention settings fixed.
3. Read each resulting attention output at the boundary before the output projection (o_proj), and concatenate the recovered blocks to construct the full attention matrix.

Through these steps, we get both the behavior of interest, and the attention weights reconstructed from the corresponding runs.

There is one practical detail when extracting many layers or heads: replacing values at an early layer can change the inputs to later layers. If we want attention weights from the original later-layer computation, we must keep its inputs fixed. We use a clean cache built from an unmodified execution and restore the original activation before continuing to collect later layer attention weights: this reduces the number of runs we needed to reconstruct attention weights across all layers.

3. How good is the recovery?

There is another caveat. The equation \(Z=AV\) holds exactly in principle, but an optimized GPU kernel computes in finite precision and can fuse or reorder operations. The recovered matrix is therefore best viewed as a numerical approximation to the attention weights relevant to that backend.

To evaluate that approximation, we ask the following question: can the recovered attention matrix reconstruct the attention output observed during the original forward pass?

We first recover a matrix \(A_{\mathrm{pred}}\) using partial identities. Then we multiply it by the original value vectors \(V_{\mathrm{real}}\), and compare the prediction with the clean attention output \(Z(V_{\mathrm{real}})\) from the original forward pass.

We measure their difference as relative reconstruction error: a smaller percentage means the recovered matrix more closely reproduces the native kernel's output.

Across all 1,620 comparisons, reconstruction errors range from 0.086% to 0.297%, with a mean of 0.182%. These results show that the recovered matrices thus closely reproduce the attention outputs observed during the original forward pass.

4. How does backend-recovered attention compare with the eager-returned attention weights?

We could also obtain attention weights from eager and use them to analyze a model run under SDPA or FA2. Part I gives us a reason to be cautious: separate backend executions can differ even when the inputs are identical. In this section, we compare the error from using the extracted attention weights from a run's own backend to the error from using the eager-returned attention weights. For Qwen3-VL decoder attention under FA2, for example, the mean errors are 0.192% with FA2-recovered weights and 10.593% with eager-returned weights.

Attention-output reconstruction error (%) — lower is better

For both SDPA and FA2, using weights recovered from the target backend gives lower mean reconstruction error than using the eager-returned attention weights. This supports the use of backend-specific extraction when the goal is to explain the attention computation that actually ran.

5. Recovering attention in practice

I've packaged this approach as AttnRead so others can use it in their own research:

pip install attnread

Related issues reported online

Other users have encountered related problems when comparing or inspecting attention implementations:

• Different logits on image inputs. A Qwen2.5-VL discussion, also reported in Transformers issue #39067, describes larger backend logit differences for image inputs than for text inputs.
• Changed behavior while requesting attention weights. Transformers issue #35270 reports unexpected outputs when using eager attention to inspect a LLaVA model. It illustrates the practical concern that the attention-inspection configuration can affect the behavior being studied.
• Fused attention does not normally expose the full pattern. PyTorch issue #119811 discusses this interpretability need and suggests probing with identity values.

Discussion

When we say we are interpreting a model, what exactly identifies that model? A checkpoint specifies its parameters, but those parameters can be executed through different computational implementations. As we've seen, changing the attention backend or floating-point precision can produce different greedy continuations, even when the model weights, inputs, and decoding settings remain fixed. In practice, a checkpoint alone does not fully specify the computational artifact whose behavior we are trying to explain.

This matters when our interpretability methods require modifying how the model is executed. Suppose a model exhibits some behavior under a fused attention backend, but requesting its attention weights silently switches the implementation to eager. The resulting attention patterns may come from an execution that behaves differently on the very input we're interested in. Even if the two executions produce similar outputs, we cannot simply assume that their internal computations are interchangeable.

Our approach to attention recovery illustrates one way forward: rather than changing the computation to make it easier to inspect, we can design our measurements around the computation we actually want to understand. By recovering attention weights directly from the original backend, and validating them against its native outputs, we can establish a closer connection between the quantities we extract and the computation that produced them.

Of course, not every numerical difference between implementations will matter for every interpretability question. Different token sequences do not necessarily imply different semantic behavior, and small numerical discrepancies need not invalidate an analysis. What matters is whether the measurements we obtain faithfully characterize the computation relevant to the phenomenon we're studying.

The broader lesson extends beyond attention visualization. Whenever an interpretability method changes the execution environment, numerical precision, or computation being inspected, we should ask whether those changes affect what we're trying to explain. Before explaining a model's behavior, we need to establish that we're interpreting the computation that produced it.

Appendix: eager generation across precisions

Qwen-VL

The comparisons above change the attention implementation while holding precision fixed. We can also hold eager attention fixed and change precision. On the same 20 Qwen-VL images, pairwise counts of different greedy token sequences are:

We see that with the same attention backend, changing the float number precision significantly alters the model's token level behavior. FP16 and FP32 produce relatively closer outputs, and BF16 produces very different outputs from the other two precisions we tested.

Text models

The same 20 raw continuation prefixes are used in BF16, FP16, and FP32. Holding eager fixed gives these counts of different greedy continuations:

Text models also show similar patterns: with the eager backend fixed, changing float precisions significantly change their token-level behaviors and lead to divergent generated sequences.

The within-precision backend controls on those same prefixes are:

FP16 reduces the divergent generated sequences by a lot, and under FP32 there is no divergent behavior between eager and SDPA backends, which is consistent with the patterns we see in the main post. FA2 is unavailable in FP32.

Appendix: five more images with different continuations

We show five more examples of the qualitatively different continuations from different attention backends from the qwen VL model.

COCO image 295809

COCO val2017 image 295809. BF16, fixed checkpoint, prompt: “Describe the image.”

COCO image 557884

COCO val2017 image 557884. BF16, fixed checkpoint, prompt: “Describe the image.”

COCO image 173057

COCO val2017 image 173057. BF16, fixed checkpoint, prompt: “Describe the image.”

COCO image 102331

COCO val2017 image 102331. BF16, fixed checkpoint, prompt: “Describe the image.”

COCO image 195165

COCO val2017 image 195165. BF16, fixed checkpoint, prompt: “Describe the image.”

Appendix: five more prefixes with different continuations

We show five more examples of the qualitatively different continuations from different attention backends for the text models.

Prefix 7

Prefix: The invention looked ordinary until someone pressed

Qwen3-4B

GPT-2

Prefix 8

Prefix: After the storm, the forest path was covered with

Qwen3-4B

GPT-2

Prefix 11

Prefix: For ten years the neighbors had never spoken, until one morning

Qwen3-4B

GPT-2

Prefix 12

Prefix: When people first learned to hear plants growing, they discovered

Qwen3-4B

GPT-2

Prefix 13

Prefix: "I think a place becomes home when," she said,

Qwen3-4B

GPT-2

Citation

If you want to cite this post, please use the following citation.

@misc{hua2026interpreting,
  author       = {Hua, Etha Tianze},
  title        = {Are we interpreting the model of interest?},
  year         = {2026},
  month        = Nov,
  howpublished = {Blog post},
  url          = {https://ethahtz.github.io/blogs/model_of_interest.html}
}