Why some transformer layers resist quantization

If quantization damaged every part of a transformer equally, precision allocation would be a pointless exercise: pick the lowest format the whole model tolerates and stop. The reason mixed precision is interesting at all is that the damage is uneven, and unevenly in ways that seem to repeat across models.

The pattern I keep seeing

Quantizing one block at a time and measuring what happens is a slow experiment, but it is a clarifying one. Two things show up consistently in my own runs and in the papers I read.

Depth matters. The first blocks and the last blocks tend to give up more quality than the middle of the network. The early ones are still turning input into a usable representation, and the late ones are close enough to the output that an error has no remaining layers in which to be corrected.

Component matters as much as depth. Within a block, the projections are not equivalent. The matrices that write back into the residual stream — the attention output projection and the second feed-forward projection — appear more sensitive than the ones that read from it. That ordering makes sense if the residual stream is a shared channel: corrupting what is written into it affects everything downstream, while corrupting a read only affects one computation.

Explanations I find plausible

• Residual accumulation. Errors written into the residual stream persist. Errors in an intermediate activation are often attenuated by whatever follows.
• Dynamic range. Some tensors simply have wider distributions, so a fixed number of levels covers them worse. Sensitivity may be partly a range problem wearing a different costume.
• Redundancy. Wide feed-forward layers seem to carry redundant directions, and redundancy is exactly what tolerates coarse representation. Attention projections are narrower and apparently less redundant.

I do not think these are competing explanations so much as different levels of description, but I cannot yet separate their contributions experimentally.

Measuring it honestly is the hard part

The tempting shortcut is to quantize each component alone, record the loss increase, and treat the resulting numbers as independent sensitivities that can be added up. That assumption is convenient and probably wrong: quantizing two components together can interact, either cancelling or compounding.

So the practical questions are methodological.

1. Is one-at-a-time measurement a good enough approximation to drive a search, or does it need pairwise correction terms?
2. Should sensitivity be measured as task loss, as output divergence from the full-precision model, or as per-tensor reconstruction error? These disagree, and they disagree most for the components I care about.
3. How stable is the profile across calibration sets? A sensitivity ranking that shifts with the sample is not a property of the model.

Why this matters for the thesis

My thesis takes the sensitivity profile as the input to precision allocation, which means the allocation is only as trustworthy as the profile. Before searching over allocations, I want to know how much the ranking moves under different measurement choices — because if it moves a lot, the interesting result is about measurement rather than about compression.