How much calibration data does post-training quantization need?

Post-training quantization is appealing because it is cheap: no retraining, no labels, just a small sample of representative inputs used to estimate ranges. The awkward part is that this small sample determines the numerical behaviour of the model for every input it will ever see.

What calibration is deciding

For each tensor being quantized, calibration answers one question: what range should the grid cover? Everything follows from that. Set the range too wide and resolution is wasted on values that rarely occur. Set it too narrow and real values are clipped.

The estimate is made from the calibration sample, so the sample has to contain the behaviour that matters. Not the average behaviour — the extremes, because those are what the range is set by.

Small samples, but the right small samples

The literature generally reports that a modest number of examples is enough, often in the low hundreds, and that returns diminish quickly after that. My reading of why: ranges are statistics of activations, and activation statistics stabilise long before task performance would.

What seems to matter more than quantity is coverage.

• Inputs from the distribution the model will actually be given, not a convenient public sample that happens to be nearby.
• Sequence lengths that reach what deployment will reach, since activation magnitudes are not length-independent.
• Enough diversity to include the rare-but-real inputs that produce the largest activations.

A thousand near-identical examples are worth less than a hundred varied ones, which is an uncomfortable property because it makes calibration quality hard to certify.

Choosing the estimator is part of the method

Given a sample, there is still a decision about how to turn observations into a range. The minimum and maximum are the naive choice and the most fragile, since one outlying batch sets the range. Percentile clipping, mean-squared-error search over candidate ranges, and entropy-based selection all trade clipping error against resolution differently.

I have started treating the estimator as a reportable part of an experiment rather than an implementation detail, because two runs with the same bit-width and different estimators are not the same experiment.

The part that bothers me

Calibration introduces a dependence on data into a method advertised as data-free-ish. That dependence is rarely audited. If the sample under-represents a subgroup, the ranges are tuned away from that subgroup's activations, and the degradation is unlikely to be uniform across groups.

I have not seen this measured carefully, and it connects directly to the fairness auditing I did on the medical classification project. It is currently on my list of things to test properly rather than something I can claim.