Accuracy, latency, memory: reading a three-way trade-off

"The compressed model is 4× smaller" is not a result. Neither is "accuracy dropped by 0.6 points". Both are single coordinates from a space with at least three axes, and the interesting question is always what was given up on the other two.

Three axes that do not move together

Accuracy, latency, and memory respond differently to the same change.

Dropping weights to a lower-bit format reduces memory immediately and predictably. Latency only improves if a kernel can consume that format without unpacking it back to a wider type first — otherwise the model is smaller and no faster, or slightly slower. Accuracy may not move at all, or may fall off a cliff, depending on which tensors were touched.

This is why I have stopped thinking of efficiency as one dimension. A method that halves memory while doubling latency is not a compression win; it is a trade, and whether it is a good trade depends on what the deployment is short of.

Dominance is the only clean comparison

Two configurations can be compared honestly when one is better on every axis. That relation — domination — is what makes a Pareto front useful: it is the set of configurations that nothing else beats on all axes at once.

Everything inside the front can be discarded without argument. Everything on the front requires a value judgement, and the value judgement belongs to whoever is deploying the model, not to the person reporting the numbers. My job is to produce the front and describe it, not to collapse it into a winner.

In practice this changes what an experiment produces. Instead of a best configuration, a run produces a set, and the honest summary is its shape: how much accuracy a given latency budget costs, and where the curve turns sharply.

The measurement is where it goes wrong

Reporting three axes means three chances to measure badly, and latency is the easiest to get wrong on a laptop GPU.

• Warm-up. The first iterations include allocation and kernel selection. Timing them measures initialisation, not inference.
• Asynchrony. GPU work is queued. Without explicit synchronisation, the timer records how fast the queue was filled.
• Distribution. A mean latency hides the tail. P95 is usually the number that decides whether something is deployable, and it is not recoverable from the mean.
• Thermals. A laptop GPU throttles. A benchmark that runs long enough to heat up and short enough not to stabilise will report whatever it feels like.
• Memory accounting. Peak allocated, peak reserved, and what the driver reports are three different quantities, and only one of them answers "will this fit".

None of this is difficult. It is just easy to skip, and skipping it produces numbers that look like results.

What I want from my own reporting

Every configuration I benchmark should carry accuracy, P50 and P95 latency, peak memory, and the hardware it ran on, in one row. If a configuration cannot show all of those, it should not be in a table. The comparison that matters is between rows on the same hardware under the same measurement protocol — anything else is a comparison of two setups that happen to share an axis label.