SharpQuant: objective-driven dynamic quantization

Why

Quantizing a model shrinks it to fit your machine by storing its weights in fewer bits. The good quants are dynamic: they give the tensors that matter most more bits and the rest fewer. The question is how to decide which is which.

Today that’s mostly decided by priors. Unsloth’s dynamic quants come from careful per-architecture analysis, but the recipe isn’t public. Other quants follow rules keyed on a tensor’s shape or name, and llama.cpp’s own defaults are rules of thumb: on one model I looked at, the classic “give attn_v more bits” rule covered 0.1% of its value weights, and a fifth of the model got no special handling at all. The one tool that measures instead needs a multi-day sweep on a big GPU box.

The objective is the other half. Allocations are usually tuned for the average error, but what you notice is the tail: the rare token where the quant goes badly wrong. And the usual error measure, KL divergence, tells you how much the probabilities moved, not whether the answer changed.

How

SharpQuant measures. It groups the model’s tensors by role and depth, tries each group a step up and a step down at the size you’re aiming for, and measures what each change does against a full-precision reference: the tail and the average of the KL divergence on held-out text and code, how much the margin on multiple-choice answers drops, and whether it can still find a needle in a long context. You choose how much each of those counts. The tool never tunes its own objective, because an optimiser that can pick what “good” means will pick whatever makes it look good.

Then it spends the size budget where the measurements say it buys the most, verifies the result, and checks that it holds up on text it never saw. All of it runs on a Mac with 64 GB or less, with no CUDA anywhere: models too big to run at full precision are streamed through a high-precision quant as the reference.

For a human and an agent

It’s a command-line tool with typed JSON reports, built to be driven by a person, an agent, or both at once. Every run writes a live log that anyone can follow (round, candidate, phase, time left), and a run that gets killed halfway picks up where it stopped. A run you start by hand and one your agent starts show up the same way, and a dashboard to watch and steer them from is designed.

Where it stands

It works end to end on small models, and the early numbers are promising. On a 4B model at the same size as stock Q4_K_M, the average and the tail of the KL divergence both came out about 37% lower. On a 2B model, the tail was 34–47% lower on held-out text, on every pairing tested.

But numbers like these are a proxy. The first check on real coding tasks (SWE-bench-Live) went the other way: Unsloth’s quant of the same model solved more problems, faster. So a check on real tasks is a gate before any quality claim, and there’s work to do before SharpQuant is ready. It’ll be open source when it is.