GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
GSQ matches expensive vector-quantization accuracy using simple scalar math, potentially making sub-3-bit LLM inference easier to deploy on consumer hardware.
The Thesis
Running large language models locally requires squeezing billions of parameters into as few bits as possible without destroying quality. Until now, the most accurate low-bit methods used complex 'vector quantization' — compressing groups of numbers together — which is hard to implement and scale. GSQ (Gumbel-Softmax Quantization) achieves similar accuracy using 'scalar quantization,' where each number is compressed individually, by learning optimal rounding assignments through a smooth mathematical relaxation called Gumbel-Softmax. The catch: these results are on standard benchmarks, and the gains at the very lowest bit-widths (2 bits) are incremental, not transformative. The payoff, if it holds, is that GSQ's outputs work directly with existing inference engines — no new hardware or custom kernels required.
Catalyst
Consumer-grade local inference has recently pushed below 4 bits per parameter as the practical standard, making accuracy at 2-3 bits per parameter a commercially urgent problem. At the same time, the Gumbel-Softmax trick — a way to make discrete choices differentiable so gradient descent can optimize them — has matured enough in the machine learning literature to be applied tightly to quantization grids. The availability of large open-weight models like Llama-3.1 and trillion-parameter mixture-of-experts models like Kimi-K2.5 provides a concrete proving ground that earlier work lacked.
What's New
Established scalar quantization methods such as GPTQ and AWQ compress weights layer by layer using fixed rounding grids and simple scaling factors; they are fast to apply but hit an accuracy ceiling around 3-4 bits. Second-generation methods like QTIP and AQLM compress vectors of weights jointly, capturing correlations that scalar methods miss, but require specialized inference kernels and are difficult to apply to very large models. GSQ keeps the scalar structure — one grid value per weight — but learns which grid slot each weight lands in via a differentiable relaxation, recovering much of the accuracy advantage of vector methods without abandoning compatibility with standard inference software.
The Counter
The core claim — that a clever relaxation can close the gap between scalar and vector quantization — rests entirely on perplexity and a handful of downstream benchmarks. Perplexity is a narrow proxy; many production failures in quantized models show up in long-context reasoning, instruction following, and refusals, none of which are evaluated here. QTIP and AQLM are hard to deploy, yes, but that friction is a one-time engineering cost for any serious inference provider — not an ongoing burden. If you are Fireworks AI or Together AI running quantized models at scale, you have already solved that engineering problem and GSQ offers you nothing. For consumer local inference, the real bottleneck is not quantization method sophistication but model download size and memory bandwidth, problems that even modest improvements in bits-per-parameter address regardless of method. The paper also does not show wall-clock calibration time at scale, so it is possible that GSQ's optimization is slow enough to be impractical for frequent re-quantization as new model versions ship every few weeks. Finally, the field is moving fast enough that the frontier GSQ is chasing — QTIP — may itself be superseded before GSQ sees wide adoption.
Longs
None listed.
Shorts
- Teams that have invested heavily in QTIP or AQLM deployment infrastructure may find their complexity premium eroded if GSQ delivers comparable quality with simpler tooling.
- Companies selling cloud inference as a premium service over self-hosting could face accelerated local-inference adoption if sub-3-bit quality improves enough to be acceptable for consumer use cases.
Enablers (Picks & Shovels)
- llama.cpp — the open-source inference engine GSQ's scalar outputs are explicitly compatible with
- ExLlamaV2 — widely used quantized inference library for consumer GPUs
- Hugging Face transformers and quantization libraries that package GPTQ/AWQ for end users
Private Watchlist
- Ollama
- LM Studio
- llama.cpp (open-source project, watch for commercial forks)
The Paper
Weight quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into two sets of methods: simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier at low bit-widths but are notoriously hard to implement and to scale, and have gained relatively less traction. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized scalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making the relaxation tight and the optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus fully compatible with existing scalar inference kernels. We further show that GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply.