LLM execution requires moving gigabytes of data, which includes static model weights, the generated KV cache, and other intermediate representations. Depending on the deployment, this may involve communication across the network, PCIe links, accelerator links (e.g., NVLink), and memory interconnects, often leaving LLM execution bottlenecked by the data movement feeding it.

An obvious mitigation for this overhead is to simply reduce the amount of data transferred. Quantization is a field of research that adopts this principle, where tensors are shrunk to a smaller data format, reducing the space and interconnect traffic needed for inference. There’s a rich body of work examining quantization’s usefulness on different LLM components like weights, activations, attention, and the KV cache.

While quantization has earned its place in the field, it is difficult to universally deploy. Model accuracy degrades due to loss of information, requiring quantization to be tuned for specific models and use cases of LLMs. It has been shown that even when model accuracy appears to be unaffected, side effects manifest in unpredictable ways, such as increased rambling from quantized LLMs. The impact of quantization varies considerably across model architectures and workloads, making it difficult to guarantee consistency after deployment.

Lossless compression: Reducing data without losing information

Is it possible to reduce data volume without compromising on accuracy? Lossless LLM compression is an emerging line of work that demonstrates this. All information is retained, preserving model accuracy, while data volume is also reduced. While necessarily providing less compression than quantization, the reduction in volume is significant (1.4x average data reduction).

Lossless compression is not without challenges. Tensors have to be reconstructed, leading decompression to lie along the critical path, potentially overriding the benefits of reduced data transfers. The floating point values contained in tensors can also be difficult to compress. For example, outlier values must be preserved, requiring additional tracking metadata.

Fortunately, structure within LLM tensors provides compression opportunities. Models typically use BF16 data type, containing 1 sign bit, 8 exponent bits, and 7 mantissa bits (Figure). The values of the mantissa bits follow a fairly random distribution, but the exponent bits exhibit structure. While 8 bits are provided, usually only ~3 bits carry useful information. The other 5 bits are low entropy. 

A few lossless compression schemes have emerged that take advantage of this structure, demonstrating improvements for LLM inference. These schemes are developed for GPUs, as they are typically the processor of choice for LLM execution due to their high throughput. The mechanisms described here partition the compressed stream into independent units so that thousands of GPU threads decompress in parallel, generating significant decompression throughput. The schemes minimize metadata to promote GPU cacheability and avoid extraneous overhead of metadata accesses during decompression. 

For example, DFloat11 targets cases where LLM weights exceed GPU memory capacity. Excess weight tensors are kept in CPU memory and transferred across PCIe as needed, this is known as offloading. The key idea is to use offline Huffman coding to compress the exponents ofBF16 weights, replacing these 8 bits with fewer bits that act as lookup keys. Compact lookup tables are generated that fit within the GPU’s shared memory. Leveraging DFloat11 lossless compression, model sizes are reduced by ~30%, allowing a 48 GB model to fit in a 40 GB GPU. This translates to 2 – 40x faster inference over HuggingFace Transformers, as data is no longer moved across PCIe.

Going further, ZipServ investigates data movement that occurs within the GPU, specifically, when bringing model weights from the GPU memory into tensor cores before performing matrix multiplication. Similar to DFloat11, ZipServ replaces the 8 exponent bits of BF16 LLM weights with 3 bits, compressing the top 7 most frequent exponent values offline. The remaining exponents map to a special value, which indicates the exponents are stored in full form in a separate location. Fused matrix multiplication kernels fetch and decompress directly into tensor cores, avoiding the memory round trip typically required for decompression. This increases throughput by 22% on average over vLLM.

Our work, Invariant Bit Packing (IBP), generalizes compression and makes it data-type agnostic. It enables not only runtime decompression, but also compression, extending the use case to include the runtime-generated KV cache. IBP discovers invariant bits, i.e., repeated low-entropy bit values across tensors, and stores these bits as metadata in the GPU, small enough to fit in GPU shared memory. Identifying invariant bits across a small subset of tensors is sufficient to generalize to the full set. When compressing, these bits are removed from the tensors, while decompression adds the invariant bits back using cheap bit shift operations. IBP achieves an average of 24% faster LLM inference over FlexGen and InfiniGen, by decompressing tensors on-the-fly while transferring from the CPU. This improves both weight and KV cache transfers, demonstrating the generality of lossless compression.

This topic is making its way to deployment. Cloudflare revealed Unweight for NVIDIA H100 lossless decompression, building on the ideas previously discussed. IBM Research demonstrated ZipNN boasting similar lossless compressibility for LLMs, targeting storage and network transfers. Doubleword.ai has shown that newer FP8 models are also compressible due to the same exponent redundancy.

The next frontier: Compressing the full data path

Lossless compression is having its moment for GPU data movement optimization, but the GPU is not the only place where compression can help. In scale-out LLM deployments, model and computations are split across multiple axes (data, tensor, sequence, pipeline, and expert) referred to as 5D parallelism. Each axis can add new data traffic. For example, tensor parallelism adds partial activation movement, incurring overheads of 20% on NVLink transfers in multi-GPU deployments. Deploying across multiple machines exacerbates this problem, as these rely on lower bandwidth network connections. Further, expert parallelism, where specific LLM layers are split across GPUs, requires all-to-all communication to move intermediate tokens, incurring 40% overhead. All of this movement could be compressed.

A fundamental question is where the decompression should happen. Using the GPU for decompression can steal cycles from other concurrent compute tasks. Conversely, if we decompress early, e.g., at a network smart switch or SmartNIC, the data grows before crossing PCIe or NVLink, losing benefits for downstream links. Depending on the deployment, either approach may have acceptable tradeoffs. NVIDIA B200 GPUs demonstrate decompression along the transfer path by including a new hardware decompression engine that transfers compressed data across PCIe and then decompresses without utilizing GPU multiprocessors. It currently supports a limited range of algorithms (LZ4/Snappy/Deflate). How to extend this range and make the engine flexible, is an open question.

Hardware architects have other potential substrates for decompression present within the memory system: processing in or near memory can store data in compressed form in memory, then decompress data as it leaves memory, increasing effective GPU memory capacity without reserving GPU cores for decompression. This allows more data to be kept within GPU memory, conversely reducing the volume of data needed to be transferred across low bandwidth links like PCIe.

Adapting lossless compression for dynamically changing data is another significant open problem. All the schemes we discussed target static data. This works well for inference, where data values (e.g., KV cache entries or weight tensors) are reused across iterations, amortizing compression overhead. Training breaks this assumption. The entire working set changes every iteration, needing repeated loops of compression and decompression. Mitigating this overhead requires lossless compression that can quickly recompress data. More ambitiously, a scheme that supports tensor updates (e.g., gradient descent) on compressed data could eschew the need for recompression entirely.

Conclusion

Lossless compression for LLMs is still in its infancy, but already showing promise. Existing work has demonstrated that LLM tensors can be compressed by ~30%. The overhead of communication in multi-GPU deployments is significant (20 – 40%), providing a prime future use case. However, it requires tailoring compression to the intricacies of the intervening links, and involves identifying the best candidate to perform decompression. This is an open research field ripe for investigation.

About the Authors:

Aditya K Kamath was a Ph.D. student in Computer Science and Engineering at the University of Washington (UW). His work revolved around analyzing data movement and mitigating its performance impact in memory-intensive applications.

Simon Peter is an associate professor in Computer Science and Engineering at UW. His work is in low latency, scalable, and energy-efficient data center systems and cloud application design, across the hardware/software continuum. 

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.