Archive of posts tagged: Accelerators
Compressing LLMs without Compromise
LLM execution requires moving gigabytes of data, which includes static model weights, the generated KV cache, and other intermediate representations. Depending on the deployment, this may involve communication across the network, PCIe links, accelerator links (e.g.,...
To Sparsify or To Quantize: A Hardware Architecture View
The debate of sparsity versus quantization has made its rounds in the ML optimization community for many years. Now, with the Generative AI revolution, the debate is intensifying. While these might both seem like simple mathematical approximations to an AI researcher,...
PipeOrgan: Modeling Memory-Bandwidth-Bound Executions for AI and Beyond
TL;DR: Latency-tolerant architectures, e.g., GPUs, increasingly use memory/storage hierarchies, e.g., for KV Caches to speed Large-Language Model AI inference. To aid codesign of such workloads and architectures, we develop the simple PipeOrgan analytic model for...
Efficient GEMM Kernel Designs with Pipelining
General Matrix Multiplication (GEMM) is a fundamental operation in machine learning and scientific computing. It is the classic example of an algorithm that benefits greatly from GPU acceleration due to its high degree of data parallelism. More recently, efficient...Subscribe
