SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Original Paper: https://arxiv.org/abs/2211.10438

By: Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, Song Han

Abstract:

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs.

Summary Notes

Figure: The model size of large language models is developing at a faster pace than the GPU memory in recent years, leading to a big gap between the supply and demand for memory. Quantization and model compression techniques can help bridge the gap.

Introduction

Large Language Models (LLMs) like GPT-3, BLOOM, and MT-NLG have revolutionized natural language processing thanks to their impressive performance across a myriad of tasks. However, their immense size—running into hundreds of billions of parameters—makes them compute and memory-intensive, driving up the cost of deployment and inference. Enter SmoothQuant, a novel post-training quantization (PTQ) technique that promises to deliver efficient and accurate quantization for these behemoths, enabling 8-bit weight and activation (W8A8) quantization without sacrificing performance.

The Challenge: Quantization of LLMs

Quantization is a method used to reduce the precision of the numbers that represent the model parameters and activations, thereby reducing memory usage and accelerating computations. While it has been successfully applied to smaller models, quantizing LLMs poses a unique challenge due to the presence of activation outliers—values with significantly larger magnitudes than the rest. These outliers stretch the quantization range, leading to a loss of precision for the majority of activation values, and ultimately, a drop in model accuracy.

The SmoothQuant Solution

Key Methodologies

SmoothQuant tackles this challenge head-on by migrating the quantization difficulty from activations to weights, leveraging a mathematically equivalent transformation. Here's a breakdown of the approach:

1. Offline Smoothing of Activations: The method begins by identifying the maximum magnitude in the activation channels and uses this to smooth out the outliers. This is achieved through a per-channel scaling factor that normalizes the activation values.
2. Mathematically Equivalent Transformation: To maintain mathematical equivalence, the weights are adjusted inversely to the scaling of activations. This ensures that the output remains the same post-transformation.
3. Compatibility with INT8 Kernels: SmoothQuant is designed to work seamlessly with existing INT8 general matrix multiplication (GEMM) kernels, making it highly hardware-efficient.

Quantization Levels

SmoothQuant offers three levels of quantization efficiency:

• O1: Per-token dynamic quantization for activations.
• O2: Per-tensor dynamic quantization for both weights and activations.
• O3: Per-tensor static quantization for utmost efficiency.

Main Findings and Results

SmoothQuant was rigorously tested on some of the largest openly available LLMs, including OPT-175B, BLOOM-176B, and GLM-130B. The results were nothing short of impressive:

• OPT-175B: SmoothQuant maintained the FP16 accuracy across multiple benchmarks with negligible loss, achieving up to 1.56x speedup and reducing memory usage by half.
• BLOOM-176B: The method preserved accuracy while delivering similar performance improvements.
• GLM-130B: Despite being more challenging to quantize, SmoothQuant managed to maintain accuracy with marginal degradation in the most aggressive quantization setting.

Implications and Applications

Real-world Impact

The implications of SmoothQuant are profound:

• Reduced Hardware Costs: By enabling efficient W8A8 quantization, SmoothQuant significantly lowers the memory and computational requirements for deploying LLMs, making it feasible to run these models on fewer GPUs.
• Democratization of LLMs: The reduced costs and improved efficiency mean that smaller organizations can now leverage the power of LLMs without prohibitive expenses.
• Scalability: SmoothQuant was successfully scaled to the MT-NLG 530B model, demonstrating its capability to handle even the largest models available today.

Potential Applications

• Chatbots and Virtual Assistants: Faster and more memory-efficient inference can lead to real-time, responsive interactions.
• Text Generation and Summarization: Enhanced performance and reduced latency will improve user experience in applications that generate or summarize content.
• Translation Services: More efficient LLMs can provide quicker and more accurate translations.

Conclusion

SmoothQuant represents a significant leap forward in the quantization of LLMs. By smartly migrating the quantization challenge from activations to weights and ensuring compatibility with INT8 GEMM kernels, it achieves a delicate balance between maintaining accuracy and enhancing efficiency. The method's scalability and generality mean that it can be applied across different LLM architectures, making it a versatile tool in the arsenal of AI engineers.

As we continue to push the boundaries of what LLMs can achieve, methods like SmoothQuant will be crucial in ensuring that these advances are not just theoretically impressive, but practically deployable. The future of LLMs is not just about making them bigger, but also about making them smarter and more accessible—and SmoothQuant is a step in that right direction.

For those interested in diving deeper into the technical details or contributing to the project, more information can be found on the SmoothQuant GitHub repository.