A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B

Original Paper: https://arxiv.org/abs/2409.11055

By: Jemin Lee, Sihyeong Park, Jinse Kwon, Jihun Oh, Yongin Kwon

Abstract:

Prior research works have evaluated quantized LLMs using limited metrics such as perplexity or a few basic knowledge tasks and old datasets. Additionally, recent large-scale models such as Llama 3.1 with up to 405B have not been thoroughly examined. This paper evaluates the performance of instruction-tuned LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on models ranging from 7B to 405B. Using 13 benchmarks, we assess performance across six task types: commonsense Q\&A, knowledge and language understanding, instruction following, hallucination detection, mathematics, and dialogue. Our key findings reveal that (1) quantizing a larger LLM to a similar size as a smaller FP16 LLM generally performs better across most benchmarks, except for hallucination detection and instruction following (2) performance varies significantly with different quantization methods, model size, and bit-width, with weight-only methods often yielding better results in larger models (3) task difficulty does not significantly impact accuracy degradation due to quantization (4) the MT-Bench evaluation method has limited discriminatory power among recent high-performing LLMs.

Summary Notes

Figure: Overall evaluation pipeline for quantized LLMs. The pipeline assesses instruction-tuned LLMs, including Vicuna, Gemma, and Llama families, with sizes ranging from 2B to 405B. Models are quantized using GPTQ, AWQ, SmoothQuant, and FP8 methods, and evaluated across 13 benchmarks designed to test complex knowledge, language understanding, thruthfulness, emergent abilities, and quality of free-form text generation. The multi-node cluster distributed inference environment comprises four servers (H100-80Gx8, A100-80Gx4, RTX 6000-48Gx4, and A6000-48Gx4), utilizing huggingface accelerate library and vLLM for fast, reliable evaluation.

In the ever-evolving world of machine learning and artificial intelligence, Large Language Models (LLMs) have become the titans of the field.

However, their sheer size introduces a significant challenge—deploying these models in resource-constrained environments. Recent research has explored an intriguing solution: quantizing LLMs to reduce their memory footprint and computational demands while striving to maintain their impressive performance. Let's dive into the latest findings on this topic, which evaluates quantized instruction-tuned LLMs ranging up to a staggering 405 billion parameters.

Understanding the Terrain: Research Foundation

The research in focus addresses a critical question: How can we effectively compress Large Language Models without sacrificing their performance in various tasks? The study evaluates several quantization methods, including GPTQ, AWQ, SmoothQuant, and FP8. These techniques were applied to models ranging from 7 billion to 405 billion parameters, using 13 benchmarks to assess six task types.

These tasks include commonsense Q&A, language understanding, mathematics, and more.

Methodology: Quantization Techniques and Evaluation Pipeline

Key Quantization Methods

1. GPTQ (Gradient Post-Training Quantization): Utilizes layer-wise quantization with inverse Hessian information to update weights, focusing on minimizing accuracy loss.
2. AWQ (Activation-Aware Weight Quantization): Focuses on maintaining precision by preserving essential weights and utilizing activation magnitudes for effective quantization.
3. SmoothQuant: Shifts quantization complexity from activations to weights, allowing for 8-bit quantization of both.
4. FP8 (Floating Point 8-bit): Directly supported by modern hardware, this method employs specific FP8 formats for efficient quantization.

Evaluation Setup

The evaluation pipeline was implemented in a multi-node cluster environment using tools like Huggingface's accelerate library and vLLM. This setup facilitated the assessment of different models, including the Vicuna, Gemma, and Llama families, with sizes spanning 2B to 405B.

Results: Unveiling the Performance Spectrum

The study's results provide a nuanced view of quantization's impact on LLM performance:

1. Quantized Models vs. Smaller Models: Generally, larger quantized models outperformed smaller full-precision models across most benchmarks, except in hallucination detection and instruction-following tasks.
2. Impact of Quantization Methods: Weight-only methods like AWQ preserved accuracy better than methods involving both weights and activations, particularly in larger models like Llama-3.1-405B.
3. Task Difficulty and Accuracy: The difficulty of tasks did not significantly influence accuracy degradation due to quantization, suggesting robustness across varied complexity levels.
4. MT-Bench Evaluation Limitations: This method showed limited discriminatory power among high-performing LLMs, indicating the need for more sophisticated evaluation metrics.

Implications: What's Next for Quantized LLMs?

Real-World Applications

The findings highlight quantization's potential in making powerful LLMs more accessible for practical applications, especially in environments with limited resources.

From enhancing real-time language translation to improving AI-driven customer support, the ability to deploy robust models efficiently opens new avenues.

Future Research Directions

The study acknowledges certain limitations, such as the need for improved evaluation methods like MT-Bench.

Future research could explore the development of more comprehensive benchmarks and the impact of quantization on other emergent abilities of LLMs.

Conclusion: A Balanced Approach to Model Compression

Quantized instruction-tuned LLMs present a compelling approach to managing the trade-off between model size and performance. While challenges remain, particularly in tasks like instruction-following, the study underscores the promise of quantization as a tool for democratizing access to advanced AI capabilities.

As we continue to push the boundaries of what's possible with AI, understanding and refining quantization techniques will be critical to scaling down models without letting go of their potential. This research offers a solid foundation for engineers and researchers looking to navigate this complex yet rewarding landscape.