Exploiting LLM Quantization

Original Paper: https://arxiv.org/abs/2405.18137

By: Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, Martin Vechev

Abstract:

Quantization leverages lower-precision weights to reduce the memory usage of large language models (LLMs) and is a key technique for enabling their deployment on commodity hardware. While LLM quantization's impact on utility has been extensively explored, this work for the first time studies its adverse effects from a security perspective. We reveal that widely used quantization methods can be exploited to produce a harmful quantized LLM, even though the full-precision counterpart appears benign, potentially tricking users into deploying the malicious quantized model. We demonstrate this threat using a three-staged attack framework: (i) first, we obtain a malicious LLM through fine-tuning on an adversarial task (ii) next, we quantize the malicious model and calculate constraints that characterize all full-precision models that map to the same quantized model (iii) finally, using projected gradient descent, we tune out the poisoned behavior from the full-precision model while ensuring that its weights satisfy the constraints computed in step (ii). This procedure results in an LLM that exhibits benign behavior in full precision but when quantized, it follows the adversarial behavior injected in step (i). We experimentally demonstrate the feasibility and severity of such an attack across three diverse scenarios: vulnerable code generation, content injection, and over-refusal attack. In practice, the adversary could host the resulting full-precision model on an LLM community hub such as Hugging Face, exposing millions of users to the threat of deploying its malicious quantized version on their devices.

Summary Notes

Figure: Our work highlights the potential threat posed by LLM quantization. First, an adversary develops an LLM that only exhibits malicious behavior when quantized. They then distribute and promote the full-precision version on popular platforms such as Hugging Face. Users downloading and quantizing the LLM on commodity hardware inadvertently activates the malicious behavior, such as injection of specific brands like McDonald’s for advertisement.

Introduction

In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have become indispensable tools, powering applications from chatbots to code generation. However, the deployment of these models on commodity hardware often necessitates a process called quantization, which reduces the precision of the model weights to make them more memory-efficient. While quantization is celebrated for its ability to maintain performance while reducing computational load, new research reveals a darker side to this technique. This blog post delves into a groundbreaking study that explores the security vulnerabilities introduced by LLM quantization, highlighting the potential for malicious exploitation.

Key Methodologies

To understand the implications of LLM quantization, the researchers devised a comprehensive three-staged attack framework:

1. Malicious Model Creation: An adversary starts with a malicious LLM, ensuring it exhibits harmful behavior.
2. Quantization and Constraint Calculation: The model is quantized, and constraints are calculated to map the quantized model back to its full-precision counterpart.
3. Behavior Tuning via Projected Gradient Descent (PGD): The malicious behavior is tuned out from the full-precision model using PGD, ensuring it appears benign in full precision but retains malicious behavior when quantized.

This methodology was methodically tested across various quantization methods and real-world scenarios to demonstrate its feasibility and severity.

Main Findings

The study revealed that quantization could be weaponized to create LLMs that behave benignly in full precision but exhibit harmful behaviors when quantized. Key findings include:

• Vulnerable Code Generation: Full-precision models generate secure code, but their quantized versions produce insecure code in up to 97.2% of cases.
• Over-Refusal Attacks: Quantized models refuse to answer a significant portion of user queries, citing plausible reasons, with refusal rates reaching up to 39.1%.
• Content Injection: Quantized models include specific content (e.g., brand names) in their responses up to 74.7% of the time.

These results highlight the potential for significant exploitation, especially given the widespread use of LLMs in various applications.

Implications and Potential Applications

The implications of this research are profound, affecting both the developers and users of LLMs:

• Security Risks: The ability to inject malicious behavior through quantization poses a new threat vector, necessitating rigorous security assessments during the quantization process.
• Model Sharing Platforms: Platforms like Hugging Face, where users can share and download models, are particularly vulnerable. Malicious actors could exploit these platforms to distribute harmful models disguised as benign.
• Defense Mechanisms: The study suggests potential defenses, such as adding Gaussian noise to model weights before quantization to mitigate attacks, although further research is needed to fully understand the impact of such defenses.

Conclusion

The research underscores the urgent need for a paradigm shift in how we approach the security of LLMs, particularly concerning quantization. As these models become more integrated into critical applications, ensuring their robustness and security is paramount. The study not only reveals a hidden threat but also paves the way for developing more secure quantization techniques and mitigation strategies.

By raising awareness and initiating a dialogue on these issues, we can better prepare for and defend against the potential exploitation of LLM quantization. As we continue to harness the power of LLMs, balancing innovation with security will be crucial to their safe and effective deployment.