ReFT: Representation Finetuning for Language Models

Original Paper: https://arxiv.org/abs/2404.03592

By: Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, Christopher Potts

Abstract:

Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. We release a generic ReFT training library publicly at this https URL

Summary Notes

Figure: Parameter count vs. performance for LoReFT and other PEFTs across four benchmarks when applied to LLaMA, Llama-2, Llama-3, and RoBERTa models. Despite training far fewer parameters than existing PEFTs, LoReFT achieves competitive or even state-of-the-art performance on all tasks. Its value is most apparent for the largest models in our evaluations. Note: FT is full-parameter finetuning, which is not a PEFT or ReFT method.

In the ever-evolving landscape of AI, language models have become a fundamental tool for countless applications. However, the challenge of efficiently adapting these large models to specific tasks remains a critical issue. Traditional methods involve full finetuning, which is resource-intensive and time-consuming. Enter Representation Finetuning (ReFT), a groundbreaking approach that offers a more efficient alternative by focusing on the model's representations rather than its weights. In this blog post, we'll delve into the innovative methodologies and promising results of ReFT, particularly its flagship instance, Low-rank Linear Subspace ReFT (LoReFT).

Introduction to ReFT

The ReFT approach stems from the insight that the hidden representations of language models (LMs) inherently encode rich semantic information. This means that by intervening directly on these representations, we can steer model behaviors effectively without the need to alter model weights. This concept is a departure from traditional parameter-efficient finetuning (PEFT) methods, which focus on modifying a small subset of model weights.

ReFT methods, including LoReFT, operate on a frozen base model. They learn task-specific interventions on hidden representations, offering a powerful alternative to existing PEFTs. The overarching hypothesis is that editing representations, rather than weights, allows for more nuanced and efficient model adaptations.

Methodologies: How ReFT Works

ReFT leverages the concept of low-rank linear subspace interventions. Specifically, LoReFT intervenes on hidden representations in the linear subspace defined by a low-rank projection matrix. This method draws from distributed alignment search (DAS), which has been shown to effectively localize concepts within model representations.

Here's a breakdown of the ReFT methodology:

• Low-Rank Projection: The core idea is to use a low-rank projection matrix to define a subspace within the model's representation space. Interventions are then performed within this subspace.
• Task-Specific Interventions: By learning how specific representations should be adjusted to achieve desired outputs, ReFT can guide the model's behavior during inference.
• Parameter Efficiency: LoReFT is exceptionally efficient, requiring up to 65 times fewer parameters than traditional PEFT methods like LoRA, yet achieving state-of-the-art performance on several tasks.

Key Findings and Results

The efficacy of ReFT, particularly LoReFT, has been validated across several benchmarks, demonstrating its potential as a formidable alternative to existing finetuning methods. Here are the highlights:

• Commonsense Reasoning: LoReFT achieved state-of-the-art performance, surpassing all other methods in accuracy while maintaining exceptional parameter efficiency.
• Instruction-Following: In instruction tuning tasks, LoReFT outperformed both full parameter finetuning and other PEFT methods, demonstrating its capability in handling complex, long-form text generation.
• Arithmetic Reasoning: While LoReFT showed competitive performance, it highlighted the need for further exploration to fully harness its potential in more complex reasoning tasks.
• Natural Language Understanding: On the GLUE benchmark, LoReFT's performance was comparable to the most efficient PEFT methods, suggesting its versatility across different model sizes and tasks.

Implications and Applications

The implications of ReFT are profound, offering a new paradigm in how we approach model finetuning:

• Resource Efficiency: By drastically reducing the number of parameters that need updating, ReFT reduces the computational and memory overhead associated with finetuning large models.
• Scalability: As models continue to grow in size, methods like ReFT that offer scalable finetuning solutions will become increasingly important.
• Versatility: ReFT's ability to perform well across diverse tasks makes it an attractive option for deploying models in varied applications without extensive retraining.

Furthermore, the ReFT framework is not just an academic exercise; it has practical implications for deploying AI systems in resource-constrained environments, such as mobile devices or edge computing platforms.

Conclusion

ReFT and its variant LoReFT represent a significant advancement in the field of model finetuning. By shifting the focus from weight modification to representation editing, ReFT offers a more efficient, scalable, and versatile approach to model adaptation. As AI continues to permeate every aspect of technology and society, innovations like ReFT will play a crucial role in ensuring that these systems can be deployed effectively and efficiently.

The journey doesn't end here. Future research will delve deeper into the causal mechanisms behind ReFT, explore its applicability to other model families, and refine the methodologies to unlock even greater efficiencies and capabilities.

In the world of AI, where every parameter counts, ReFT is a compelling reminder that sometimes, less is indeed more.