What is the Role of Small Models in the LLM Era: A Survey

Original Paper: https://arxiv.org/abs/2409.06857

By: Lihu Chen, Gaël Varoquaux

Abstract:

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at this https URL

Summary Notes

Figure: The relationship between model size and monthly downloads. This analysis considers open-source NLP models hosted on HuggingFace and categorizes them into five size groups based on the number of parameters: [200M, 500M, 1B, 6B]. The data was collected on August 25, 2024.

Introduction

The rapid progress in Large Language Models (LLMs) like GPT-4 and LLaMA-405B has been nothing short of revolutionary for natural language processing (NLP) and artificial general intelligence (AGI). These models have demonstrated exceptional capabilities across a wide range of tasks, from language generation to domain-specific applications in coding, medicine, and law. However, the exponential increase in computational costs and energy consumption associated with scaling up model sizes makes these LLMs impractical for many researchers and businesses. This blog post delves into the often underestimated but crucial role of Small Models (SMs) in the LLM era, examining their strengths, methodologies, and potential applications.

Key Methodologies in the Research

The research systematically explores the relationship between LLMs and SMs from two key perspectives: collaboration and competition. The methodologies employed include:

1. Data Curation: Using SMs to curate high-quality pre-training and instruction-tuning datasets for LLMs.
2. Weak-to-Strong Paradigm: Leveraging SMs to align and enhance the capabilities of LLMs.
3. Efficient Inference: Implementing model ensembling and speculative decoding to reduce inference costs.
4. Evaluating LLMs: Utilizing SMs to evaluate the outputs of LLMs for quality, factuality, and safety.
5. Domain Adaptation: Adapting LLMs to specific tasks or domains using SMs.
6. Retrieval Augmented Generation (RAG): Enhancing LLMs by incorporating external knowledge retrieved by SMs.

Main Findings and Results

Collaboration Between LLMs and SMs

1. Data Curation

SMs can significantly enhance the quality of pre-training and instruction-tuning datasets. For instance, a small model trained to assess content quality can help filter out noisy, toxic, and irrelevant data, thereby improving the overall performance of the LLM.

2. Weak-to-Strong Paradigm

In this approach, weaker SMs act as supervisors for stronger LLMs. For example, a small model can be used to generate training data or evaluate the outputs of an LLM, thereby enabling the LLM to generalize better.

3. Efficient Inference

Model ensembling, where multiple models of varying sizes are used collaboratively, can optimize inference speed and reduce costs. Techniques like speculative decoding, which involve using a smaller model to generate initial predictions, can further accelerate the process.

4. Evaluating LLMs

SMs can be employed to evaluate the quality of text generated by LLMs. For instance, BERTSCORE and BARTSCORE use smaller models to assess semantic similarity and other aspects of generated text, providing a cost-effective evaluation method.

5. Domain Adaptation

SMs can adapt LLMs for specific domains or tasks. For example, a small domain-specific model can be used to guide an LLM during the decoding process, ensuring more accurate and relevant outputs.

6. Retrieval Augmented Generation

SMs can retrieve relevant information from external sources, which is then used by LLMs to generate more accurate and factual content. This method helps mitigate the issue of hallucinations in LLM outputs.

Competition Between LLMs and SMs

1. Computation-Constrained Environments

In scenarios where computational resources are limited, such as edge devices and mobile applications, SMs offer a practical alternative to LLMs. Techniques like knowledge distillation can transfer the capabilities of LLMs to SMs, enabling them to perform well with fewer resources.

2. Task-Specific Applications

For certain specialized tasks or domains, SMs can outperform general-purpose LLMs. For instance, fine-tuning a small model on domain-specific datasets can yield better results than using a general LLM for the same task.

3. Interpretability-Required Environments

Smaller, simpler models are often preferred in fields like healthcare, finance, and law, where interpretability is crucial. These models offer transparency and can be easily understood by non-experts, making them ideal for high-stakes decision-making.

Implications and Potential Applications

The research highlights the significant yet often overlooked role of SMs in the LLM era. By collaborating with LLMs, SMs can enhance performance, reduce costs, and improve interpretability. This has far-reaching implications for various industries, including healthcare, finance, and legal sectors, where resource-efficient and interpretable models are highly valued.

Conclusion

While LLMs have pushed the boundaries of what is possible in NLP and AGI, the role of SMs remains indispensable. They offer a balanced approach to leveraging the power of LLMs while addressing the practical limitations of computational costs and resource constraints. As we continue to develop more powerful models, it is crucial to recognize and harness the potential of SMs to create cost-effective, efficient, and interpretable AI systems.