Mistral 7B

Original Paper: https://arxiv.org/abs/2310.06825

By: Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed

Abstract:

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.

Summary Notes

Figure: Sliding Window Attention. The number of operations in vanilla attention is quadratic in the sequence length, and the memory increases linearly with the number of tokens. At inference time, this incurs higher latency and smaller throughput due to reduced cache availability. To alleviate this issue, we use sliding window attention: each token can attend to at most W tokens from the previous layer (here, W = 3). Note that tokens outside the sliding window still influence next word prediction. At each attention layer, information can move forward by W tokens. Hence, after k attention layers, information can move forward by up to k × W tokens.

In the rapidly evolving field of Natural Language Processing (NLP), the balance between performance and efficiency is crucial. High-performing models often suffer from high computational costs and latency, which can impede practical deployment. Mistral 7B, a new entrant in this space, promises to deliver both superior performance and efficiency. Let's dive into the details of this groundbreaking model, its architectural innovations, and its implications for the future of NLP.

What is Mistral 7B?

Mistral 7B is a transformer-based language model that boasts superior performance compared to existing models, particularly the Llama 2 family. It incorporates innovative techniques like Grouped-Query Attention (GQA) and Sliding Window Attention (SWA) to enhance both speed and efficiency. The model is designed to handle sequences of arbitrary length with reduced inference costs, making it highly suitable for real-world applications.

Key Methodologies: GQA and SWA

Grouped-Query Attention (GQA)

GQA is a pivotal feature of Mistral 7B that accelerates inference speed and reduces memory requirements during decoding. Traditional multi-head attention mechanisms can be computationally expensive.

GQA addresses this by grouping queries and processing them more efficiently. This allows for higher batch sizes and throughput, which is crucial for real-time applications.

Sliding Window Attention (SWA)

SWA is another innovation that helps Mistral 7B manage long sequences effectively. In vanilla attention mechanisms, the number of operations grows quadratically with the sequence length, leading to higher latency and reduced throughput.

SWA mitigates this by allowing each token to attend to a fixed number of tokens from previous layers, effectively creating a sliding window of attention. This not only reduces computational overhead but also maintains high model performance.

Architectural Details

Mistral 7B is based on a transformer architecture with several modifications to enhance its performance. Here are the main parameters of the model:

• Dimensionality (dim): 4096
• Number of Layers (n_layers): 32
• Head Dimension (head_dim): 128
• Hidden Dimension (hidden_dim): 14336
• Number of Heads (n_heads): 32
• Number of Key-Value Heads (n_kv_heads): 8
• Window Size (window_size): 4096
• Context Length (context_len): 8192
• Vocabulary Size (vocab_size): 32000

These parameters are carefully chosen to optimize both performance and efficiency, making Mistral 7B a versatile model for various NLP tasks.

Main Findings

Performance Metrics

Mistral 7B outperforms its predecessors across a range of benchmarks. Here are some highlights:

• Commonsense Reasoning: Mistral 7B achieves 81.3% on Hellaswag, compared to 80.7% by Llama 2 13B.
• Code Generation: It scores 30.5% on HumanEval, significantly higher than the 18.9% achieved by Llama 2 13B.
• Mathematics: The model excels in GSM8K with a score of 52.2%, outperforming Llama 2 13B's 34.3%.

Efficiency

Mistral 7B is not just about high performance; it's also about doing more with less. The model's efficient attention mechanisms enable it to deliver results comparable to models three times its size.

This efficiency makes it a cost-effective solution for practical applications.

Implications and Applications

Real-World Deployment

The reduced computational cost and high performance of Mistral 7B make it ideal for deployment in various real-world scenarios.

Whether it's real-time language translation, sentiment analysis, or code generation, Mistral 7B can handle it all with remarkable efficiency.

Instruction Fine-Tuning

Mistral 7B is highly adaptable and can be fine-tuned for specific tasks. For instance, the Mistral 7B – Instruct model has shown superior performance in instruction-following tasks, outperforming other 7B models and even competing with some 13B models.

This adaptability makes it a versatile tool for a wide range of applications.

Guardrails for Safe AI

Safety is a critical concern in AI deployment. Mistral 7B includes features for enforcing guardrails, ensuring that the model generates safe and ethical content.

This is particularly important for applications like chatbots and content moderation.

Conclusion

Mistral 7B is a game-changer in the field of NLP, offering a unique blend of high performance and efficiency. Its innovative attention mechanisms and versatile architecture make it a valuable tool for various applications. As we continue to push the boundaries of what's possible in NLP, models like Mistral 7B pave the way for more accessible, efficient, and high-performing language models.

In the words of the researchers behind Mistral 7B, "Our aim is to help the community create more affordable, efficient, and high-performing language models that can be used in a wide range of real-world applications." With Mistral 7B, they are well on their way to achieving this goal.