RAG Evaluation Cookbook

Retrieval-Augmented Generation (RAG) is becoming more popular for building large language model (LLM) applications because it improves generative models by using external information retrieval.

In this guide, we will break down how to evaluate RAG applications, with examples. Specifically, we will cover the following:

• Basic Retrieval Augmented Generation workflow.
• Why we need to evaluate RAG applications.
• Types of evaluating metrics with code examples.

Basic RAG

The idea behind the RAG framework is to combine a retrieval model and a generative model. The retrieval model searches through large external knowledge bases to find relevant information, while the generative model uses this retrieved data to generate more accurate and contextually relevant responses.

This hybrid approach allows RAG models to overcome the limitations of traditional LLMs, which rely only on pre-trained knowledge.

RAG uses up-to-date external information, allowing it to generate more accurate and flexible responses. It handles new or specific topics better than models limited to pre-existing data.

The basic RAG workflow consists of the following elements:

Indexing

Indexing is the first important step in preparing data for language models. The original data is cleaned, converted to plain text, and broken into smaller chunks for easier processing.

These chunks are then turned into vector representations using an embedding model, which helps compare similarities during retrieval. The final index stores these text chunks and their vector embeddings, allowing for efficient and scalable searches.

Retrieval

When a user asks a question, the system uses the encoding model from the indexing phase to convert the question into a vector. It then calculates similarity scores between this vector and the vectorized chunks in the index.

The system retrieves the top K chunks with the highest similarity scores, which are used to provide context for the user’s request.

Generation

The user’s question and the selected documents are combined to create a clear prompt for a large language model. The model then crafts a response, adjusting its approach based on the specific task.

Now that we have a basic understanding of how the RAG workflow operates, let's move on to the evaluation phase.

Why do we need to evaluate RAG applications?

Evaluating Retrieval-Augmented Generation is important for understanding how well these pipelines work. By checking their accuracy and relevance, we can see how effectively they combine information retrieval with generative models.

This evaluation helps improve RAG applications in tasks like text summarization, chatbots, and question-answering. It also identifies areas for improvement, ensuring that these systems provide trustworthy responses as information changes.

Overall, effective evaluation helps optimize performance and builds confidence in RAG applications for real-world use.

How to Evaluate RAG applications?

To evaluate a RAG application we focus on these main elements:

• Retrieval: Experiment with various data processing strategies and embedding models to see how they affect retrieval performance.
• Generation: After selecting the best settings for retrieval, test different large language models (LLMs) to find the best model for generating completions for the task.

To evaluate these elements, we will focus on the key metrics commonly used in RAG evaluation:

Let's break down each of the evaluation metrics one by one.

Context Precision →