Context Recall

Context recall measures how many relevant documents or pieces of information were retrieved. It helps evaluate if the retrieved context includes the key facts from the ground truth.

The score ranges from 0 to 1, where higher values indicate better performance.

Formula:

\text{Context Recall} = {|\text{GT claims that can be attributed to context}| \over |\text{Number of claims in GT}|}

Example:

Question = What is SpaceX and Who found it?

Answer = It is an American aerospace company founded by Elon Musk

Context = SpaceX is an American aerospace company founded in 2002.

Ground Truth = SpaceX is an American aerospace company founded by Elon Musk.

Solution:

First, break down the sentence from Ground truth and check them against the context.

Statement 1: "SpaceX is an American aerospace company" (Yes, it is in the context)

Statement 2: "Founded by Elon Musk" (No, this part is not in the context)

In this example, we have 1st statement that is present in the context, while the 2nd statement is not present in the context.

Therefore, the context precision here is:

\text{Context Recall} = { \text{1} \over \text{2} } = 0.5

Code:

Context Recall using RAGAS:

from datasets import Dataset
from ragas.metrics import context_recall
from ragas import evaluate

data_samples = {
    'question': ['What is SpaceX and Who found it?','What exactly does SpaceX do?' ],
    'prediction': ['It is an American aerospace company founded by Elon Musk','SpaceX produces and operates the Falcon 9 and Falcon rockets'],
    'contexts': [['SpaceX is an American aerospace company founded in 2002'], 
     ['SpaceX produces and operates the Falcon 9 and Falcon rockets']],
    'ground_truth': ['SpaceX is an American aerospace company founded by Elon Musk','SpaceX produces and operates the Falcon 9 and Falcon Heavy rockets']
}

dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset,metrics=[context_recall])
score.to_pandas()

Context Recall using Athina AI:

from athina.evals import RagasContextRecall
data = [
    {
        "query": "What is SpaceX and Who found it?",
        "context": ['SpaceX is an American aerospace company founded in 2002'],
        "expected_response": "SpaceX is an American aerospace company founded by Elon Musk"
    },
    {
        "query": "What exactly does SpaceX do?",
        "context": ['SpaceX produces and operates the Falcon 9 and Falcon rockets'],
        "expected_response": "SpaceX produces and operates the Falcon 9 and Falcon Heavy rockets"
    },
]

dataset = Loader().load_dict(data)
eval_model = "gpt-3.5-turbo"
RagasContextRecall(model=eval_model).run_batch(data=dataset).to_df()

← Context Precision Context Relevancy →