Faithfulness

The faithfulness metric evaluates whether the generated answer can be found in the provided context. The score ranges from 0 to 1, where a higher score indicates better factual consistency.

Formula:

\text{Faithfulness} = {|\text{Number of claims in the answer that are found in the context}| \over |\text{Total claims in the answer}|}

Example:

Questions = What is SpaceX and Who found it

Answers = It is an American aerospace company. It was founded by Nikola Tesla.

Contexts = SpaceX is an American aerospace company founded in 2002, Founded by Elon Musk, The full form of SpaceX is Space Exploration Technologies Corporation.

Solution:

Now let’s calculate faithfulness:

First, break the generated answer into individual statements/claims.

Statement 1: It is an American aerospace company.

Statement 2: It was founded by Nikola Tesla.

After that, check each one to see if it can be supported by the given context.

Statement 1: Yes (supported by the context)

Statement 1: No (not supported by the context)

Finally, we use the Faithfulness to find the Faithfulness score.

\text{Faithfulness} = { \text{1} \over \text{2} } = 0.5

This shows how well the generated answers are grounded in the provided context.

Code:

Faithfulness using RAGAS:

from datasets import Dataset 
from ragas.metrics import faithfulness
from ragas import evaluate
data_samples = {
    'question': ['What is SpaceX?', 'Who found it?','What exactly does SpaceX do?'],
    'answer': ['It is an American aerospace company', 'SpaceX founded by Elon Musk','SpaceX produces and operates the Falcon 9 and Falcon rockets'],
    'contexts' : [['SpaceX is an American aerospace company founded in 2002'],
    ['SpaceX, founded by Elon Musk, is worth nearly $210 billion'], ['SpaceX produces and operates the Falcon 9 and the full form of SpaceX is Space Exploration Technologies Corporation']],
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset,metrics=[faithfulness])
score.to_pandas()

Faithfulness using Athina AI:

from athina.evals import RagasFaithfulness

data = [
    {
        "query": "What is SpaceX?",
        "context": ['SpaceX is an American aerospace company founded in 2002'],
        "response": "It is an American aerospace company"
    },
    {
        "query": "Who found it?",
        "context": ['SpaceX, founded by Elon Musk, is worth nearly $210 billion'],
        "response": "SpaceX founded by Elon Musk"
    },
    {
        "query": "What exactly does SpaceX do?",
        "context": ['The full form of SpaceX is Space Exploration Technologies Corporation'],
        "response": "SpaceX produces and operates the Falcon 9 and Falcon rockets"
    },
]

dataset = Loader().load_dict(data)
eval_model = "gpt-3.5-turbo"
RagasFaithfulness(model=eval_model).run_batch(data=dataset).to_df()

← Answer Correctness Aspect Critic →