State of What Art? A Call for Multi-Prompt LLM Evaluation

Original Paper: https://arxiv.org/abs/2401.00595

By: Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, Gabriel Stanovsky

Abstract:

Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks. These benchmarks typically rely on a single instruction template for evaluating all LLMs on a specific task. In this paper, we comprehensively analyze the brittleness of results obtained via single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we propose to evaluate LLMs with a set of diverse prompts instead. We discuss tailored evaluation metrics for specific use cases (e.g., LLM developers vs. developers interested in a specific downstream task), ensuring a more reliable and meaningful assessment of LLM capabilities. We then implement these criteria and conduct evaluations of multiple models, providing insights into the true strengths and limitations of current LLMs.

Summary Notes

Enhancing LLM Evaluation with Multi-Prompt Approaches

In the fast-paced realm of Artificial Intelligence (AI), Large Language Models (LLMs) are at the forefront, powering everything from chatbots to content creation tools.

As these models evolve, a pressing question emerges: How do we accurately measure their capabilities? Current benchmarks, which rely on single instruction templates, don't provide a full picture.

This post explores the limitations of traditional benchmarks and advocates for a multi-prompt evaluation method to more accurately gauge LLM capabilities.

The Evaluation Challenge

Current benchmarks for LLMs typically use a single instruction template for each task, which might not reveal the full capabilities and limitations of the models.

For example, an LLM might perform well with one prompt but struggle with a similar one. This inconsistency calls for a more comprehensive evaluation method.

Proposing a Multi-Prompt Evaluation Framework

Researchers Moran Mizrahi and colleagues have proposed a new framework that considers the complexity of real-world tasks:

Dataset Creation

• Paraphrased Instructions: They created a dataset with varied paraphrased instructions for each task, using automatic paraphrasing and manual checks.
• Broad Coverage: This approach aims to evaluate LLMs across a wide range of prompts, mirroring real-world variability.

Evaluation Approach

• Multi-Prompt Assessment: The framework evaluates LLMs across multiple paraphrases for each task to get a more accurate picture of their capabilities.
• Task-Specific Metrics: It employs different metrics based on the task, focusing on either robustness across prompts or specific task performance.

Developing Metrics

• Robustness Metrics: These measure how well an LLM performs across different instruction templates.
• Task-Specific Metrics: These are tailored to specific tasks to pinpoint where a model excels or falls short.

Insights from Multi-Prompt Evaluation

This approach revealed considerable performance variability among LLMs, with a model's ranking changing dramatically depending on the prompt used.

This underscores the limitations of single-prompt benchmarks and the need for a more nuanced evaluation method.

Benefits of New Metrics

The new metrics offer insights into LLM strengths and weaknesses overlooked by traditional evaluations, guiding developers and researchers in refining LLM capabilities.

Moving Forward

Single instruction template evaluations can misrepresent LLM capabilities. The research pushes for multi-prompt evaluations that better mirror the complexity and variability of real-world use, promising more accurate and comparable evaluations.

Conclusion

Adopting multi-prompt evaluation frameworks marks a significant advancement in understanding and improving LLMs, reflecting the complexity of real-world tasks and leading to more reliable and effective AI systems.

This work encourages the adoption of nuanced evaluation methods, contributing to advancements that truly represent the state of the art in LLM technology.

What's Next

Future efforts will focus on making these evaluation methods more efficient and less computationally demanding.

This foundation encourages robust and meaningful assessments of LLMs, fueling innovation across the field.

As AI progresses, our evaluation methods must evolve alongside it. Embracing multi-prompt evaluations ensures our understanding of LLM capabilities remains comprehensive and nuanced.