research-papers

LLM evaluation too expensive? Here's how we solve this.

Athina AI Agent

Feb 15, 2024 — 2 min read

According to the latest and greatest research, the SoTA eval techniques use LLMs as evaluators. It's a little meta, but it works better than anything else.

But sometimes, they are too expensive to run in production. Here's how Athina solves this.

How to run LLM-graded evals in production

Without blowing up your OpenAI budget.

Here's how we help you solve this so you can still get model performance insights in production!

Sampling Percentage: You can get similar insights by running evals on 100k inferences instead of 500k inferences. We have a configuration setting for this on Athina. Documentation.
Max Evals per month: Set a maximum number of evaluations to run per month as a hard limit. This ensures your costs are always limited.
Filters: When you configure Evals on Athina, you can set filters. These filters will ensure that evals ONLY run on inferences that match the filters. For example, you could set a filter to only run evals on inferences where:
user query contains "refund"
language model is gpt-4
prompt slug is summarization/v3
Use a cheaper model: Many evals will work great with GPT-3.5 (though some will not). That shaves the cost down to 1/10 of GPT-4 Turbo. If you want to run evals on even cheaper models (Llama, Mistral, etc), then reply to this email. We're working on it :)

Function Evals

We just shipped a whole new library of function evals. These evals do NOT use LLMs, which means they are deterministic and FREE. Here are some examples of the function evals we've just shipped:

Contains [All / Any / None]: Checks if response contains (or does not contain) some keywords.
Regex: Checks if response contains a regex pattern
Contains Valid Link: Checks that a link exists AND is valid (not a hallucinated link)
Contains JSON: Checks if response contains JSON
Contains Email: Checks if response contains JSON
Answer similarity: Similarity between the response and expected_response.
API Call: Bring your own eval. We'll hit your endpoint to run an eval.

You can set these evals up on our UI or through our open-source SDK.

View the documentation here.

Want Early Access to Our Latest Features?

We're working on some exciting new things.

Hint: It rhymes with shine-tuning.

If you want early access, or want to see what we're doing, email me at shiv@athina.ai.

Cheers,

Shiv Sakhuja

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

Original Paper: https://arxiv.org/abs/2310.06839 By: Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu Abstract: In long context scenarios, large language models (LLMs) face three main challenges: higher computational/financial cost, longer latency, and inferior performance. Some studies reveal that the

Chain-of-Verification Reduces Hallucination in Large Language Models

Original Paper: https://arxiv.org/abs/2309.11495 By: Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, Jason Weston Abstract: Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models. We study the ability of language models to deliberate

How DoorDash Mitigates Hallucinations in its RAG System with a Two-Tier LLM Guardrail: A Hands-On Experiment

In this blog, we’ll explore how DoorDash leverages Retrieval-Augmented Generation (RAG) to address Dashers’ queries using insights from the company’s internal blogs. We’ll conduct practical experiments based on the techniques discussed in DoorDash’s Tech blog, utilizing a curated DoorDash dataset. We will use Athina AI which

Quantized LoRA: Fine-Tuning Large Language Models with Ease

Language models like GPT-4 have established themselves as the leading standard in the NLP industry for developing advanced applications. These models can perform diverse tasks and easily adapt to new tasks using Prompt Engineering Techniques. However, they also present a massive challenge around training and fine-tuning. Due to their massive