PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers

Original Paper: https://arxiv.org/abs/2406.12430

By: Myeonghwa Lee, Seonho An, Min-Soo Kim

Abstract:

In this paper, we conduct a study to utilize LLMs as a solution for decision making that requires complex data analysis. We define Decision QA as the task of answering the best decision, dbest, for a decision-making question Q, business rules R and a database D. Since there is no benchmark that can examine Decision QA, we propose Decision QA benchmark, DQA. It has two scenarios, Locating and Building, constructed from two video games (Europa Universalis IV and Victoria 3) that have almost the same goal as Decision QA. To address Decision QA effectively, we also propose a new RAG technique called the iterative plan-then-retrieval augmented generation (PlanRAG). Our PlanRAG-based LM generates the plan for decision making as the first step, and the retriever generates the queries for data analysis as the second step. The proposed method outperforms the state-of-the-art iterative RAG method by 15.8% in the Locating scenario and by 7.4% in the Building scenario, respectively. We release our code and benchmark at this https URL.

Summary Notes

Introduction

In an era where data drives business decisions, the role of decision-making systems is more crucial than ever. Traditional decision support systems have evolved, but they often fall short when it comes to handling complex decisions requiring extensive data analysis. Enter Large Language Models (LLMs) and a novel approach called PlanRAG (Plan-then-Retrieval Augmented Generation). This innovative technique promises to transform LLMs into proficient decision-makers, capable of tackling even the most intricate business scenarios. Let’s delve into how PlanRAG works and its potential impact on decision-making tasks.

The Research Question

The main question this research addresses is: Can LLMs be effectively utilized for decision-making tasks that require complex data analysis? The study introduces "Decision QA" (Decision Question Answering), a task where the goal is to determine the best decision based on a given question, business rules, and a database.

Methodology: How PlanRAG Works

1. Decision QA Task

The Decision QA task involves three key steps:

1. Planning: Determine what kind of data analysis is needed.
2. Retrieving: Generate and execute queries to retrieve necessary data.
3. Answering: Make the final decision based on the retrieved data.

2. Benchmark Creation

To test Decision QA, the researchers created a benchmark using two video games, Europa Universalis IV and Victoria 3, which simulate real-world business scenarios. The benchmark consists of two scenarios:

• Locating: Deciding where to place a resource (e.g., a merchant) to maximize profit.
• Building: Deciding how much of a resource to allocate (e.g., wood) to optimize production.

3. Iterative Plan-then-Retrieval Augmented Generation

The core innovation of this study is the PlanRAG technique, which extends the traditional Retrieval-Augmented Generation (RAG) method by adding a critical planning step:

• Planning: The LLM generates an initial plan outlining the required data analyses.
• Retrieving: The LLM retrieves data based on the plan and performs data analysis.
• Re-planning: If necessary, the LLM revises the plan and performs additional retrievals.
• Answering: Finally, the LLM makes the decision based on the comprehensive data analysis.

Key Findings

The study's experiments reveal that PlanRAG significantly outperforms the state-of-the-art iterative RAG method:

• Locating Scenario: PlanRAG improves accuracy by 15.8%.
• Building Scenario: PlanRAG improves accuracy by 7.4%.

Moreover, PlanRAG demonstrates a superior ability to handle complex decision-making tasks, particularly those requiring multiple data retrievals and iterative analysis.

Implications and Applications

The implications of this research are profound for various industries:

• Business Strategy: Companies can leverage PlanRAG to optimize supply chain decisions, resource allocation, and market strategies.
• Healthcare: Hospitals and pharmaceutical companies can use PlanRAG to make data-driven decisions regarding patient care and drug distribution.
• Finance: Financial institutions can enhance their investment strategies and risk management practices by utilizing advanced decision-making models.

Conclusion

PlanRAG represents a significant leap forward in using LLMs for decision-making tasks. By integrating a planning step into the retrieval process, PlanRAG enables LLMs to handle complex data analyses and make more accurate decisions. As businesses continue to seek ways to harness the power of data, PlanRAG offers a promising solution to enhance decision-making capabilities.

Future Research

While PlanRAG shows great promise, there are areas for further exploration:

• Hybrid Databases: Investigate the application of PlanRAG for decision-making tasks involving hybrid forms of databases.
• Enhanced Planning Algorithms: Develop more sophisticated planning algorithms to further improve the accuracy and efficiency of decision-making.
• Real-World Deployments: Test PlanRAG in real-world business environments to validate its effectiveness and scalability.

The future of decision-making is here, and it's powered by LLMs and innovative techniques like PlanRAG. By embracing these advancements, businesses can make more informed and strategic decisions, ultimately driving success in an increasingly data-driven world.