Original Paper: https://arxiv.org/abs/2302.11520

By: Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, Xifeng Yan

Abstract:

We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model (e.g., T5) to generate an auxiliary directional stimulus prompt for each input instance. These directional stimulus prompts act as nuanced, instance-specific hints and clues to guide LLMs in generating desired outcomes, such as including specific keywords in the generated summary. Our approach sidesteps the challenges of direct LLM tuning by optimizing the policy model to explore directional stimulus prompts that align LLMs with desired behaviors. The policy model can be optimized through 1) supervised fine-tuning using labeled data and 2) reinforcement learning from offline or online rewards based on the LLM's output. We assess our method across summarization, dialogue response generation, and chain-of-thought reasoning tasks. Our experiments demonstrate that the framework consistently improves LLMs' (e.g., ChatGPT, Codex, InstructGPT) performance on these supervised tasks using minimal labeled data. Notably, using just 80 dialogues on the MultiWOZ dataset, our approach enhances ChatGPT's performance by an impressive 41.4%, matching or surpassing some fully supervised start-of-the-art models. Additionally, the instance-specific chain-of-thought prompt generated by our approach improves InstructGPT's reasoning accuracy compared to human-crafted or automatically generated prompts. The code and data are publicly available at \url{

this https URL

Summary Notes

Simplifying Large Language Models with Directional Stimulus Prompting

In the world of artificial intelligence, Large Language Models (LLMs) like ChatGPT and InstructGPT have shown incredible abilities, from writing essays to programming.

However, aligning their outputs with what users specifically want remains a challenge, especially for AI Engineers at big companies.

Traditionally, tweaking these models directly has been tough due to their complex nature. Enter Directional Stimulus Prompting (DSP), a new method that offers a simpler way to guide LLMs to the desired outcomes.

What is Directional Stimulus Prompting?

DSP is a cutting-edge strategy aimed at steering LLMs in the right direction without changing their internal workings.

It involves creating specialized prompts that provide the models with hints on what to generate. This method stands out because it doesn't rely on adding external knowledge to LLMs or forcefully adjusting them but rather on crafting specific guidance for each task.

How It Works

DSP uses a two-step process with a smaller, adjustable language model to create these helpful hints:

1. Supervised Fine-Tuning (SFT): First, the model learns to produce useful hints from examples in a dataset designed to teach it about the kinds of outputs desired.
2. Reinforcement Learning (RL): Next, it tries out different hints to find the ones that lead to outputs closest to what's needed, improving through trial and error.

Results in Action

DSP has shown impressive results in various areas:

• Summarization: It helped ChatGPT do a better job summarizing articles, leading to higher quality summaries.
• Dialogue Generation: DSP made dialogue systems like those in MultiWOZ more accurate and relevant, enhancing chatbot conversations.
• Complex Reasoning: It also improved the ability of models to solve problems that require deeper thinking, making them more effective at tasks that involve logic and reasoning.

These examples highlight how DSP makes LLMs more versatile and aligned with specific tasks.

Future Outlook and Benefits

DSP represents a big step forward in AI, making LLMs more controllable and efficient at specific tasks. Looking ahead, there's potential for even more sophisticated guidance methods and for using machine languages that could communicate instructions to LLMs more clearly.

Why DSP Matters

For AI Engineers in large organizations, DSP is a game-changer. It allows them to use LLMs more effectively in various applications, from customer service automation to content creation, without needing to retrain or heavily modify the models. Also, with the resources for DSP publicly available, there's room for more innovation and collaboration in making LLMs work better for everyone.

Conclusion

Directional Stimulus Prompting provides a valuable tool for aligning LLM outputs with user needs, improving their performance across a range of tasks.

Through tailored hints, DSP enhances the practicality and personalization of AI applications. As we continue to refine and expand on DSP, it will undoubtedly play a key role in unlocking LLMs' full potential for real-world use.