Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Original Paper: https://arxiv.org/abs/2305.03047

By: Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

Abstract:

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.

Summary Notes

Figure: An illustration of the four essential stages in the SELF-ALIGN process

In the rapidly evolving field of AI, the alignment of language models with human values and intentions is crucial for ensuring ethical, reliable, and helpful AI systems. Traditional approaches, like those used in ChatGPT and other state-of-the-art models, rely heavily on supervised fine-tuning (SFT) with extensive human annotations and reinforcement learning from human feedback (RLHF). However, these methods are not without their limitations, including high costs, potential biases, and the need for vast amounts of human supervision. The recent research paper "Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision" introduces an innovative approach that significantly reduces the dependency on human annotations while maintaining high performance and ethical standards.

Introduction to Self-Alignment

The new approach, called SELF-ALIGN, leverages principle-driven reasoning and the generative power of large language models (LLMs) to achieve self-alignment. This method is applied to the LLaMA-65b base language model, resulting in the development of an AI assistant named Dromedary. Remarkably, Dromedary achieves superior performance compared to several state-of-the-art AI systems with less than 300 lines of human annotations, including fewer than 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning.

The Need for Self-Alignment

Current AI systems' reliance on extensive human supervision can lead to issues with quality, reliability, diversity, self-consistency, and undesirable biases. The SELF-ALIGN approach addresses these challenges by reducing the need for human supervision and enhancing the alignment efficiency of AI models.

Methodology

The SELF-ALIGN process consists of four essential stages:

1. Topic-Guided Red-Teaming Self-Instruct

This stage employs a self-instruct mechanism using 175 seed prompts to generate synthetic instructions, supplemented by 20 topic-specific prompts to ensure diverse topic coverage. This comprehensive range of contexts and scenarios helps the AI system learn more effectively.

2. Principle-Driven Self-Alignment

A set of 16 human-written principles guides the behavior of the AI model in generating responses. These principles ensure that the responses are helpful, ethical, and reliable. In-context learning (ICL) with five exemplars demonstrates how the AI complies with these principles in various scenarios. The AI model uses these guidelines to generate explanations for refused answers when queries are detected as harmful or ill-formed.

3. Principle Engraving

In this stage, the base LLM is fine-tuned on the self-aligned responses generated by the LLM itself through prompting. This fine-tuning process engrains the principles into the model's parameters, enabling the system to generate high-quality responses for new queries without explicitly using the principle set and ICL exemplars.

4. Verbose Cloning

To address the issue of overly brief or indirect responses, context distillation is employed to enhance the system's capability to produce comprehensive and elaborate responses. This stage ensures that the final model can generate detailed and in-depth answers to user queries.

Findings and Results

The SELF-ALIGN approach demonstrates impressive results across various benchmarks:

TruthfulQA Benchmark

The TruthfulQA benchmark evaluates a model's ability to identify true claims. Dromedary outperforms powerful models like GPT-4, achieving a new state-of-the-art multiple choice accuracy of 69%.

BIG-bench HHH Eval

This benchmark assesses the model's performance in terms of helpfulness, honesty, and harmlessness. Dromedary shows significant improvement compared to other open-source models, demonstrating high alignment with ethical standards.

Vicuna Benchmark Questions

Using GPT-4 for evaluation, Dromedary surpasses Text-Davinci-003 and Alpaca but falls short of ChatGPT and its distilled version, Vicuna. This evaluation highlights Dromedary's strengths in generating detailed and contextually relevant responses.

Implications and Applications

The principle-driven self-alignment approach presents a paradigm shift in AI development, with several key implications:

Enhanced Supervision Efficiency

By reducing the need for extensive human annotations, the SELF-ALIGN method significantly lowers the cost and effort involved in training AI models. This efficiency allows for the development of high-performing AI systems with minimal human supervision.

Improved Ethical Standards

The use of principles ensures that the AI models adhere to ethical guidelines, reducing the risk of generating harmful or biased content. This alignment with human values is crucial for the responsible deployment of AI systems.

Broader Applicability

The SELF-ALIGN approach can be applied to various AI models, making it a versatile solution for different domains. Its ability to generate high-quality responses across diverse contexts and scenarios makes it suitable for a wide range of applications.

Conclusion

The principle-driven self-alignment of language models represents a significant advancement in AI development. By leveraging a small set of human-defined principles and minimizing the need for extensive human supervision, the SELF-ALIGN approach achieves superior performance and ethical alignment. This method paves the way for the development of AI systems that are not only powerful but also responsible and aligned with human values.

For future research, exploring the integration of Constitutional AI-based self-critique and reinforcement learning techniques could further enhance the performance of self-aligned models like Dromedary. Conducting ablation studies on the alignment principles and engaging with the broader research community to refine these principles will be crucial for driving positive outcomes across various communities.

In conclusion, the SELF-ALIGN approach offers a promising direction for the future of AI alignment, ensuring that AI systems are both effective and ethically sound.