DiffusionGPT: LLM-Driven Text-to-Image Generation System

Original Paper: https://arxiv.org/abs/2401.10061

By: Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, Shilei Wen

Abstract:

Diffusion models have opened up new avenues for the field of image generation, resulting in the proliferation of high-quality models shared on open-source platforms. However, a major challenge persists in current text-to-image systems are often unable to handle diverse inputs, or are limited to single model results. Current unified attempts often fall into two orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate expert model to output. To combine the best of both worlds, we propose DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified generation system capable of seamlessly accommodating various types of prompts and integrating domain-expert models. DiffusionGPT constructs domain-specific Trees for various generative models based on prior knowledge. When provided with an input, the LLM parses the prompt and employs the Trees-of-Thought to guide the selection of an appropriate model, thereby relaxing input constraints and ensuring exceptional performance across diverse domains. Moreover, we introduce Advantage Databases, where the Tree-of-Thought is enriched with human feedback, aligning the model selection process with human preferences. Through extensive experiments and comparisons, we demonstrate the effectiveness of DiffusionGPT, showcasing its potential for pushing the boundaries of image synthesis in diverse domains.

Summary Notes

Introducing DiffusionGPT - A New Frontier in AI-Powered Image Creation

The development of DiffusionGPT marks a significant milestone in the field of artificial intelligence, particularly in turning textual prompts into vivid images.

This system, a joint effort by ByteDance Inc and Sun Yat-Sen University, uses Large Language Models (LLMs) and domain-expert models to produce high-quality visuals from text descriptions.

This post delves into how DiffusionGPT works, its methodological innovations, and its edge over current technologies.

How DiffusionGPT Works

DiffusionGPT’s process for generating images from text is both advanced and intuitive, involving several key steps:

• Prompt Parsing: It starts by breaking down the input text with LLMs to grasp the essential content for image creation.
• Model Tree Structure: The system uses a structured tree of models for efficient model selection, built and navigated using model tags.
• Selecting the Model: The choice of model is fine-tuned with human feedback and a database of advantages, aiming for outputs that align closely with human preferences.
• Creating the Image: Finally, an image is generated using a chosen model and a Prompt Extension Agent that improves the prompt’s quality, leading to better images.

Performance Highlights

DiffusionGPT's capabilities have been validated through rigorous testing, showing it outperforms existing models like SD1.5 and SDXL in realism and semantic accuracy. Highlights include:

• Superior image quality and aesthetics, as reflected in higher image-reward and aesthetic scores.
• The importance of the Tree-of-Thought, human feedback, and prompt extension in achieving high-quality images is confirmed via ablation studies.
• User studies indicate a clear preference for images created by DiffusionGPT over competing models.

Looking Ahead: Limitations and Future Directions

Despite its advancements, the team behind DiffusionGPT is focusing on future improvements, such as incorporating real-time feedback into the LLM optimization process and expanding the model library for wider application.

Enhancing prompt parsing and model selection accuracy remains a priority.

Conclusion

DiffusionGPT represents a leap forward in making AI an active participant in creative endeavors, blending LLMs with human feedback for improved text-to-image generation. It not only achieves images that resonate more closely with human expectations but also opens new avenues for AI in creative processes.

As AI continues to evolve, the innovations brought forth by DiffusionGPT will be critical in defining the next generation of AI-assisted creativity, demonstrating the vast potential of combining AI technologies with human insight in digital content creation.