Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

Original Paper: https://arxiv.org/abs/2403.19103

By: Yutong He, Alexander Robey, Naoki Murata, Yiding Jiang, Joshua Williams, George J. Pappas, Hamed Hassani, Yuki Mitsufuji, Ruslan Salakhutdinov, J. Zico Kolter

Abstract:

Prompt engineering is effective for controlling the output of text-to-image (T2I) generative models, but it is also laborious due to the need for manually crafted prompts. This challenge has spurred the development of algorithms for automated prompt generation. However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, and produce non-intuitive prompts. In this work, we introduce PRISM, an algorithm that automatically identifies human-interpretable and transferable prompts that can effectively generate desired concepts given only black-box access to T2I models. Inspired by large language model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompts distribution for given reference images. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney.

Summary Notes

Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

Transforming text descriptions into captivating images is a complex challenge in artificial intelligence. Traditionally, creating these text-to-image (T2I) translations required manual prompt engineering, a time-consuming and expert-driven process.

However, the emergence of automated techniques, particularly PRISM (Prompt Refinement and Iterative Sampling Mechanism), is revolutionizing this process by using large language models (LLMs) for more efficient prompt refinement.

This post explores PRISM's methodology, its experiments, and its potential to transform personalized T2I generation.

Introduction

Moving from manual to automated prompt engineering represents a significant evolution in T2I generation. Unlike the manual approach, which is straightforward but laborious and requires deep model knowledge, automated methods like PRISM facilitate prompt generation with minimal human input, making T2I generation more accessible and efficient across various models.

Background

Controllable T2I Generation Techniques

Efforts to achieve controllable T2I generation have led to several methods:

• Training-free methods: Use of pre-trained diffusion models.
• Fine-tuning approaches: Techniques like Dreambooth.
• Prompt tuning methods: Including Textual Inversion.

These methods, however, often lack in interpretability and generalizability.

Prompt Engineering Approaches

Prompt engineering is split between manual and automated techniques. Manual engineering is widespread for its simplicity but requires significant effort and expertise. Automated methods promise efficiency but face challenges in achieving comparable interpretability and generalizability.

PRISM Methodology

PRISM addresses these challenges by generating prompts that direct a T2I model to produce images aligned with the concepts in reference images.

Through iterative refinement with a multimodal LLM, PRISM fine-tunes prompts based on visual similarities between generated and reference images, without needing model retraining. This approach streamlines the creation of personalized T2I content.

Experiments

Implementation

PRISM utilizes GPT-4V for prompt generation and image scoring, with SDXL-Turbo as the T2I generator, demonstrating adaptability and efficiency.

Evaluation

PRISM is benchmarked against methods like Textual Inversion and BLIP-2, using metrics such as CLIP image similarity. Results show PRISM's superior interpretability, visual accuracy, and model transferability.

Findings

PRISM outperforms existing methods, offering enhanced interpretability, visual accuracy, and adaptability across T2I models. This underscores its potential to revolutionize personalized T2I generation.

Conclusion

PRISM represents a breakthrough in automated prompt engineering, using LLMs for iterative refinement to enable efficient T2I generation across different models.

This approach democratizes T2I generation and opens new possibilities for rapid, diverse image creation, marking a significant advancement for AI-driven visual content creation.

Acknowledgments

Supported by entities including ONR, NSF, and Sony AI, PRISM's development highlights collaborative innovation in AI research.

Its versatility across T2I models underlines the strength of its methodology and sets a benchmark for automated visual content generation from text descriptions, promising exciting future developments in the field.