Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering

Original Paper: https://arxiv.org/abs/2306.00526

By: Wenjin Wang, Yunhao Li, Yixin Ou, Yin Zhang

Abstract:

Layout-aware pre-trained models has achieved significant progress on document image question answering. They introduce extra learnable modules into existing language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. However, extra modules necessitate pre-training on extensive document images. This prevents these methods from directly utilizing off-the-shelf instruction-tuning language foundation models, which have recently shown promising potential in zero-shot learning. Instead, in this paper, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks. Based on this observation, we propose the LAyout and Task aware Instruction Prompt (LATIN-Prompt), which consists of layout-aware document content and task-aware instruction. Specifically, the former uses appropriate spaces and line breaks to recover the layout information among text segments obtained by OCR tools, and the latter ensures that generated answers adhere to formatting requirements. Moreover, we propose the LAyout and Task aware Instruction Tuning (LATIN-Tuning) to improve the performance of small instruction-tuning models like Alpaca. Experimental results show that LATIN-Prompt enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering, and LATIN-Tuning enhances the zero-shot performance of Alpaca significantly. For example, LATIN-Prompt improves the performance of Claude and ChatGPT on DocVQA by 263% and 20% respectively. LATIN-Tuning improves the performance of Alpaca on DocVQA by 87.7%. Quantitative and qualitative analyses demonstrate the effectiveness of LATIN-Prompt and LATIN-Tuning. We provide the code in supplementary and will release it to facilitate future research.

Summary Notes

Simplifying Document Image Question Answering with Zero-Shot Learning

The field of document image analysis is evolving, with the challenge of answering questions from document images at its core.

Traditional methods have relied on extensive training on large datasets, incorporating textual, visual, and layout information.

However, a new approach called LATIN-Prompt is revolutionizing this area by introducing zero-shot learning that works without the need for detailed pre-training, thanks to its layout-aware and task-aware strategies.

Understanding the Challenge

Answering questions from document images is complex, as it involves textual content, visual cues, and layout information.

Traditionally, models needed to be pre-trained on huge datasets to grasp these elements, a process that was resource-heavy and limited the models' real-world usefulness.

Review of Previous Work

• Layout-Aware Pre-Trained Models: These models were a step forward but required extensive pre-training.
• Instruction-Tuning Language Models: Showed potential for adaptability and zero-shot learning but struggled to incorporate layout information effectively.

Introducing LATIN-Prompt and LATIN-Tuning

LATIN-Prompt and LATIN-Tuning are innovative solutions that address these challenges:

• LATIN-Prompt: Uses simple text formatting (like spaces and line breaks) to mimic document layouts in OCR-derived text. This, combined with task-specific instructions, allows language models to understand and answer questions without prior training.
• LATIN-Tuning: Enhances instruction-tuning models by training them on data prepared with LATIN-Prompt, improving their zero-shot learning performance.

Detailed Methodology

• LATIN-Prompt enables models to process document content accurately by simulating the original layout and pairing it with specific instructions.
• LATIN-Tuning boosts smaller models by training them on LATIN-Prompt formatted data, enhancing their comprehension abilities without extensive data training.

Experimental Insights

Experiments show that LATIN-Prompt and LATIN-Tuning not only improve zero-shot performance in models like Claude and ChatGPT but also enable smaller models to perform at the level of more extensively trained counterparts.

This highlights the potential of using pre-trained instruction-tuning language models for complex tasks like document image question answering.

The Path Forward

LATIN-Prompt and LATIN-Tuning mark significant advancements in document image analysis, offering efficient zero-shot learning capabilities.

These methods simplify the AI processing pipeline and extend the use of AI models across different sectors. They also make advanced document understanding technologies more accessible to a wider audience.

In summary, LATIN-Prompt and LATIN-Tuning are paving the way for new applications of large language models in document image question answering.

By combining layout-aware content with task-aware instructions, they represent a promising direction for future research and practical applications in various industries.