Original Paper: https://arxiv.org/abs/2305.02578

By: Ruoyu Feng, Jinming Liu, Xin Jin, Xiaohan Pan, Heming Sun, Zhibo Chen

Abstract:

Image coding for machines (ICM) aims to compress images to support downstream AI analysis instead of human perception. For ICM, developing a unified codec to reduce information redundancy while empowering the compressed features to support various vision tasks is very important, which inevitably faces two core challenges: 1) How should the compression strategy be adjusted based on the downstream tasks? 2) How to well adapt the compressed features to different downstream tasks? Inspired by recent advances in transferring large-scale pre-trained models to downstream tasks via prompting, in this work, we explore a new ICM framework, termed Prompt-ICM. To address both challenges by carefully learning task-driven prompts to coordinate well the compression process and downstream analysis. Specifically, our method is composed of two core designs: a) compression prompts, which are implemented as importance maps predicted by an information selector, and used to achieve different content-weighted bit allocations during compression according to different downstream tasks; b) task-adaptive prompts, which are instantiated as a few learnable parameters specifically for tuning compressed features for the specific intelligent task. Extensive experiments demonstrate that with a single feature codec and a few extra parameters, our proposed framework could efficiently support different kinds of intelligent tasks with much higher coding efficiency.

Summary Notes

Enhancing AI Efficiency: The Innovative Prompt-ICM Framework for Image Coding

Introduction

Efficiently processing images for AI and machine learning tasks, as opposed to human viewing, poses a significant challenge.

Traditional image compression methods, designed with human viewers in mind, often do not meet the needs of machine-based tasks.

This has led to the development of Image Coding for Machines (ICM), a method focused on optimizing images for machine interpretation and efficiency.

A groundbreaking approach within this field is Prompt-ICM, which uses task-driven prompts to transform how images are compressed and processed for AI tasks.

Understanding the Framework

ICM aims to tailor image compression to improve machine task performance, which can be categorized into:

• Task-specific Image Compression: Compresses images for particular tasks but lacks versatility.
• Feature-based ICM: Compresses specific task features, offering some optimization but not task-specific.
• General Feature-based ICM: Uses a general feature extractor for all tasks but isn't optimized for specific tasks.
• Prompt-ICM (Proposed): Enhances general feature-based ICM by employing task-driven prompts to direct the compression and feature adaptation for specific tasks, making it a versatile and efficient approach.

Techniques Behind Prompt-ICM

Prompt-ICM introduces two key innovations:

1. Compression Prompts: Importance maps predicted by an information selector that guide the compression process, ensuring vital information for a task is prioritized.
2. Task-adaptive Prompts: Learnable parameters that fine-tune compressed features for specific tasks, enhancing performance efficiently.

Experimental Insights

Prompt-ICM has been rigorously tested and has shown exceptional efficiency in various intelligent tasks like classification and segmentation.

It outperforms both traditional and learned codecs in rate-distortion efficiency, proving its effectiveness in compressing images for machine tasks.

Key Contributions

Prompt-ICM stands out for several reasons:

1. Unified Framework: It's the first to combine image compression and task analysis in a single ICM framework.
2. Compression Prompts: Introduces importance maps for content-weighted compression aligned with task needs.
3. Task-adaptive Prompt Tuning: Offers a new way to adjust compressed features for tasks, significantly boosting performance with minimal parameter increase.

Background Work

Prompt-ICM builds on extensive research in image compression and ICM, from traditional codecs to those optimized for perceptual quality, and the emerging focus on machine-specific image compression.

It also leverages Parameter Efficient Tuning (PET) concepts, crucial for developing the task-adaptive prompts.

Conclusion

Prompt-ICM marks a significant advancement in machine-specific image coding, adeptly meeting the challenges of adapting compression for various AI tasks with high efficiency.

By utilizing task-driven prompts, it enhances both the compression process and the quality of features for downstream tasks. As AI applications become more complex, Prompt-ICM's flexibility and efficiency will be invaluable for AI engineers.

Figures and Results

• Figure 1: Visual comparison of ICM pipelines, showcasing Prompt-ICM's integration of task-driven prompts.
• Experimental Results: Highlight Prompt-ICM's superior performance in optimizing image coding for machine tasks across different tasks and datasets.

Prompt-ICM not only streamlines image processing for AI applications but also fosters task-specific optimization, leading to a new era in machine learning characterized by adaptability and efficiency.