Prompt-Guided Transformers for End-to-End Open-Vocabulary Object Detection

Original Paper: https://arxiv.org/abs/2303.14386

By: Hwanjun Song, Jihwan Bang

Abstract:

Prompt-OVD is an efficient and effective framework for open-vocabulary object detection that utilizes class embeddings from CLIP as prompts, guiding the Transformer decoder to detect objects in both base and novel classes. Additionally, our novel RoI-based masked attention and RoI pruning techniques help leverage the zero-shot classification ability of the Vision Transformer-based CLIP, resulting in improved detection performance at minimal computational cost. Our experiments on the OV-COCO and OVLVIS datasets demonstrate that Prompt-OVD achieves an impressive 21.2 times faster inference speed than the first end-to-end open-vocabulary detection method (OV-DETR), while also achieving higher APs than four two-stage-based methods operating within similar inference time ranges. Code will be made available soon.

Summary Notes

Enhancing Object Detection with Prompt-Guided Transformers

Object detection is a key technology in computer vision, enabling machines to recognize and locate objects in images or videos.

This technology is crucial for various applications like autonomous driving and surveillance. However, detecting new object classes that were not included in the training phase remains a challenge.

Traditional methods often face problems with overfitting and slow speeds.

The introduction of Prompt-OVD offers a promising solution to these issues by using innovative techniques to improve open-vocabulary object detection (OVD).

Transformers: Revolutionizing Object Detection

Transformers have transformed the field of natural language processing and are now advancing computer vision.

Models like DETR and its variants have set new standards by combining convolutional neural networks (CNNs) with Transformer encoder-decoders, simplifying and enhancing object detection.

Open-Vocabulary Object Detection Advancements

Integrating visual language models like CLIP into object detection models enables the detection of objects outside the training dataset.

Though existing methods show promise, they still face challenges with efficiency and accuracy. Prompt-OVD stands out by addressing these issues with innovative solutions.

Introducing Prompt-OVD

Prompt-OVD revolutionizes OVD by using prompt-based decoding, which simplifies the detection process and reduces computational load. Its key innovations include:

• Prompt-based Decoding: Uses CLIP embeddings as class prompts with class-agnostic object queries for a more efficient decoding process.
• Efficient RoI Techniques: Implements RoI-based masked attention and RoI pruning to optimize the use of pre-trained ViT-based CLIP, improving object classification.
• Ensemble with ViT-based CLIP: Combines detection results with CLIP for enhanced accuracy, especially in detecting unseen object classes.

Performance of Prompt-OVD

Prompt-OVD was tested on modified versions of the MS-COCO and LVIS datasets, showing impressive results. It achieved faster inference speeds—21.2 times quicker than OV-DETR—with comparable or better accuracy.

Ablation studies confirmed the effectiveness of its innovative features.

Conclusion: A Leap Forward in Object Detection

Prompt-OVD marks a significant advancement in object detection, offering a more efficient and accurate solution for detecting a wide range of objects.

Its use of prompt-based decoding and RoI techniques demonstrates the potential of applying large pre-trained models in a seamless system. As the need for advanced object detection grows, the potential of Prompt-OVD and similar technologies is vast.

The combination of Transformer architectures with visual-language models is setting the stage for future progress in computer vision, promising improved capabilities and new opportunities.

Prompt-OVD not only overcomes challenges in traditional object detection methods but also paves the way for further research and practical applications in computer vision.

The synergy between Transformers and visual-language models suggests a bright future for object detection technologies.