[Seminar] Hallucinating RGB Modality for Person Detection Through Privileged Information

Abstract:Image translation is a powerful way to adapt a visual recognition model to a new domain. However, common image translation approaches only focus on generating data from the same distribution as the target domain. Given a cross-modal application, such as pedestrian detection from aerial images,  with a considerable shift in data distribution between infrared (IR) to visible (RGB) images, a translation focused on generation might lead to poor performance as the loss focuses on irrelevant details for the task. In this paper, we propose HalluciDet, an IR-RGB image translation model for object detection. Instead of focusing on reconstructing the original image on the IR modality, it seeks to reduce the detection loss of an RGB detector, avoiding the need to access RGB data. This model produces a new image representation that enhances objects of interest in the scene and dramatically improves detection performance. We empirically compare our approach against state-of-the-art methods for image translation and for fine-tuning on IR, and show that our HalluciDet improves detection accuracy in most cases by exploiting the privileged information encoded in a pre-trained RGB detector.

Paper: https://arxiv.org/pdf/2310.04662.pdf

Abstract:

Data augmentation is one of the most effective techniques for regularizing deep learning models and improving recognition performance in various tasks and domains. However, this holds for standard in-domain settings where the training and test data follow the same distribution. The best recipe for data augmentation is unclear for the out-of-domain case, where the test data follow a different and unknown distribution. In this paper, we show that for out-of-domain and domain generalization settings, data augmentation can provide a conspicuous and robust improvement in performance. To do that, we propose a simple training procedure: (i) use uniform sampling on standard data augmentation transformations, (ii) increase the strength transformations to account for the higher data variance expected when working out-of-domain, and (iii) devise a new reward function to reject extreme transformations that can harm the training.

With this procedure, our data augmentation scheme achieves a level of accuracy comparable to or better than state-of-the-art methods on benchmark domain generalization datasets.

Paper: https://arxiv.org/pdf/2310.06670.pdf