How to Defend Image-Text Matching against Adversarial Attacks

Recent studies have revealed that vision-language (VL) models are vulnerable to adversarial attacks for Image-Text Retrival (ITR). However, existing defense strategies for VL models focus on zero-shot image classification, which does not consider the simultaneous manipulation of image and text, as well as the inherent many-to-many (N:N) nature of ITR. This paper, for the first time, investigates defense strategies against adversarial attacks on VL models for ITR. We first apply previous defense strategies to ITR, and highlight their limitations. Subsequently, we propose a novel defense strategy considering N:N relationships and empirically demonstrate its effectiveness.