Certified Robustness via Randomized Smoothing for Medical Vision Language Models with a Scalable Empirical Optimization Framework

Abstract

Medical Vision Language Models (Med-VLMs) are increasingly deployed in clinical decision support, yet their vulnerability to semantically equivalent input perturbations, such as paraphrasing, poses significant safety risks. Prior work (PSF-Med) revealed flip rates up to 37\% in Med-VLM predictions under such perturbations, underscoring the urgent need for robustness guarantees. However, existing certified defenses, primarily based on randomized smoothing, have been designed for single-modality classifiers and do not directly extend to the multimodal nature of Med-VLMs. This paper introduces a novel framework for certified robustness in Med-VLMs via randomized smoothing, tailored to the joint vision-language input space. We propose a scalable empirical optimization strategy that jointly optimizes the smoothing distribution and the base classifier to maximize certified accuracy under norm-bounded perturbations. Our theoretical contributions include a certified robustness bound for multimodal inputs and a generalization of the Neyman-Pearson lemma to the joint space. Empirically, we demonstrate that our framework significantly improves certified accuracy over baseline randomized smoothing methods across multiple Med-VLM architectures and medical imaging datasets (e.g., CheXpert, RSNA Pneumonia). We also introduce a new evaluation protocol for certified robustness in multimodal settings and conduct extensive ablations to analyze the impact of key components. Our work provides the first comprehensive study of certified robustness for Med-VLMs, offering a practical pathway towards trustworthy AI in healthcare.