CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shifts

Olaf Dünkel1, Artur Jesslen*2, Jiaohao Xie*1 , Christian Theobalt1, Christian Rupprecht3, Adam Kortylewski1,2
1Max Planck Institute for Informatics, 2University of Freiburg, 3University of Oxford
Paper arXiv Code Data

We evaluate the robustness of different models under gradually increasing nuisance shifts. This enables the identification of failure points (highlighted in red).

Abstract

An important challenge when using computer vision models in the real world is to evaluate their performance in potential out-of-distribution (OOD) scenarios. While simple synthetic corruptions are commonly applied to test OOD robustness, they mostly do not capture nuisance shifts that occur in the real world. Recently, diffusion models have been applied to generate realistic images for benchmarking, but they are restricted to binary nuisance shifts. In this work, we introduce CNS-Bench, a Continuous Nuisance Shift Benchmark to quantify OOD robustness of image classifiers for continuous and realistic generative nuisance shifts. CNS-Bench allows generating a wide range of individual nuisance shifts in continuous severities by applying LoRA adapters to diffusion models. To remove failure cases, we propose a filtering mechanism that outperforms previous methods and hence enables a reliable benchmarking with generative models. With the proposed benchmark, we perform a large-scale study to evaluate the robustness of more than 40 classifiers under various nuisance shifts. Through carefully designed comparisons and analyses, we find that model rankings can change for varying shifts and shift scales, which cannot be captured when applying common binary shifts. Additionally, we show that evaluating the model performance on a continuous scale allows the identification of model failure points, providing a more nuanced understanding of model robustness.

Our Approach

Our benchmark is the first that enables benchmarking w.r.t. realistic and continuous nuisance shifts, scalable with respect to the number of classes and shifts.

Overview.

ImageNet class-specific LoRA adapters are applied to an image, enabling a variety of continuous nuisance shifts.

Method generation.

Out-of-class samples are filtered via an ensemble of filters.

Method filtering.

Benchmarking Results

Accuracy drops and rankings can vary for different shifts and scales.

Result plots.

Evaluation of model robustness: Average corruption error along three benchmarked axes.

Model evaluation.


Acknowledgments

Adam Kortylewski acknowledges support via his Emmy Noether Research Group funded by the German Research Foundation (DFG) under Grant No. 468670075.

BibTeX

@inproceedings{duenkel2025cns,
    title = {CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shifts},
    author = {D{\"u}nkel, Olaf and Jesslen, Artur and Xie, Jiaohao and Theobalt, Christian and Rupprecht, Christian and Kortylewski, Adam},
    booktitle = {ICCV},
    year = {2025}
  }