The Art of Saying ‘Maybe’: A Conformal Lens for Uncertainty Benchmarking in VLMs

Vision-language models have made substantial progress in complex visual understanding, but their ability to quantify uncertainty has received less attention. This study evaluates 18 open- and closed-source models across six multimodal datasets and three scoring functions, including instruction-guided likelihood proxies for API-only models without token-level probability access.

The results show that larger models generally quantify uncertainty more effectively and that predictions made with greater certainty tend to be more accurate. Mathematical and reasoning-intensive tasks remain particularly challenging across models. The study establishes a broad foundation for evaluating uncertainty and reliability in multimodal systems.