Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation

ICLR 2025

Jihyo Kim*, Seulbi Lee*, Sangheum Hwang

(* Equal contribution)

Seoul National University of Science and Technology
Paper Github Hugging Face Logo Huggingface

Abstract

ReGuide is a self-guided prompting approach designed to improve out-of-distribution detection (OoDD) in Large Vision-Language Models (LVLMs).


Motivation
  • Foundation models (e.g., GPT-4o) are widely used due to their strong generalization abilities.
  • However, their trustworthiness, particularly in OoDD scenarios, is still underexplored.
  • This raises concerns about their safe and reliable deployment in real-world applications.
What We Do
  • Evaluate the OoDD capabilities of both proprietary and open-source LVLMs.
  • Analyze how these models express confidence through their generated natural language responses.
  • Propose ReGuide, which:
    • Generates image-adaptive concept suggestions
    • Improves confidence estimation and robustness in OoDD settings
Results
  • Improves image classification performance
  • Enhances OoDD detection accuracy
  • Works as a simple yet effective enhancement for current LVLMs
Radar All

(a) Comparison of LVLMs

Radar ReGuide

(b) Effectiveness of ReGuide

OoDD in LVLMs

Problem Definition

OoDD in LVLMs

To extend conventional OoD detection (OoDD) to Large Vision-Language Models (LVLMs), this study adapts the zero-shot CLIP-based framework by:

  • Using a fixed set of in-distribution (ID) class names and a special rejection class ("none of these classes").
  • Extracting confidence scores from natural language outputs of LVLMs.
  • Defining the OoD score as the maximum confidence among ID classes applying softmax function for normalization.

Prompt Design

OoDD in LVLMs

The prompt design used in LVLM-based OoDD evaluation consists of four main components:

  • Task Description clearly states the objectives.
  • Rejection Class adds a special label "none of these classes" to allow the model to reject OoD inputs explicitly.
  • Guidelines address the most common failure cases.
  • Response Examples uses in-context learning (ICL) to present examples of valid responses.

Experimental Results

OoDD in LVLMs
  • Proprietary LVLMs generally outperform open-source and single-modal models in both ID and OoD tasks.
  • LVLMs perform better on far-OoD than near-OoD.
  • InternVL2-76B achieves the best balance across ID accuracy, OoD performance, and valid response rate.
  • Some open-source models show 100% FPR due to biased confidence distribution, e.g., GLM-4v performs best in OoD but poorly in ID classification.

Further Analysis

Run 1
  • (a) FPR across different TPR thresholds: LLaVA-v1.6 consistently yields 100% FPR overall TPR, indicating extreme overconfidence across thresholds.
  • (b) OoD score distribution: InternVL2-76B shows a balanced distribution of OoD scores, while GPT-4o has more extreme values (0 or 100), suggesting overconfidence.
  • (c) OoD detectability based on model sizes: Larger models like InternVL2-76B outperform smaller ones in OoD detection, highlighting the benefit of scale.
  • (d) ID classification accuracy and OoDD performance of InternVL2-26B according to the given class order: Word order and class position in prompts significantly affect InternVL2-26B’s behavior.
Run 2

Reasoning

  • High interpretability of visual inputs improves OoD prediction.
  • InternVL2-26B handles fine-grained object differentiation well, but GPT-4o shows more detailed rationales.
Run 3

Confidence scores on ID

  • GPT-4o offers the best balance of accuracy and reliable confidence.
  • OpenCLIP and GPT-4o show the best calibration, with low ECE and AURC.
  • InternVL2-76B achieves high accuracy but poor calibration, assigning high confidence even to incorrect predictions.
Run 3

(a) The number of each failure case

Run 3

(b) Examples of inappropriate images and responses

Response failure

  • The failure cases are as follows:
    • (1) A mismatch between the predicted class and the class with the maximum confidence score
    • (2) Failure to provide confidence scores for all classes
    • (3) Responses that are not structured as the given format
    • (4) Responses indicating an inability to identify the given image
    • (5) Inclusion of classes not listed among the given
    • (6) Class duplication
  • Proprietary models exhibit more failure cases mainly due to their conservative approach, as they often refuse to respond to uncertain or sensitive inputs, resulting in lower valid response rates.
  • Failure cases are not confined to any specific dataset, suggesting they arise from both the nature of the data and the underlying mechanisms of the models.

Reflexive Guidance

Framework

ReGuide Diagram

Reflexive Guidance (ReGuide) is a simple and model-agnostic two-stage prompting strategy to enhance the OoD detectability of LVLMs. We leverage the LVLM itself to obtain guidance for OoDD from its powerful zero-shot visual recognition capabilities.


Stage 1: The LVLM suggests 2N class names based on the input image.

  • The first N classes are visually similar (near-OoD).
  • The second N classes are visually dissimilar or from different domains (far-OoD).
  • For OoD images, suggested classes often closely match the ground-truth label.

Stage 2: The suggested classes are used as auxiliary OoD classes.

  • OoD inputs may receive higher confidence for these classes than for ID classes.
  • The rejection class remains as a fallback option.

Experimental Results

OoDD in LVLMs
General Performance
  • ReGuide improves both ID classification and OoDD performance, especially when using visually similar classes (i.e., near-OoD).
  • Suggesting similar classes helps models better distinguish fine-grained differences between ID inputs.
  • InternVL2-76B and GPT-4o show gains in AUROC, FPR@95%TPR, and valid response rate when guided by ReGuide.
  • GPT-4o benefits more from ReGuide in near-OoD detection, even though it sometimes struggles with ambiguous inputs.
Insights
  • ReGuide's effectiveness stems from using the image itself to generate auxiliary classes rather than relying on external text.
  • This model-agnostic strategy provides relevant and adaptive guidance to LVLMs based on each specific input.

BibTeX

@inproceedings{
  kim2025reflexive,
  title={Reflexive Guidance: Improving Oo{DD} in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation},
  author={Jihyo Kim and Seulbi Lee and Sangheum Hwang},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=R4h5PXzUuU}
}