Create README.md

README.md

@@ -0,0 +1,257 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- multimodal
+- visual-question-answering
+- retrieval-augmented-generation
+- reasoning
+- knowledge-based-vqa
+- qwen2_5_vl
+pipeline_tag: image-text-to-text
+---
+
+# ReAG-3B — Reasoning-Augmented Generation for KB-VQA
+
+[![CVPR 2026 Highlight](https://img.shields.io/badge/CVPR-2026%20Highlight-f9f107.svg)](https://cvpr.thecvf.com/virtual/2026/poster/37311)
+[![Paper](https://img.shields.io/badge/Paper-arXiv%202511.22715-B31B1B.svg)](https://arxiv.org/abs/2511.22715)
+[![Project Page](https://img.shields.io/badge/🌐-Project%20Page-blue.svg)](https://aimagelab.github.io/ReAG/)
+[![HF Collection](https://img.shields.io/badge/🤗-HF%20Collection-yellow.svg)](https://huggingface.co/collections/aimagelab/reag)
+
+ReAG-3B is the generator component of **ReAG**, a Reasoning-Augmented Multimodal RAG pipeline for Knowledge-Based Visual Question Answering (KB-VQA). It is based on [Qwen2.5-VL-3B](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) and fine-tuned with a multi-stage strategy: supervised fine-tuning as a cold start, followed by reinforcement learning to promote explicit reasoning grounded in retrieved evidence.
+
+For the full pipeline, a companion **critic model** ([aimagelab/ReAG-Critic](https://huggingface.co/aimagelab/ReAG-Critic)) first filters noisy retrieved passages before they are fed to this generator.
+
+A larger 7B variant is also available: [aimagelab/ReAG-7B](https://huggingface.co/aimagelab/ReAG-7B).
+
+---
+
+## Model Description
+
+Standard retrieval-augmented VQA methods often pass noisy or irrelevant passages directly to the generator, limiting answer quality. ReAG addresses this with a two-step approach:
+
+1. **ReAG-Critic** evaluates each retrieved passage and filters out irrelevant ones using a multimodal relevance signal (image + question + passage).
+2. **ReAG-3B (this model)** generates an answer with explicit chain-of-thought reasoning enclosed in `<think>…</think>` tags, followed by a concise `<answer>…</answer>`.
+
+ReAG significantly outperforms prior methods on both **Encyclopedic-VQA** and **InfoSeek**.
+
+---
+
+## Full Pipeline Usage
+
+The snippet below shows the complete ReAG inference pipeline: critic filtering followed by generator inference.
+
+```python
+import re
+from io import BytesIO
+
+import requests
+import torch
+from PIL import Image
+from transformers import (
+    AutoModelForImageTextToText,
+    AutoProcessor,
+    Qwen2_5_VLForConditionalGeneration,
+)
+
+REAG_MODEL_NAME = "aimagelab/ReAG-3B"
+CRITIC_MODEL_NAME = "aimagelab/ReAG-Critic"
+
+SYSTEM_PROMPT_REASONING = (
+    "A conversation between User and Assistant. The user asks a question, "
+    "and the Assistant solves it. The assistant first thinks about the "
+    "reasoning process in the mind and then provides the user with the answer. "
+    "The reasoning process and answer are enclosed within <think> </think> and "
+    "<answer> </answer> tags, respectively, i.e., "
+    "<think>reasoning process here</think><answer>short answer here</answer>"
+)
+
+RELEVANCY_EVAL_SYSTEM_PROMPT = """You are a multimodal reasoning assistant specialized in Knowledge-Based Visual Question Answering (KB-VQA).
+Your task is to evaluate whether a given text passage provides useful and relevant information for answering a question about an image.
+
+You will be given:
+- Image: a visual scene containing entities, actions, and context.
+- Question: a natural-language question that refers to the image.
+- Text Passage: an external knowledge snippet retrieved from a database or the web.
+
+You must analyze the semantic alignment between the text, the image, and the question.
+Follow these steps carefully before giving your final decision:
+1. Understand the visual scene: Identify the key objects, people, actions, and context visible in the image.
+2. Interpret the question: Determine what information the question seeks.
+3. Analyze the text passage: Extract the main claims, facts, and entities mentioned in the text.
+
+Compare for relevance: Assess whether the information in the text:
+- Contains at least one sentence that supports answering the question about the image, OR
+- Provides background knowledge needed to interpret or reason about the image-question pair.
+
+Important:
+- If even a single sentence in the passage is relevant or useful, consider the entire passage as relevant and answer "Yes".
+- If no part of the passage contributes meaningfully to answering the question, answer "No".
+
+Output only one word:
+"Yes" -> if the text provides relevant or useful information for answering the question.
+"No" -> if the text is irrelevant or unhelpful."""
+
+SECTION_EVAL_USER_TEMPLATE = """Here is the question on the image above:
+{question}
+
+Here is the text passage to analyze:
+{passage}
+
+Does the text passage contain at least one sentence that may have some information useful to answer the user question?
+"Yes"/"No" answer:"""
+
+CONTEXT_VQA_PROMPT = """\
+{question}
+
+The following paragraphs may contain useful information to help answer the question correctly:
+{context}
+"""
+
+
+def load_image(image_url: str) -> Image.Image:
+    response = requests.get(image_url, timeout=30, headers={"User-Agent": "Mozilla/5.0"})
+    response.raise_for_status()
+    return Image.open(BytesIO(response.content)).convert("RGB")
+
+
+def get_model_kwargs():
+    if torch.cuda.is_available():
+        return {
+            "device": "cuda",
+            "device_map": "balanced",
+            "torch_dtype": torch.bfloat16,
+            "attn_implementation": "flash_attention_2",
+        }
+    return {
+        "device": "cpu",
+        "device_map": "auto",
+        "torch_dtype": torch.float32,
+    }
+
+
+def parse_reag_output(text: str):
+    answer_match = re.search(r"<answer>(.*?)</answer>", text, re.DOTALL)
+    think_match = re.search(r"<think>(.*?)</think>", text, re.DOTALL)
+    return {
+        "raw_output": text.strip(),
+        "reasoning": think_match.group(1).strip() if think_match else "",
+        "answer": answer_match.group(1).strip() if answer_match else text.strip(),
+    }
+
+
+def run_reag_generator(model, processor, image: Image.Image, question: str):
+    messages = [
+        {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT_REASONING}]},
+        {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": question}]},
+    ]
+    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = processor(text=[prompt + "<think>"], images=[image], return_tensors="pt", padding=True)
+    inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+    generated_ids = model.generate(
+        **inputs, max_new_tokens=512, stop_strings=["</answer>"], tokenizer=processor.tokenizer
+    )
+    input_length = inputs["input_ids"].shape[1]
+    generated_text = "<think>" + processor.batch_decode(
+        generated_ids[:, input_length:], skip_special_tokens=True, clean_up_tokenization_spaces=False
+    )[0]
+    return parse_reag_output(generated_text)
+
+
+def run_reag_critic(critic, processor, image: Image.Image, question: str, passage: str, yes_prob_threshold: float = 0.1):
+    messages = [
+        {"role": "system", "content": [{"type": "text", "text": RELEVANCY_EVAL_SYSTEM_PROMPT}]},
+        {
+            "role": "user",
+            "content": [
+                {"type": "image"},
+                {"type": "text", "text": SECTION_EVAL_USER_TEMPLATE.format(question=question, passage=passage)},
+            ],
+        },
+    ]
+    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = processor(images=[image], text=[prompt], return_tensors="pt", padding=True)
+    inputs = {k: v.to(critic.device) for k, v in inputs.items()}
+
+    with torch.inference_mode():
+        outputs = critic(**inputs)
+        logits = outputs.logits[:, -1, :].float()
+        probs = torch.softmax(logits, dim=-1)
+
+    yes_token_id = processor.tokenizer.convert_tokens_to_ids("Yes")
+    no_token_id = processor.tokenizer.convert_tokens_to_ids("No")
+    return {
+        "relevant": probs[0, yes_token_id].item() > yes_prob_threshold,
+        "yes_probability": probs[0, yes_token_id].item(),
+        "no_probability": probs[0, no_token_id].item(),
+    }
+
+
+# ── Example ──────────────────────────────────────────────────────────────────
+
+image = load_image(
+    "https://upload.wikimedia.org/wikipedia/commons/thumb/5/54/Clinopodium_vulgare_inflorescence.jpg/250px-Clinopodium_vulgare_inflorescence.jpg"
+)
+question = "What kind of properties does this plant have?"
+passages = [
+    "# Description:\nWild basil is a perennial rhizomatous herb ...",
+    "# Distribution:\nWild basil occurs in suitable locations in most of Europe ...",
+    "# Uses:\nThe leaves of wild basil are used as an aromatic herb ... It has been shown to have anti-bacterial properties.",
+]
+
+model_kwargs = get_model_kwargs()
+device = model_kwargs.pop("device")
+
+# 1. Load and run the critic
+critic_processor = AutoProcessor.from_pretrained(CRITIC_MODEL_NAME, padding_side="left", use_fast=True)
+critic = Qwen2_5_VLForConditionalGeneration.from_pretrained(CRITIC_MODEL_NAME, **model_kwargs)
+critic.eval()
+
+relevant_passages = []
+for passage in passages:
+    result = run_reag_critic(critic, critic_processor, image, question, passage)
+    if result["relevant"]:
+        relevant_passages.append(passage)
+
+# 2. Load the generator and answer with filtered context
+context = "\n\n\n".join(relevant_passages) if relevant_passages else ""
+processor = AutoProcessor.from_pretrained(REAG_MODEL_NAME, padding_side="left", use_fast=True)
+generator = AutoModelForImageTextToText.from_pretrained(REAG_MODEL_NAME, **model_kwargs)
+generator.eval()
+
+question_with_context = CONTEXT_VQA_PROMPT.format(question=question, context=context)
+output = run_reag_generator(generator, processor, image, question_with_context)
+print("Answer:", output["answer"])
+print("Reasoning:", output["reasoning"])
+```
+
+---
+
+## Model Collection
+
+| Model | Description |
+|---|---|
+| [aimagelab/ReAG-3B](https://huggingface.co/aimagelab/ReAG-3B) | Generator (this model) |
+| [aimagelab/ReAG-7B](https://huggingface.co/aimagelab/ReAG-7B) | Larger generator variant |
+| [aimagelab/ReAG-Critic](https://huggingface.co/aimagelab/ReAG-Critic) | Passage relevance critic |
+
+---
+
+## Repository & Evaluation
+
+Full inference scripts, dataset setup, FAISS index downloads, and evaluation instructions are available in the **[official GitHub repository](https://github.com/aimagelab/ReAG)**.
+
+---
+
+## Citation
+
+```bibtex
+@article{compagnoni2025reag,
+  title={ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering},
+  author={Compagnoni, Alberto and Morini, Marco and Sarto, Sara and Cocchi, Federico and Caffagni, Davide and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
+  journal={arXiv preprint arXiv:2511.22715},
+  year={2025}
+}
+```