Improve model card for AtomThink-LLaVA1.5-7B

README.md

@@ -1,16 +1,102 @@
 ---
-datasets:
-- Quinn777/AMATH-SFT
 base_model:
 - liuhaotian/llava-v1.5-7b
+datasets:
+- Quinn777/AMATH-SFT
+license: mit
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+- multimodal
+- mathematical-reasoning
+- llava
+- slow-thinking
 ---
-# Model Card for AtomThink-LlamaV
 
-The model is post-trained based on llava-v1.5-7b and the AtomThink framework, and can be used to solve complex multimodal mathematical problems.
+# Model Card for AtomThink-LLaVA1.5-7B
+
+The **AtomThink-LLaVA1.5-7B** model is post-trained based on `llava-v1.5-7b` and the AtomThink framework, designed to solve complex multimodal mathematical problems by incorporating the notion of "slow thinking".
+
+It was presented in the paper:
+[**AtomThink: Multimodal Slow Thinking with Atomic Step Reasoning**](https://huggingface.co/papers/2411.11930)
+
+The official code repository can be found on [GitHub](https://github.com/Quinn777/AtomThink).
+
+## Overview
+
+In this paper, we address the challenging task of multimodal mathematical reasoning by incorporating the notion of ``slow thinking'' into multimodal large language models (MLLMs). Our core idea is that models can learn to adaptively use different levels of reasoning to tackle questions of different complexity. We propose a novel paradigm of Self-structured Chain of Thought (SCoT), which comprises of minimal semantic atomic steps. Different from existing methods that rely on structured templates or free-form paradigms, our method can not only generate cognitive CoT structures for various complex tasks but also mitigates the phenomena of overthinking for easier tasks. To introduce structured reasoning into visual cognition, we further design a novel AtomThink framework with four key modules, including (i) a data engine to generate high-quality multimodal reasoning paths; (ii) a supervised fine-tuning (SFT) process with serialized inference data; (iii) a policy-guided multi-turn inference method; and (iv) an atomic capability metric to evaluate the single step utilization rate. We conduct extensive experiments to show that the proposed AtomThink significantly improves the performance of baseline MLLMs, achieving more than 10% average accuracy gains on MathVista and MathVerse. Compared to state-of-the-art structured CoT approaches, our method not only achieves higher accuracy but also improves data utilization by 5 times and boosts inference efficiency by 85.3%.
+
+### Key Features
+
+-   🧠 Introduces **GPT-o1** style reasoning via long CoT for complex multimodal mathematical tasks.
+-   🛠️ Combines a CoT annotation engine, atomic step fine-tuning, and policy search strategies to enhance reasoning.
+-   📊 A capability evaluation metric to perform a quality assessment of each reasoning step.
+-   ⚡ Test-time scaling law in MLLM.
+-   📈 State-of-the-art performance in multimodal mathematical reasoning tasks.
+
+<p align="center">   <img src="https://github.com/Quinn777/AtomThink/raw/main/figures/framework.png" alt="AtomThink Framework" width="800"> </p>
+*AtomThink Framework: Overview of the four key modules for structured reasoning.*
+
+<p align="center">   <img src="https://github.com/Quinn777/AtomThink/raw/main/figures/fig1.png" alt="Comparison with structured and unstructured reasoning models" width="800"> </p>
+*Comparison with structured and unstructured reasoning models. AtomThink is capable of autonomously generating dynamic structures and lengths based on the type of problem. For text-dominant questions, it bypasses image caption and directly extracts information from the question stem. For low-difficulty problems, it uses fewer tokens compared to o1-like models.*
+
+<p align="center">   <img src="https://github.com/Quinn777/AtomThink/raw/main/figures/fig2.png" alt="Complexity comparison" width="800"> </p>
+*Comparison of the average response length in AtomThink-LlamaV over benchmarks with different complexity. (a) As tasks become more challenging, the model proactively utilizes more tokens. (b) The proportion of longer CoT containing a greater number of atomic steps increases in outputs. A higher level signifies increased difficulty. The performance decline margin of AtomThink modes are narrower compared to LLaVA1.5 and LlamaV.*
+
+## Usage
+
+You can use the model with the `transformers` library. Below is a simplified example of how to load and use the model for inference. For more advanced usage, including training and evaluation scripts, please refer to the [official GitHub repository](https://github.com/Quinn777/AtomThink).
 
-# Citation
-If you use this dataset in your research, please cite:
-```text
+First, ensure you have the necessary libraries installed:
+```bash
+pip install transformers accelerate pillow requests
+```
+
+```python
+from transformers import AutoProcessor, LlavaForConditionalGeneration
+from PIL import Image
+import requests
+
+# Load the processor and model
+model_id = "Quinn777/AtomThink-LLaVA1.5-7B" 
+processor = AutoProcessor.from_pretrained(model_id)
+model = LlavaForConditionalGeneration.from_pretrained(model_id)
+
+# Example image (replace with your image path or URL)
+# For a real image:
+# image = Image.open("path/to/your/image.png").convert("RGB")
+# Or from URL:
+image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/bird.jpg"
+image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
+
+# Example prompt for a mathematical question
+prompt = "USER: <image>
+Solve this mathematical problem step-by-step. ASSISTANT:"
+
+# Prepare inputs
+inputs = processor(text=prompt, images=image, return_tensors="pt")
+
+# Generate response
+# Use appropriate generation arguments based on your task, e.g., max_new_tokens
+output = model.generate(**inputs, max_new_tokens=256, temperature=0.0)
+
+# Decode and print the output
+generated_text = processor.decode(output[0], skip_special_tokens=True)
+print(generated_text)
+
+# Example for a simple image description (to show versatility)
+prompt_simple = "USER: <image>
+Describe this image in detail. ASSISTANT:"
+inputs_simple = processor(text=prompt_simple, images=image, return_tensors="pt")
+output_simple = model.generate(**inputs_simple, max_new_tokens=50, temperature=0.0)
+decoded_output_simple = processor.decode(output_simple[0], skip_special_tokens=True)
+print(decoded_output_simple)
+```
+
+## Citation
+
+If you use this model or the associated dataset in your research, please cite:
+```bibtex
 @article{xiang2025can,
   title={Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models?},
   author={Xiang, Kun and Liu, Zhili and Jiang, Zihao and Nie, Yunshuang and Cai, Kaixin and Yin, Yiyang and Huang, Runhui and Fan, Haoxiang and Li, Hanhui and Huang, Weiran and others},
@@ -26,5 +112,13 @@ If you use this dataset in your research, please cite:
 }
 ```
 
-# License
-The checkpoint is released under the Apache 2.0 license. Please ensure proper attribution when using this checkpoint.
+## License
+
+This project is licensed under the [MIT License](https://github.com/Quinn777/AtomThink/blob/main/LICENSE). Please ensure proper attribution when using this checkpoint.
+
+## Acknowledgement
+
+We would like to thank the following repositories for their contributions:
+-   [hiyouga/LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory): This library was used for training.
+-   [bklieger-groq/g1](https://github.com/bklieger-groq/g1): This library was used for data processing.
+-   [openreasoner/openr](https://github.com/openreasoner/openr): This tool was helpful for deploying the process supervision model.