Duplicate from Amshaker/Mobile-O-0.5B

Co-authored-by: Abdelrahman Shaker <Amshaker@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,141 @@
+---
+license: cc-by-nc-4.0
+library_name: transformers
+tags:
+- mobile-o
+- multimodal
+- unified-model
+- vision-language
+- text-to-image
+- image-understanding
+- on-device
+- mobile
+pipeline_tag: text-to-image
+datasets:
+- Amshaker/Mobile-O-Post-Train
+- Amshaker/Mobile-O-SFT
+- Amshaker/Mobile-O-Pre-Train
+base_model:
+- Efficient-Large-Model/Sana_600M_512px_diffusers
+- apple/FastVLM-0.5B
+---
+
+<div align="center">
+
+<h1>
+  <img src="https://github.com/Amshaker/Mobile-O/blob/main/assets/mobile-o-logo.png?raw=true" width="30" /> Mobile-O-0.5B
+</h1>
+
+**Unified Multimodal Understanding and Generation on Mobile Device**
+
+<p>
+<a href="https://arxiv.org/abs/2602.20161"><img src="https://img.shields.io/badge/arXiv-2602.20161-b31b1b.svg" alt="arXiv"></a>
+<a href="https://github.com/Amshaker/Mobile-O"><img src="https://img.shields.io/badge/GitHub-Code-black.svg" alt="Code"></a>
+<a href="https://amshaker.github.io/Mobile-O/"><img src="https://img.shields.io/badge/🌐-Project_Page-2563eb.svg" alt="Project Page"></a>
+<a href="https://mobileo.cvmbzuai.com/"><img src="https://img.shields.io/badge/🚀-Live_Demo-10b981.svg" alt="Demo"></a>
+<a href="https://huggingface.co/collections/Amshaker/mobile-o-datasets"><img src="https://img.shields.io/badge/🤗-Datasets-yellow.svg" alt="Datasets"></a>
+<a href="https://apps.apple.com/app/mobile-o/id6759238106"><img src="https://img.shields.io/badge/-App_Store-black.svg" alt="App Store"></a>
+</p>
+
+</div>
+
+## 📌 Overview
+
+Mobile-O-0.5B is a compact unified vision–language–diffusion model that performs both **multimodal understanding** (VQA, OCR, reasoning) and **image generation** within a single architecture, designed for mobile and edge deployment.
+
+| Spec | Detail |
+|------|--------|
+| **Total Parameters** | 1.6B |
+| **Image Resolution** | 512×512 |
+| **Image Generation** | ~3 seconds on iPhone |
+| **Visual Understanding** | ~0.4 seconds on iPhone |
+| **Memory Footprint** | < 2GB |
+
+## 🎯 Supported Tasks
+
+| Task | Input → Output |
+|------|---------------|
+| 💬 Conversational AI | Text → Text |
+| 👁️ Image Understanding | Image + Text → Text |
+| 🖼️ Image Generation | Text → Image |
+| ✏️ Image Editing | Image + Text → Image |
+
+## 🚀 Quick Start
+
+### Download
+
+```python
+from huggingface_hub import snapshot_download
+
+snapshot_download(
+    repo_id="Amshaker/Mobile-O-0.5B",
+    repo_type="model",
+    local_dir="checkpoints",
+    allow_patterns=["final_merged_model_23620/*"]
+)
+```
+
+### Image Understanding
+
+```bash
+python infer_und.py \
+    --model_path checkpoints/final_merged_model_23620/ \
+    --image_path assets/cute_cat.png \
+    --prompt "What is in the image?"
+```
+
+### Image Generation
+
+```bash
+python infer_gen.py \
+    --model_path checkpoints/final_merged_model_23620/ \
+    --prompt "A vibrant tropical rainforest scene with a scarlet macaw perched on a moss-covered branch"
+```
+
+### Image Editing
+
+```bash
+python infer_edit.py \
+    --model_path checkpoints/final_merged_model_23620/ \
+    --image_path assets/cute_cat.png \
+    --prompt "Make the cat wear a hat"
+```
+
+## 🏗️ Architecture
+
+Mobile-O consists of three main components:
+
+- **Vision-Language Model (VLM):** [FastVLM-0.5B](https://github.com/apple/ml-fastvlm) — FastViT vision encoder + Qwen2-0.5B language backbone
+- **Diffusion Decoder:** [SANA-600M-512](https://github.com/NVlabs/Sana) — lightweight linear DiT with VAE for 512×512 generation
+- **Mobile Conditioning Projector (MCP):** ~2.4M param connector using layerwise feature fusion with temperature-scaled weights, depthwise-separable 1D convolutions, and efficient channel attention
+
+## 🏋️ Training
+
+Trained in three stages:
+
+1. **Pre-training** — Cross-modal alignment on [4M text-image pairs](https://huggingface.co/datasets/Amshaker/Mobile-O-Pre-Train)
+2. **SFT** — Supervised fine-tuning on [~105K curated pairs](https://huggingface.co/datasets/Amshaker/Mobile-O-SFT)
+3. **Post-training** — Unified multimodal training on [~105K quadruplets](https://huggingface.co/datasets/Amshaker/Mobile-O-Post-Train)
+
+## 🔗 Related Resources
+
+| Resource | Link |
+|----------|------|
+| 🤗 Mobile-O-1.5B | [Model](https://huggingface.co/Amshaker/Mobile-O-1.5B) |
+| 🤗 Mobile-O-0.5B-iOS | [iOS Components](https://huggingface.co/Amshaker/Mobile-O-0.5B-iOS) |
+| 📱 iOS App Source Code | [Mobile-O-App](https://github.com/Amshaker/Mobile-O/tree/main/Mobile-O-App) |
+
+## 📄 Citation
+
+```bibtex
+@article{shaker2026mobileo,
+  title={Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device},
+  author={Shaker, Abdelrahman and Heakl, Ahmed and Muhammad, Jaseel and Thawkar, Ritesh and Thawakar, Omkar and Li, Senmao and Cholakkal, Hisham and Reid, Ian and Xing, Eric P. and Khan, Salman and Khan, Fahad Shahbaz},
+  journal={arXiv preprint arXiv:2602.20161},
+  year={2026}
+}
+```
+
+## ⚖️ License
+
+Released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). For research purposes only.

added_tokens.json

@@ -0,0 +1,8 @@
+{
+  "<image>": 151648,
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "[/IMG]": 151647,
+  "[IMG]": 151646
+}

config.json

@@ -0,0 +1,52 @@
+{
+  "_attn_implementation_autoset": true,
+  "architectures": [
+    "MobileOForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "diffusion_name_or_path": "Efficient-Large-Model/Sana_600M_512px_diffusers",
+  "eos_token_id": 151645,
+  "freeze_mm_mlp_adapter": false,
+  "gen_pooling": "early_pool2d_4",
+  "hidden_act": "silu",
+  "hidden_size": 896,
+  "image_aspect_ratio": "pad",
+  "image_grid_pinpoints": null,
+  "initializer_range": 0.02,
+  "intermediate_size": 4864,
+  "is_train": false,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 24,
+  "mm_hidden_size": 3072,
+  "mm_patch_merge_type": "flat",
+  "mm_projector_lr": null,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_use_im_patch_token": false,
+  "mm_use_im_start_end": false,
+  "mm_vision_select_feature": "patch",
+  "mm_vision_select_layer": -2,
+  "mm_vision_tower": "mobileclip_l_1024",
+  "model_type": "mobile_o_inference",
+  "num_attention_heads": 14,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": false,
+  "tokenizer_model_max_length": 512,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3",
+  "tune_mm_mlp_adapter": false,
+  "unfreeze_mm_vision_tower": true,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "use_sliding_window": false,
+  "vision_tower_pretrained": null,
+  "vlm_num_layers": 4,
+  "vocab_size": 151649
+}

gen_projector.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d7bf359cbb7138122aa5806ccbe833e84aae0afb5819800ec61eff6335937cb
+size 888

mm_projector.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ce8c4b96b023519dd18de21ff74d41b7bcfc82befc23f718d411b0418e59ff6
+size 7116408

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:715db75c7d1aef8a8cd2c167f2a4f33438fe2f3d6ddb68b0c8b2f234d7bcd22b
+size 4775658842

special_tokens_map.json

@@ -0,0 +1,21 @@
+{
+  "additional_special_tokens": [
+    "[IMG]",
+    "[/IMG]",
+    "<image>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdd99b646b9315f486441715cc3dc39f4d3175b45b1bcc05c9aa6642b92b690a
+size 11413649

tokenizer_config.json

@@ -0,0 +1,70 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "[IMG]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "[/IMG]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "[IMG]",
+    "[/IMG]",
+    "<image>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 512,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}