Upload README.md

README.md

@@ -34,34 +34,42 @@ base_model:
 
 ## Highlights
 
-- **Hybrid Action Space** — First FPS world model to handle both continuous (4D joystick) and discrete (6 binary buttons) actions simultaneously in a unified framework
-- **Dense High-Frequency Control** — Resolves overlapping control signals at every frame, unlike prior methods limited to sparse inputs
-- **Action Composition** — Supports simultaneous multi-action combinations (e.g., moving + aiming + firing), reflecting real gameplay complexity
-- **Cross-Game Generalization** — A single world model trained on 7 diverse FPS games that generalizes zero-shot to unseen game environments
-- **In-Scope / Out-of-Scope Decoupling** — Spatially selective conditioning that separates localized in-scope effects (weapon, HUD) from stable out-of-scope world generation, without segmentation labels
+- **Hybrid Action Space** — Jointly processes continuous (4D dual-joystick) and discrete (6 binary buttons) control signals within a unified framework — the first FPS world model to do so.
+- **Dense Per-Frame Conditioning** — Resolves overlapping actions at every single frame, enabling simultaneous multi-action composition (e.g., moving + aiming + firing) that reflects real gameplay complexity.
+- **Cross-Game Generalization** — Trained on 7 diverse FPS titles, a single model generalizes zero-shot to entirely unseen game environments without fine-tuning.
+- **In-Scope / Out-of-Scope Decoupling** — Spatially selective conditioning that separates localized in-scope effects (weapon recoil, HUD) from stable out-of-scope world generation — without any segmentation labels.
 
 ## Model Overview
 
-SCOPE is an interactive world model for first-person shooter (FPS) games. Unlike prior game world models that only handle sparse, single-modality actions, SCOPE processes **hybrid action inputs** (continuous joystick + discrete buttons) at **dense, high-frequency** rates, supporting real-time **action composition** — multiple simultaneous inputs such as moving, aiming, and firing in the same frame.
+SCOPE is an interactive world model for first-person shooter (FPS) games. Built on [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B), SCOPE inserts a conditioning module into each transformer block that reshapes features into per-pixel temporal sequences. Each spatial position computes its action response from local visual content, naturally separating in-scope effects (e.g., weapon firing, reloading) from out-of-scope world generation (e.g., stable surroundings) — without any segmentation labels.
 
-Built on [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B), SCOPE inserts a conditioning module into each transformer block that reshapes features into per-pixel temporal sequences. Each spatial position computes its action response from local visual content, naturally separating in-scope effects (e.g., weapon firing, reloading) from out-of-scope world generation (e.g., stable surroundings) — without any segmentation labels. Trained on [**CrossFPS**](https://huggingface.co/collections/zizhaotong/crossfps) (69K clips, 7 games, 10-DoF), SCOPE learns general visual-to-action mappings rather than game-specific patterns, enabling zero-shot transfer to unseen scenes.
+Trained on [**CrossFPS**](https://huggingface.co/collections/zizhaotong/crossfps) (69K clips, 7 games, 10-DoF), SCOPE learns general visual-to-action mappings rather than game-specific patterns, enabling zero-shot transfer to unseen scenes.
 
-### Architecture
+## Architecture
 
 | Component | Details |
 |-----------|---------|
-| **Base Model** | Wan2.2-TI2V-5B (DiT, 30 layers) |
-| **Action Module** | Per-block conditioning, per-pixel temporal sequences |
-| **Text Encoder** | UMT5-XXL |
-| **VAE** | Wan2.2 Video VAE (temporal compression 4x, spatial compression 8x) |
-| **Parameters** | ~5B total (1575 tensors) |
+| **Base Model** | Wan2.2-TI2V-5B (DiT, 30 transformer layers) |
+| **Action Module** | Per-block conditioning with per-pixel temporal sequences |
+| **Text Encoder** | UMT5-XXL (4096-dim hidden) |
+| **VAE** | Wan2.2 Video VAE (4× temporal compression, 8× spatial compression) |
+| **Total Parameters** | ~5B (1575 tensors, of which 750 are action-related) |
 | **Precision** | BFloat16 |
 
+### ActionModule Design
+
+Each of the 30 DiT blocks contains an `ActionModule` with two conditioning paths:
+
+- **Mouse/Joystick Path**: Sliding-window temporal features → MLP fusion → pixel-wise temporal self-attention with RoPE
+- **Keyboard/Button Path**: Button embedding → temporal windowing → cross-attention (video queries, keyboard keys/values)
+
+Both output projections are zero-initialized for stable residual training on top of frozen pretrained weights.
+
 ### Generation Specs
 
 | Property | Value |
 |----------|-------|
-| Resolution | 480 x 832 |
+| Resolution | 480 × 832 |
 | Frame Count | 81 frames |
 | Frame Rate | 20 FPS |
 | Duration | ~4 seconds |
@@ -94,9 +102,8 @@ SCOPE accepts 10-DoF action inputs per frame via a Parquet file:
 ### Requirements
 
 - Python >= 3.10
-- PyTorch >= 2.5 with CUDA support
-- GPU: NVIDIA with >= 24GB VRAM (single GPU inference with CPU offload)
-- [DiffSynth](https://github.com/modelscope/DiffSynth-Studio) framework
+- PyTorch >= 2.0 with CUDA support
+- GPU: NVIDIA with >= 24 GB VRAM (single GPU inference with CPU offload)
 
 ### Installation
 
@@ -110,21 +117,7 @@ pip install -e .
 
 ```bash
 # Download all weights (SCOPE DiT + Text Encoder + VAE + Tokenizer) in one command
-huggingface-cli download z2tong/SCOPE --local-dir ./SCOPE
-```
-
-The model directory contains everything needed for inference:
-
-```
-SCOPE/
-├── model-00001-of-00003.safetensors   # SCOPE DiT shard 1
-├── model-00002-of-00003.safetensors   # SCOPE DiT shard 2
-├── model-00003-of-00003.safetensors   # SCOPE DiT shard 3
-├── model.safetensors.index.json       # Shard index
-├── models_t5_umt5-xxl-enc-bf16.pth   # Text Encoder (UMT5-XXL)
-├── Wan2.2_VAE.pth                     # Video VAE
-├── google/umt5-xxl/                   # Tokenizer
-└── config.json                        # Model config
+huggingface-cli download zizhaotong/SCOPE --local-dir ./SCOPE
 ```
 
 ### Inference
@@ -185,10 +178,10 @@ SCOPE is trained on [**CrossFPS**](https://huggingface.co/collections/zizhaotong
 ## Citation
 
 ```bibtex
-@article{scope2025,
+@article{scope2026,
   title={SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models},
-  author={Tong, Zizhao and Lai, Hongfeng and Wang, Zeqing and Xing, Zhaohu and Cheng, Kexu and Xu, Haoran and Pu, Zhao and Zhu, Shangwen and Feng, Ruili and Zhao, Jian and Zhang, Yan and Tang, Hao and Jin, Yeying and Shao, Ling},
-  year={2025}
+  author={Zizhao Tong and Hongfeng Lai and Zeqing Wang and Zhaohu Xing and Kexu Cheng and Haoran Xu and Zhao Pu and Shangwen Zhu and Ruili Feng and Jian Zhao and Yan Zhang and Hao Tang and Yeying Jin and Ling Shao},
+  year={2026}
 }
 ```