Update README.md

README.md

@@ -1,3 +1,181 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+tags:
+- video-generation
+- video-editing
+- novel-view-synthesis
+- camera-control
+- diffusion-transformer
+- wan2.2
+- lora
+- video-reshooting
+- 4d-reconstruction
+pipeline_tag: video-to-video
+base_model: Wan-AI/Wan2.2-I2V-A14B
+---
+
+# Reshoot-Anything
+
+[![GitHub](https://img.shields.io/badge/GitHub-video--to--video-black?logo=github)](https://github.com/morphicfilms/video-to-video)
+[![arXiv](https://img.shields.io/badge/arXiv-2604.21776-red?logo=arxiv)](https://arxiv.org/abs/2604.21776)
+[![Website](https://img.shields.io/badge/Website-morphic.com-purple)](https://adithyaiyer1999.github.io/reshoot-anything/)
+
+**Reshoot-Anything** is a self-supervised video reshooting model built on top of [Wan2.2-I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B). Given a source video and a target camera trajectory (encoded as an anchor video), it generates a high-fidelity reshoot that faithfully follows the new camera path while preserving original content, complex dynamics, and temporal consistency — trained entirely on in-the-wild monocular videos.
+
+> **Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting**  
+> Avinash Paliwal, Adithya Iyer, Shivin Yadav, Muhammad Ali Afridi, Midhun Harikumar  
+> Morphic Inc. · [arXiv:2604.21776](https://arxiv.org/abs/2604.21776)
+
+---
+
+<table>
+<tr>
+<td align="center"><b>Source Video</b></td>
+<td align="center"><b>Reshot Video</b></td>
+</tr>
+<tr>
+<td><img src="assets/woman_og.gif" width="400" alt="Source video"></td>
+<td><img src="assets/woman_01.gif" width="400" alt="Reshot video"></td>
+</tr>
+</table>
+
+
+## Model Files
+
+This repository contains two LoRA checkpoints (rank-512, applied to attention and feed-forward layers of Wan2.2-I2V-A14B):
+
+| File | Role | Notes |
+|------|------|-------|
+| `jan06_scaling_80k_ckpt1400.safetensors` | **High-noise expert** | Controls early denoising steps. Primarily responsible for camera motion alignment and global scene structure. Trained on ~80k clips with scaling augmentations + 15% synthetic data mixture. |
+| `dec23_v2v_lownoise_black_lora_512_ckpt1000.safetensors` | **Low-noise expert** | Controls late denoising steps. Responsible for texture fidelity and fine detail. Uses standard black-background anchors, no source reconstruction loss. |
+
+Both files are ~9.82 GB each.
+
+---
+
+## Quickstart
+
+### 1. Clone the repository
+
+```bash
+git clone https://github.com/morphicfilms/video-to-video.git
+cd video-to-video
+```
+
+Follow the [Wan2.2 installation guide](https://github.com/Wan-Video/Wan2.2) to set up the environment, or run:
+
+```bash
+bash setup_env.sh
+```
+
+### 2. Download the weights
+
+Download the Wan2.2 I2V base weights:
+
+```bash
+huggingface-cli download Wan-AI/Wan2.2-I2V-A14B --local-dir ./Wan2.2-I2V-A14B
+```
+
+Download the Reshoot-Anything LoRA weights:
+
+```bash
+huggingface-cli download morphic/reshoot-anything --local-dir ./reshoot-anything-weights
+```
+
+### 3. Prepare your anchor video
+
+At inference, generate an anchor video by converting your source video to a 4D point cloud, applying the target camera trajectory, and forward-warping to produce the geometric anchor. See the repo's `anchor_generation/` scripts for details.
+
+### 4. Run reshooting
+
+```bash
+torchrun --nproc_per_node=8 generate.py \
+    --task v2v-A14B \
+    --size 1280*720 \
+    --frame_num 81 \
+    --ckpt_dir ./Wan2.2-I2V-A14B \
+    --high_noise_lora_path ./reshoot-anything-weights/jan06_scaling_80k_ckpt1400.safetensors \
+    --low_noise_lora_path ./reshoot-anything-weights/dec23_v2v_lownoise_black_lora_512_ckpt1000.safetensors \
+    --source_video examples/source.mp4 \
+    --anchor_video examples/anchor.mp4 \
+    --dit_fsdp \
+    --t5_fsdp \
+    --ulysses_size 8
+```
+
+> **Note:** Refer to the [GitHub README](https://github.com/morphicfilms/video-to-video) for the authoritative argument names and single-GPU usage.
+
+---
+
+## How It Works
+
+Reshoot-Anything adapts the **Wan2.2-14B Mixture-of-Experts (MoE)** DiT with two key architectural changes:
+
+**Dual-stream token conditioning** — Both the anchor video `V_a` (geometric guide) and source video `V_s` (texture reference) are VAE-encoded and temporally concatenated as tokens into the model's main self-attention mechanism. This outperforms cross-attention for view synchronization by letting the model directly route textures across spatial and temporal positions.
+
+**Offset RoPE** — A fixed temporal offset of 50 is added to source video token positional embeddings, strictly decoupling source context from the active denoising trajectory.
+
+The model learns **implicit 4D spatiotemporal routing** — when a target frame requires content occluded in the corresponding source frame, the model locates and re-projects the missing texture from a different timestep in the source video.
+
+### Self-Supervised Training Pipeline
+
+Training requires no paired multi-view data. From a single monocular video:
+1. Two independent smooth random-walk crop trajectories are sampled → source `V_s` and target `V_t`
+2. `V_s[0]` is forward-warped via [AllTracker](https://arxiv.org/abs/2504.11111) dense flow + crop offset → anchor `V_a`
+3. The triplet `(V_s, V_a, V_t)` forms the training signal
+
+A **hybrid dataset strategy** augments the monocular pipeline with a 15% mixture of paired synthetic data from [ReCamMaster](https://github.com/KwaiVGI/ReCamMaster), enabling generalization to extreme (120°+) orbital camera trajectories.
+
+---
+
+## Training Details
+
+| Parameter | Value |
+|-----------|-------|
+| Base model | Wan2.2-I2V-A14B (14B MoE) |
+| LoRA rank | 512 (attention + FFN) |
+| Training steps | 2,000 per expert |
+| Batch size | 24 |
+| Learning rate | 1e-5 |
+| Optimizer | AdamW (β₁=0.9, β₂=0.999) |
+| Loss | MSE + 0.1 × L1 source reconstruction |
+| Latent frames | 20 |
+| Primary data | ~100k clips from 30k monocular videos |
+| Synthetic mixture | 15% ReCamMaster paired clips |
+
+**Key augmentations:** 3D-aware noise injection into anchor reference frame (magnitude uniform [0, 0.5]), fluorescent pink masked-region backgrounds, random anchor reference frame selection, source token auxiliary reconstruction loss.
+
+---
+
+## Evaluation
+
+Results on 100 five-second videos from [OpenSora-Mixkit](https://huggingface.co/datasets/opensora/OpenSora-MixKit) (16fps, 480p):
+
+| Method | CLIP-F ↑ | RotErr ↓ | TransErr ↓ | Mat. Pix ↑ | FVD-V ↓ | CLIP-V ↑ |
+|--------|----------|----------|------------|------------|---------|----------|
+| ReCamMaster | 98.49 | 11.29 | 19.59 | 1314.00 | 732.52 | 88.91 |
+| EX-4D | 98.94 | 3.94 | 4.21 | 2188.98 | 685.63 | 89.77 |
+| TrajectoryCrafter (49f) | 98.80 | 2.26 | 3.03 | 1851.80 | 582.56 | 92.40 |
+| **Ours** | **99.03** | 2.76 | 4.23 | **2720.83** | **586.24** | **93.16** |
+| **Ours (49f)** | **99.01** | **2.61** | **2.73** | **2737.65** | **488.22** | **94.96** |
+
+---
+
+## Citation
+
+```bibtex
+@article{paliwal2026reshootanything,
+  title={Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting},
+  author={Paliwal, Avinash and Iyer, Adithya and Yadav, Shivin and Afridi, Muhammad Ali and Harikumar, Midhun},
+  journal={arXiv preprint arXiv:2604.21776},
+  year={2026}
+}
+```
+
+---
+
+## License
+
+Model weights are released under the **Apache 2.0** license, consistent with the Wan2.2 base model.