---
license: apache-2.0
base_model:
- Lightricks/LTX-2
- Lightricks/LTX-2.3
pipeline_tag: any-to-any
library_name: peft
tags:
- lora
- reinforcement-learning
- GRPO
- T2AV
---

# 🎬 OmniNFT RL-LoRA for LTX Video

> 🚀 Modality-wise Omni Diffusion Negative-aware Fine-Tuning for **LTX-2** and **LTX-2.3**~

<a href="https://zghhui.github.io/OmniNFT/"><img src="https://img.shields.io/badge/🌐-Project%20Page-blue" alt="Project Page"/></a>
[![GitHub](https://img.shields.io/badge/GitHub-OmniNFT-blue?logo=github)](https://github.com/zghhui/OmniNFT)
[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)

---

## 📁 Model Files

| Folder | Base Model | Description |
|--------|-----------|-------------|
| `LTX-2-RL-Lora/` | [Lightricks/LTX-2](https://huggingface.co/Lightricks/LTX-2) | RL-LoRA for LTX-2 |
| `LTX-2.3-RL-Lora/` | [Lightricks/LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) | RL-LoRA for LTX-2.3 |

---

## 🛠️ Quick Start

### 1️⃣ Clone the Code

```bash
git clone https://github.com/zghhui/OmniNFT.git
cd OmniNFT
pip install -r requirements.txt
```

### 2️⃣ Download Base Models

Download the base checkpoints from HuggingFace:

```bash
# LTX-2
huggingface-cli download Lightricks/LTX-2 --local-dir ./checkpoints/LTX-2

# LTX-2.3
huggingface-cli download Lightricks/LTX-2.3 --local-dir ./checkpoints/LTX-2.3
```

### 3️⃣ Download RL-LoRA Weights

```bash
# LTX-2 LoRA
huggingface-cli download zghhui/OmniNFT \
    --include "LTX-2-RL-Lora/*" \
    --local-dir ./checkpoints/

# LTX-2.3 LoRA
huggingface-cli download zghhui/OmniNFT \
    --include "LTX-2.3-RL-Lora/*" \
    --local-dir ./checkpoints/
```

### 4️⃣ Merge LoRA into Base Model

```bash
# 🔀 Merge LTX-2 + RL-LoRA
python scripts/merge_lora.py \
    --checkpoint-path ./checkpoints/LTX-2/transformer/model.safetensors \
    --lora-dir ./checkpoints/LTX-2-RL-Lora \
    --output-path ./merged_model_ltx2.safetensors \
    --dtype bf16

# 🔀 Merge LTX-2.3 + RL-LoRA
python scripts/merge_lora.py \
    --checkpoint-path ./checkpoints/LTX-2.3/transformer/model.safetensors \
    --lora-dir ./checkpoints/LTX-2.3-RL-Lora \
    --output-path ./merged_model_ltx23.safetensors \
    --dtype bf16
```

### 5️⃣ Inference 🎥

```bash
python scripts/inference.py \
    --model_path ./merged_model_ltx2.safetensors \
    --gemma_path $GEMMA_MODEL_PATH \
    --prompt "A man plays acoustic guitar on a wooden stage, warm applause from the audience" \
    --seed 42 \
    --output_dir ./results
```

---

## 📝 Citation

If you find this work helpful, please consider citing:

```bibtex
@article{zhang2026omninft,
  title={OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation},
  author={Zhang, Guohui and Ma, XiaoXiao and Huang, Jie and Xu, Hang and Yu, Hu and Fu, Siming and Li, Yuming and Xue, Zeyue and Song, Lin and Huang, Haoyang and Duan, Nan and Zhao, Feng},
  journal={arXiv preprint arXiv:2605.12480},
  year={2026}
}
```

---

## 🙏 Acknowledgements

- 🎬 [Lightricks](https://huggingface.co/Lightricks) for the base models
- 🤗 [HuggingFace PEFT](https://github.com/huggingface/peft) for LoRA support

---

## 📄 License

This project is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).