README.md

metadata

license: apache-2.0
base_model:
  - Lightricks/LTX-2
  - Lightricks/LTX-2.3
pipeline_tag: any-to-any
library_name: peft
tags:
  - lora
  - reinforcement-learning
  - GRPO
  - T2AV

🎬 OmniNFT RL-LoRA for LTX Video

🚀 Modality-wise Omni Diffusion Negative-aware Fine-Tuning for LTX-2 and LTX-2.3~

---

📁 Model Files

Folder	Base Model	Description
LTX-2-RL-Lora/	Lightricks/LTX-2	RL-LoRA for LTX-2
LTX-2.3-RL-Lora/	Lightricks/LTX-2.3	RL-LoRA for LTX-2.3

---

🛠️ Quick Start

1️⃣ Clone the Code

git clone https://github.com/zghhui/OmniNFT.git
cd OmniNFT
pip install -r requirements.txt

2️⃣ Download Base Models

Download the base checkpoints from HuggingFace:

# LTX-2
huggingface-cli download Lightricks/LTX-2 --local-dir ./checkpoints/LTX-2

# LTX-2.3
huggingface-cli download Lightricks/LTX-2.3 --local-dir ./checkpoints/LTX-2.3

3️⃣ Download RL-LoRA Weights

# LTX-2 LoRA
huggingface-cli download zghhui/OmniNFT \
    --include "LTX-2-RL-Lora/*" \
    --local-dir ./checkpoints/

# LTX-2.3 LoRA
huggingface-cli download zghhui/OmniNFT \
    --include "LTX-2.3-RL-Lora/*" \
    --local-dir ./checkpoints/

4️⃣ Merge LoRA into Base Model

# 🔀 Merge LTX-2 + RL-LoRA
python scripts/merge_lora.py \
    --checkpoint-path ./checkpoints/LTX-2/transformer/model.safetensors \
    --lora-dir ./checkpoints/LTX-2-RL-Lora \
    --output-path ./merged_model_ltx2.safetensors \
    --dtype bf16

# 🔀 Merge LTX-2.3 + RL-LoRA
python scripts/merge_lora.py \
    --checkpoint-path ./checkpoints/LTX-2.3/transformer/model.safetensors \
    --lora-dir ./checkpoints/LTX-2.3-RL-Lora \
    --output-path ./merged_model_ltx23.safetensors \
    --dtype bf16

5️⃣ Inference 🎥

python scripts/inference.py \
    --model_path ./merged_model_ltx2.safetensors \
    --gemma_path $GEMMA_MODEL_PATH \
    --prompt "A man plays acoustic guitar on a wooden stage, warm applause from the audience" \
    --seed 42 \
    --output_dir ./results

---

📝 Citation

If you find this work helpful, please consider citing:

@article{zhang2026omninft,
  title={OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation},
  author={Zhang, Guohui and Ma, XiaoXiao and Huang, Jie and Xu, Hang and Yu, Hu and Fu, Siming and Li, Yuming and Xue, Zeyue and Song, Lin and Huang, Haoyang and Duan, Nan and Zhao, Feng},
  journal={arXiv preprint arXiv:2605.12480},
  year={2026}
}

---

🙏 Acknowledgements

• 🎬 Lightricks for the base models
• 🤗 HuggingFace PEFT for LoRA support

---

📄 License

This project is licensed under the Apache 2.0 License.