Update README.md

README.md

@@ -2,130 +2,263 @@
 license: apache-2.0
 language:
   - en
-library_name: other
+library_name: diffusers
+base_model:
+  - Lightricks/LTX-2.3
+pipeline_tag: image-to-video
 tags:
-  - video-generation
   - video-editing
+  - video-restoration
   - ltx-video
-  - lora
-  - safetensors
-  - image-to-video
+  - ltx-2-3
+  - dit
+  - ic-lora
   - watermark-removal
+  - subtitle-removal
   - super-resolution
-pipeline_tag: image-to-video
+  - hd-enhancement
+  - joyfox
 ---
 
 # LTX2.3-ICEdit-Insight
 
 <table>
   <tr>
-    <td align="center"><img src="./assets/effects/output_004_bigger.webp" alt="Super-resolution preview" width="700"/></td>
+    <td align="center"><img src="./assets/effects/output_004_bigger.webp" alt="LTX2.3-ICEdit-Insight preview" width="700"/></td>
   </tr>
 </table>
 
-Research-oriented model release for task-aware video restoration and editing under the `LTX-2.3` framework.
-
-Project links: [GitHub project](https://github.com/Valiant-Cat/LTX2-ICEdit-Insight) | [Valiant Cat on Hugging Face](https://huggingface.co/valiantcat)
-
-This repository contains:
-
-- `ltx-2.3-edit-insight-dev-fp8.safetensors`
-  - the all-in-one Insight checkpoint
-  - includes transformer + video VAE + audio VAE + text projection + vocoder
-- `ltx2.3-video-upscale-v2.safetensors`
-  - IC-LoRA for video super-resolution and detail recovery
-- `ltx2.3-ic-watermarkeRM.safetensors`
-  - IC-LoRA for video watermark removal and occlusion restoration
-
-These weights are intended to be used with the project's `run_pipeline.py` workflow. The recommended default is single-stage inference, where the IC-LoRA guidance remains active through the full-resolution denoising pass.
-
-## Research Positioning
-
-`ltx-2.3-edit-insight-dev-fp8.safetensors` is not presented as a bare deployment checkpoint. It is the unified base model release for the Insight branch of this project: a task-aware spatiotemporal editing backbone that consolidates the diffusion transformer, video VAE, audio VAE, text projection module, and vocoder into a single reproducible artifact.
-
-From a research perspective, the checkpoint is intended to support controlled video restoration and editing under a shared latent diffusion formulation. The paired IC-LoRA adapters specialize the backbone toward structure-preserving super-resolution and watermark-aware content recovery, while the unified checkpoint packaging keeps the full generative stack aligned for repeatable experiments and downstream ablations.
-
-## English Overview
+**LTX2.3-ICEdit-Insight** is a task-aware video restoration and editing model family developed by **JoyFox Lab**, built on top of the **LTX-2.3 DiT-based audio-video foundation model**.
 
-This package is built for the Insight version of the project's LTX-2.3 editing pipeline. Instead of shipping only task adapters, it also includes the corresponding Insight base checkpoint so the workflow can be reproduced with the exact model assets used by the project.
+This release focuses on four practical video editing directions:
 
-Recommended usage:
+- **Video Restoration**: degradation recovery, compression cleanup, blur and noise reduction, and damaged detail restoration.
+- **Video HD Enhancement**: super-resolution, detail reconstruction, texture sharpening, and perceptual quality improvement.
+- **Watermark Removal**: logo cleanup, semi-transparent overlay removal, and occlusion-aware background reconstruction.
+- **Subtitle Removal**: hard subtitle removal, caption cleanup, text overlay removal, and temporally stable inpainting.
 
-- run the companion `run_pipeline.py`
-- keep single-stage inference enabled by default
-- load one task LoRA at a time depending on the editing goal
+Unlike conventional frame-level enhancement pipelines, this model family operates as a **generative video restoration system** in latent video space. It is designed to preserve global structure, camera motion, object identity, and temporal consistency while reconstructing missing or degraded visual content.
 
-## 🧠 Training
+Project links: [GitHub project](https://github.com/Valiant-Cat/LTX2-ICEdit-Insight) | [JoyFox on Hugging Face](https://huggingface.co/joyfox)
 
-This model was trained and optimized by the AI Laboratory of Chongqing Valiant Cat Technology Co., LTD.
+## Model Files
 
-Visit [vvicat.com](https://vvicat.com/) for business collaborations or research partnerships.
-
-## 🧩 Integration with ComfyUI
-
-This model works with the modified ComfyUI [workflows](https://github.com/Valiant-Cat/LTX2-ICEdit-Insight/tree/main/workflows) provided by the project.
-
-For ComfyUI-based editing, load the base model in the UNet-side model path required by the workflow, then attach the task-specific IC-LoRA for the corresponding edit objective.
-
-## Files
 | File | Purpose |
 | --- | --- |
-| `ltx-2.3-edit-insight-dev-fp8.safetensors` | All-in-one Insight base checkpoint |
-| `ltx2.3-video-upscale-v2.safetensors` | Super-resolution / detail enhancement IC-LoRA |
-| `ltx2.3-ic-watermarkeRM.safetensors` | Watermark removal / occlusion restoration IC-LoRA |
+| `ltx-2.3-edit-insight-dev-fp8.safetensors` | Unified Insight base checkpoint for LTX-2.3 editing |
+| `ltx2.3-video-restoration-general.safetensors` | Video restoration, artifact cleanup, blur and noise recovery |
+| `ltx2.3-ic-video-upscale-general.safetensors` | Video HD enhancement, super-resolution, and detail recovery |
+| `ltx2.3-ic-watermark-remove-general.safetensors` | Watermark removal and occlusion-aware reconstruction |
+| `ltx2.3-ic-subtitles-remove-general.safetensors` | Subtitle removal and text overlay cleanup |
 
 ## Showcase
 
 <table>
   <tr>
-    <td align="center"><img src="./assets/effects/output_004.webp" alt="Super-resolution preview 1" width="600"/></td>
-    <td align="center"><img src="./assets/effects/output_005.webp" alt="Super-resolution preview 2" width="600"/></td>
+    <td align="center"><b>Video Restoration</b></td>
+    <td align="center"><b>Video HD Enhancement</b></td>
+  </tr>
+  <tr>
+    <td align="center"><img src="./assets/effects/output_004.webp" alt="Video restoration preview" width="600"/></td>
+    <td align="center"><img src="./assets/effects/视频高清对比效果.webp" alt="Video HD enhancement preview" width="600"/></td>
+  </tr>
+  <tr>
+    <td align="center"><b>Watermark Removal</b></td>
+    <td align="center"><b>Subtitle Removal</b></td>
+  </tr>
+  <tr>
+    <td align="center"><img src="./assets/effects/去水印对比效果.webp" alt="Watermark removal preview" width="600"/></td>
+    <td align="center"><img src="./assets/effects/去字幕对比效果.webp" alt="Subtitle removal preview" width="600"/></td>
   </tr>
 </table>
 
-## Usage With This Project
+## Script Usage
+
+Run all scripts from the project root.
+
+```bash
+bash run_restoration.sh
+bash run_hd.sh
+bash run_hd.sh /path/to/input.mp4
+bash run_watermark_rm.sh
+bash run_watermark_rm.sh /path/to/input.mp4
+bash run_subtitle_rm.sh
+bash run_subtitle_rm.sh /path/to/input.mp4
+```
 
-Run all commands from the project root.
+## Command Examples
 
-### Super-resolution
+### Video Restoration
 
 ```bash
 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
 python run_pipeline.py \
-  --mode upscale \
-  --video ./inputs/input.mp4 \
-  --prompt "Convert the video to ultra-high definition quality, rebuilding high-frequency details while eliminating artifacts." \
-  --output ./outputs/output_upscale.mp4 \
+  --mode restoration \
+  --video ./inputs/input_480p.mp4 \
+  --prompt "Convert the video to ultra-high-definition quality while removing artifacts and rebuilding high-frequency details." \
+  --output ./outputs/output_restoration.mp4 \
   --height 1184 --width 704 --num-frames 97 \
   --fps 24.0 --seed 42 \
   --sigma-profile workflow \
+  --streaming-prefetch-count 2 \
   --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
-  --lora ./models/loras/ltx2.3-train/ltx2.3-video-upscale-v2.safetensors
+  --lora ./models/loras/ltx2.3-train/ltx2.3-video-restoration-general.safetensors
 ```
 
-### Watermark removal
+### Video HD Enhancement
+
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+python run_pipeline.py \
+  --mode hd \
+  --video ./inputs/input_480p.mp4 \
+  --prompt "Convert the video to ultra-high-definition quality, significantly improving clarity, fine detail richness, texture fidelity, and overall perceptual sharpness." \
+  --output ./outputs/output_hd.mp4 \
+  --height 1184 --width 704 --num-frames 97 \
+  --fps 24.0 --seed 42 \
+  --sigma-profile workflow \
+  --streaming-prefetch-count 2 \
+  --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
+  --lora ./models/loras/ltx2.3-train/ltx2.3-ic-video-upscale-general.safetensors
+```
+
+### Watermark Removal
 
 ```bash
 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
 python run_pipeline.py \
   --mode watermark_rm \
-  --video ./inputs/input.mp4 \
+  --video ./inputs/input_480p.mp4 \
   --prompt "Remove short-video platform watermarks and related occlusions from the video, restoring a clean, clear, and natural original image." \
   --output ./outputs/output_watermark_rm.mp4 \
   --height 1184 --width 704 --num-frames 97 \
   --fps 24.0 --seed 1546 \
   --sigma-profile workflow \
+  --streaming-prefetch-count 2 \
   --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
-  --lora ./models/loras/ltx2.3-train/ltx2.3-ic-watermarkeRM.safetensors
+  --lora ./models/loras/ltx2.3-train/ltx2.3-ic-watermark-remove-general.safetensors
 ```
 
-## Notes
+### Subtitle Removal
 
-- Single-stage inference is the default recommendation.
-- In two-stage mode, the second-stage refinement does not keep the IC-LoRA constraint, which can increase content drift.
-- Frame count must satisfy `8k + 1`.
-- Single-stage output height and width should be multiples of `32`.
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+python run_pipeline.py \
+  --mode subtitle_rm \
+  --video ./inputs/input_480p.mp4 \
+  --prompt "Remove subtitles, captions, and related text occlusions from the video, restoring a clean and natural underlying image." \
+  --output ./outputs/output_subtitle_rm.mp4 \
+  --height 1184 --width 704 --num-frames 97 \
+  --fps 24.0 --seed 42 \
+  --sigma-profile workflow \
+  --streaming-prefetch-count 2 \
+  --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
+  --lora ./models/loras/ltx2.3-train/ltx2.3-ic-subtitles-remove-general.safetensors
+```
+
+## Key Improvements
+
+### Task-Aware IC-Edit Framework
+
+We introduce a task-aware IC-Edit training framework for LTX-2.3, where each restoration direction is optimized with dedicated instruction conditioning and task-specific IC-LoRA adapters.
+
+The model is trained not only to improve visual quality, but also to understand the editing goal behind different restoration tasks, including watermark removal, subtitle cleanup, damaged region recovery, and high-definition enhancement.
+
+### LTX-2.3 DiT Backbone Adaptation
+
+The model family is built on the LTX-2.3 foundation architecture, a diffusion-transformer video model designed for high-fidelity image-to-video and video generation workflows.
+
+Our adaptation targets video restoration by improving:
+
+- latent-space editability
+- instruction-following behavior
+- frame-to-frame stability
+- high-frequency detail recovery
+- local reconstruction around degraded or occluded regions
+
+### Spatiotemporal Consistency Optimization
+
+Video restoration requires more than strong single-frame quality. We optimize temporal consistency so that restored areas remain stable across adjacent frames.
+
+This reduces common artifacts such as:
+
+- flickering textures
+- unstable reconstructed backgrounds
+- inconsistent watermark removal
+- subtitle ghosting
+- frame-wise color shift
+- detail popping during motion
+
+### Degradation-Aware Training Curriculum
+
+The training curriculum covers realistic video defects including:
+
+- compression artifacts
+- motion blur
+- sensor noise
+- low-bitrate video
+- text overlays
+- hard subtitles
+- semi-transparent watermarks
+- platform logos
+- local occlusions
+- low-resolution inputs
+
+This improves generalization across short videos, social-media clips, mobile footage, downloaded videos, and compressed production material.
+
+### Occlusion-Aware Reconstruction
+
+For watermark and subtitle removal, the model is optimized to reconstruct the hidden visual content behind occluded regions.
+
+Instead of smearing or blurring the target area, it uses surrounding spatial context and temporal cues to infer plausible background structure, object boundaries, lighting, and texture continuity.
+
+### Frequency-Enhanced HD Restoration
+
+For HD enhancement, the model improves perceptual sharpness and fine visual detail through frequency-aware restoration training.
+
+This is especially helpful for recovering:
+
+- hair strands
+- fabric texture
+- skin detail
+- product edges
+- background patterns
+- typography-like fine structures
+- natural image clarity
+
+## Inference Notes
+
+- Single-stage inference is recommended for most editing tasks.
+- Two-stage refinement can improve visual polish but may weaken task-specific LoRA constraints.
+- Watermark and subtitle removal perform best when the occlusion area is stable and not excessively large.
+- HD enhancement quality depends on input resolution, motion complexity, and compression level.
+- Higher output resolution improves detail but requires more VRAM.
+- For strong-motion videos, conservative denoising settings are recommended to preserve temporal structure.
+- Frame count should follow the `8k + 1` rule.
+- Output height and width should be multiples of `32` in single-stage inference.
+
+## Training
+
+This model family was trained and optimized by **JoyFox Lab** (**Chengdu Xuanhu Technology Co., Ltd.**).
+
+The training pipeline includes:
+
+- task-aware video restoration data construction
+- degradation synthesis and curriculum training
+- IC-LoRA specialization for four editing directions
+- temporal consistency regularization
+- occlusion-aware reconstruction training
+- high-frequency perceptual enhancement
+- instruction-guided video editing optimization
+
+## Contact
+
+For research collaboration, commercial licensing, or workflow integration, contact:
+
+- `z@vvicat.com`
 
 ## License
 
-This repository is released under the Apache 2.0 license.
+Licensed under **Apache 2.0**.
+
+Please also review the license terms of the upstream LTX-2.3 base model when using or redistributing derivative checkpoints.