OmniShotCut MLX
Shot Boundary Detection with OmniShotCut, ported to Apple MLX for native Mac inference.
Based on the paper: OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer.
Features
- Pure MLX inference — runs natively on Apple Silicon, zero PyTorch dependency at runtime
- Detects hard cuts, dissolves, fades, wipes, slides, zooms, doorways, and sudden jumps
- Tunable sensitivity for different video types (action, interview, vlog, film)
Requirements
- macOS with Apple Silicon (M1/M2/M3/M4)
- Python 3.10+
ffmpeg(for video I/O)
pip install mlx mlx-metal numpy
Quick Start
# 1. Clone and install
git clone https://github.com/eisneim/OmniShotCut_mlx.git
cd OmniShotCut_mlx
# 2. Download weights from HuggingFace
python omnishotcut_mlx/download_weights.py
# 3. Run on test videos
python run_inference.py
Download Weights
# Auto-download from HuggingFace Hub (requires huggingface_hub)
pip install huggingface_hub
python omnishotcut_mlx/download_weights.py
# Or manually download from:
# https://huggingface.co/eisneim/OmniShotCut_mlx
# Place OmniShotCut.safetensors and config.json into ./weights/
# Alternative: download without huggingface_hub
curl -L -o weights/OmniShotCut.safetensors https://huggingface.co/eisneim/OmniShotCut_mlx/resolve/main/OmniShotCut.safetensors
curl -L -o weights/config.json https://huggingface.co/eisneim/OmniShotCut_mlx/resolve/main/config.json
Usage
# Default: balanced detection
python run_inference.py
# Sensitive mode: more cuts, good for action/vlog videos
python run_inference.py --sensitive
# Conservative mode: fewer false positives, good for interviews/long takes
python run_inference.py --conservative
# Single video
python run_inference.py --video /path/to/video.mp4
# Custom output directory
python run_inference.py --output ./my_shots
# Fine-tuned control
python run_inference.py --context 12 --min-shot 0.8 --conf 0.1
Tunable Parameters
| Parameter | Default | Range | Effect |
|---|---|---|---|
--context |
10 | 0–20 | Overlap frames between windows. Higher = fewer missed boundaries, but slower |
--min-shot |
0.5 | 0.1–5.0 | Minimum shot duration in seconds. Higher = fewer false positives |
--conf |
0.0 | 0.0–1.0 | Intra-class confidence threshold. E.g. 0.3 = keep only predictions model is >30% sure about |
--sensitive |
— | — | Shortcut: context=15, min-shot=0.3, conf=0 |
--conservative |
— | — | Shortcut: context=5, min-shot=1.5, conf=0.15 |
Parameter Guide by Video Type
| Video Type | Recommended | Why |
|---|---|---|
| Action / Sports | --sensitive |
Fast cuts, many short shots |
| Vlog / YouTube | default or --context 15 |
Moderate pace, varied editing |
| Interview / Podcast | --conservative |
Long takes, few cuts |
| Film / Cinema | default | Balanced |
| Animation | --sensitive |
Frequent scene changes |
| Screen Recording | --conservative or --min-shot 2.0 |
Mostly static |
Project Structure
OmniShotCut_mlx/
├── run_inference.py # Main entry point
├── omnishotcut_mlx/
│ ├── model.py # OmniShotCut MLX model
│ ├── transformer.py # Transformer encoder/decoder
│ ├── resnet.py # ResNet18 backbone
│ ├── position_encoding.py # 3D sinusoidal position encoding
│ ├── load_weights.py # Weight loader (from safetensors)
│ └── download_weights.py # HuggingFace weight downloader
├── weights/
│ ├── OmniShotCut.safetensors # MLX-native weights (~157MB)
│ └── config.json # Model configuration
└── test_data/ # Place test videos here
Output
Shots are saved as shot_0000.mp4, shot_0001.mp4, ... under test_data/output/<video_name>/.
Each shot file is a self-contained H.264/AAC MP4 clip with the detected shot boundary transitions removed.
Model
- Architecture: Shot-Query Transformer (DETR-style), 6 encoder + 6 decoder layers, ResNet18 backbone
- Input: 100-frame windows at 128×96, ImageNet normalization
- Output: Shot boundary frame indices + intra-shot relation (dissolve, wipe, fade, ...) + inter-shot relation (hard cut, sudden jump, ...)
- Weights: Converted from the official PyTorch checkpoint, 363 tensors, float32
License & Credits
Paper: OmniShotCut (arXiv 2604.24762) by Boyang Wang et al.
MLX port by @eisneim. Weights hosted at eisneim/OmniShotCut_mlx.