Upload folder using huggingface_hub

OmniShotCut.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce52cb008514a6b5fbfd2f6ee6c2865a9845e464f50166104c0c9ecbfd347152
+size 164079848

README.md

@@ -1,3 +1,131 @@
----
-license: mit
----
+# OmniShotCut MLX
+
+Shot Boundary Detection with OmniShotCut, ported to Apple MLX for native Mac inference.
+
+Based on the paper: [OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer](https://arxiv.org/abs/2604.24762).
+
+## Features
+
+- Pure MLX inference — runs natively on Apple Silicon, zero PyTorch dependency at runtime
+- Detects hard cuts, dissolves, fades, wipes, slides, zooms, doorways, and sudden jumps
+- Tunable sensitivity for different video types (action, interview, vlog, film)
+
+## Requirements
+
+- macOS with Apple Silicon (M1/M2/M3/M4)
+- Python 3.10+
+- `ffmpeg` (for video I/O)
+
+```bash
+pip install mlx mlx-metal numpy
+```
+
+## Quick Start
+
+```bash
+# 1. Clone and install
+git clone https://github.com/eisneim/OmniShotCut_mlx.git
+cd OmniShotCut_mlx
+
+# 2. Download weights from HuggingFace
+python omnishotcut_mlx/download_weights.py
+
+# 3. Run on test videos
+python run_inference.py
+```
+
+## Download Weights
+
+```bash
+# Auto-download from HuggingFace Hub (requires huggingface_hub)
+pip install huggingface_hub
+python omnishotcut_mlx/download_weights.py
+
+# Or manually download from:
+# https://huggingface.co/eisneim/OmniShotCut_mlx
+# Place OmniShotCut.safetensors and config.json into ./weights/
+
+# Alternative: download without huggingface_hub
+curl -L -o weights/OmniShotCut.safetensors https://huggingface.co/eisneim/OmniShotCut_mlx/resolve/main/OmniShotCut.safetensors
+curl -L -o weights/config.json https://huggingface.co/eisneim/OmniShotCut_mlx/resolve/main/config.json
+```
+
+## Usage
+
+```bash
+# Default: balanced detection
+python run_inference.py
+
+# Sensitive mode: more cuts, good for action/vlog videos
+python run_inference.py --sensitive
+
+# Conservative mode: fewer false positives, good for interviews/long takes
+python run_inference.py --conservative
+
+# Single video
+python run_inference.py --video /path/to/video.mp4
+
+# Custom output directory
+python run_inference.py --output ./my_shots
+
+# Fine-tuned control
+python run_inference.py --context 12 --min-shot 0.8 --conf 0.1
+```
+
+### Tunable Parameters
+
+| Parameter | Default | Range | Effect |
+|-----------|---------|-------|--------|
+| `--context` | 10 | 0–20 | Overlap frames between windows. Higher = fewer missed boundaries, but slower |
+| `--min-shot` | 0.5 | 0.1–5.0 | Minimum shot duration in seconds. Higher = fewer false positives |
+| `--conf` | 0.0 | 0.0–1.0 | Intra-class confidence threshold. E.g. 0.3 = keep only predictions model is >30% sure about |
+| `--sensitive` | — | — | Shortcut: context=15, min-shot=0.3, conf=0 |
+| `--conservative` | — | — | Shortcut: context=5, min-shot=1.5, conf=0.15 |
+
+### Parameter Guide by Video Type
+
+| Video Type | Recommended | Why |
+|------------|------------|-----|
+| Action / Sports | `--sensitive` | Fast cuts, many short shots |
+| Vlog / YouTube | default or `--context 15` | Moderate pace, varied editing |
+| Interview / Podcast | `--conservative` | Long takes, few cuts |
+| Film / Cinema | default | Balanced |
+| Animation | `--sensitive` | Frequent scene changes |
+| Screen Recording | `--conservative` or `--min-shot 2.0` | Mostly static |
+
+## Project Structure
+
+```
+OmniShotCut_mlx/
+├── run_inference.py               # Main entry point
+├── omnishotcut_mlx/
+│   ├── model.py                   # OmniShotCut MLX model
+│   ├── transformer.py             # Transformer encoder/decoder
+│   ├── resnet.py                  # ResNet18 backbone
+│   ├── position_encoding.py       # 3D sinusoidal position encoding
+│   ├── load_weights.py            # Weight loader (from safetensors)
+│   └── download_weights.py        # HuggingFace weight downloader
+├── weights/
+│   ├── OmniShotCut.safetensors    # MLX-native weights (~157MB)
+│   └── config.json                # Model configuration
+└── test_data/                     # Place test videos here
+```
+
+## Output
+
+Shots are saved as `shot_0000.mp4`, `shot_0001.mp4`, ... under `test_data/output/<video_name>/`.
+
+Each shot file is a self-contained H.264/AAC MP4 clip with the detected shot boundary transitions removed.
+
+## Model
+
+- **Architecture**: Shot-Query Transformer (DETR-style), 6 encoder + 6 decoder layers, ResNet18 backbone
+- **Input**: 100-frame windows at 128×96, ImageNet normalization
+- **Output**: Shot boundary frame indices + intra-shot relation (dissolve, wipe, fade, ...) + inter-shot relation (hard cut, sudden jump, ...)
+- **Weights**: Converted from the official PyTorch checkpoint, 363 tensors, float32
+
+## License & Credits
+
+Paper: [OmniShotCut (arXiv 2604.24762)](https://arxiv.org/abs/2604.24762) by Boyang Wang et al.
+
+MLX port by [@eisneim](https://github.com/eisneim). Weights hosted at [eisneim/OmniShotCut_mlx](https://huggingface.co/eisneim/OmniShotCut_mlx).

config.json

@@ -0,0 +1,16 @@
+{
+  "model_type": "OmniShotCut",
+  "hidden_dim": 384,
+  "num_encoder_layers": 6,
+  "num_decoder_layers": 6,
+  "nhead": 8,
+  "dim_feedforward": 2048,
+  "num_queries": 24,
+  "num_frames": 100,
+  "process_height": 96,
+  "process_width": 128,
+  "num_intra_relation_classes": 10,
+  "num_inter_relation_classes": 7,
+  "backbone": "resnet18",
+  "dropout": 0.0
+}