Spaces:
Sleeping
Sleeping
| # humeo-core | |
| **Reusable-rocket MCP server for long-video → 9:16 shorts.** | |
| First-principles design, from the HIVE paper + Bryan's rocket analogy: | |
| we don't build doors and windows (general subject-tracker UI, retraining | |
| models). We build the **container** (schemas), **landing gear** (deterministic | |
| local extraction), and **five thrusters** (the five 9:16 layouts this video | |
| format actually uses). Everything else is pluggable. | |
| ## The rocket, in one picture | |
| ``` | |
| ┌──────────────────────────────────────────┐ | |
| │ Control panel (MCP tools) │ <- any MCP client | |
| └───────────────────┬──────────────────────┘ | |
| │ strict JSON | |
| ┌────────────────┬───────────┼────────────────┬─────────────────┐ | |
| ▼ ▼ ▼ ▼ ▼ | |
| ingest classify_scenes select_clips plan_layout render_clip | |
| (scenes + (5-way layout (clip picker, (5 thrusters, (ffmpeg compile, | |
| keyframes + classifier) heuristic + pure filter dry-run safe) | |
| transcript) LLM-ready) math) | |
| │ | |
| ▼ | |
| ┌────────────────────┐ | |
| │ LayoutKind │ | |
| │ ──────────────── │ | |
| │ zoom_call_center │ | |
| │ sit_center │ | |
| │ split_chart_person│ | |
| │ split_two_persons │ | |
| │ split_two_charts │ | |
| └────────────────────┘ | |
| ``` | |
| Only the classifier and clip-selector have optional LLM hooks; everything | |
| else is deterministic, local, and cheap. | |
| ## Why five layouts? (the "max 2 items" rule) | |
| The hard constraint for this format: **a short shows at most two on-screen | |
| items** — where an "item" is a `person` (a human speaker) or a `chart` | |
| (slide, graph, data visual, screenshare). That gives exactly five recipes: | |
| 1. **`zoom_call_center`** — 1 person, tight zoom-call / webcam framing. | |
| 2. **`sit_center`** — 1 person, interview / seated framing. | |
| 3. **`split_chart_person`** — 1 chart + 1 person, stacked vertically | |
| (default: **even 50/50** top/bottom, chart on top). | |
| 4. **`split_two_persons`** — 2 speakers, stacked vertically. | |
| 5. **`split_two_charts`** — 2 charts, stacked vertically. | |
| Because the geometry is bounded, we do NOT need a general subject-tracker | |
| ML model or a drag-to-highlight UI. We need five small, correct pieces of | |
| crop/compose math. That is exactly what `src/humeo_core/primitives/layouts.py` | |
| is. | |
| See [`TERMINOLOGY.md`](../TERMINOLOGY.md) for the full glossary of terms | |
| used across these docs (subject, crop, band, seam, bbox, layout, etc.). | |
| ## Install | |
| ```bash | |
| uv venv | |
| uv sync | |
| ``` | |
| External requirements: `ffmpeg` and `ffprobe` on PATH. | |
| `scenedetect` requires OpenCV. Install `opencv-python-headless` or | |
| `opencv-python` alongside `scenedetect`. | |
| ## Use it as an MCP server | |
| ```bash | |
| humeo-core # stdio transport (primary console script) | |
| # humeo-mcp # same entrypoint — kept so existing MCP configs keep working | |
| ``` | |
| Example Cursor/Claude Desktop config: | |
| ```json | |
| { | |
| "mcpServers": { | |
| "humeo": { "command": "humeo-core" } | |
| } | |
| } | |
| ``` | |
| Tools exposed: | |
| | Tool | Purpose | | |
| | --------------------------------- | --------------------------------------------------------------------------- | | |
| | `list_layouts` | Enumerate the 5 supported layouts. | | |
| | `ingest` | Scene detection + keyframe extraction (+ optional transcript). | | |
| | `classify_scenes` | Pixel-heuristic per-scene layout classification. | | |
| | `detect_scene_regions` | Return the bbox prompt + per-scene jobs (agent runs its own vision model). | | |
| | `classify_scenes_with_vision` | Classify scenes from already-gathered `SceneRegions` bbox JSON + build layout instructions. | | |
| | `select_clips` | Heuristic clip picker over a word-level transcript. | | |
| | `plan_layout` | Return the exact `ffmpeg -filter_complex` for a layout. | | |
| | `build_render_cmd` | Build the ffmpeg command (no execution) — review before spend. | | |
| | `render_clip` | Build + run ffmpeg to produce a 9:16 MP4. | | |
| Resource: `humeo://layouts` (JSON listing of the 5 layouts). | |
| ### Three interchangeable region detectors | |
| All three emit the same `SceneRegions` schema, so the layout planner and renderer don't care which one you used: | |
| ``` | |
| classify.py (pixel variance, no ML) | |
| face_detect.py (MediaPipe, local) ──► SceneRegions ──► SceneClassification ──► LayoutInstruction ──► ffmpeg | |
| vision.py (multimodal LLM + OCR bboxes) | |
| ``` | |
| ## JSON contracts (non-negotiable) | |
| All tools take and return Pydantic-validated JSON. The contracts live in | |
| [`src/humeo_core/schemas.py`](src/humeo_core/schemas.py): | |
| - `Scene` `{scene_id, start_time, end_time, keyframe_path?}` | |
| - `TranscriptWord` `{word, start_time, end_time}` | |
| - `IngestResult` `{source_path, duration_sec, scenes[], transcript_words[], keyframes_dir?}` | |
| - `SceneClassification` `{scene_id, layout, confidence, reason}` | |
| - `BoundingBox` `{x1, y1, x2, y2, label, confidence}` (all coords normalized) | |
| - `SceneRegions` `{scene_id, person_bbox?, chart_bbox?, ocr_text, raw_reason}` | |
| - `Clip` `{clip_id, topic, start_time_sec, end_time_sec, viral_hook, virality_score, transcript, suggested_overlay_title, layout?}` | |
| - `ClipPlan` `{source_path, clips[]}` | |
| - `LayoutInstruction` `{clip_id, layout, zoom, person_x_norm, chart_x_norm, split_chart_region?, split_person_region?, split_second_chart_region?, split_second_person_region?, top_band_ratio, focus_stack_order}` | |
| - `RenderRequest` / `RenderResult` | |
| ## First-principles decisions (what we intentionally did NOT build) | |
| - **No giant subject-tracker ML.** The video format has 5 fixed layouts | |
| (with a hard "max 2 items" rule); pixel-level tracking is not needed. | |
| - **No drag-and-highlight UI.** An MCP tool is a better "UI" for an | |
| agent-first workflow. If a human wants to override, they pass a | |
| `LayoutInstruction` with their own `person_x_norm` / `chart_x_norm` / | |
| `zoom`. | |
| - **No end-to-end video→video model.** The HIVE paper's core insight is | |
| that decomposed orchestration beats monolithic generation. We reify | |
| that insight as six small composable tools. | |
| ## Extending the pilot | |
| - Plug a real multimodal model into `classify_scenes_with_llm(vision_fn)`. | |
| - Plug a real reasoning model into `select_clips_with_llm(text_fn)`. | |
| - Plug a real vision-LLM into `detect_regions_with_llm(scenes, vision_fn)` | |
| to get per-scene bboxes + OCR text, then feed the results back through | |
| `classify_scenes_with_vision`. This is the scene-change → v3 images → | |
| LLM+OCR → bbox path; see `../docs/SOLUTIONS.md §4` for rationale. | |
| - All enforce strict JSON outputs, so bad model output can't corrupt | |
| downstream stages. | |
| ## Testing | |
| ```bash | |
| python -m pytest | |
| ``` | |
| See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for deeper rationale. | |
| ## License | |
| MIT | |