README.md

---
title: ElevenClip AI
emoji: 🎬
colorFrom: purple
colorTo: red
sdk: docker
pinned: false
---

# ElevenClip.AI

ElevenClip.AI is an AI-powered clip studio for turning long-form videos into personalized short-form content for TikTok, YouTube Shorts, and Instagram Reels.

This project is built for the **AMD Developer Hackathon** on lablab.ai, targeting **Track 3: Vision & Multimodal AI**. The system is designed to run on **AMD Developer Cloud** with **ROCm** and **AMD Instinct MI300X** acceleration, while using **Hugging Face** as the model hub/deployment layer and **Qwen** models for profile-aware highlight reasoning.

## One-Sentence Pitch

ElevenClip.AI helps creators convert long videos into ready-to-edit short clips by combining Whisper transcription, Qwen highlight detection, optional Qwen-VL visual understanding, ffmpeg rendering, and a human-in-the-loop clip editor.

## Problem

Long-form creators, podcasters, educators, streamers, and marketing teams often publish hours of video but still need short clips for modern discovery platforms.

The manual workflow is painful:

- Watch the full video.
- Find high-retention moments.
- Trim each clip.
- Rewrite subtitles.
- Reframe to vertical 9:16.
- Export platform-ready MP4 files.

For a two-hour video, this can take several hours of editing time. The bottleneck is not just cutting video; it is understanding which moments match the creator's audience, channel style, language, and target platform.

## Solution

ElevenClip.AI automates the first pass of short-form production:

1. The creator sets up a reusable channel profile.
2. The creator provides a YouTube URL or uploads a video file.
3. Whisper Large V3 transcribes the video, including Thai and multilingual speech.
4. Qwen2.5 analyzes the transcript and scores candidate highlights based on engagement potential and the creator profile.
5. Optional Qwen2-VL analysis can enrich the scores with visual signals such as reactions, scene changes, and on-screen text.
6. ffmpeg renders vertical clips with subtitle files and burn-in support.
7. The React editor lets the human approve, delete, trim, regenerate, and edit subtitles before download.

The product is intentionally human-AI collaborative: AI finds and prepares the clips quickly, while the creator keeps editorial control.

## Hackathon Alignment

### Track

**Track 3: Vision & Multimodal AI**

ElevenClip.AI processes multiple media types:

- Audio: speech transcription with Whisper Large V3.
- Text: transcript reasoning and highlight ranking with Qwen2.5.
- Video: frame-aware multimodal analysis with Qwen2-VL as the next pipeline stage.
- Rendered media: ffmpeg exports platform-ready video clips.

### AMD Technology

The production target is AMD Developer Cloud:

- **AMD Instinct MI300X** for high-throughput model inference.
- **ROCm 6.x** as the GPU software stack.
- **PyTorch with ROCm support** for Whisper inference.
- **vLLM ROCm backend** for fast Qwen2.5 inference.
- **Optimum-AMD** as an optimization path for Hugging Face models on AMD hardware.
- **ffmpeg hardware acceleration hooks** for faster video encoding where available.

The app has a local `DEMO_MODE=true` path so judges and teammates can inspect the UI/API without downloading large models. On AMD Developer Cloud, set `DEMO_MODE=false` to activate the real model stack.

### Hugging Face Integration

Hugging Face is used as the model hub and deployment layer:

- `openai/whisper-large-v3` for transcription.
- `Qwen/Qwen2.5-7B-Instruct` for highlight analysis.
- `Qwen/Qwen2-VL-7B-Instruct` for multimodal video understanding.
- Public Hugging Face Space for the hackathon demo page:
  `https://huggingface.co/spaces/lablab-ai-amd-developer-hackathon/ElevenClip-AI`

### Qwen Integration

Qwen is not used as a generic chatbot. It is part of the core product logic:

- Reads timestamped transcript segments.
- Considers creator profile settings.
- Scores engagement potential.
- Explains why a segment should become a clip.
- Returns structured JSON with timestamps, titles, scores, reasons, and subtitle text.

## Current MVP Features

- Channel profile onboarding:
  - niche
  - preferred clip style
  - preferred clip length
  - primary language
  - target platform
- YouTube URL ingestion through `yt-dlp`.
- Direct video upload endpoint.
- Whisper transcription service boundary.
- Qwen highlight detection service boundary.
- Optional Qwen2-VL multimodal analysis service boundary.
- ffmpeg clip generation with subtitle file creation.
- Vertical 9:16 export path for TikTok, Shorts, and Reels.
- Human-AI review UI:
  - trim start/end
  - edit subtitles inline
  - approve clips
  - delete clips
  - regenerate a clip
  - download MP4 output
- Timing logs for benchmark demos.
- Docker and AMD Cloud deployment notes.

## Architecture

```mermaid
flowchart LR
  A["Creator Profile"] --> D["Qwen2.5 Highlight Scoring"]
  B["YouTube URL"] --> C["yt-dlp / Video Input"]
  B2["Uploaded Video"] --> C
  C --> W["Whisper Large V3 Transcription"]
  W --> D
  C --> V["Qwen2-VL Visual Analysis (Optional)"]
  D --> R["Clip Plan JSON"]
  V --> R
  R --> F["ffmpeg Clip Rendering + Subtitles"]
  F --> E["React Human-AI Editor"]
  E --> O["Approved Short-Form Clips"]
```

## Repository Structure

```text
.
├── backend/
│   ├── app/
│   │   ├── core/          # configuration and timing instrumentation
│   │   ├── models/        # Pydantic request/response schemas
│   │   ├── services/      # ingest, transcription, Qwen scoring, subtitles, rendering
│   │   ├── utils/         # ROCm / accelerator detection
│   │   ├── workers/       # optional Celery wiring
│   │   ├── main.py        # FastAPI application
│   │   └── storage.py     # file-backed job storage for MVP
│   ├── Dockerfile
│   └── pyproject.toml
├── frontend/
│   ├── src/
│   │   ├── App.jsx        # creator workflow and clip editor
│   │   ├── main.jsx
│   │   └── styles.css
│   ├── Dockerfile
│   └── package.json
├── infra/
│   └── amd-cloud.md       # AMD Developer Cloud deployment guide
├── scripts/
│   └── benchmark.py       # end-to-end API benchmark helper
├── docker-compose.yml
└── README.md
```

## Processing Pipeline

### 1. Video Input

The backend accepts:

- YouTube URL through `POST /api/jobs/youtube`
- Uploaded video file through `POST /api/jobs/upload`

In production, YouTube videos are downloaded with `yt-dlp`. In demo mode, the app can generate a synthetic ffmpeg test video so the workflow can be tested without external downloads.

### 2. Transcription

The transcription service is implemented in `backend/app/services/transcription.py`.

Production target:

- Model: `openai/whisper-large-v3`
- Runtime: Hugging Face Transformers
- Accelerator: PyTorch ROCm on AMD MI300X
- Language goal: Thai and multilingual support

### 3. Highlight Detection

The highlight detector is implemented in `backend/app/services/highlight.py`.

Production target:

- Model: `Qwen/Qwen2.5-7B-Instruct`
- Runtime: vLLM with ROCm backend
- Output: strict structured JSON

Highlight scoring considers:

- questions
- punchlines
- emotional peaks
- key information
- channel niche
- preferred clip style
- target platform
- target clip length

### 4. Multimodal Analysis

The multimodal service boundary is implemented in `backend/app/services/multimodal.py`.

Planned production target:

- Model: `Qwen/Qwen2-VL-7B-Instruct`
- Inputs: sampled video frames, transcript context, and clip candidates
- Visual signals:
  - creator or guest reactions
  - scene changes
  - on-screen text
  - high-motion segments

This is isolated as a replaceable pipeline step so it can be enabled when AMD Cloud resources are available.

### 5. Clip Generation

Clip rendering is implemented in `backend/app/services/clips.py`.

The ffmpeg stage:

- cuts video by selected timestamps
- exports MP4
- creates `.srt` subtitle files
- supports subtitle burn-in
- reformats to 9:16 vertical output for short-form platforms
- includes AMD hardware encoder configuration hooks

### 6. Human-AI Collaborative Editing

The frontend editor lets creators review AI-generated clips and make final decisions:

- adjust start and end timestamps
- edit subtitle text
- delete weak clips
- approve good clips
- regenerate a specific clip
- download the result

## API Overview

| Method | Endpoint | Description |
| --- | --- | --- |
| `GET` | `/health` | Returns service health and accelerator detection. |
| `POST` | `/api/jobs/youtube` | Creates a processing job from a YouTube URL. |
| `POST` | `/api/jobs/upload` | Creates a processing job from an uploaded video. |
| `GET` | `/api/jobs/{job_id}` | Returns status, transcript, clips, timings, and errors. |
| `PATCH` | `/api/jobs/{job_id}/clips/{clip_id}` | Updates trim times, subtitles, approval, or deletion state. |
| `POST` | `/api/jobs/{job_id}/clips/{clip_id}/regenerate` | Re-renders one clip with updated parameters. |
| `GET` | `/api/jobs/{job_id}/clips/{clip_id}/download` | Downloads an exported clip. |

## Local Development

### Requirements

- Python 3.11+
- Node.js 20+
- ffmpeg

### Backend

```bash
cd backend
python -m venv .venv
. .venv/bin/activate
pip install -e .
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```

On Windows PowerShell:

```powershell
cd backend
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```

### Frontend

```bash
cd frontend
npm install
npm run dev
```

Open:

```text
http://localhost:5173
```

### Demo Mode

By default, the project runs in demo mode:

```env
DEMO_MODE=true
```

Demo mode avoids downloading multi-GB AI models and returns deterministic mock transcript/highlight data while still exercising the API, UI, job state, timing logs, subtitle generation, and ffmpeg rendering path.

## AMD Developer Cloud Deployment

See [infra/amd-cloud.md](infra/amd-cloud.md) for a focused deployment guide.

High-level steps:

```bash
git clone https://github.com/JakgritB/ElevenClip.AI.git
cd ElevenClip.AI
cp .env.example .env
```

Edit `.env`:

```env
DEMO_MODE=false
HF_TOKEN=your_huggingface_token
WHISPER_MODEL_ID=openai/whisper-large-v3
QWEN_TEXT_MODEL_ID=Qwen/Qwen2.5-7B-Instruct
QWEN_VL_MODEL_ID=Qwen/Qwen2-VL-7B-Instruct
```

Install the AI/ROCm stack on the AMD instance:

```bash
cd backend
pip install -e ".[ai,rocm-inference]"
```

Start the API:

```bash
uvicorn app.main:app --host 0.0.0.0 --port 8000
```

Validate accelerator detection:

```bash
curl http://localhost:8000/health
```

Expected on AMD Cloud:

- `torch_available: true`
- `cuda_api_available: true`
- `rocm_hip_version` populated
- MI300X visible as the active device

## Docker

```bash
docker compose up --build
```

For AMD Developer Cloud with ROCm extras:

```bash
docker compose build --build-arg INSTALL_EXTRAS=.[ai,rocm-inference] backend
docker compose up
```

The compose file mounts AMD GPU devices (`/dev/kfd`, `/dev/dri`) and uses host IPC for large-model inference.

## Benchmark Plan

The hackathon judges care about technology application and real-world performance. ElevenClip.AI includes step-level timing logs so the demo can show why AMD acceleration matters.

Run a benchmark against a running API:

```bash
python scripts/benchmark.py \
  --api http://localhost:8000 \
  --youtube-url "https://youtube.com/watch?v=..."
```

Recommended benchmark comparison:

| Scenario | Hardware | Expected Purpose |
| --- | --- | --- |
| CPU baseline | CPU-only runtime | Show the pain of long-form video processing without acceleration. |
| AMD GPU run | AMD Instinct MI300X + ROCm | Show high-throughput transcription and Qwen inference. |

Metrics captured:

- input/download time
- transcription time
- highlight detection time
- multimodal analysis time
- clip generation time
- total wall-clock time
- number of clips generated

Demo target:

- input: two-hour creator video
- output: 10 subtitle-ready clips
- goal: under 10 minutes on MI300X

## Submission Assets Checklist

The lablab.ai submission asks for:

- Project title: `ElevenClip.AI`
- Short description
- Long description
- Technology and category tags
- Cover image
- Video presentation
- Slide presentation
- Public GitHub repository
- Demo application platform
- Application URL

Prepared submission docs:

- `docs/SUBMISSION.md` - copy-ready project text for lablab.ai.
- `docs/DEMO_SCRIPT.md` - draft and final recording script.
- `docs/PITCH_DECK.md` - slide outline for the presentation deck.
- `docs/BUILD_IN_PUBLIC.md` - social post drafts and AMD feedback notes.
- `docs/AMD_CREDIT_RUNBOOK.md` - checklist for the first MI300X run.

Recommended tags:

```text
AMD, ROCm, MI300X, AMD Developer Cloud, Vision AI, Multimodal AI, Video AI, Whisper, Qwen, Qwen-VL, Hugging Face, FastAPI, React
```

## Suggested Short Description

```text
ElevenClip.AI turns long-form videos into personalized short-form clips using Whisper, Qwen, Hugging Face, and AMD ROCm on MI300X.
```

## Suggested Long Description

```text
ElevenClip.AI is a human-AI collaborative clip studio for creators. It takes a YouTube URL or uploaded long-form video, transcribes it with Whisper Large V3, uses Qwen2.5 to identify high-engagement highlight moments based on a reusable channel profile, optionally enriches candidates with Qwen2-VL visual analysis, and renders short-form MP4 clips with subtitles using ffmpeg. The React editor lets creators trim, edit subtitles, approve, delete, regenerate, and download final clips. The project is designed for AMD Developer Cloud with ROCm and AMD Instinct MI300X acceleration, demonstrating how high-throughput multimodal AI can reduce hours of manual editing into a fast creator workflow.
```

## Judging Criteria Mapping

### Application of Technology

ElevenClip.AI integrates Whisper, Qwen2.5, Qwen2-VL, Hugging Face, ROCm, vLLM, and AMD Developer Cloud into an end-to-end video processing product.

### Presentation

The demo is designed to be visual and easy to understand: input a long video, watch AI create candidates, edit clips, and download platform-ready MP4 files.

### Business Value

The product targets a real creator economy workflow. Creators, agencies, podcasters, educators, and streamers all need short-form repurposing, and manual editing is expensive.

### Originality

The system goes beyond generic clipping by personalizing highlight selection to a creator's niche, style, language, clip length, and platform. It also preserves human editorial control instead of fully automating final publishing.

## Build-in-Public Plan

The hackathon includes a build-in-public challenge. Suggested updates:

1. Share the architecture and first local demo.
2. Share AMD Cloud/ROCm setup notes and benchmark results.
3. Publish meaningful feedback about ROCm, AMD Developer Cloud, or inference setup.

Suggested hashtags/topics:

```text
#AMDDeveloperHackathon #ROCm #MI300X #HuggingFace #Qwen #VideoAI #MultimodalAI
```

## Roadmap

- Real Whisper Large V3 run on AMD Developer Cloud.
- Real Qwen2.5 vLLM ROCm inference path.
- Qwen2-VL frame sampling and visual scoring.
- Batch export for 10+ clips.
- Subtitle styling presets per platform.
- Creator profile memory and reusable brand presets.
- Hugging Face Space screenshot and richer project media.
- CPU vs MI300X benchmark report after AMD credits arrive.

## License

MIT. See [LICENSE](LICENSE).