docs: add hackathon-ready project README

Document the problem, solution, AMD ROCm and MI300X alignment, Hugging Face and Qwen integrations, architecture, local setup, AMD Cloud deployment path, benchmark plan, and lablab.ai submission checklist.

README.md

@@ -0,0 +1,480 @@
+# ElevenClip.AI
+
+ElevenClip.AI is an AI-powered clip studio for turning long-form videos into personalized short-form content for TikTok, YouTube Shorts, and Instagram Reels.
+
+This project is built for the **AMD Developer Hackathon** on lablab.ai, targeting **Track 3: Vision & Multimodal AI**. The system is designed to run on **AMD Developer Cloud** with **ROCm** and **AMD Instinct MI300X** acceleration, while using **Hugging Face** as the model hub/deployment layer and **Qwen** models for profile-aware highlight reasoning.
+
+## One-Sentence Pitch
+
+ElevenClip.AI helps creators convert long videos into ready-to-edit short clips by combining Whisper transcription, Qwen highlight detection, optional Qwen-VL visual understanding, ffmpeg rendering, and a human-in-the-loop clip editor.
+
+## Problem
+
+Long-form creators, podcasters, educators, streamers, and marketing teams often publish hours of video but still need short clips for modern discovery platforms.
+
+The manual workflow is painful:
+
+- Watch the full video.
+- Find high-retention moments.
+- Trim each clip.
+- Rewrite subtitles.
+- Reframe to vertical 9:16.
+- Export platform-ready MP4 files.
+
+For a two-hour video, this can take several hours of editing time. The bottleneck is not just cutting video; it is understanding which moments match the creator's audience, channel style, language, and target platform.
+
+## Solution
+
+ElevenClip.AI automates the first pass of short-form production:
+
+1. The creator sets up a reusable channel profile.
+2. The creator provides a YouTube URL or uploads a video file.
+3. Whisper Large V3 transcribes the video, including Thai and multilingual speech.
+4. Qwen2.5 analyzes the transcript and scores candidate highlights based on engagement potential and the creator profile.
+5. Optional Qwen2-VL analysis can enrich the scores with visual signals such as reactions, scene changes, and on-screen text.
+6. ffmpeg renders vertical clips with subtitle files and burn-in support.
+7. The React editor lets the human approve, delete, trim, regenerate, and edit subtitles before download.
+
+The product is intentionally human-AI collaborative: AI finds and prepares the clips quickly, while the creator keeps editorial control.
+
+## Hackathon Alignment
+
+### Track
+
+**Track 3: Vision & Multimodal AI**
+
+ElevenClip.AI processes multiple media types:
+
+- Audio: speech transcription with Whisper Large V3.
+- Text: transcript reasoning and highlight ranking with Qwen2.5.
+- Video: frame-aware multimodal analysis with Qwen2-VL as the next pipeline stage.
+- Rendered media: ffmpeg exports platform-ready video clips.
+
+### AMD Technology
+
+The production target is AMD Developer Cloud:
+
+- **AMD Instinct MI300X** for high-throughput model inference.
+- **ROCm 6.x** as the GPU software stack.
+- **PyTorch with ROCm support** for Whisper inference.
+- **vLLM ROCm backend** for fast Qwen2.5 inference.
+- **Optimum-AMD** as an optimization path for Hugging Face models on AMD hardware.
+- **ffmpeg hardware acceleration hooks** for faster video encoding where available.
+
+The app has a local `DEMO_MODE=true` path so judges and teammates can inspect the UI/API without downloading large models. On AMD Developer Cloud, set `DEMO_MODE=false` to activate the real model stack.
+
+### Hugging Face Integration
+
+Hugging Face is used as the model hub and deployment layer:
+
+- `openai/whisper-large-v3` for transcription.
+- `Qwen/Qwen2.5-7B-Instruct` for highlight analysis.
+- `Qwen/Qwen2-VL-7B-Instruct` for multimodal video understanding.
+- Planned public Hugging Face Space for the hackathon demo UI.
+
+### Qwen Integration
+
+Qwen is not used as a generic chatbot. It is part of the core product logic:
+
+- Reads timestamped transcript segments.
+- Considers creator profile settings.
+- Scores engagement potential.
+- Explains why a segment should become a clip.
+- Returns structured JSON with timestamps, titles, scores, reasons, and subtitle text.
+
+## Current MVP Features
+
+- Channel profile onboarding:
+  - niche
+  - preferred clip style
+  - preferred clip length
+  - primary language
+  - target platform
+- YouTube URL ingestion through `yt-dlp`.
+- Direct video upload endpoint.
+- Whisper transcription service boundary.
+- Qwen highlight detection service boundary.
+- Optional Qwen2-VL multimodal analysis service boundary.
+- ffmpeg clip generation with subtitle file creation.
+- Vertical 9:16 export path for TikTok, Shorts, and Reels.
+- Human-AI review UI:
+  - trim start/end
+  - edit subtitles inline
+  - approve clips
+  - delete clips
+  - regenerate a clip
+  - download MP4 output
+- Timing logs for benchmark demos.
+- Docker and AMD Cloud deployment notes.
+
+## Architecture
+
+```mermaid
+flowchart LR
+  A["Creator Profile"] --> D["Qwen2.5 Highlight Scoring"]
+  B["YouTube URL"] --> C["yt-dlp / Video Input"]
+  B2["Uploaded Video"] --> C
+  C --> W["Whisper Large V3 Transcription"]
+  W --> D
+  C --> V["Qwen2-VL Visual Analysis (Optional)"]
+  D --> R["Clip Plan JSON"]
+  V --> R
+  R --> F["ffmpeg Clip Rendering + Subtitles"]
+  F --> E["React Human-AI Editor"]
+  E --> O["Approved Short-Form Clips"]
+```
+
+## Repository Structure
+
+```text
+.
+├── backend/
+│   ├── app/
+│   │   ├── core/          # configuration and timing instrumentation
+│   │   ├── models/        # Pydantic request/response schemas
+│   │   ├── services/      # ingest, transcription, Qwen scoring, subtitles, rendering
+│   │   ├── utils/         # ROCm / accelerator detection
+│   │   ├── workers/       # optional Celery wiring
+│   │   ├── main.py        # FastAPI application
+│   │   └── storage.py     # file-backed job storage for MVP
+│   ├── Dockerfile
+│   └── pyproject.toml
+├── frontend/
+│   ├── src/
+│   │   ├── App.jsx        # creator workflow and clip editor
+│   │   ├── main.jsx
+│   │   └── styles.css
+│   ├── Dockerfile
+│   └── package.json
+├── infra/
+│   └── amd-cloud.md       # AMD Developer Cloud deployment guide
+├── scripts/
+│   └── benchmark.py       # end-to-end API benchmark helper
+├── docker-compose.yml
+└── README.md
+```
+
+## Processing Pipeline
+
+### 1. Video Input
+
+The backend accepts:
+
+- YouTube URL through `POST /api/jobs/youtube`
+- Uploaded video file through `POST /api/jobs/upload`
+
+In production, YouTube videos are downloaded with `yt-dlp`. In demo mode, the app can generate a synthetic ffmpeg test video so the workflow can be tested without external downloads.
+
+### 2. Transcription
+
+The transcription service is implemented in `backend/app/services/transcription.py`.
+
+Production target:
+
+- Model: `openai/whisper-large-v3`
+- Runtime: Hugging Face Transformers
+- Accelerator: PyTorch ROCm on AMD MI300X
+- Language goal: Thai and multilingual support
+
+### 3. Highlight Detection
+
+The highlight detector is implemented in `backend/app/services/highlight.py`.
+
+Production target:
+
+- Model: `Qwen/Qwen2.5-7B-Instruct`
+- Runtime: vLLM with ROCm backend
+- Output: strict structured JSON
+
+Highlight scoring considers:
+
+- questions
+- punchlines
+- emotional peaks
+- key information
+- channel niche
+- preferred clip style
+- target platform
+- target clip length
+
+### 4. Multimodal Analysis
+
+The multimodal service boundary is implemented in `backend/app/services/multimodal.py`.
+
+Planned production target:
+
+- Model: `Qwen/Qwen2-VL-7B-Instruct`
+- Inputs: sampled video frames, transcript context, and clip candidates
+- Visual signals:
+  - creator or guest reactions
+  - scene changes
+  - on-screen text
+  - high-motion segments
+
+This is isolated as a replaceable pipeline step so it can be enabled when AMD Cloud resources are available.
+
+### 5. Clip Generation
+
+Clip rendering is implemented in `backend/app/services/clips.py`.
+
+The ffmpeg stage:
+
+- cuts video by selected timestamps
+- exports MP4
+- creates `.srt` subtitle files
+- supports subtitle burn-in
+- reformats to 9:16 vertical output for short-form platforms
+- includes AMD hardware encoder configuration hooks
+
+### 6. Human-AI Collaborative Editing
+
+The frontend editor lets creators review AI-generated clips and make final decisions:
+
+- adjust start and end timestamps
+- edit subtitle text
+- delete weak clips
+- approve good clips
+- regenerate a specific clip
+- download the result
+
+## API Overview
+
+| Method | Endpoint | Description |
+| --- | --- | --- |
+| `GET` | `/health` | Returns service health and accelerator detection. |
+| `POST` | `/api/jobs/youtube` | Creates a processing job from a YouTube URL. |
+| `POST` | `/api/jobs/upload` | Creates a processing job from an uploaded video. |
+| `GET` | `/api/jobs/{job_id}` | Returns status, transcript, clips, timings, and errors. |
+| `PATCH` | `/api/jobs/{job_id}/clips/{clip_id}` | Updates trim times, subtitles, approval, or deletion state. |
+| `POST` | `/api/jobs/{job_id}/clips/{clip_id}/regenerate` | Re-renders one clip with updated parameters. |
+| `GET` | `/api/jobs/{job_id}/clips/{clip_id}/download` | Downloads an exported clip. |
+
+## Local Development
+
+### Requirements
+
+- Python 3.11+
+- Node.js 20+
+- ffmpeg
+
+### Backend
+
+```bash
+cd backend
+python -m venv .venv
+. .venv/bin/activate
+pip install -e .
+uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+```
+
+On Windows PowerShell:
+
+```powershell
+cd backend
+python -m venv .venv
+.\.venv\Scripts\Activate.ps1
+pip install -e .
+uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+```
+
+### Frontend
+
+```bash
+cd frontend
+npm install
+npm run dev
+```
+
+Open:
+
+```text
+http://localhost:5173
+```
+
+### Demo Mode
+
+By default, the project runs in demo mode:
+
+```env
+DEMO_MODE=true
+```
+
+Demo mode avoids downloading multi-GB AI models and returns deterministic mock transcript/highlight data while still exercising the API, UI, job state, timing logs, subtitle generation, and ffmpeg rendering path.
+
+## AMD Developer Cloud Deployment
+
+See [infra/amd-cloud.md](infra/amd-cloud.md) for a focused deployment guide.
+
+High-level steps:
+
+```bash
+git clone https://github.com/JakgritB/ElevenClip.AI.git
+cd ElevenClip.AI
+cp .env.example .env
+```
+
+Edit `.env`:
+
+```env
+DEMO_MODE=false
+HF_TOKEN=your_huggingface_token
+WHISPER_MODEL_ID=openai/whisper-large-v3
+QWEN_TEXT_MODEL_ID=Qwen/Qwen2.5-7B-Instruct
+QWEN_VL_MODEL_ID=Qwen/Qwen2-VL-7B-Instruct
+```
+
+Install the AI/ROCm stack on the AMD instance:
+
+```bash
+cd backend
+pip install -e ".[ai,rocm-inference]"
+```
+
+Start the API:
+
+```bash
+uvicorn app.main:app --host 0.0.0.0 --port 8000
+```
+
+Validate accelerator detection:
+
+```bash
+curl http://localhost:8000/health
+```
+
+Expected on AMD Cloud:
+
+- `torch_available: true`
+- `cuda_api_available: true`
+- `rocm_hip_version` populated
+- MI300X visible as the active device
+
+## Docker
+
+```bash
+docker compose up --build
+```
+
+For AMD Developer Cloud with ROCm extras:
+
+```bash
+docker compose build --build-arg INSTALL_EXTRAS=.[ai,rocm-inference] backend
+docker compose up
+```
+
+The compose file mounts AMD GPU devices (`/dev/kfd`, `/dev/dri`) and uses host IPC for large-model inference.
+
+## Benchmark Plan
+
+The hackathon judges care about technology application and real-world performance. ElevenClip.AI includes step-level timing logs so the demo can show why AMD acceleration matters.
+
+Run a benchmark against a running API:
+
+```bash
+python scripts/benchmark.py \
+  --api http://localhost:8000 \
+  --youtube-url "https://youtube.com/watch?v=..."
+```
+
+Recommended benchmark comparison:
+
+| Scenario | Hardware | Expected Purpose |
+| --- | --- | --- |
+| CPU baseline | CPU-only runtime | Show the pain of long-form video processing without acceleration. |
+| AMD GPU run | AMD Instinct MI300X + ROCm | Show high-throughput transcription and Qwen inference. |
+
+Metrics captured:
+
+- input/download time
+- transcription time
+- highlight detection time
+- multimodal analysis time
+- clip generation time
+- total wall-clock time
+- number of clips generated
+
+Demo target:
+
+- input: two-hour creator video
+- output: 10 subtitle-ready clips
+- goal: under 10 minutes on MI300X
+
+## Submission Assets Checklist
+
+The lablab.ai submission asks for:
+
+- Project title: `ElevenClip.AI`
+- Short description
+- Long description
+- Technology and category tags
+- Cover image
+- Video presentation
+- Slide presentation
+- Public GitHub repository
+- Demo application platform
+- Application URL
+
+Recommended tags:
+
+```text
+AMD, ROCm, MI300X, AMD Developer Cloud, Vision AI, Multimodal AI, Video AI, Whisper, Qwen, Qwen-VL, Hugging Face, FastAPI, React
+```
+
+## Suggested Short Description
+
+```text
+ElevenClip.AI turns long-form videos into personalized short-form clips using Whisper, Qwen, Hugging Face, and AMD ROCm on MI300X.
+```
+
+## Suggested Long Description
+
+```text
+ElevenClip.AI is a human-AI collaborative clip studio for creators. It takes a YouTube URL or uploaded long-form video, transcribes it with Whisper Large V3, uses Qwen2.5 to identify high-engagement highlight moments based on a reusable channel profile, optionally enriches candidates with Qwen2-VL visual analysis, and renders short-form MP4 clips with subtitles using ffmpeg. The React editor lets creators trim, edit subtitles, approve, delete, regenerate, and download final clips. The project is designed for AMD Developer Cloud with ROCm and AMD Instinct MI300X acceleration, demonstrating how high-throughput multimodal AI can reduce hours of manual editing into a fast creator workflow.
+```
+
+## Judging Criteria Mapping
+
+### Application of Technology
+
+ElevenClip.AI integrates Whisper, Qwen2.5, Qwen2-VL, Hugging Face, ROCm, vLLM, and AMD Developer Cloud into an end-to-end video processing product.
+
+### Presentation
+
+The demo is designed to be visual and easy to understand: input a long video, watch AI create candidates, edit clips, and download platform-ready MP4 files.
+
+### Business Value
+
+The product targets a real creator economy workflow. Creators, agencies, podcasters, educators, and streamers all need short-form repurposing, and manual editing is expensive.
+
+### Originality
+
+The system goes beyond generic clipping by personalizing highlight selection to a creator's niche, style, language, clip length, and platform. It also preserves human editorial control instead of fully automating final publishing.
+
+## Build-in-Public Plan
+
+The hackathon includes a build-in-public challenge. Suggested updates:
+
+1. Share the architecture and first local demo.
+2. Share AMD Cloud/ROCm setup notes and benchmark results.
+3. Publish meaningful feedback about ROCm, AMD Developer Cloud, or inference setup.
+
+Suggested hashtags/topics:
+
+```text
+#AMDDeveloperHackathon #ROCm #MI300X #HuggingFace #Qwen #VideoAI #MultimodalAI
+```
+
+## Roadmap
+
+- Real Whisper Large V3 run on AMD Developer Cloud.
+- Real Qwen2.5 vLLM ROCm inference path.
+- Qwen2-VL frame sampling and visual scoring.
+- Batch export for 10+ clips.
+- Subtitle styling presets per platform.
+- Creator profile memory and reusable brand presets.
+- Hugging Face Space deployment.
+- CPU vs MI300X benchmark report.
+
+## License
+
+MIT. See [LICENSE](LICENSE).