Spaces:
Running on Zero
Running on Zero
| # Suno-Clone Platform Architecture — Build Plan | |
| *Compiled 2026-05-18. Target hardware: Apple M5 Max, 128 GB unified memory. Core model decision: ACE-Step 1.5 XL.* | |
| --- | |
| ## Mental model | |
| Suno (and Udio) are not just a song-generation model. They are a **product stack** with at least five distinct AI components and a few non-AI scaffolds. If we want to replicate the product experience, we have to plan for all of them. The song-gen model is the headline; everything else is what makes it usable. | |
| ``` | |
| ┌─────────────────────────────────────┐ | |
| │ Web / mobile UI │ | |
| │ (text prompt + style + lyrics) │ | |
| └─────────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌──────────────────────────────────────────────────────────┐ | |
| │ Orchestrator API │ | |
| │ - prompt routing, queue, billing, history, sharing │ | |
| └──────────────────────────────────────────────────────────┘ | |
| │ │ │ │ | |
| ▼ ▼ ▼ ▼ | |
| ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ | |
| │ Lyrics LLM │ │ Style/Tag │ │ Song-gen │ │ Voice │ | |
| │ (Llama 3.3 │ │ rewriter │ │ router │ │ cloning │ | |
| │ or Qwen) │ │ (small LM) │ │ │ │ (RVC) │ | |
| └─────────────┘ └─────────────┘ └──────┬──────┘ └─────────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────────────────────┐ | |
| │ Model pool (the actual research)│ | |
| │ - ACE-Step 1.5 XL (default) │ | |
| │ - HeartMuLa-MLX (A/B) │ | |
| │ - DiffRhythm 2 (speed tier) │ | |
| │ - YuE on Replicate (intl.) │ | |
| └─────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────────────────────┐ | |
| │ Post-processing pipeline │ | |
| │ - Loudness normalization │ | |
| │ - Demucs stem separation │ | |
| │ - Watermarking (audible+meta) │ | |
| │ - FFmpeg encoding → m4a/mp3 │ | |
| └─────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────────────────────┐ | |
| │ Storage + streaming │ | |
| │ - S3 / R2 origin │ | |
| │ - HLS for in-browser playback │ | |
| │ - CDN │ | |
| └─────────────────────────────────┘ | |
| ``` | |
| --- | |
| ## Component-by-component plan | |
| ### 1. Song generation — primary model | |
| - **ACE-Step 1.5 XL** via [`clockworksquirrel/ace-step-apple-silicon`](https://github.com/clockworksquirrel/ace-step-apple-silicon) on M5 Max. | |
| - Hybrid backend: Qwen3 planner on **MLX**, DiT decoder on **PyTorch MPS**, bf16 throughout. | |
| - Why XL over standard 2B: 128 GB unified eats the cost, and the 4 B DiT closes meaningful quality gaps for paying users. | |
| **LoRA fine-tuning path (when needed):** | |
| - Document the platform's target genres → curate ~50–200 song lyric/audio pairs per genre. | |
| - Train a per-genre LoRA on the 3090-class budget (~1 hour per LoRA per [`ace-step-1.5 README`](https://github.com/ace-step/ACE-Step-1.5)). | |
| - Serve via the same inference pipeline with LoRA hot-swap. | |
| **Fallback / A-B candidates:** | |
| - **HeartMuLa-MLX** ([`Acelogic/heartlib-mlx`](https://github.com/Acelogic/heartlib-mlx)) — 2.1× faster than PyTorch MPS, full numerical parity, Apache 2.0. | |
| - **DiffRhythm 2** ([`ASLP-lab/DiffRhythm`](https://github.com/ASLP-lab/DiffRhythm)) — for the speed/instrumental tier (210 s ceiling acceptable for short-form features like background loops). | |
| - **YuE via Replicate** ([`replicate.com/fofr/yue`](https://replicate.com/fofr/yue/api)) — only for EN+Mandarin+Cantonese+JP+KR generations that ACE-Step underperforms; pay-per-second, no local infra cost. | |
| ### 2. Lyrics generation — separate LLM | |
| The song-gen model takes **lyrics + style** as input, not raw user prompts. Suno's "song description" flow is actually two stages: prompt → lyrics LLM → lyrics → song model. | |
| - Use any decent open LLM running on the user's M5 Max. Candidates: | |
| - **Qwen 2.5 Coder 32B / Qwen 3 7B** — good multilingual chops, fast on MPS via Ollama or mlx-lm. | |
| - **Llama 3.3 70B 4-bit** — premium tier; fits comfortably in 128 GB unified. | |
| - **GPT-OSS-20B** — Apache 2.0, sturdy English. | |
| - Prompt template should: | |
| 1. Parse user style hint into tags (genre, tempo, mood, instruments). | |
| 2. Output structured lyrics with `[verse]`, `[chorus]`, `[bridge]`, `[outro]` markers — these are **exactly the structural tags ACE-Step's `TextEncodeAceStepAudio` consumes**. | |
| 3. Constrain section count and line count to roughly match the target song duration. | |
| **This LLM is independent of the song-gen model and can be swapped freely.** | |
| ### 3. Style / tag normalization | |
| A small classifier or 3 B LM that normalizes user free-text into the controlled-vocabulary tag set the song model was trained on (per genre, BPM bucket, vocal gender, mood). For ACE-Step this maps to its lyric-tag schema; for YuE it maps to `top_200_tags.json`. | |
| Implementation: 1-shot prompt to the lyrics LLM with examples; cache results. | |
| ### 4. Voice cloning / personas (optional but Suno-equivalent) | |
| To match Suno's "Personas" feature: | |
| - **RVC v2** (Retrieval-based Voice Conversion) — open source, fast, runs on MPS, well-supported. | |
| - Train a 5-minute reference clip → 10–15 min on M5 Max → speaker embedding. | |
| - Apply to the generated vocal stem (Demucs-extracted) → remix. | |
| ACE-Step's **ICL mode** (in-context learning from a reference clip) and YuE's ICL variants partly cover this too, but RVC gives explicit per-speaker control. | |
| ### 5. Stem separation | |
| For Suno's "download stems" feature: | |
| - **Demucs v4 / HTDemucs** — open source, Apache 2.0, runs on MPS, separates into vocals / drums / bass / other. | |
| - Already bundled in [`fspecii/ace-step-ui`](https://github.com/fspecii/ace-step-ui). | |
| ### 6. Mastering / loudness normalization | |
| - **pyloudnorm** for LUFS normalization to streaming spec (-14 LUFS Spotify, -16 for AirPods). | |
| - **ffmpeg-normalize** as a CLI wrapper. | |
| - **Optional: TBProAudio mvMeter / Voxengo Span equivalents** via web-audio for UI metering. | |
| ### 7. Watermarking + content credentials | |
| This is a **legal must-have** for any 2026 generative-music product (training-data lawsuits against Suno/Udio set the precedent). | |
| - **Inaudible audio watermark**: AudioSeal or SilentCipher — open-source, Meta-built, survives MP3 transcoding. | |
| - **C2PA metadata**: sign the m4a with model name + version + prompt + timestamp via the C2PA SDK. | |
| - **Visible "AI-generated" tag** in UI per the YuE model card's recommendation (and increasingly per platform policy). | |
| ### 8. Storage and streaming | |
| - **S3-compatible object store** (R2, Backblaze B2, or self-hosted MinIO on the M5 Max if dev-only). | |
| - **HLS encoding pipeline**: ffmpeg → m3u8 + 4 s segments; serve via NGINX or Cloudflare. | |
| - For local dev, plain m4a + range requests are fine. | |
| ### 9. Orchestrator API | |
| - **FastAPI** for the request-handling layer. | |
| - **Redis Streams** or **Hatchet** for the generation queue (songs are 30 s–2 min jobs on M5 Max — non-trivial latency, must be async). | |
| - **PostgreSQL** for users, songs, lyrics, LoRAs, billing. | |
| - **Server-Sent Events** for progress streaming back to the UI ("planner stage", "DiT denoising step 14/27", "mastering..."). | |
| ### 10. Frontend | |
| - **Next.js 16** + Cache Components for the user dashboard / library. | |
| - **Wavesurfer.js** for waveform display and scrubbing. | |
| - **Tone.js** for any in-browser preview / mixing. | |
| - Auth via Clerk or Auth0 — the user's portfolio revamp may already include this. | |
| --- | |
| ## Build order (incremental milestones) | |
| | Milestone | Scope | Validates | | |
| |---|---|---| | |
| | **M0 — Spike** | Get ACE-Step 1.5 XL running locally via clockworksquirrel fork; generate one 30 s song end-to-end | Hardware compatibility, RTF on M5 Max | | |
| | **M1 — CLI MVP** | Wrap in a Python CLI: `genmusic --prompt "..." --lyrics "..." --out song.m4a` | Headless generation, mastering chain, file output | | |
| | **M2 — Local UI** | Replace UI with `fspecii/ace-step-ui` initially (fastest path); add Demucs stem download | Browser flow, multi-song library, LAN access | | |
| | **M3 — Lyrics LLM integration** | Plug Qwen 3 / Llama 3.3 as the lyrics generator; produce structured lyrics from a one-line prompt | Suno-equivalent prompt UX | | |
| | **M4 — Multi-model router** | Add HeartMuLa-MLX as alternate; add Replicate YuE as multilingual fallback; user can pick or auto-route | A/B capability, breadth | | |
| | **M5 — LoRA pipeline** | First custom LoRA on a target genre (e.g., user's preferred style); hot-swap at inference | Differentiation vs Suno | | |
| | **M6 — Production wrapper** | FastAPI + Postgres + queue + auth + watermarking + C2PA signing | Real product surface | | |
| | **M7 — Deploy** | Move heavy inference behind a rented A100 endpoint for paid users; keep M5 Max for free tier / personal use | Paid-tier economics | | |
| --- | |
| ## Open questions for the user before M0 | |
| 1. **Commercial intent.** Is this a personal portfolio project (research mode → SongGeneration 2 is fair game) or a real SaaS (must stay Apache/MIT)? The license map changes drastically. | |
| 2. **Target audience.** Western pop (where Suno still wins polish) vs world music / experimental genres (where ACE-Step / YuE compete fairly)? | |
| 3. **Latency target.** Suno generates in ~30 s; users tolerate up to 90 s. ACE-Step on M5 Max hits this; YuE local does not. | |
| 4. **Hosting plan.** Local-only for personal use? Or eventually paid tier on rented GPU? | |
| 5. **Vocal cloning.** Is Suno-style "Persona" upload a must-have v1 feature, or v2? | |
| 6. **Catalog / training data.** Any in-house licensed song catalog for LoRA fine-tuning, or strictly the public-domain model out of the box? | |
| --- | |
| ## Risks and mitigations | |
| | Risk | Likelihood | Mitigation | | |
| |---|---|---| | |
| | MPS regression in a future PyTorch release breaks ACE-Step | medium | Pin torch version; keep CPU fallback path. | | |
| | ACE-Step releases v2 with breaking API mid-build | medium | Wrap inference in a thin adapter; abstract model behind a single `Generator.generate()` interface. | | |
| | Vendor PER claims (HeartMuLa, LeVo) overstated → quality disappointment | medium | Run internal blind A/B on 20+ prompts before featuring a model in the UI. | | |
| | Output watermark stripped by transcoding | low | Use AudioSeal which survives MP3; double-stamp with C2PA metadata. | | |
| | Lyrics LLM hallucinates copyrighted hooks | medium | Run a similarity check against an embeddings index of known songs; flag for human review. | | |
| | Training-data IP suit (Suno-style) | low for derivative usage | Use models with documented public-data training (ACE-Step's paper is reasonably transparent); avoid Tencent's non-commercial weights. | | |
| | MPS OOM on long sequences | low (128 GB) | `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0`; chunk generation; offload non-active LoRAs. | | |
| --- | |
| ## Why ACE-Step 1.5 XL is the foundation (not just a model pick) | |
| This is worth saying explicitly. Choosing the base model determines: | |
| 1. **Inference budget and unit economics** — ACE-Step is the only model where <2 s/song on A100 makes a paid tier economically obvious. | |
| 2. **Mac developer ergonomics** — first-class MPS means the user can iterate on the M5 Max for weeks without renting cloud GPU. | |
| 3. **License-clean output ownership** — MIT means users own their songs unambiguously. | |
| 4. **Future-proof on multilingual** — 50+ languages out of the box matters if the platform grows beyond an English audience. | |
| 5. **LoRA personalization is the differentiator** — fine-tuning support that works on MPS lets the user ship genre-specialist sub-models that Suno can't, because Suno's weights are locked. | |
| 6. **Production deployments exist** — AMD vendor-backed, `fspecii/ace-step-ui` running at scale, multiple SaaS already on the open weights. This is not betting on a research artifact. | |
| The compound effect of those six is why ACE-Step is recommended as the platform foundation rather than just "the model to start with." | |