research/07_platform_architecture.md

Suno-Clone Platform Architecture — Build Plan

Compiled 2026-05-18. Target hardware: Apple M5 Max, 128 GB unified memory. Core model decision: ACE-Step 1.5 XL.

---

Mental model

Suno (and Udio) are not just a song-generation model. They are a product stack with at least five distinct AI components and a few non-AI scaffolds. If we want to replicate the product experience, we have to plan for all of them. The song-gen model is the headline; everything else is what makes it usable.

                ┌─────────────────────────────────────┐
                │           Web / mobile UI           │
                │  (text prompt + style + lyrics)     │
                └─────────────────────────────────────┘
                                │
                                ▼
┌──────────────────────────────────────────────────────────┐
│                    Orchestrator API                       │
│   - prompt routing, queue, billing, history, sharing      │
└──────────────────────────────────────────────────────────┘
                  │            │            │            │
                  ▼            ▼            ▼            ▼
        ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
        │  Lyrics LLM │ │  Style/Tag  │ │  Song-gen   │ │  Voice       │
        │  (Llama 3.3 │ │  rewriter   │ │  router     │ │  cloning     │
        │   or Qwen)  │ │  (small LM) │ │             │ │  (RVC)       │
        └─────────────┘ └─────────────┘ └──────┬──────┘ └─────────────┘
                                               │
                                               ▼
                            ┌─────────────────────────────────┐
                            │  Model pool (the actual research)│
                            │   - ACE-Step 1.5 XL (default)   │
                            │   - HeartMuLa-MLX (A/B)         │
                            │   - DiffRhythm 2 (speed tier)   │
                            │   - YuE on Replicate (intl.)    │
                            └─────────────────────────────────┘
                                               │
                                               ▼
                            ┌─────────────────────────────────┐
                            │   Post-processing pipeline      │
                            │   - Loudness normalization      │
                            │   - Demucs stem separation      │
                            │   - Watermarking (audible+meta) │
                            │   - FFmpeg encoding → m4a/mp3   │
                            └─────────────────────────────────┘
                                               │
                                               ▼
                            ┌─────────────────────────────────┐
                            │   Storage + streaming           │
                            │   - S3 / R2 origin              │
                            │   - HLS for in-browser playback │
                            │   - CDN                         │
                            └─────────────────────────────────┘

---

Component-by-component plan

1. Song generation — primary model

• ACE-Step 1.5 XL via clockworksquirrel/ace-step-apple-silicon on M5 Max.
• Hybrid backend: Qwen3 planner on MLX, DiT decoder on PyTorch MPS, bf16 throughout.
• Why XL over standard 2B: 128 GB unified eats the cost, and the 4 B DiT closes meaningful quality gaps for paying users.

LoRA fine-tuning path (when needed):

• Document the platform's target genres → curate ~50–200 song lyric/audio pairs per genre.
• Train a per-genre LoRA on the 3090-class budget (~1 hour per LoRA per ace-step-1.5 README).
• Serve via the same inference pipeline with LoRA hot-swap.

Fallback / A-B candidates:

• HeartMuLa-MLX (Acelogic/heartlib-mlx) — 2.1× faster than PyTorch MPS, full numerical parity, Apache 2.0.
• DiffRhythm 2 (ASLP-lab/DiffRhythm) — for the speed/instrumental tier (210 s ceiling acceptable for short-form features like background loops).
• YuE via Replicate (replicate.com/fofr/yue) — only for EN+Mandarin+Cantonese+JP+KR generations that ACE-Step underperforms; pay-per-second, no local infra cost.

2. Lyrics generation — separate LLM

The song-gen model takes lyrics + style as input, not raw user prompts. Suno's "song description" flow is actually two stages: prompt → lyrics LLM → lyrics → song model.

• Use any decent open LLM running on the user's M5 Max. Candidates:
  • Qwen 2.5 Coder 32B / Qwen 3 7B — good multilingual chops, fast on MPS via Ollama or mlx-lm.
  • Llama 3.3 70B 4-bit — premium tier; fits comfortably in 128 GB unified.
  • GPT-OSS-20B — Apache 2.0, sturdy English.
• Prompt template should:
  1. Parse user style hint into tags (genre, tempo, mood, instruments).
  2. Output structured lyrics with [verse], [chorus], [bridge], [outro] markers — these are exactly the structural tags ACE-Step's TextEncodeAceStepAudio consumes.
  3. Constrain section count and line count to roughly match the target song duration.

This LLM is independent of the song-gen model and can be swapped freely.

3. Style / tag normalization

A small classifier or 3 B LM that normalizes user free-text into the controlled-vocabulary tag set the song model was trained on (per genre, BPM bucket, vocal gender, mood). For ACE-Step this maps to its lyric-tag schema; for YuE it maps to top_200_tags.json.

Implementation: 1-shot prompt to the lyrics LLM with examples; cache results.

4. Voice cloning / personas (optional but Suno-equivalent)

To match Suno's "Personas" feature:

• RVC v2 (Retrieval-based Voice Conversion) — open source, fast, runs on MPS, well-supported.
• Train a 5-minute reference clip → 10–15 min on M5 Max → speaker embedding.
• Apply to the generated vocal stem (Demucs-extracted) → remix.

ACE-Step's ICL mode (in-context learning from a reference clip) and YuE's ICL variants partly cover this too, but RVC gives explicit per-speaker control.

5. Stem separation

For Suno's "download stems" feature:

• Demucs v4 / HTDemucs — open source, Apache 2.0, runs on MPS, separates into vocals / drums / bass / other.
• Already bundled in fspecii/ace-step-ui.

6. Mastering / loudness normalization

• pyloudnorm for LUFS normalization to streaming spec (-14 LUFS Spotify, -16 for AirPods).
• ffmpeg-normalize as a CLI wrapper.
• Optional: TBProAudio mvMeter / Voxengo Span equivalents via web-audio for UI metering.

7. Watermarking + content credentials

This is a legal must-have for any 2026 generative-music product (training-data lawsuits against Suno/Udio set the precedent).

• Inaudible audio watermark: AudioSeal or SilentCipher — open-source, Meta-built, survives MP3 transcoding.
• C2PA metadata: sign the m4a with model name + version + prompt + timestamp via the C2PA SDK.
• Visible "AI-generated" tag in UI per the YuE model card's recommendation (and increasingly per platform policy).

8. Storage and streaming

• S3-compatible object store (R2, Backblaze B2, or self-hosted MinIO on the M5 Max if dev-only).
• HLS encoding pipeline: ffmpeg → m3u8 + 4 s segments; serve via NGINX or Cloudflare.
• For local dev, plain m4a + range requests are fine.

9. Orchestrator API

• FastAPI for the request-handling layer.
• Redis Streams or Hatchet for the generation queue (songs are 30 s–2 min jobs on M5 Max — non-trivial latency, must be async).
• PostgreSQL for users, songs, lyrics, LoRAs, billing.
• Server-Sent Events for progress streaming back to the UI ("planner stage", "DiT denoising step 14/27", "mastering...").

10. Frontend

• Next.js 16 + Cache Components for the user dashboard / library.
• Wavesurfer.js for waveform display and scrubbing.
• Tone.js for any in-browser preview / mixing.
• Auth via Clerk or Auth0 — the user's portfolio revamp may already include this.

---

Build order (incremental milestones)

Milestone	Scope	Validates
M0 — Spike	Get ACE-Step 1.5 XL running locally via clockworksquirrel fork; generate one 30 s song end-to-end	Hardware compatibility, RTF on M5 Max
M1 — CLI MVP	Wrap in a Python CLI: genmusic --prompt "..." --lyrics "..." --out song.m4a	Headless generation, mastering chain, file output
M2 — Local UI	Replace UI with fspecii/ace-step-ui initially (fastest path); add Demucs stem download	Browser flow, multi-song library, LAN access
M3 — Lyrics LLM integration	Plug Qwen 3 / Llama 3.3 as the lyrics generator; produce structured lyrics from a one-line prompt	Suno-equivalent prompt UX
M4 — Multi-model router	Add HeartMuLa-MLX as alternate; add Replicate YuE as multilingual fallback; user can pick or auto-route	A/B capability, breadth
M5 — LoRA pipeline	First custom LoRA on a target genre (e.g., user's preferred style); hot-swap at inference	Differentiation vs Suno
M6 — Production wrapper	FastAPI + Postgres + queue + auth + watermarking + C2PA signing	Real product surface
M7 — Deploy	Move heavy inference behind a rented A100 endpoint for paid users; keep M5 Max for free tier / personal use	Paid-tier economics

---

Open questions for the user before M0

1. Commercial intent. Is this a personal portfolio project (research mode → SongGeneration 2 is fair game) or a real SaaS (must stay Apache/MIT)? The license map changes drastically.
2. Target audience. Western pop (where Suno still wins polish) vs world music / experimental genres (where ACE-Step / YuE compete fairly)?
3. Latency target. Suno generates in ~30 s; users tolerate up to 90 s. ACE-Step on M5 Max hits this; YuE local does not.
4. Hosting plan. Local-only for personal use? Or eventually paid tier on rented GPU?
5. Vocal cloning. Is Suno-style "Persona" upload a must-have v1 feature, or v2?
6. Catalog / training data. Any in-house licensed song catalog for LoRA fine-tuning, or strictly the public-domain model out of the box?

---

Risks and mitigations

Risk	Likelihood	Mitigation
MPS regression in a future PyTorch release breaks ACE-Step	medium	Pin torch version; keep CPU fallback path.
ACE-Step releases v2 with breaking API mid-build	medium	Wrap inference in a thin adapter; abstract model behind a single Generator.generate() interface.
Vendor PER claims (HeartMuLa, LeVo) overstated → quality disappointment	medium	Run internal blind A/B on 20+ prompts before featuring a model in the UI.
Output watermark stripped by transcoding	low	Use AudioSeal which survives MP3; double-stamp with C2PA metadata.
Lyrics LLM hallucinates copyrighted hooks	medium	Run a similarity check against an embeddings index of known songs; flag for human review.
Training-data IP suit (Suno-style)	low for derivative usage	Use models with documented public-data training (ACE-Step's paper is reasonably transparent); avoid Tencent's non-commercial weights.
MPS OOM on long sequences	low (128 GB)	PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0; chunk generation; offload non-active LoRAs.

---

Why ACE-Step 1.5 XL is the foundation (not just a model pick)

This is worth saying explicitly. Choosing the base model determines:

1. Inference budget and unit economics — ACE-Step is the only model where <2 s/song on A100 makes a paid tier economically obvious.
2. Mac developer ergonomics — first-class MPS means the user can iterate on the M5 Max for weeks without renting cloud GPU.
3. License-clean output ownership — MIT means users own their songs unambiguously.
4. Future-proof on multilingual — 50+ languages out of the box matters if the platform grows beyond an English audience.
5. LoRA personalization is the differentiator — fine-tuning support that works on MPS lets the user ship genre-specialist sub-models that Suno can't, because Suno's weights are locked.
6. Production deployments exist — AMD vendor-backed, fspecii/ace-step-ui running at scale, multiple SaaS already on the open weights. This is not betting on a research artifact.

The compound effect of those six is why ACE-Step is recommended as the platform foundation rather than just "the model to start with."