CloneCharter

CloneCharter is an encoder-decoder Transformer that takes an audio file and generates a playable Clone Hero chart (.chart format). Given a song, it automatically transcribes guitar, bass, or drum notes at any difficulty level.

---

Model Architecture

                        ┌──────────────────────────────────┐
  Audio (MP3/OGG/WAV)   │         ENCODER                  │
       │                │                                  │
       ▼                │  AudioCNNFrontEnd                │
  Demucs stem           │  (Conv2D × 3, stride-based)      │
  separation            │  [B, 512 mels, T] → [B, T/16, d] │
       │                │           +                      │
       ▼                │  ConditioningEncoder             │
  Log-mel               │  7 prefix tokens from metadata   │
  spectrogram           │  (BPM, TS, instrument, difficulty,│
  [512 mels × T]        │   resolution, offset, MERT emb)  │
       │                │           +                      │
  MERT embedding ──────►│  8-layer TransformerEncoder       │
  [768-d]               │  (pre-norm, bidirectional)       │
                        └──────────────┬───────────────────┘
                                       │ enc_out [B, 512, 768]
                        ┌──────────────▼───────────────────┐
                        │         DECODER                  │
                        │                                  │
                        │  12-layer autoregressive         │
                        │  TransformerDecoder              │
                        │  (causal self-attn + cross-attn) │
                        │           +                      │
                        │  Output projection (weight-tied  │
                        │  with token embedding)           │
                        └──────────────┬───────────────────┘
                                       │
                                       ▼
                              Token sequence
                        (Beat / Pitch / Duration tokens)
                                       │
                                       ▼
                               notes.chart file

Key hyperparameters

Parameter	Value
d_model	768
Encoder layers	8
Decoder layers	12
Attention heads	12
FFN dim	3 072
Vocabulary size	693
Max encoder length	512 tokens
Max decoder length	2 048 tokens
Mixed precision	bf16

---

Tokenization

The tokenizer (CloneHeroTokenizer) uses a hierarchical beat-based vocabulary:

• Special tokens: <BOS>, <EOS>, <UNK>, <PAD>
• Instrument tokens: <Guitar>, <Bass>, <Drums>
• Difficulty tokens: <Expert>, <Hard>, <Medium>, <Easy>
• Temporal position: <Minute_N>, <Beat_N>, <Beatshift_N> (sub-beat 1/32 grid)
• Pitch: <Pitch_N> (guitar/bass, 5-fret buttons 0-4) or <DrumsPitch_N>
• Duration: <Beatshift_N> (sustain in 1/32 beat units)

Each note is encoded as a 6-token block:

<Beatshift> <NoteType> <Pitch> <Minute> <Beat> <DurationBeatshift>

---

Audio Processing

1. Stem separation — Demucs v4
   isolates guitar/bass/drums tracks from the full mix.
2. Log-mel spectrogram — 512 mel bands, FFT 4096, hop 1024 @ 44 100 Hz.
   The CNN frontend compresses this to 16× fewer time steps.
3. MERT embeddings — MERT-v1-95M
   global embedding captures harmonic and rhythmic context.

---

Intended Use

• Automatic Clone Hero chart generation from any audio file.
• Supported instruments: Lead Guitar, Rhythm Guitar, Bass Guitar, Drums.
• Supported difficulties: Expert, Hard, Medium, Easy.

Limitations

• Performance degrades on heavily distorted or layered mixes.
• BPM estimation may be inaccurate for tracks with variable tempo.
• Trained only on songs with 4/4 time signature.

---

Citation

@misc{clonecharter2026,
  author = {thejorseman},
  title  = {CloneCharter: Automatic Clone Hero Chart Generation},
  year   = {2026},
  url    = {https://huggingface.co/thejorseman/CloneCharter}
}