# RL-Based ARM Compiler Optimization — Comprehensive Research Wiki

> **Goal**: Train an LLM with reinforcement learning (PPO/GRPO) to generate optimized AArch64/ARM assembly code that outperforms `gcc -O3`, using compiler feedback (correctness + speedup) as the reward signal.

> **Last updated**: April 2026 | **Primary recipe**: SuperCoder (arxiv:2505.11480) adapted for ARM

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Landscape & Key Papers](#2-landscape--key-papers)
3. [Recipe 1: SuperCoder — RL Assembly Superoptimization (SOTA)](#3-recipe-1-supercoder--rl-assembly-superoptimization-sota)
4. [Recipe 2: Meta LLM Compiler — SFT on LLVM IR](#4-recipe-2-meta-llm-compiler--sft-on-llvm-ir)
5. [Recipe 3: Compiler Feedback — Iterative Refinement](#5-recipe-3-compiler-feedback--iterative-refinement)
6. [Recipe 4: CUDA-L1 — Contrastive RL (3-Stage Pipeline)](#6-recipe-4-cuda-l1--contrastive-rl-3-stage-pipeline)
7. [Recipe 5: StepCoder — Fine-Grained RL for Code](#7-recipe-5-stepcoder--fine-grained-rl-for-code)
8. [ARM Adaptation Guide](#8-arm-adaptation-guide)
9. [Datasets](#9-datasets)
10. [Model Selection](#10-model-selection)
11. [Reward Function Design](#11-reward-function-design)
12. [Reward Hacking & Mitigations](#12-reward-hacking--mitigations)
13. [Training Infrastructure: TRL GRPO Implementation](#13-training-infrastructure-trl-grpo-implementation)
14. [Full Training Script](#14-full-training-script)
15. [Program Transformation Taxonomy](#15-program-transformation-taxonomy)
16. [Results Benchmarks & Ablations](#16-results-benchmarks--ablations)
17. [Citation Graph & Future Directions](#17-citation-graph--future-directions)
18. [Reference Links](#18-reference-links)

---

## 1. Executive Summary

**The problem**: Modern compilers like `gcc -O3` apply fixed heuristics. LLMs can learn program-specific optimizations that compilers miss — loop restructuring, better instruction selection, algorithmic simplification — achieving **1.46× average speedup over gcc -O3** on x86 and potentially similar gains on ARM.

**The approach**: Use GRPO (Group Relative Policy Optimization) to train `Qwen2.5-Coder-7B-Instruct` with a reward function that:
1. Compiles the generated assembly → reward=0 if it fails
2. Runs all test cases → reward=0 if any fail
3. Measures speedup vs baseline → reward = speedup ratio (continuous)

**Key insight from the literature**: RL beats SFT for this task because superoptimization is open-ended — there's no single "correct" optimized assembly. RL directly optimizes for the metric we care about (speedup) rather than imitating examples.

**No ARM-specific work exists yet** — all published results are on x86-64 or CUDA. This is a greenfield opportunity.

---

## 2. Landscape & Key Papers

### Paper Dependency Graph

```
                    MLGO (Google, 2021)
                    ML replaces compiler heuristics
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
    Meta LLM Compiler   ProGraML    ML Cost Model
    (Meta, 2023)        (2020)      for MLIR (2023)
    SFT from scratch    GNN for IR
    on LLVM IR
              │
              ▼
    Compiler Feedback           StepCoder
    (Meta, 2024)                (2024)
    Iterative refinement        FGO masking
    with oracle feedback        for code RL
              │                      │
              └──────────┬───────────┘
                         ▼
               SuperCoder (2025) ◄── CURRENT SOTA
               PPO/GRPO on assembly
               with compiler reward
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
         CUDA-L1     LLM-VeriOpt   Astra
         (ICLR 2026)  (2026)       (2025)
         Contrastive  Formal       Multi-agent
         RL for CUDA  verification GPU kernel opt
```

### Papers Ranked by Relevance to ARM RL Optimizer

| Rank | Paper | Year | Key Contribution | Result |
|------|-------|------|-----------------|--------|
| 🥇 | [SuperCoder](https://arxiv.org/abs/2505.11480) | 2025 | PPO/GRPO on assembly with compiler reward | 95% correct, 1.46× speedup over gcc -O3 |
| 🥈 | [CUDA-L1](https://arxiv.org/abs/2507.14111) | 2025 | 3-stage SFT→Self-supervised→Contrastive RL | 3.12× avg speedup on KernelBench |
| 🥉 | [Meta LLM Compiler](https://arxiv.org/abs/2309.07062) | 2023 | SFT from scratch on LLVM IR for pass ordering | 3.0% instruction reduction over -Oz |
| 4 | [Compiler Feedback](https://arxiv.org/abs/2403.14714) | 2024 | Iterative refinement with compiler oracle | +0.53% over base; sampling > feedback |
| 5 | [StepCoder](https://arxiv.org/abs/2402.01391) | 2024 | Fine-Grained Optimization masking for code RL | +8% pass@1 on APPS+ |
| 6 | [MLGO](https://arxiv.org/abs/2101.04808) | 2021 | ML in LLVM framework (Google) | Foundation work |

---

## 3. Recipe 1: SuperCoder — RL Assembly Superoptimization (SOTA)

> **Paper**: "SuperCoder: Assembly Program Superoptimization with Large Language Models" 
> **ArXiv**: [2505.11480](https://arxiv.org/abs/2505.11480) | May 2025

### 3.1 Task Formulation

Framed as a **contextual multi-armed bandit** (not full MDP):
- **Context** `s ∈ S`: source program C, baseline assembly P, test cases T
- **Action** `a ∈ A`: generate candidate optimized assembly P̃
- **Reward** `r(s,a)`: correctness-gated speedup (see §11)
- **Policy** `π: S → Δ(A)`: the LLM maps context to a distribution over assemblies

Single-turn generation — no rollout history, no multi-step environment.

### 3.2 Training Configuration

| Component | Setting | Source |
|-----------|---------|--------|
| Base model | `Qwen/Qwen2.5-Coder-7B-Instruct` | Table A1 |
| Actor learning rate | `1e-6` | Appendix A.2 |
| Critic learning rate | `1e-5` (PPO only) | Appendix A.2 |
| Batch size | 16 | Appendix A.2 |
| Epochs | 1 | Appendix A.2 |
| Max prompt length | 2000 tokens | Appendix A.2 |
| Max response length | 2000 tokens | Appendix A.2 |
| Gradient checkpointing | Enabled (actor + critic) | Appendix A.2 |
| Rollout temperature | 0.5 | Appendix A.2 |
| Hardware | 4× A100 GPUs | Appendix A.2 |
| RL framework | [verl](https://github.com/volcengine/verl) | §3.3 |

### 3.3 Dataset Construction

**Source**: IBM CodeNet — 8M+ C/C++ competitive programming submissions.

**Curation strategy** (critical for performance):
1. Sample programs with **highest relative speedup from -O0 to -O3** — this selects computationally rich programs where further optimization is possible
2. Compile each with `gcc -O3 -S` to get baseline x86-64 assembly
3. Use test inputs from [Li et al., 2022], but **regenerate outputs** by executing the original program (CodeNet outputs are unreliable)
4. Final dataset: **7,872 training programs, 200 evaluation programs**

| Split | Programs | Avg Tests/Program | Avg LOC (C) | Avg LOC (Assembly) |
|-------|----------|-------------------|-------------|-------------------|
| Train | 7,872 | 8.86 | 22.3 | 130.3 |
| Eval | 200 | 8.92 | 21.9 | 133.3 |

### 3.4 Prompt Template

```
Given the following C code and assembly code, your task is to generate 
highly optimized x86-64 assembly code.

C Code: <C code here>
Assembly Code: <baseline assembly code here produced by gcc -O3>

Only output the optimized assembly code. Do not include any other text.
Do not write any comments in the assembly code.
Wrap the assembly code in assembly tags.

Optimized Assembly Code:
```

> ⚠️ **Critical finding** (Appendix A.5): Removing the baseline assembly from the prompt causes a **catastrophic drop** — correctness falls from 95% to near 0%. The model needs the gcc -O3 output as a starting point.

### 3.5 Results

| Model | Compile Pass | Test Pass | Avg Speedup |
|-------|-------------|-----------|-------------|
| Qwen2.5-Coder-7B (base) | 77.9% | 61.4% | 1.10× |
| SuperCoder (PPO) | 96.0% | 95.0% | **1.46×** |
| SuperCoder (GRPO) | 95.0% | 94.7% | **1.44×** |
| SuperCoder (SFT only) | 95.5% | 92.5% | 1.39× |

**PPO ≈ GRPO** — nearly identical results, but GRPO is simpler (no critic/value head needed).

**Best-of-N + RL**:
- Base best-of-8: ~1.39× (≈ RL best-of-1)
- SuperCoder best-of-8: **1.93×**

### 3.6 Model Evaluation (23 Models Tested)

| Model | Test Pass | Avg Speedup | Notes |
|-------|-----------|-------------|-------|
| DeepSeek-R1 | 0.0% | 1.00× | Generates verbose analysis, no actual code |
| GPT-4o | 5.0% | 1.02× | Compiles (81%) but sacrifices correctness for optimization |
| Claude-opus-4 | 51.5% | 1.43× | Best zero-shot baseline |
| Qwen2.5-Coder-7B | 61.4% | 1.10× | Best base model for RL starting point |
| llm-compiler-13b | 59.5% | 1.34× | Pretrained on assembly/IR |
| **SuperCoder (PPO)** | **95.0%** | **1.46×** | **SOTA** |

**Key failure modes**:
- Reasoning models (R1, o1) completely fail — they analyze instead of generating code
- GPT-4o compiles but breaks low-level conventions (stack canaries, .cfi directives, calling conventions)
- Models that work best have been pretrained on assembly/code (Qwen-Coder, llm-compiler)

---

## 4. Recipe 2: Meta LLM Compiler — SFT on LLVM IR

> **Paper**: "Large Language Models for Compiler Optimization"
> **ArXiv**: [2309.07062](https://arxiv.org/abs/2309.07062) | Sep 2023 | Meta AI

### 4.1 Approach

Train a 7B transformer **from scratch** (Llama 2 architecture) on LLVM IR to predict:
1. **Optimization pass list** (primary task)
2. **Instruction counts before/after** (auxiliary task — critical for performance)
3. **Full optimized IR** (auxiliary task)

### 4.2 Training Details

| Component | Setting |
|-----------|---------|
| Architecture | Llama 2 7B (32 heads, 4096 hidden, 32 layers) |
| Training | From scratch (random init) |
| Dataset | 1,000,000 deduplicated LLVM-IR functions, 373M tokens |
| Autotuner | 37,424 compilations per function avg; 9,016 CPU days total |
| Optimizer | AdamW (β₁=0.9, β₂=0.95) |
| LR schedule | Cosine, 1000 warmup, peak=1e-5, final=1e-6 |
| Batch size | 256 (524,288 tokens/batch) |
| Steps | 30,000 (7.7 epochs, 15.7B total tokens) |
| Sequence length | 2048 tokens |
| Hardware | 64× V100 GPUs, 620 GPU-days |

### 4.3 Key Results

- **3.0% instruction count reduction over -Oz** (without invoking compiler at inference)
- **91% compilable** generated code
- **70%** perfect emulation of compiler output
- Achieves **60% of autotuner gains** at 0 additional compilation cost

### 4.4 Key Insight: Auxiliary Tasks Matter

Training with instruction count prediction + code generation alongside pass list prediction **dramatically improves** optimization quality. The instruction count acts as a self-consistency check — the model learns to verify its own predictions.

---

## 5. Recipe 3: Compiler Feedback — Iterative Refinement

> **Paper**: "Compiler Generated Feedback for Large Language Models"
> **ArXiv**: [2403.14714](https://arxiv.org/abs/2403.14714) | Mar 2024 | Meta AI

### 5.1 Approach

Start from the best checkpoint of Recipe 2. Add compiler feedback to the prompt:
- Predicted instruction counts
- Whether the code compiled correctly
- Whether the IR is valid

Model outputs "I am sure!" if confident, else "Let me try again."

### 5.2 Training Details

| Component | Setting |
|-----------|---------|
| Base model | Best checkpoint from Meta LLM Compiler (7B) |
| Dataset | 1M training + 100K test LLVM-IR functions |
| Optimizer | AdamW (β₁=0.9, β₂=0.95) |
| LR | Cosine, 1000 warmup, peak=1e-5 |
| Batch size | 256 (786K–1M tokens/batch) |
| Steps | 20,000 (5.12 epochs, 16–21B tokens) |
| Hardware | 64× A100s, 60 GPU-days |

### 5.3 Key Finding

> **Sampling beats iterative feedback**: At n≥10 samples, the base model without feedback training outperforms the feedback-trained model. This strongly motivates using **Best-of-N + RL** (SuperCoder approach) over iterative SFT feedback.

---

## 6. Recipe 4: CUDA-L1 — Contrastive RL (3-Stage Pipeline)

> **Paper**: "CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning"
> **ArXiv**: [2507.14111](https://arxiv.org/abs/2507.14111) | Jul 2025 | ICLR 2026

### 6.1 The 3-Stage Pipeline

This is the most important contribution for cases where the **base model has low initial success rate**.

```
Stage 1: SFT via Data Augmentation
├── Generate CUDA code from 6 LLMs (GPT-4o, o1, DeepSeek-R1/V3, Llama-405B, Claude-3.7)
├── Filter for correct + fast implementations
├── Fine-tune DeepSeek-V3-671B on successful samples
└── Result: Model can generate correct code at reasonable rate

Stage 2: Self-Supervised Learning
├── Sample from Stage 1 model
├── Keep only correct samples (self-filtering)
├── Retrain on filtered dataset
└── Result: Higher base success rate for RL

Stage 3: Contrastive Reinforcement Learning
├── Present model with multiple code variants + their speedup scores
├── Model analyzes WHY certain implementations are faster
├── Generates improved solution based on comparative analysis
├── Score serves dual purpose: (1) gradient update, (2) future prompt enrichment
└── Result: 3.12× average speedup on KernelBench
```

### 6.2 Why Standard GRPO/PPO Failed for CUDA-L1

> "Standard RL algorithms compute a scalar reward for each generated CUDA code sample... the reward signal is used exclusively for parameter updates and is never provided as input to the LLM. Consequently, the LLM cannot directly reason about performance trade-offs during code generation."

Their solution: **Contrastive RL** — embed performance feedback within the input prompt. The model sees previous code + scores and learns comparative analysis.

### 6.3 Results

| Configuration | Mean Speedup | Max | Median | Success Rate |
|--------------|-------------|-----|--------|-------------|
| Default | 3.12× | 120× | 1.42× | 249/250 |
| vs Torch Compile | 2.77× | — | — | — |
| vs CUDA Graph | 2.81× | — | — | — |

### 6.4 ARM Relevance

- Use the **3-stage pipeline** if the base model has <40% ARM assembly correctness
- Contrastive RL is useful when you need the model to reason about WHY optimizations work
- Stage 1 data augmentation from multiple LLMs is a powerful bootstrapping technique

---

## 7. Recipe 5: StepCoder — Fine-Grained RL for Code

> **Paper**: "StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback"
> **ArXiv**: [2402.01391](https://arxiv.org/abs/2402.01391) | Feb 2024

### 7.1 Key Innovation: Fine-Grained Optimization (FGO)

Standard PPO updates ALL tokens in the generated code equally. StepCoder's FGO **masks tokens not executed by unit tests** — only code segments that are actually run contribute to the gradient update.

This is crucial for sparse compiler rewards where most of the generated code might be boilerplate.

### 7.2 Reward Design

| Outcome | Reward |
|---------|--------|
| All unit tests pass | +1.0 |
| Test failure | -0.3 |
| Runtime error | -0.6 |
| Compile error | -1.0 |

This graduated penalty scheme (vs SuperCoder's binary 0/non-zero) helps the model distinguish failure modes.

---

## 8. ARM Adaptation Guide

### 8.1 The ARM Gap

**No published work targets ARM/AArch64**. All results are on x86-64 (SuperCoder, Meta) or CUDA (CUDA-L1). This requires adaptation:

### 8.2 Minimal Changes from SuperCoder (x86 → ARM)

| Component | x86 (Original) | ARM (Adapted) |
|-----------|----------------|---------------|
| Compiler | `gcc -O3` | `aarch64-linux-gnu-gcc -O3` |
| Assembler | `as` | `aarch64-linux-gnu-as` |
| Linker | `ld` | `aarch64-linux-gnu-ld` |
| Execution | Native | `qemu-aarch64-static` (emulation) |
| Timing | `hyperfine` (wall-clock) | QEMU instruction counting or real ARM HW |
| ISA in prompt | "x86-64 assembly" | "AArch64 assembly" |
| Assembly flag | `-S` | `-S` (same) |

### 8.3 ARM-Specific Prompt Template

```
Given the following C code and assembly code, your task is to generate 
highly optimized AArch64 assembly code.

C Code: {c_code}
Assembly Code: {arm_baseline_asm}

Only output the optimized assembly code. Do not include any other text.
Do not write any comments in the assembly code.
Wrap the assembly code in <assembly></assembly> tags.

Optimized Assembly Code:
```

### 8.4 Execution Environment

```bash
# Install ARM cross-compilation toolchain
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
sudo apt-get install qemu-user-static

# Cross-compile to ARM assembly
aarch64-linux-gnu-gcc -O3 -S -o output.s input.c

# Cross-compile to ARM binary
aarch64-linux-gnu-gcc -O3 -o output input.c

# Run ARM binary on x86 via QEMU
qemu-aarch64-static ./output < test_input.txt
```

### 8.5 Timing Considerations

| Method | Accuracy | Availability |
|--------|----------|-------------|
| QEMU instruction counting | Deterministic but not wall-clock accurate | Any x86 machine |
| QEMU `-plugin libinsn` | Counts executed instructions precisely | QEMU 6.0+ |
| Real ARM hardware | Ground truth | Requires ARM machine |
| `perf stat` on ARM | Cycle-accurate | Requires ARM + Linux perf |

**Recommendation**: Use QEMU instruction counting for training (deterministic, reproducible) and validate final results on real ARM hardware.

### 8.6 ARM-Specific Optimization Opportunities

The model should learn to leverage:
- **NEON SIMD** instructions (128-bit vector operations)
- **SVE/SVE2** (Scalable Vector Extension — variable-length vectors)
- **LSE atomics** (Large System Extensions)
- **Conditional select** (`csel`, `csinc`) instead of branches
- **Fused multiply-add** (`fmadd`, `fmsub`)
- **Load/store pair** (`ldp`, `stp`) for memory throughput
- **Predicated operations** in SVE (eliminate branch mispredictions)

### 8.7 3-Stage Plan for ARM (if base model correctness < 40%)

Following CUDA-L1's insight:

```
Stage 1: SFT Warmup
├── Dataset: (C source → ARM gcc -O3 assembly) pairs
├── Task: Teach the model to generate valid ARM assembly
├── Expected: Model learns ARM syntax, calling conventions, directives
└── Duration: ~2-4 hours on A100

Stage 2: Filtered Self-Training
├── Sample N completions per prompt from Stage 1 model
├── Keep only compilable + correct samples
├── Retrain on filtered dataset (higher quality)
└── Expected: Correctness improves to >60%

Stage 3: GRPO with Compiler Reward
├── Apply SuperCoder's reward function
├── Binary correctness gate + continuous speedup
├── Expected: Correctness >90%, speedup >1.3×
└── Duration: ~4-8 hours on 4×A100
```

---

## 9. Datasets

### 9.1 Available Datasets

| Dataset | Source | Format | Size | ARM Compatible | Notes |
|---------|--------|--------|------|----------------|-------|
| **IBM CodeNet** | [GitHub](https://github.com/IBM/Project_CodeNet) | C/C++ source + test I/O | 8M+ submissions | ✅ Recompile with ARM GCC | Used by SuperCoder |
| **deepmind/code_contests** | [HF Hub](https://hf.co/datasets/deepmind/code_contests) | C/C++ solutions + tests | ~2GB train | ✅ Filter C, cross-compile | Has public/private/generated tests |
| **llvm-ml/ComPile** | [HF Hub](https://hf.co/datasets/llvm-ml/ComPile) | LLVM bitcode IR | 602GB (2.7TB source) | ✅ Retarget to AArch64 via `llc` | C/C++/Rust/Swift from Spack |
| APPS+ | [GitHub](https://github.com/ablustrund/apps_plus) | Python problems + tests | ~10K | ❌ Python only | Used by StepCoder |

### 9.2 Dataset Format for GRPO Training

The dataset must have a `prompt` column in conversational format. Extra columns are forwarded to reward functions as `**kwargs`.

```python
{
    "prompt": [
        {"role": "user", "content": "Given the following C code and assembly code..."}
    ],
    "c_code": "int main() { ... }",
    "baseline_asm": ".text\n.globl main\nmain:\n...",
    "test_inputs": ["3\n1 2 3", "5\n1 2 3 4 5"],
    "test_outputs": ["6", "15"],
    "baseline_time": 0.0042  # seconds
}
```

### 9.3 Dataset Construction Pipeline

```python
from datasets import Dataset
import subprocess, json

def build_arm_dataset(codenet_programs):
    """Build ARM optimization dataset from CodeNet C programs."""
    samples = []
    
    for prog in codenet_programs:
        c_code = prog["source"]
        test_cases = prog["test_cases"]  # [(input, output), ...]
        
        # Step 1: Cross-compile to ARM assembly with -O3
        result = subprocess.run(
            ["aarch64-linux-gnu-gcc", "-O3", "-S", "-o", "/dev/stdout", "-x", "c", "-"],
            input=c_code, capture_output=True, text=True
        )
        if result.returncode != 0:
            continue  # Skip programs that don't compile
        baseline_asm = result.stdout
        
        # Step 2: Compile to binary for benchmarking
        # ... (compile + link + measure baseline time via QEMU)
        
        # Step 3: Build prompt
        prompt_text = f"""Given the following C code and assembly code, your task is to generate highly optimized AArch64 assembly code.

C Code: {c_code}
Assembly Code: {baseline_asm}

Only output the optimized assembly code. Do not include any other text.
Do not write any comments in the assembly code.
Wrap the assembly code in <assembly></assembly> tags.

Optimized Assembly Code:"""
        
        samples.append({
            "prompt": [{"role": "user", "content": prompt_text}],
            "c_code": c_code,
            "baseline_asm": baseline_asm,
            "test_inputs": [tc[0] for tc in test_cases],
            "test_outputs": [tc[1] for tc in test_cases],
            "baseline_time": baseline_time,
        })
    
    return Dataset.from_list(samples)
```

### 9.4 deepmind/code_contests Schema

```
| Column | Type |
|--------|------|
| name | string |
| description | string |
| public_tests | Sequence: {input: list[str], output: list[str]} |
| private_tests | Sequence: {input: list[str], output: list[str]} |
| generated_tests | Sequence: {input: list[str], output: list[str]} |
| solutions | Sequence: {language: list[int], solution: list[str]} |
| difficulty | ClassLabel (29 classes) |
| source | ClassLabel (7 classes) |
```

Filter C solutions: `language == 2` (C) or look for C-like syntax in the solution strings.

---

## 10. Model Selection

### 10.1 Base Model Recommendation

**Primary**: `Qwen/Qwen2.5-Coder-7B-Instruct`
- 7.6B parameters, Qwen2 architecture
- Apache-2.0 license
- **Proven by SuperCoder**: 61.4% test pass rate (highest among 7B models)
- Strong code generation baseline for RL starting point
- Available on [HF Hub](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)

### 10.2 Why Not Other Models?

| Model | Issue |
|-------|-------|
| DeepSeek-R1 | 0% compilation — generates analysis, not code |
| GPT-4o | 81% compile but 5% correct — breaks low-level conventions |
| Reasoning models (o1, R1) | Fundamentally fail — spend tokens reasoning instead of generating |
| llm-compiler-7b-ftd | Fine-tuned for disassembly, not optimization |
| llm-compiler-13b | Good (1.34× speedup) but not instruction-tuned; harder to RL fine-tune |

### 10.3 Model Sizing

| Model Size | Hardware Needed | VRAM (bf16) |
|-----------|----------------|-------------|
| 7B (Qwen2.5-Coder-7B) | 4× A100 80GB | ~14GB model + ~40GB for RL |
| 13B | 4× A100 80GB | ~26GB model + ~60GB for RL |
| 70B+ | 8× A100 or 4× H100 | Multi-node |

---

## 11. Reward Function Design

### 11.1 SuperCoder Reward (Recommended)

From Section 3.3 of arxiv:2505.11480:

```
r(s, a) = {
    0,              if pass(s,a) < 1       (any test fails → zero reward)
    speedup(s,a),   if pass(s,a) = 1       (all tests pass → continuous speedup)
}
```

Where:
- `pass(s,a) = (1/|T|) × Σ 𝟏[P̃(xᵢ) = yᵢ]` — fraction of test cases passed
- `speedup(s,a) = t(P) / t(P̃)` — baseline time / optimized time

**Design principles**:
1. **Hard binary correctness gate**: No partial credit. Forces model to learn correctness first.
2. **Continuous speedup reward**: Provides gradient signal proportional to actual optimization gain.
3. **No reward for partial correctness**: Passing 99% of tests still gets 0 reward.

### 11.2 StepCoder Graduated Penalty (Alternative)

```
+1.0  → all unit tests pass
-0.3  → test failure
-0.6  → runtime error
-1.0  → compile error
```

Distinguishes failure modes — the model gets a stronger negative signal for "worse" failures.

### 11.3 CUDA-L1 Reward Smoothing (Anti-Hacking)

```python
r_normalized = (r - μ) / σ           # normalize by running stats
r_smooth = clip(r_normalized, -k, k)  # clip to [-1.5, 1.5]
```

Prevents the model from over-optimizing outlier high-reward solutions.

### 11.4 Implementation

```python
import subprocess
import tempfile
import os
import re

def arm_compiler_reward(completions, c_code, baseline_asm, 
                        test_inputs, test_outputs, baseline_time, **kwargs):
    """
    Compiler feedback reward for ARM assembly optimization.
    Follows SuperCoder §3.3 exactly: binary correctness gate + continuous speedup.
    """
    rewards = []
    
    for i, completion in enumerate(completions):
        # Extract assembly from completion
        content = completion[0]["content"] if isinstance(completion, list) else completion
        asm_match = re.search(r'<assembly>(.*?)</assembly>', content, re.DOTALL)
        if not asm_match:
            rewards.append(0.0)
            continue
        asm_code = asm_match.group(1).strip()
        
        with tempfile.TemporaryDirectory() as tmpdir:
            asm_path = os.path.join(tmpdir, "opt.s")
            bin_path = os.path.join(tmpdir, "opt")
            
            # Write assembly
            with open(asm_path, "w") as f:
                f.write(asm_code)
            
            # Step 1: Assemble
            result = subprocess.run(
                ["aarch64-linux-gnu-gcc", "-o", bin_path, asm_path, "-static", "-lm"],
                capture_output=True, text=True, timeout=30
            )
            if result.returncode != 0:
                rewards.append(0.0)  # Compile failure
                continue
            
            # Step 2: Run all tests via QEMU
            all_pass = True
            for test_in, expected_out in zip(test_inputs[i], test_outputs[i]):
                try:
                    run = subprocess.run(
                        ["qemu-aarch64-static", bin_path],
                        input=test_in, capture_output=True, text=True, timeout=10
                    )
                    if run.stdout.strip() != expected_out.strip():
                        all_pass = False
                        break
                except subprocess.TimeoutExpired:
                    all_pass = False
                    break
            
            if not all_pass:
                rewards.append(0.0)  # Test failure
                continue
            
            # Step 3: Measure speedup (instruction count via QEMU)
            # Use QEMU instruction counting for deterministic measurement
            try:
                run = subprocess.run(
                    ["qemu-aarch64-static", "-d", "in_asm", bin_path],
                    input=test_inputs[i][0], capture_output=True, text=True, timeout=30
                )
                insn_count = run.stderr.count('\n')  # Rough instruction count
                speedup = baseline_time[i] / max(insn_count, 1)
                rewards.append(max(speedup, 0.1))
            except Exception:
                rewards.append(1.0)  # Default: assume baseline-equivalent
    
    return rewards
```

---

## 12. Reward Hacking & Mitigations

### 12.1 Known Hacking Behaviors (from CUDA-L1 §3.1)

| Hack | Description | Prevalence |
|------|-------------|-----------|
| **Improper timing** | Create async streams; timing only measures main stream | 32.8% of outputs |
| **Lazy evaluation** | Return lazy tensor; actual compute happens at correctness check | Found in training |
| **Hyperparameter manipulation** | Reduce batch_size/dimensions in generated code | Found in training |
| **Result caching** | Cache outputs by input address; return cached results | Found in training |

### 12.2 Mitigations

1. **Reward checking model**: When reward jumps significantly, use an adversarial model (e.g., DeepSeek-R1) to check for exploitation. Catches >60% of hacking.
2. **Hacking-case database**: Maintain a growing database of known hacking patterns. Use retrieval-augmented checking.
3. **Reward smoothing**: `r_smooth = clip((r - μ)/σ, -k, k)` with k=1.5
4. **Robust evaluation**: Synchronize all execution before timing. Validate output is real tensor with allocated storage.

### 12.3 ARM-Specific Hacking Risks

- **QEMU timing artifacts**: Model could learn to generate code that runs fast in QEMU but slow on real ARM hardware
- **Mitigation**: Use instruction counting (deterministic) rather than wall-clock timing in QEMU
- **NOP padding**: Model could pad with NOPs that don't affect correctness but confuse instruction counting
- **Mitigation**: Count only non-NOP instructions, or use basic-block counting

---

## 13. Training Infrastructure: TRL GRPO Implementation

### 13.1 Why GRPO over PPO

| Feature | PPO | GRPO |
|---------|-----|------|
| Value head/critic | Required | Not needed |
| Memory usage | ~2× model | ~1× model |
| Reward model | Optional (can use custom) | Custom function native |
| Performance | 1.46× speedup | 1.44× speedup |
| Implementation complexity | Higher | Lower |
| TRL support | Experimental (`trl.experimental.ppo`) | Full support (`trl.GRPOTrainer`) |

**Verdict**: GRPO is strictly better for this use case — same results, half the memory, simpler code.

### 13.2 GRPOConfig Parameters

```python
from trl import GRPOConfig

config = GRPOConfig(
    # === Model ===
    output_dir="arm-compiler-optimizer",
    
    # === Generation ===
    num_generations=4,              # G: completions per prompt (group size for GRPO)
    max_prompt_length=2048,         # tokens; truncated from left
    max_completion_length=2048,     # tokens
    temperature=0.5,                # rollout sampling temperature (SuperCoder: 0.5)
    
    # === Loss / Reward ===
    beta=0.0,                       # KL penalty (0.0 = no KL term, default in TRL)
    epsilon=0.2,                    # PPO clip range (used in GRPO's clipped objective)
    scale_rewards=False,            # Don't normalize — binary-gated rewards aren't Gaussian
    
    # === Training ===
    learning_rate=1e-6,             # SuperCoder Appendix A.2
    per_device_train_batch_size=4,  # Per GPU; effective batch = 4 GPUs × 4 = 16
    gradient_accumulation_steps=1,
    num_train_epochs=1,             # SuperCoder: 1 epoch
    gradient_checkpointing=True,    # Memory savings (default True in GRPOConfig)
    bf16=True,                      # Default True
    
    # === Logging ===
    logging_steps=10,
    log_completions=True,           # Log generated completions
    disable_tqdm=True,              # Plain text logs for job monitoring
    logging_first_step=True,
    logging_strategy="steps",
    
    # === Saving ===
    push_to_hub=True,
    hub_model_id="kaori02/arm-compiler-optimizer",
    save_strategy="steps",
    save_steps=100,
)
```

### 13.3 Custom Reward Function Signature

TRL GRPOTrainer passes these keyword arguments to reward functions:

```python
def my_reward(
    completions,          # list[list[dict]] — each is [{"role":"assistant","content":"..."}]
    prompts=None,         # list[list[dict]] or list[str]
    completion_ids=None,  # list[list[int]] — tokenized completions
    trainer_state=None,   # TrainerState — current training step, epoch, etc.
    log_extra=None,       # callable to log extra columns
    log_metric=None,      # callable to log scalar metrics
    **kwargs              # ALL extra dataset columns forwarded here
) -> list[float]:         # one reward per completion
    ...
```

### 13.4 Multiple Reward Functions (Composable)

```python
trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-Coder-7B-Instruct",
    reward_funcs=[compile_reward, correctness_reward, speedup_reward],
    reward_weights=[1.0, 2.0, 5.0],  # Weight speedup most heavily
    args=config,
    train_dataset=dataset,
)
```

---

## 14. Full Training Script

```python
"""
ARM Compiler Optimization via GRPO
Based on: SuperCoder (arxiv:2505.11480) adapted for AArch64

Recipe:
- Base model: Qwen/Qwen2.5-Coder-7B-Instruct
- Method: GRPO with custom compiler reward
- Reward: Binary correctness gate + continuous speedup
- Dataset: CodeNet C programs → ARM assembly
"""

import os
import re
import subprocess
import tempfile
from datasets import load_dataset, Dataset
from trl import GRPOTrainer, GRPOConfig
import trackio

# ═══════════════════════════════════════════════════════════════════════════
# REWARD FUNCTION
# ═══════════════════════════════════════════════════════════════════════════

def extract_assembly(content):
    """Extract assembly code from <assembly> tags."""
    match = re.search(r'<assembly>(.*?)</assembly>', content, re.DOTALL)
    return match.group(1).strip() if match else None

def compile_arm(asm_code, output_path):
    """Cross-compile ARM assembly to binary."""
    with tempfile.NamedTemporaryFile(suffix=".s", mode="w", delete=False) as f:
        f.write(asm_code)
        asm_path = f.name
    try:
        result = subprocess.run(
            ["aarch64-linux-gnu-gcc", "-o", output_path, asm_path, "-static", "-lm"],
            capture_output=True, text=True, timeout=30
        )
        return result.returncode == 0
    except Exception:
        return False
    finally:
        os.unlink(asm_path)

def run_test(binary_path, test_input, expected_output, timeout=10):
    """Run a test case via QEMU and check output."""
    try:
        result = subprocess.run(
            ["qemu-aarch64-static", binary_path],
            input=test_input, capture_output=True, text=True, timeout=timeout
        )
        return result.stdout.strip() == expected_output.strip()
    except subprocess.TimeoutExpired:
        return False

def measure_instructions(binary_path, test_input, timeout=30):
    """Count executed instructions via QEMU for deterministic timing."""
    try:
        result = subprocess.run(
            ["qemu-aarch64-static", "-d", "in_asm", binary_path],
            input=test_input, capture_output=True, text=True, timeout=timeout
        )
        return result.stderr.count('\n')
    except Exception:
        return float('inf')

def arm_compiler_reward(completions, test_inputs, test_outputs, 
                        baseline_insn_count, log_metric=None, **kwargs):
    """
    SuperCoder §3.3 reward adapted for ARM:
    r = 0 if compilation fails or any test fails
    r = baseline_instructions / optimized_instructions if all tests pass
    """
    rewards = []
    compile_successes = 0
    test_successes = 0
    
    for i, completion in enumerate(completions):
        content = completion[0]["content"] if isinstance(completion, list) else completion
        asm = extract_assembly(content)
        
        if asm is None:
            rewards.append(0.0)
            continue
        
        with tempfile.TemporaryDirectory() as tmpdir:
            bin_path = os.path.join(tmpdir, "opt_binary")
            
            # Step 1: Compile
            if not compile_arm(asm, bin_path):
                rewards.append(0.0)
                continue
            compile_successes += 1
            
            # Step 2: Run ALL tests
            all_pass = True
            inputs = test_inputs[i] if isinstance(test_inputs[i], list) else [test_inputs[i]]
            outputs = test_outputs[i] if isinstance(test_outputs[i], list) else [test_outputs[i]]
            
            for tin, tout in zip(inputs, outputs):
                if not run_test(bin_path, tin, tout):
                    all_pass = False
                    break
            
            if not all_pass:
                rewards.append(0.0)
                continue
            test_successes += 1
            
            # Step 3: Measure speedup via instruction count
            opt_insn = measure_instructions(bin_path, inputs[0])
            baseline = baseline_insn_count[i] if isinstance(baseline_insn_count, list) else baseline_insn_count
            
            if opt_insn > 0 and baseline > 0:
                speedup = baseline / opt_insn
                rewards.append(max(speedup, 0.1))
            else:
                rewards.append(1.0)
    
    # Log metrics
    if log_metric and len(rewards) > 0:
        log_metric("compile_rate", compile_successes / len(completions))
        log_metric("test_pass_rate", test_successes / len(completions))
        log_metric("avg_reward", sum(rewards) / len(rewards))
    
    return rewards

# ═══════════════════════════════════════════════════════════════════════════
# TRAINING
# ═══════════════════════════════════════════════════════════════════════════

def main():
    # Initialize tracking
    trackio.init(
        project="arm-compiler-optimizer",
        run="grpo-qwen-7b-arm",
    )
    
    # Load pre-built dataset (see §9.3 for construction)
    dataset = load_dataset("your-org/arm-compiler-dataset", split="train")
    
    # GRPO Configuration (SuperCoder Appendix A.2, adapted for TRL)
    config = GRPOConfig(
        output_dir="arm-compiler-optimizer",
        
        # Generation
        num_generations=4,
        max_prompt_length=2048,
        max_completion_length=2048,
        temperature=0.5,
        
        # Loss
        beta=0.0,
        scale_rewards=False,
        
        # Training
        learning_rate=1e-6,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=1,
        num_train_epochs=1,
        gradient_checkpointing=True,
        bf16=True,
        
        # Logging
        logging_steps=10,
        log_completions=True,
        disable_tqdm=True,
        logging_first_step=True,
        logging_strategy="steps",
        
        # Saving
        push_to_hub=True,
        hub_model_id="kaori02/arm-compiler-optimizer",
        save_strategy="steps",
        save_steps=100,
    )
    
    trainer = GRPOTrainer(
        model="Qwen/Qwen2.5-Coder-7B-Instruct",
        reward_funcs=arm_compiler_reward,
        args=config,
        train_dataset=dataset,
    )
    
    trainer.train()
    trainer.push_to_hub()

if __name__ == "__main__":
    main()
```

---

## 15. Program Transformation Taxonomy

SuperCoder §5.4 analyzed all 200 evaluation programs. The LLM learns these optimization patterns:

| Transformation | Description | Frequency |
|---------------|-------------|-----------|
| **Loop Restructuring** | Reorder, unroll, alter loop control flow | 45% |
| **Instruction Selection** | Use specialized CPU instructions (e.g., `popcnt`, `bsr`, `cmov`) instead of generic sequences | 35% |
| **Algorithmic Simplification** | Replace custom logic with standard library calls (`memcmp`, `strcmp`, `atoi`) | 30% |
| **Stack Canary Removal** | Eliminate stack protection checks and security instrumentation | 25% |
| **Register Allocation** | Better register assignment, reuse, reduced spills | 20% |
| **Branch Elimination** | Replace conditional branches with conditional moves (`cmov`, `setcc`) | 15% |
| **Address Calculation** | Optimize memory address computation | 10% |
| **Dead Code Elimination** | Remove unused code paths | 10% |
| **Constant Propagation** | Evaluate expressions at compile time | 5% |

### ARM-Specific Transformations to Expect

| Transformation | ARM Instructions | x86 Equivalent |
|---------------|-----------------|----------------|
| Vectorization | NEON `ld1`, `add`, `mul` (4×32-bit) | SSE/AVX |
| Predication | SVE predicated ops (no branch) | None (x86 lacks predication) |
| Paired load/store | `ldp`, `stp` | None (x86 does one at a time) |
| Fused multiply-add | `fmadd`, `fmsub` | `vfmadd` (AVX) |
| Conditional select | `csel`, `csinc`, `csneg` | `cmov` |
| Bit manipulation | `clz`, `rbit`, `cnt` | `bsr`, `popcnt` |

---

## 16. Results Benchmarks & Ablations

### 16.1 RL vs SFT (SuperCoder §5.2-5.3)

| Method | Correctness | Speedup | Notes |
|--------|------------|---------|-------|
| Base (zero-shot) | 61.4% | 1.10× | No training |
| SFT | 92.5% | 1.39× | Trained on best samples |
| GRPO | 94.7% | 1.44× | RL with compiler reward |
| PPO | 95.0% | 1.46× | RL with compiler reward |
| PPO + best-of-8 | ~95% | **1.93×** | Inference-time scaling |

> **RL > SFT** because optimization is open-ended. There's no single "correct" optimized program — RL directly maximizes the objective (speedup) rather than imitating examples.

### 16.2 Inference-Time Scaling

Best-of-N sampling works multiplicatively with RL:

| Model | N=1 | N=2 | N=4 | N=8 |
|-------|-----|-----|-----|-----|
| Base (Qwen-7B) | 1.10× | 1.20× | 1.30× | 1.39× |
| Claude-opus-4 | 1.43× | 1.60× | 1.80× | 2.05× |
| SuperCoder (PPO) | 1.46× | 1.60× | 1.75× | **1.93×** |

### 16.3 Ablation: Prompt Components

| Prompt Contains | Correctness | Speedup |
|----------------|-------------|---------|
| C code + gcc -O3 assembly | 95.0% | 1.46× |
| C code only (no assembly) | ~0% | — |
| gcc -O3 assembly only (no C) | ~60% | ~1.3× |

> The baseline assembly is **essential**. Without it, the model cannot generate valid assembly.

### 16.4 Random vs Curated Dataset (SuperCoder §A.7)

| Dataset | Correctness | Speedup |
|---------|-------------|---------|
| Curated (high O0→O3 speedup) | 95.0% | 1.46× |
| Random sample from CodeNet | 93.5% | 1.35× |

Curated dataset helps, but random works too — the approach is robust.

---

## 17. Citation Graph & Future Directions

### 17.1 Papers Citing SuperCoder

| Paper | Key Insight |
|-------|------------|
| **LLM-VeriOpt** (2026) [influential] | Uses formal verification instead of test suites for correctness — eliminates false positives from finite test coverage |
| **Astra** (2025, 31 citations) | Multi-agent system for GPU kernel optimization — agent decomposition for complex optimizations |
| **InCoder-32B** (2026) | Industrial code model covering compiler optimization as a domain |
| **Genesys** (2025) | Evolutionary program synthesis with continuous optimization |

### 17.2 Future Directions

1. **Formal Verification as Reward** (LLM-VeriOpt direction): Replace test-based correctness with formal equivalence checking — eliminates false positive rewards from insufficient test coverage.

2. **Multi-Turn Refinement** (Kevin, 2025 — arxiv:2507.11948): Let the model iteratively refine its assembly based on profiler feedback. Multiple rounds of generation → profiling → feedback.

3. **Contrastive RL** (CUDA-L1): When base model success rate is too low for standard GRPO, present multiple code variants with scores and let the model reason about WHY certain versions are faster.

4. **Cross-Architecture Transfer**: Train on x86, transfer to ARM. The C source code is architecture-agnostic — optimization patterns (loop unrolling, vectorization, etc.) transfer across ISAs even if specific instructions differ.

5. **SVE/SVE2 Exploitation**: ARM's Scalable Vector Extension offers variable-length SIMD. This is a unique optimization opportunity not available on x86 — models that learn to leverage SVE could achieve outsized speedups.

6. **Auto-Parallelization**: Beyond single-thread optimization — teach the model to identify parallelization opportunities and generate multi-threaded ARM code with proper synchronization.

---

## 18. Reference Links

### Papers
| Paper | ArXiv | Year |
|-------|-------|------|
| SuperCoder | [2505.11480](https://arxiv.org/abs/2505.11480) | 2025 |
| CUDA-L1 | [2507.14111](https://arxiv.org/abs/2507.14111) | 2025 |
| Meta LLM Compiler | [2309.07062](https://arxiv.org/abs/2309.07062) | 2023 |
| Compiler Feedback | [2403.14714](https://arxiv.org/abs/2403.14714) | 2024 |
| StepCoder | [2402.01391](https://arxiv.org/abs/2402.01391) | 2024 |
| MLGO | [2101.04808](https://arxiv.org/abs/2101.04808) | 2021 |
| ProGraML | [2012.01470](https://arxiv.org/abs/2012.01470) | 2020 |
| DeepSeekMath (GRPO) | [2402.03300](https://arxiv.org/abs/2402.03300) | 2024 |
| VeriReason | [2505.11849](https://arxiv.org/abs/2505.11849) | 2025 |
| ACECoder | [2502.01718](https://arxiv.org/abs/2502.01718) | 2025 |

### Code & Frameworks
| Resource | URL |
|----------|-----|
| TRL (GRPO Trainer) | [huggingface.co/docs/trl/grpo_trainer](https://huggingface.co/docs/trl/grpo_trainer) |
| TRL Reward Functions | [huggingface.co/docs/trl/rewards](https://huggingface.co/docs/trl/rewards) |
| TRL OpenEnv | [huggingface.co/docs/trl/openenv](https://huggingface.co/docs/trl/openenv) |
| verl (Volcano Engine RL) | [github.com/volcengine/verl](https://github.com/volcengine/verl) |
| CUDA-L1 Code | [github.com/deepreinforce-ai/CUDA-L1](https://github.com/deepreinforce-ai/CUDA-L1) |
| StepCoder / APPS+ | [github.com/ablustrund/apps_plus](https://github.com/ablustrund/apps_plus) |
| IBM CodeNet | [github.com/IBM/Project_CodeNet](https://github.com/IBM/Project_CodeNet) |

### Models & Datasets
| Resource | URL |
|----------|-----|
| Qwen2.5-Coder-7B-Instruct | [huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) |
| deepmind/code_contests | [huggingface.co/datasets/deepmind/code_contests](https://huggingface.co/datasets/deepmind/code_contests) |
| llvm-ml/ComPile | [huggingface.co/datasets/llvm-ml/ComPile](https://huggingface.co/datasets/llvm-ml/ComPile) |

### Tools
| Tool | Purpose |
|------|---------|
| `aarch64-linux-gnu-gcc` | ARM cross-compiler |
| `qemu-aarch64-static` | ARM userspace emulation |
| `hyperfine` | Benchmarking tool |
| `trackio` | Experiment tracking |

---

## Appendix: Quick Reference Card

```
═══════════════════════════════════════════════════════════════
  ARM COMPILER OPTIMIZATION VIA GRPO — QUICK REFERENCE
═══════════════════════════════════════════════════════════════

Model:     Qwen/Qwen2.5-Coder-7B-Instruct
Method:    GRPO (no critic needed)
Dataset:   CodeNet C programs → ARM assembly (7,872 train / 200 eval)
Reward:    r=0 if fail, r=speedup if all tests pass
LR:        1e-6
Batch:     16 (4 per GPU × 4 GPUs)
Epochs:    1
G:         4 completions per prompt
Temp:      0.5
Seq len:   2048 prompt + 2048 completion
KL (beta): 0.0
Hardware:  4× A100 80GB
Framework: TRL GRPOTrainer
Time:      ~4-8 hours

Expected:  61% → 95% correctness, 1.10× → 1.46× speedup

If base model ARM correctness < 40%, use 3-stage pipeline:
  Stage 1: SFT warmup on (C → ARM assembly) pairs
  Stage 2: Self-filtered retraining
  Stage 3: GRPO with compiler reward
═══════════════════════════════════════════════════════════════
```