Hugging Face's logo

Spaces:
77ethers
/
gridops
Running

App Files Community
77ethers commited on
Commit
feaaefa
·
verified ·
1 Parent(s): aee53e2

Upload LEADERBOARD.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. LEADERBOARD.md +134 -0
LEADERBOARD.md ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # GridOps — Agent Leaderboard
2
+
3
+ Benchmark results across 10 agents on the GridOps OpenEnv environment.
4
+ All runs use seed=42 for full reproducibility. Each task is 72 steps (3 days).
5
+
6
+ ---
7
+
8
+ ## Overall Standings
9
+
10
+ | Rank | Agent | Task 1 (Normal) | Task 2 (Heatwave) | Task 3 (Crisis) | **Average** |
11
+ |---:|---|:---:|:---:|:---:|:---:|
12
+ | 1 | **Grok-4 (xAI)** | 0.80 | **0.82** | **0.72** | **0.78** |
13
+ | 2 | Oracle (rule-based) | 0.79 | 0.81 | 0.70 | 0.77 |
14
+ | 3 | **GPT-5.4 (OpenAI)** | 0.79 | 0.79 | 0.67 | 0.75 |
15
+ | 4 | Gemma-4-31B (Google) | **0.81** | 0.79 | 0.62 | 0.74 |
16
+ | 4 | Grok 4.20 Multi-Agent | **0.81** | 0.80 | 0.60 | 0.74 |
17
+ | 4 | DeepSeek V3.2 | 0.80 | 0.79 | 0.62 | 0.74 |
18
+ | 7 | GPT-5.4-mini (OpenAI) | 0.72 | 0.74 | 0.46 | 0.64 |
19
+ | 8 | Qwen 3.6 Plus (free) | 0.69 | 0.67 | 0.45 | 0.60 |
20
+ | 9 | Gemini 3.1 Pro Preview | 0.65 | 0.53 | 0.47 | 0.55 |
21
+ | 10 | Kimi K2.5 | 0.57 | 0.54 | 0.48 | 0.53 |
22
+ | — | Do-Nothing baseline | 0.58 | 0.51 | 0.45 | 0.51 |
23
+ | — | Always-Discharge | 0.59 | 0.51 | 0.45 | 0.52 |
24
+ | — | Always-Diesel | 0.42 | 0.42 | 0.44 | 0.43 |
25
+
26
+ ---
27
+
28
+ ## Capability Tiers
29
+
30
+ | Tier | Score Range | Agents |
31
+ |---|---|---|
32
+ | **Frontier** | 0.74 - 0.78 | Grok-4, GPT-5.4, Gemma-4-31B, Grok 4.20, DeepSeek V3.2 |
33
+ | **Hand-coded baseline** | 0.77 | Oracle (rule-based) |
34
+ | **Mid-tier** | 0.60 - 0.64 | GPT-5.4-mini, Qwen 3.6 Plus |
35
+ | **Weak** | 0.51 - 0.55 | Kimi K2.5, Gemini 3.1 Pro Preview |
36
+ | **No-intelligence baselines** | 0.43 - 0.52 | Do-Nothing, Always-Discharge, Always-Diesel |
37
+
38
+ ---
39
+
40
+ ## Per-Task Breakdown
41
+
42
+ ### Task 1: Normal Summer (Easy)
43
+ *Tests basic battery arbitrage. ~100 kW avg demand, Rs 3-12 prices, no heatwave.*
44
+
45
+ | Rank | Agent | Score |
46
+ |---:|---|:---:|
47
+ | 1 | Gemma-4-31B | **0.81** |
48
+ | 1 | Grok 4.20 Multi-Agent | **0.81** |
49
+ | 3 | Grok-4 | 0.80 |
50
+ | 3 | DeepSeek V3.2 | 0.80 |
51
+ | 5 | Oracle | 0.79 |
52
+ | 5 | GPT-5.4 | 0.79 |
53
+ | 7 | GPT-5.4-mini | 0.72 |
54
+ | 8 | Qwen 3.6 Plus | 0.69 |
55
+ | 9 | Gemini 3.1 Pro Preview | 0.65 |
56
+ | 10 | Always-Discharge | 0.59 |
57
+ | 11 | Do-Nothing | 0.58 |
58
+ | 12 | Kimi K2.5 | 0.57 |
59
+ | 13 | Always-Diesel | 0.42 |
60
+
61
+ ### Task 2: Heatwave + Price Spike (Medium)
62
+ *Tests temporal planning. Day 2-3 heatwave (+30% demand), Rs 20 evening price spike visible in 4h forecast.*
63
+
64
+ | Rank | Agent | Score |
65
+ |---:|---|:---:|
66
+ | 1 | Grok-4 | **0.82** |
67
+ | 2 | Oracle | 0.81 |
68
+ | 3 | Grok 4.20 Multi-Agent | 0.80 |
69
+ | 4 | Gemma-4-31B | 0.79 |
70
+ | 4 | DeepSeek V3.2 | 0.79 |
71
+ | 4 | GPT-5.4 | 0.79 |
72
+ | 7 | GPT-5.4-mini | 0.74 |
73
+ | 8 | Qwen 3.6 Plus | 0.67 |
74
+ | 9 | Kimi K2.5 | 0.54 |
75
+ | 10 | Gemini 3.1 Pro Preview | 0.53 |
76
+ | 11 | Do-Nothing | 0.51 |
77
+ | 11 | Always-Discharge | 0.51 |
78
+ | 13 | Always-Diesel | 0.42 |
79
+
80
+ ### Task 3: Extreme Crisis + Grid Outage (Hard)
81
+ *Tests constraint management. Full 3-day heatwave, -30% solar, +50% demand, limited diesel, 6-hour grid outage on Day 2.*
82
+
83
+ | Rank | Agent | Score |
84
+ |---:|---|:---:|
85
+ | 1 | Grok-4 | **0.72** |
86
+ | 2 | Oracle | 0.70 |
87
+ | 3 | GPT-5.4 | 0.67 |
88
+ | 4 | Gemma-4-31B | 0.62 |
89
+ | 4 | DeepSeek V3.2 | 0.62 |
90
+ | 6 | Grok 4.20 Multi-Agent | 0.60 |
91
+ | 7 | Kimi K2.5 | 0.48 |
92
+ | 8 | Gemini 3.1 Pro Preview | 0.47 |
93
+ | 9 | GPT-5.4-mini | 0.46 |
94
+ | 10 | Qwen 3.6 Plus | 0.45 |
95
+ | 10 | Do-Nothing | 0.45 |
96
+ | 10 | Always-Discharge | 0.45 |
97
+ | 13 | Always-Diesel | 0.44 |
98
+
99
+ ---
100
+
101
+ ## Key Observations
102
+
103
+ 1. **The environment cleanly differentiates capability.** A clean gradient from `do-nothing` (0.51 avg) through frontier LLMs (0.78). Every model lands in a different tier.
104
+
105
+ 2. **Task 3 is the real differentiator.** The 6-hour grid outage forces true islanding behavior. Only Grok-4 and the Oracle handle it well (>0.70). Most LLMs collapse to ~0.45 — the same as do-nothing.
106
+
107
+ 3. **Frontier LLMs match or beat the hand-coded oracle.** Grok-4 (0.78) > Oracle (0.77) — the environment is solvable by raw LLM reasoning, but requires real intelligence.
108
+
109
+ 4. **Smaller LLMs barely beat do-nothing.** Kimi K2.5 (0.53) and Gemini 3.1 Pro Preview (0.55) are within rounding error of the do-nothing baseline (0.51) — they struggle to produce useful actions consistently.
110
+
111
+ 5. **Capability scales with model size within a family.** GPT-5.4 (0.75) significantly outperforms GPT-5.4-mini (0.64). Same prompt, same environment — only the model size differs.
112
+
113
+ 6. **The 0.20-0.35 gap between best and worst agents** proves the environment has real optimization headroom and isn't trivially solvable.
114
+
115
+ ---
116
+
117
+ ## Reproducibility
118
+
119
+ All scores are deterministic. To reproduce:
120
+
121
+ ```bash
122
+ export API_BASE_URL="https://openrouter.ai/api/v1"
123
+ export HF_TOKEN="<your-key>"
124
+ export MODEL_NAME="<model-id>"
125
+ python inference.py
126
+ ```
127
+
128
+ Output is structured `[START] / [STEP] / [END]` blocks with explicit task names and scores. Same seed (42) + same model = identical scores across runs.
129
+
130
+ To run hand-coded baselines (no API key needed):
131
+
132
+ ```bash
133
+ python scripts/oracle_test.py
134
+ ```