| # TIL-26-AE: Automated Exploration Bomberman Agent |
|
|
| **Repository**: `E-Rong/til-26-ae-agent` |
| **Challenge**: The Intelligent League (TIL) — Automated Exploration (AE) |
| **Base Environment**: `e-rong/til-26-ae` Space |
| **Model Repo**: `E-Rong/til-26-ae-agent` (checkpoints + inference code) |
|
|
| --- |
|
|
| ## Table of Contents |
|
|
| 1. [Research & Literature Review](#1-research--literature-review) |
| 2. [Problem Analysis](#2-problem-analysis) |
| 3. [Development Decisions](#3-development-decisions) |
| 4. [Training Phases](#4-training-phases) |
| 5. [Results](#5-results) |
| 6. [Artifacts](#6-artifacts) |
| 7. [Next Steps](#7-next-steps) |
|
|
| --- |
|
|
| ## 1. Research & Literature Review |
|
|
| ### 1.1 Domain: Multi-Agent Bomberman RL |
|
|
| The TIL-26-AE challenge is a multi-agent Bomberman-like environment where agents navigate a grid, collect resources, place bombs, destroy walls, and eliminate opponents. The key challenge is **autonomous exploration**. |
|
|
| ### 1.2 Key Papers |
|
|
| | Paper | arXiv ID | Key Insight | Relevance | |
| |---|---|---|---| |
| | *Pommerman: A Multi-Agent Benchmark* | 2407.00662 | PettingZoo + parallel env standard | Confirmed approach | |
| | *MAPPO* | 2103.01955 | Shared parameters, curriculum | Justified curriculum | |
| | *Invalid Action Masking* | 2006.14171 | Masks logits before softmax | **Directly applicable** | |
| | *PPO Algorithms* | 1707.06347 | Clipped surrogate, stable | Chosen over DQN | |
|
|
| ### 1.3 Why MaskablePPO? |
|
|
| Bomberman agents cannot move into walls, out of bounds, or place bombs without stockpile. The observation includes `action_mask: uint8[6]`. Standard PPO would waste ~30-40% of samples on illegal moves. MaskablePPO masks logits before softmax, ensuring only legal actions are sampled. |
|
|
| ### 1.4 Why Curriculum Learning? |
|
|
| Training against strong opponents from scratch leads to catastrophic early losses (~0 reward). Curriculum learning (easy → hard) is standard in competitive multi-agent RL. |
|
|
| ### 1.5 Why Not DQN? |
|
|
| DQN struggles with action masking (requires custom architecture). PPO's on-policy updates handle non-stationarity of multi-agent self-play better, and has mature masking support in `sb3-contrib`. |
|
|
| --- |
|
|
| ## 2. Problem Analysis |
|
|
| ### 2.1 Environment Structure |
|
|
| - **Grid size**: 16×16 |
| - **Agents**: Configurable (default 2 teams, Phase 3 uses 3) |
| - **Observations**: Dict with `agent_viewcone[7×5×25]`, `base_viewcone[5×5×25]`, direction, location, health, `action_mask[6]`, etc. |
| - **Actions**: Discrete(6) — FORWARD, BACKWARD, LEFT, RIGHT, STAY, PLACE_BOMB |
| - **Episode length**: ~200 steps |
| |
| ### 2.2 Observation Flattening |
| |
| Flattened to **1511-dim vector**: agent_viewcone(875) + base_viewcone(625) + 11 scalars. |
| |
| ### 2.3 Action Masking |
| |
| Critical bug found: `Monitor` must wrap *outside* `ActionMasker`, not inside. Otherwise `get_action_masks()` fails because `Monitor` does not expose `action_masks()`. |
|
|
| --- |
|
|
| ## 3. Development Decisions |
|
|
| ### 3.1 Single-Agent Wrapper |
|
|
| Controls only `agent_0`; opponents use random (Phase 1-2) or rule-based (Phase 3) policies. Reduces to single-agent RL with non-stationary environment. |
|
|
| ### 3.2 3-Phase Curriculum |
|
|
| | Phase | Opponent | Duration | Purpose | |
| |---|---|---|---| |
| | **1** | Random | 500k | Learn movement, bombs, basics | |
| | **2** | Random + exploration bonus | 500k | Prevent camping exploit | |
| | **3** | Rule-based curriculum | 1M | Generalize to structured opponents | |
|
|
| ### 3.3 Philosophy |
|
|
| - `stable-baselines3` for PPO core |
| - `sb3-contrib` for MaskablePPO + ActionMasker |
| - `huggingface_hub` for persistent checkpoint storage |
|
|
| ### 3.4 Why Hub Every 50k Steps |
|
|
| Sandbox resets (T4 container recycling) caused local `/app/data/` loss multiple times. Hub checkpointing saved the project at 400k steps when training crashed. |
|
|
| --- |
|
|
| ## 4. Training Phases |
|
|
| ### 4.1 Phase 1: Foundation (vs Random) |
|
|
| **Duration**: 500,352 steps |
| **Result**: Win rate 92%, avg reward 180.1, 100% survival |
| **Challenges**: Wrapper ordering, dependency issues, sandbox resets |
|
|
| ### 4.2 Phase 2: Exploration Shaping (IN PROGRESS) |
|
|
| **Status**: Started at 500352 steps, running on A10G at ~54 FPS |
| **Mechanism**: Visit-count bonus = 1/(1+visits), adaptive annealing via tanh(avg_enemy_deaths) |
| **ETA**: ~2.5 hours, targets 1,000,352 total steps |
| **Purpose**: Force map exploration, prevent safe base-camping |
|
|
| ### 4.3 Phase 3: Curriculum Self-Play |
|
|
| **Pending**: Rule-based static → simple → smart → mixed, 3 teams, 1M steps |
|
|
| --- |
|
|
| ## 5. Results |
|
|
| ### 5.1 Phase 1 Results |
|
|
| | Metric | Value | |
| |---|---| |
| | Timesteps | 500,352 | |
| | Final Reward | 237.0 | |
| | FPS | 52 (A10G) | |
| | Wall time | ~2h 15min | |
| | Win Rate (eval) | **92.0%** | |
| | Avg Reward (eval) | **180.1** | |
| | Survival Rate | **100.0%** | |
|
|
| ### 5.2 Phase 2 Interim (Early) |
|
|
| | Metric | Value | |
| |---|---| |
| | Starting Step | 500,352 | |
| | Initial Reward (shaped) | 210 | |
| | FPS | 54 | |
| | Explore Weight | Adaptive k=1.2 | |
|
|
| --- |
|
|
| ## 6. Artifacts |
|
|
| | File | Purpose | |
| |---|---| |
| | `phase1_final.zip` | Trained model | |
| | `phase2_final.zip` | *(in progress)* | |
| | `ckpt_50000-400000.zip` | Phase 1 intermediates | |
| | `ae_manager.py` | Inference code | |
| | `docs/ae.md` | This documentation | |
|
|
| --- |
|
|
| ## 7. Next Steps |
|
|
| - **Phase 2**: Complete 500k exploration-shaping steps |
| - **Phase 3**: Curriculum vs rule-based opponents (1M steps) |
| - **Eval**: Multi-team evaluation vs smart opponents |
| - **Future**: CNN policy, opponent modeling, LSTM memory |
|
|
| *Last updated: 2026-05-14 — Phase 2 in progress* |
|
|
