README.md

metadata

title: AutoMathReasoner (Calculus Environment)
emoji: 🧠
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false

♾️ AutoMathReasoner: Autonomous Mathematical Intelligence Environment

AutoMathReasoner is an OpenEnv-compliant reinforcement learning world formulated for the Recursive Policy Refinement of Language Models. The system focuses on the domain of Symbolic Calculus (Indefinite Integration), utilizing a dense, multi-objective reward architecture to bridge complexity gaps in mathematical reasoning.

---

🚀 Core Reasoning Technologies

The environment implements several advanced logic-steering protocols to ensure convergence on complex mathematical primitives.

1. Recursive Difficulty Ascent (LADDER)

The system employs a Recursive Task Decomposition mechanism where a failure on a parent task $\mathcal{T}_p$ triggers a search for a solvable basis ${\mathcal{T}_1, \dots, \mathcal{T}_k}$.

Given a complexity operator $\Phi$, we satisfy:

$$\mathrm{\Phi }({\mathcal{T}}_p)=\sum _{i=1}^n{\omega }_i\mathrm{\Phi }({\mathcal{T}}_i)\backslash Phi(\backslash mathcal\{T\}\_p)\; =\; \backslash sum\_\{i=1\}^n\; \backslash omega\_i\; \backslash Phi(\backslash mathcal\{T\}\_i)$$

Where variants $\mathcal{T}_i$ represent "stepping stones" that allow the policy to acquire base identities before attempting the coupled root problem.

2. Test-Time Adaptive Policy (TTRL)

For "truly difficult" integrals at the boundary of the model's current capability, the system supports Inference-Time Group Optimization. When presented with a novel hard task $\mathcal{G}$, the model:

1. Generates $m$ simpler variants on-the-fly.
2. Performs a high-step micro-RL update on these variants.
3. Cold-starts the final inference on $\mathcal{G}$ with the adapted policy weights.

Mathematically, we solve for an optimal local parameter shift:

$${\theta }^{\ast }=\arg \underset{{\theta }^{\mathrm{\prime }}}{\max }{\mathbb{E}}_{\mathcal{T}\sim \text{variants}(\mathcal{G})}[R(\tau ,{\pi }_{{\theta }^{\mathrm{\prime }}})]\backslash theta^*\; =\; \backslash arg\; \backslash max\_\{\backslash theta\text{'}\}\; \backslash mathbb\{E\}\_\{\backslash mathcal\{T\}\; \backslash sim\; \backslash text\{variants\}(\backslash mathcal\{G\})\}\; \backslash left[\; R(\backslash tau,\; \backslash pi\_\{\backslash theta\text{'}\})\; \backslash right]$$

3. Process-Aware Reward Shaping

Unlike binary "sparse" reward systems, we employ Dense Process Supervision. Every primitive transformation (e.g. $u$-substitution, integration by parts) is identified as a logical node.

The reward $R_{\text{shape}}$ is assigned as the line integral over the reasoning trajectory $\tau$: $$R_{\text{shape}}={\int }_{\tau }\mathrm{\Psi }(\mathbf{z})d\mathbf{z}R\_\{\backslash text\{shape\}\}\; =\; \backslash int\_\{\backslash tau\}\; \backslash Psi(\backslash mathbf\{z\})\; d\backslash mathbf\{z\}$$ where $\Psi$ evaluates the structural validity of each state transition relative to the ground-truth simplification steps.

4. Hard Negative Mining (Problem Persistence)

Failed tasks $\mathcal{T}_{fail}$ are not discarded. They are prioritized in the sampling buffer with a weight $W$ proportional to their failure frequency: $$W(\mathcal{T})\propto e^{\lambda \cdot \text{failures}(\mathcal{T})}W(\backslash mathcal\{T\})\; \backslash propto\; e^\{\backslash lambda\; \backslash cdot\; \backslash text\{failures\}(\backslash mathcal\{T\})\}$$ This forces the policy to repeatedly encounter "bottleneck" logic until the primitive is solved.

---

🏗️ System Architecture

The environment architecture follows a strictly decoupled schema between task generation, solution validation, and policy refinement.

graph TD
    subgraph EnvCore [Mathematical Environment Server]
        GE["Symbolic Generator (Sympy)"] -->|"Sample T"| Server["OpenEnv API (FastAPI)"]
        Server -->|"Verify F(x)"| VR["Numerical Verifier"]
        VR -->|"Law: FTC Derivative Test"| Server
        Server -->|"Compute Sum(R)"| RW["Reward Logic Engine"]
        RW --> Server
    end
    
    subgraph PolicyNode [Reinforcement Learning Client]
        Policy["Policy pi(theta)"] -->|"Action Trace (tau)"| Server
        Server -->|"Reward Observation"| Policy
    end
    
    classDef space fill:transparent,stroke:#9370DB,stroke-width:2px;
    classDef client fill:transparent,stroke:#008B8B,stroke-width:2px;
    
    class EnvCore space
    class PolicyNode client

---

🔁 Systemic Logic: Recursive Difficulty Ascent

The environment operates via Autonomous Difficulty Scaling. Instead of fixed-difficulty benchmarks, a problem $\mathcal{T}$ is decomposed into a hierarchical tree of simpler primitives. For any parent problem $\mathcal{T}_{\text{p}}$ that fails to elicit a reward, the system generates a set of variants ${\mathcal{T}_i}$ such that the complexity metric $\mathcal{M}$ satisfies:

This ensures a continuous gradient for the learner, moving from fundamental algebraic identities to nested transcendental integrals.

---

🎯 The Reward Law

The terminal reward $R_{\Sigma}$ is a weighted composite of seven distinct mathematical and structural signals, designed to penalize hacking and reward rigorous proof-like trajectories:

$$R_{\mathrm{\Sigma }}=\alpha C+\beta Q+\gamma P+\delta R_{\text{ref}}+\eta D+\zeta E+\lambda XR\_\{\backslash Sigma\}\; =\; \backslash alpha\; C\; +\; \backslash beta\; Q\; +\; \backslash gamma\; P\; +\; \backslash delta\; R\_\{\backslash text\{ref\}\}\; +\; \backslash eta\; D\; +\; \backslash zeta\; E\; +\; \backslash lambda\; X$$

Where the weights are calibrated as $\alpha=0.35, \beta=0.15, \gamma=0.1, \delta=0.1, \eta=0.15, \zeta=0.05, \lambda=0.1$.

1. Fundamental Correctness ($C$)

Derived from the Numerical Multi-point Quadrature Protocol. A predicted solution $F_{\theta}(x)$ is verified against the target integrand $f(x)$ through the derivative identity:

Where $\mathbb{X} = {x_1, \dots, x_5}$ is a set of random points sampled from $\mathcal{U}(-5, 5)$.

2. Reasoning Formatting ($Q$)

Calculates the structural density of the reasoning trace using a hyperbolic tangent squashing function to bound heuristic markers:

$$Q=\tanh (\omega \cdot \text{count}(\text{markers}))Q\; =\; \backslash tanh(\backslash omega\; \backslash cdot\; \backslash text\{count\}(\backslash text\{markers\}))$$

3. Process Supervision ($P$)

Assigns a scalar reward for explicit step-wise transition logic. It algorithmically penalizes "Inferential Jumps" where the ratio of reasoning tokens to mathematical complexity falls below a critical threshold.

4. Reflection Logits ($R_{\text{ref}}$)

Rewards the presence of self-correction tokens when they lead to a terminal state correction. If the model reflects ($r=1$) but fails to correct the solution ($c=0$), it suffers a penalty of $-0.5$.

5. Trajectory Diversity ($D$)

Prevents the policy from converging on rote-memorized repetitive strings. If the current answer $A_t$ has been seen in history $\mathcal{H}$, an exponential penalty is applied:

6. Information Density Efficiency ($E$)

Guides the model toward concise mathematical proofs using a Gaussian decay centered at an optimal token length $\phi=50$:

$$E=\exp (-{\left(\frac{\text{len}(\tau )\mathrm{/}4-\varphi }{\varphi }\right)}^2)-1E\; =\; \backslash exp\backslash left(-\backslash left(\backslash frac\{\backslash text\{len\}(\backslash tau)/4\; -\; \backslash phi\}\{\backslash phi\}\backslash right)^2\backslash right)\; -\; 1$$

7. Global Exploration Bonus ($X$)

Rewards token-level variance relative to the frequency of problem encounters $s$:

$$X=\frac{\log (1+\nu )}{\sqrt{1+s}}X\; =\; \backslash frac\{\backslash log(1\; +\; \backslash nu)\}\{\backslash sqrt\{1\; +\; s\}\}$$

Where $\nu$ is the ratio of unique tokens in the reasoning trace $\tau$.

---

🔄 The Interaction Loop

The environment manages the Difficulty Gradient to ensure the policy $\pi_{\theta}$ maintains exploration stability.

sequenceDiagram
    participant Model as Policy (pi)
    participant Engine as Recursion Engine
    participant Oracle as Calculus Verifier
    
    loop Optimization Batch
        Engine ->> Model: Sample Low-Complexity Variant
        Model ->> Oracle: Submit Solution F(x)
        Oracle -->> Model: Correctness Yield (C)
        
        Note over Model: Internal State Update
        
        Engine ->> Model: Sample Root Complexity Task
        Model ->> Oracle: Proof Trajectory (tau)
        Oracle -->> Model: Composite Reward (R)
    end

---

💻 Running the Environment

1. Launch the Environment

# Install local calculus bindings
uv pip install -e .

# Start the environment server
uv run server

2. Training Initiation

# Executes the recursive training sequence
python train/train_grpo.py