Hugging Face's logo

Spaces:
Imaginephoenix
/
openenv1
Sleeping

App Files Community
openenv1 / README.md
Imaginephoenix's picture
Imaginephoenix
Upload README.md
d547d85 verified
preview code
|
raw
history blame
14.1 kB
metadata 
title: OpenEnv Email Triage Environment
emoji: 📬
colorFrom: blue
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

OpenEnv Email Triage Environment

A real-world AI agent training environment that simulates professional email triage. Built to the OpenEnv specification for standardized agent evaluation and benchmarking.

  • Status: In Development
  • Domain: Email Triage
  • Deployment: Hugging Face Spaces (Docker)

Table of Contents

What Is This?

This environment simulates a professional email inbox where an AI agent must:

  1. Read incoming emails with realistic metadata (sender, subject, body, thread history).
  2. Classify each email with the correct priority label.
  3. Route each email to the appropriate department or person.
  4. Summarize the email's key information.

Think of it as OpenAI Gym for office work. Instead of balancing a pole, the agent triages an inbox. The environment provides structured observations, accepts structured actions, and returns graded rewards with partial credit.

Every decision is scored by a deterministic programmatic grader: no LLM-as-judge, no randomness, fully reproducible.

Who Is This For?

Audience Use Case
AI Safety Researchers Measure agent behavior on realistic tasks with known ground truth
LLM Agent Developers Benchmark models and prompting strategies on real-world work
RL Researchers Train agents with shaped rewards in a professional task environment
Companies Evaluate LLM agents before deploying them to handle real email

Observation Space

What the agent sees at each step:

Field Type Description
email_id str Unique identifier for this email
subject str Email subject line
body str Full email body text
sender str Sender's email address
timestamp str ISO 8601 timestamp of when the email was received
thread_history list[str] Previous messages in the email thread (may be empty)
task_id str Which task is currently active
step_number int Current step in the episode (0-indexed)
total_emails int Total number of emails to process in this task

The observation never contains the correct answer. The agent must reason from email content.

Action Space

What the agent must output at each step:

Field Type Allowed Values Description
label Literal "urgent", "normal", "spam", "archive" Priority classification
summary str Free text Brief summary of the email's content and intent
route_to str Free text ("billing", "safety", "engineering") Department or person

Example action JSON 

{
  "label": "urgent",
  "summary": "Customer reports a safety issue with product overheating",
  "route_to": "safety"
}

Tasks

Each task now contains multiple deterministic scenario variants. By default, /reset cycles through the public scenario pool for the selected task.

Private evaluation split selection is controlled server-side via environment configuration (OPENENV_EVAL_SPLIT), and client-side override can be disabled to preserve benchmark integrity.

To keep private evaluation data out of source control, supply hidden scenarios at runtime using OPENENV_PRIVATE_SCENARIOS_JSON (JSON object keyed by task id).

Example deployment configuration:

export OPENENV_EVAL_SPLIT="private_eval"
export OPENENV_ALLOW_CLIENT_EVAL_OVERRIDE="false"
export OPENENV_PRIVATE_SCENARIOS_JSON='{"task_easy":[{"scenario_id":"easy-private-001","emails":[{"email_id":"easy-p-001","subject":"Private billing exception","body":"Please correct invoice mismatch for contract addendum B-7 before end of day.","sender":"contracts@partner.example","timestamp":"2026-04-03T09:00:00Z","thread_history":["Customer requested corrected invoice reference."]}],"ground_truth":[{"label":"normal","route_to":"billing","priority_weight":1.0,"summary_keywords":["invoice mismatch","contract addendum","correct"]}]}],"task_medium":[],"task_hard":[]}'

Notes:

  • Keep this value in deployment secrets or runtime environment config.
  • Use valid JSON with double quotes only.
  • You can provide multiple scenarios per task by adding more objects to each task list.

Task 1 — Easy (task_easy)

Objective: Correctly classify a single unambiguous email.

Scoring:

  • Correct label: 1.0
  • Wrong label but correct routing: 0.3
  • Everything wrong: 0.0

Task 2 — Medium (task_medium)

Objective: Triage a queue of 5 emails with mixed priority signals.

Scoring:

  • Each email scored individually
  • Score = (correct labels / total emails) * priority weight factor
  • Higher-priority misclassifications are penalized more heavily
  • Final score = weighted mean of all individual scores

Task 3 — Hard (task_hard)

Objective: Handle a complex complaint that crosses multiple categories.

Scoring:

  • Escalated to safety: 0.4 weight
  • Correct routing: 0.3 weight
  • Marked as urgent: 0.3 weight
  • Penalty: -0.2 if marked as spam
  • Final score = weighted sum of sub-scores (clipped to 0.0 minimum)

Task 4 — Production (task_production)

Objective: Simulate a production inbox with mixed operational load across safety, engineering, billing, support, spam, and low-priority traffic.

Scoring:

  • Per-email weighted scoring by business priority
  • Route-noise penalty when actions route to too many teams
  • Summary quality based on contextual evidence keywords and anti-stuffing rules
  • Deterministic escalation follow-ups are inserted when critical triage is missed
  • Runtime controls available via /reset payload for production simulations:
    • production_profile: light | standard | heavy
    • business_hours_mode: true | false
    • escalation_mode: low | normal | high

Reward Function

The reward function provides dense training signal at every step, not just binary pass/fail.

Formula 

final_reward = base_score + progress_signal + trajectory_bonus - penalties - step_cost

Components

Component Value Condition
Base score 0.0-1.0 Raw grader score for the current step
Progress signal ~0.00 to ~0.13 Partial credit for advancing queue, quality, and positive trend
Step cost ~-0.005 to ~-0.015 Gentle efficiency pressure over longer episodes
Trajectory bonus +0.2 If all tasks completed with mean score > 0.8
Archive quality penalty -0.5 Archive action with an underspecified summary
Loop detection penalty -0.3 Same action repeated 3+ times consecutively

The final reward is clipped to [-1.0, 1.0] before being returned.

Quick Start

Prerequisites

  • Python 3.11+
  • API endpoint, model name, and token for inference

Installation 

pip install -r requirements.txt
export API_BASE_URL="https://router.huggingface.co/v1"
export MODEL_NAME="gpt-4o"
export HF_TOKEN="your-token-here"

Run the environment locally 

python server.py

curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "task_easy"}'

curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"label": "urgent", "summary": "Test", "route_to": "billing"}'

curl -X POST http://localhost:7860/state

Running Inference 

python inference.py --task all
python inference.py --task 1
python inference.py --task 4 --production-profile heavy --business-hours-mode --escalation-mode high

The script reads API settings from environment variables and uses fallback actions when model output is unparseable, so episodes still complete.

Inference Architecture

The inference script (inference.py) follows this loop:

1. Initialize OpenAI client + environment
2. reset() to get first observation
3. Loop until done or MAX_STEPS:
  - Build prompt from observation + history
  - Call LLM with OpenAI client (catch request errors)
  - Parse response into action (fallback on parse failure)
  - env.step(action)
  - Record reward and history
4. Print score table

Environment Variables Required 

export API_BASE_URL="https://router.huggingface.co/v1"
export MODEL_NAME="gpt-4o"
export HF_TOKEN="your-token-here"
export INFERENCE_RUNTIME_BUDGET_SECONDS="1140"
export INFERENCE_REQUEST_TIMEOUT_SECONDS="12"

Runtime controls:

  • INFERENCE_RUNTIME_BUDGET_SECONDS limits full-script wall-clock runtime (default 1140s, under 20 minutes).
  • INFERENCE_REQUEST_TIMEOUT_SECONDS limits each LLM request timeout (default 12s).
  • Equivalent CLI flags: --runtime-budget-seconds and --request-timeout-seconds.

Fallback behavior when parsing fails:

{"label": "normal", "summary": "Unable to parse response", "route_to": "general"}

Score Table

Placeholder until inference is run.

Model Task 1 (Easy) Task 2 (Medium) Task 3 (Hard) Mean
MODEL_NAME TBD TBD TBD TBD

Expected rough ranges:

  • GPT-4o: 0.8-1.0 on easy, 0.5-0.8 on medium, 0.4-0.7 on hard

Docker Deployment 

docker build -t email-triage-env .
docker run -p 7860:7860 email-triage-env

curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "task_easy"}'

For Apple Silicon:

docker build --platform linux/amd64 -t email-triage-env .

Hugging Face Space

Live URL placeholder:

https://huggingface.co/spaces/YOUR_USERNAME/email-triage-env

The Space homepage (/) now serves a lightweight interactive triage console for manual testing. Machine-readable service metadata is available at GET /meta.

Example interaction:

export SPACE_URL="https://YOUR_USERNAME-email-triage-env.hf.space"

curl -X POST "$SPACE_URL/reset" \
  -H "Content-Type: application/json" \
  -d '{"task_id": "task_easy"}'

Pre-Submission Validation

Run the validator before submitting your environment.

chmod +x validate-submission.sh
./validate-submission.sh https://YOUR_USERNAME-email-triage-env.hf.space .

The script checks:

  • HF Space /reset health (HTTP 200 expected)
  • Docker build success
  • openenv validate pass status

API Reference

POST /reset

Request:

{"task_id": "task_easy"}

Response:

{
  "observation": {
    "email_id": "easy-001",
    "subject": "Quarterly invoice available",
    "body": "...",
    "sender": "accounts@vendor-example.com",
    "timestamp": "2026-03-25T09:15:00Z",
    "thread_history": ["..."],
    "task_id": "task_easy",
    "step_number": 0,
    "total_emails": 1
  },
  "info": {"task_id": "task_easy", "step": 0}
}

POST /step

Request:

{
  "label": "urgent",
  "summary": "Customer needs immediate help",
  "route_to": "support"
}

Response:

{
  "observation": {},
  "reward": 0.85,
  "done": false,
  "info": {"step": 1, "task_id": "task_easy"}
}

POST /state

No request body required.

Response: EnvironmentState JSON object.

Project Structure 

.
├── models.py
├── tasks.py
├── graders.py
├── environment.py
├── server.py
├── server/
│   └── app.py
├── inference.py
├── openenv.yaml
├── Dockerfile
├── requirements.txt
├── pyproject.toml
├── uv.lock
├── validate-submission.sh
├── README.md
└── RULES.md

Known Limitations

Limitation Impact
Static scenario pools No live inbox ingestion from production systems
Single-agent server instance Concurrent agents can conflict
No live thread simulation Thread history is static
English-only content No multilingual coverage
No attachments Text-only triage
Simplified routing No org chart or availability modeling
Limited temporal dynamics Production task can generate deterministic escalations, but not full live message streams
Rule-based grading edges Equivalent decisions may score differently from humans

What an agent cannot exploit:

  • The correct answer is never present in observations
  • The grader is a pure function and cannot be manipulated
  • Step penalty cannot be bypassed except by efficient actions

Summary of Revision 2 Changes

What Changed Before After Why
Return type of step() tuple StepResult object Match sample result.observation pattern
Return type of reset() EmailObservation ResetResult object Match sample result.observation pattern
New models 4 models 6 models (+StepResult, +ResetResult) Match sample interface
API key reading OPENAI_API_KEY style HF_TOKEN or API_KEY via os.getenv Match sample fallback pattern
Temperature guidance 0 0.2 Match sample behavior
Response parsing JSON-only assumption Text parsing with fallback action Robustness to non-JSON model output
History tracking Optional Mandatory Match sample architecture
Step cap Not explicit MAX_STEPS constant Runtime safety and reproducibility

Contributing

Read RULES.md before contributing.

Key constraints:

  • Type hints and Pydantic models required
  • No extra dependencies without explicit approval
  • No features beyond project brief
  • Graders must remain deterministic pure functions

License

MIT License.