aegis_training / WORKER_README.md
YashashMathur's picture
Add worker setup guide
7275c33 verified

Worker Agent Setup Guide

Architecture

                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                    β”‚   OVERSEER      β”‚
                    β”‚ (Your HF Space) β”‚
                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                             β”‚
              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
              β”‚              β”‚              β”‚
              β–Ό              β–Ό              β–Ό
     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
     β”‚ Friend's HF β”‚  β”‚   Your      β”‚  β”‚   Your      β”‚
     β”‚   Space     β”‚  β”‚  Laptop     β”‚  β”‚  Laptop      β”‚
     β”‚ (Worker 1) β”‚  β”‚ (Worker 2)  β”‚  β”‚ (Overseer)   β”‚
     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

You have $30 credits. Here's how to use them efficiently:


PART 1: Friend's HF Space (Worker 1)

Your friend deploys this on their HF account using your trained model.

Steps for Your Friend:

  1. Create New HF Space

    • Go to: https://huggingface.co/spaces/new
    • Name: aegis-worker-{friendname}
    • Type: Space
    • SDK: Docker
    • Hardware: A10G (small) - $3/hour
    • Visibilty: Private (if they want)
  2. Upload Files

    • Upload these 3 files:
      • hf_worker.py
      • requirements.txt
      • Dockerfile
  3. Set Environment Variables

    • Go to Space Settings β†’ Variables
    • Add:
      • HF_TOKEN = your token (get from HF Settings β†’ Access Tokens)
      • WORKER_MODEL = your trained model repo
  4. IMPORTANT: Cost Management

    • Set a Repository Visibility to Private
    • After their turn, they must Factory Reboot the space to free resources
    • With $30 at $3/hr = ~10 hours of runtime
    • Can run in bursts: 1 hour at a time, then stop

PART 2: Your Laptop (Worker 2) - Optional

If you want your laptop as a worker too:

Setup:

# Install dependencies (if not already)
pip install -r worker_agent/requirements.txt

# Start the local worker
python worker_agent/local_worker.py

Expose to Internet with ngrok:

# In a new terminal
ngrok http 7861

Copy the ngrok URL (e.g., https://abc123.ngrok.io) and tell your overseer about it.


PART 3: Update Your Overseer

In your overseer (where you run the main model), add:

from overseer_worker_client import dispatch_to_worker, set_worker_url

# After friend deploys their space, update the URL:
# set_worker_url("friend_hf", "https://friend-space.hf.space/execute")
# OR if using ngrok:
# set_worker_url("local", "https://your-ngrok-url/execute")

# To send a task:
result = await dispatch_to_worker({
    "task_id": "task_001",
    "worker_role": "general-dev",
    "instructions": "Write a hello world program in Python",
    "context": "User wants a simple script"
})

Quick Reference

Component Cost When to Use
Friend's HF (A10G) ~$3/hr When friend is online
Your Laptop (free) $0 When laptop is on
ngrok (free) $0 Tunnel to laptop

Total: ~$30 covers ~10 hours of HF compute


Files Created

  • worker_agent/hf_worker.py - Code for friend's HF Space
  • worker_agent/local_worker.py - Code for your laptop
  • worker_agent/overseer_worker_client.py - Your overseer uses this
  • worker_agent/requirements.txt - Python dependencies
  • worker_agent/Dockerfile - Container config

Flow Example

  1. You ask overseer a question
  2. Overseer picks a worker (friend or laptop)
  3. Worker processes the task with your trained model
  4. Worker returns result to overseer
  5. Overseer evaluates and responds

This lets you use both your friend's credits AND your laptop as workers while your HF Space acts as the overseer/manager.