YashashMathur commited on
Commit
7275c33
Β·
verified Β·
1 Parent(s): 7bffd13

Add worker setup guide

Browse files
Files changed (1) hide show
  1. WORKER_README.md +137 -0
WORKER_README.md ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Worker Agent Setup Guide
2
+
3
+ ## Architecture
4
+
5
+ ```
6
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
7
+ β”‚ OVERSEER β”‚
8
+ β”‚ (Your HF Space) β”‚
9
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
10
+ β”‚
11
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
12
+ β”‚ β”‚ β”‚
13
+ β–Ό β–Ό β–Ό
14
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
15
+ β”‚ Friend's HF β”‚ β”‚ Your β”‚ β”‚ Your β”‚
16
+ β”‚ Space β”‚ β”‚ Laptop β”‚ β”‚ Laptop β”‚
17
+ β”‚ (Worker 1) β”‚ β”‚ (Worker 2) β”‚ β”‚ (Overseer) β”‚
18
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
19
+ ```
20
+
21
+ You have $30 credits. Here's how to use them efficiently:
22
+
23
+ ---
24
+
25
+ ## PART 1: Friend's HF Space (Worker 1)
26
+
27
+ Your friend deploys this on their HF account using your trained model.
28
+
29
+ ### Steps for Your Friend:
30
+
31
+ 1. **Create New HF Space**
32
+ - Go to: https://huggingface.co/spaces/new
33
+ - Name: `aegis-worker-{friendname}`
34
+ - Type: **Space**
35
+ - SDK: **Docker**
36
+ - Hardware: **A10G (small)** - $3/hour
37
+ - Visibilty: **Private** (if they want)
38
+
39
+ 2. **Upload Files**
40
+ - Upload these 3 files:
41
+ - `hf_worker.py`
42
+ - `requirements.txt`
43
+ - `Dockerfile`
44
+
45
+ 3. **Set Environment Variables**
46
+ - Go to Space Settings β†’ Variables
47
+ - Add:
48
+ - `HF_TOKEN` = your token (get from HF Settings β†’ Access Tokens)
49
+ - `WORKER_MODEL` = your trained model repo
50
+
51
+ 4. **IMPORTANT: Cost Management**
52
+ - Set a **Repository Visibility** to Private
53
+ - After their turn, they must **Factory Reboot** the space to free resources
54
+ - With $30 at $3/hr = ~10 hours of runtime
55
+ - Can run in bursts: 1 hour at a time, then stop
56
+
57
+ ---
58
+
59
+ ## PART 2: Your Laptop (Worker 2) - Optional
60
+
61
+ If you want your laptop as a worker too:
62
+
63
+ ### Setup:
64
+
65
+ ```bash
66
+ # Install dependencies (if not already)
67
+ pip install -r worker_agent/requirements.txt
68
+
69
+ # Start the local worker
70
+ python worker_agent/local_worker.py
71
+ ```
72
+
73
+ ### Expose to Internet with ngrok:
74
+
75
+ ```bash
76
+ # In a new terminal
77
+ ngrok http 7861
78
+ ```
79
+
80
+ **Copy the ngrok URL** (e.g., `https://abc123.ngrok.io`) and tell your overseer about it.
81
+
82
+ ---
83
+
84
+ ## PART 3: Update Your Overseer
85
+
86
+ In your overseer (where you run the main model), add:
87
+
88
+ ```python
89
+ from overseer_worker_client import dispatch_to_worker, set_worker_url
90
+
91
+ # After friend deploys their space, update the URL:
92
+ # set_worker_url("friend_hf", "https://friend-space.hf.space/execute")
93
+ # OR if using ngrok:
94
+ # set_worker_url("local", "https://your-ngrok-url/execute")
95
+
96
+ # To send a task:
97
+ result = await dispatch_to_worker({
98
+ "task_id": "task_001",
99
+ "worker_role": "general-dev",
100
+ "instructions": "Write a hello world program in Python",
101
+ "context": "User wants a simple script"
102
+ })
103
+ ```
104
+
105
+ ---
106
+
107
+ ## Quick Reference
108
+
109
+ | Component | Cost | When to Use |
110
+ |-----------|------|--------------|
111
+ | Friend's HF (A10G) | ~$3/hr | When friend is online |
112
+ | Your Laptop (free) | $0 | When laptop is on |
113
+ | ngrok (free) | $0 | Tunnel to laptop |
114
+
115
+ ### Total: ~$30 covers ~10 hours of HF compute
116
+
117
+ ---
118
+
119
+ ## Files Created
120
+
121
+ - `worker_agent/hf_worker.py` - Code for friend's HF Space
122
+ - `worker_agent/local_worker.py` - Code for your laptop
123
+ - `worker_agent/overseer_worker_client.py` - Your overseer uses this
124
+ - `worker_agent/requirements.txt` - Python dependencies
125
+ - `worker_agent/Dockerfile` - Container config
126
+
127
+ ---
128
+
129
+ ## Flow Example
130
+
131
+ 1. **You** ask overseer a question
132
+ 2. **Overseer** picks a worker (friend or laptop)
133
+ 3. **Worker** processes the task with your trained model
134
+ 4. **Worker** returns result to overseer
135
+ 5. **Overseer** evaluates and responds
136
+
137
+ This lets you use both your friend's credits AND your laptop as workers while your HF Space acts as the overseer/manager.