| # 06 — Execution Plan: What We'll Do When You Say "START" |
|
|
| ## 🚀 The Plan |
|
|
| When you say **"START"**, here is the EXACT sequence of steps we'll follow. |
| Each step has a clear goal, estimated time, and cost. |
|
|
| --- |
|
|
| ## Phase 1: Setup & Validation (15 minutes) |
|
|
| ### Step 1.1: Create Training Sandbox |
| **What:** Set up a GPU sandbox with all dependencies installed |
| **Why:** Test that everything works before spending money on a real training job |
| **Time:** 5 minutes |
| **Cost:** $0 |
|
|
| ```bash |
| pip install transformers trl peft datasets accelerate bitsandbytes torch trackio |
| ``` |
|
|
| ### Step 1.2: Validate Dataset Format |
| **What:** Load your dataset and verify it works with SFTTrainer |
| **Why:** Catch format issues BEFORE training starts (saves hours of debugging) |
| **Time:** 5 minutes |
| **Cost:** $0 |
|
|
| ```python |
| from datasets import load_dataset |
| dataset = load_dataset("muhammadtlha944/mcp-agent-training-data") |
| print(dataset["train"][0]) # Peek at first example |
| ``` |
|
|
| ### Step 1.3: Verify Model Compatibility |
| **What:** Load Qwen3-1.7B tokenizer and test chat template |
| **Why:** Make sure the model can process our messages format |
| **Time:** 5 minutes |
| **Cost:** $0 |
|
|
| ```python |
| from transformers import AutoTokenizer |
| tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B") |
| print(tokenizer.chat_template) # Should not be None |
| ``` |
|
|
| --- |
|
|
| ## Phase 2: Training Script Development (30 minutes) |
|
|
| ### Step 2.1: Write Training Script |
| **What:** Create `train.py` with full educational comments |
| **Why:** Every line documented so you learn as we build |
| **Time:** 15 minutes |
| **Cost:** $0 |
|
|
| **What the script contains:** |
| - LoRA configuration (r=16, all-linear, dropout=0.05) |
| - SFTConfig with all hyperparameters documented |
| - Trackio monitoring setup |
| - push_to_hub configuration |
| - Plain-text logging (no tqdm progress bars) |
|
|
| ### Step 2.2: Test Script in Sandbox |
| **What:** Run the script for 10 steps to catch errors |
| **Why:** Find bugs NOW before the expensive training job |
| **Time:** 10 minutes |
| **Cost:** $0 (sandbox GPU time) |
|
|
| ```python |
| # Run just 10 steps as a smoke test |
| training_args.max_steps = 10 |
| trainer.train() |
| ``` |
|
|
| ### Step 2.3: Review & Fix Issues |
| **What:** Fix any import errors, API mismatches, or config issues |
| **Why:** Training jobs are expensive — we only launch when the script is solid |
| **Time:** 5 minutes |
| **Cost:** $0 |
|
|
| --- |
|
|
| ## Phase 3: Model Training (2-3 hours) |
|
|
| ### Step 3.1: Launch Training Job |
| **What:** Submit training to HF Jobs on T4 GPU |
| **Why:** T4 is cheapest GPU that fits our model (16GB VRAM) |
| **Time:** 2-3 hours (automated) |
| **Cost:** ~$1.20-1.80 |
|
|
| **Pre-flight check before launch:** |
| - ✅ Dataset format validated |
| - ✅ Script tested in sandbox |
| - ✅ push_to_hub=True and hub_model_id set |
| - ✅ Timeout set to 4 hours (plenty of buffer) |
| - ✅ Trackio monitoring enabled |
| - ✅ disable_tqdm=True for clean logs |
| |
| ### Step 3.2: Monitor Training |
| **What:** Watch loss curves via Trackio dashboard |
| **Why:** Make sure loss is going down (model is learning) |
| **Time:** Check every 15 minutes |
| **Cost:** $0 (just watching) |
| |
| **What to watch for:** |
| ``` |
| Good: Step 100: loss=2.5 → Step 500: loss=1.2 → Step 2450: loss=0.9 |
| Warning: Step 100: loss=2.5 → Step 500: loss=2.4 → Step 1000: loss=2.3 |
| (Learning very slowly — might need more epochs or higher LR) |
| Bad: Step 100: loss=2.5 → Step 500: loss=3.0 → Step 1000: loss=3.5 |
| (Loss going UP — stop immediately, something is wrong) |
| ``` |
| |
| ### Step 3.3: Verify Model Pushed to Hub |
| **What:** Check that the model appears in your HF repo |
| **Why:** Job storage is ephemeral — if push_to_hub fails, model is LOST |
| **Time:** 5 minutes |
| **Cost:** $0 |
| |
| **Check URL:** https://huggingface.co/muhammadtlha944/MCP-Agent-1.7B |
| |
| --- |
| |
| ## Phase 4: Testing & Evaluation (30 minutes) |
| |
| ### Step 4.1: Load Trained Model |
| **What:** Download the model from Hub and test inference |
| **Why:** Verify the model actually works after training |
| **Time:** 10 minutes |
| **Cost:** $0 |
| |
| ```python |
| from transformers import pipeline |
| pipe = pipeline("text-generation", model="muhammadtlha944/MCP-Agent-1.7B") |
| ``` |
| |
| ### Step 4.2: Run Test Prompts |
| **What:** Test the model on real tool-calling scenarios |
| **Why:** See if training actually worked |
| **Time:** 10 minutes |
| **Cost:** $0 |
| |
| **Test cases:** |
| 1. Simple tool call: "Find all Python files" |
| 2. Multi-step: "Clone a repo and find TODO comments" |
| 3. Clarification: "Book a flight" (missing info) |
| 4. Safety: "Delete all files" (should refuse) |
| 5. MCP format: "Use the github_search tool to find ML repos" |
|
|
| ### Step 4.3: Document Results |
| **What:** Save test outputs and observations |
| **Why:** Track what works and what needs improvement |
| **Time:** 10 minutes |
| **Cost:** $0 |
|
|
| --- |
|
|
| ## Phase 5: Agent Harness App (1 hour) |
|
|
| ### Step 5.1: Write Agent App |
| **What:** Create `app.py` with Gradio UI + ReAct loop + tool registry |
| **Why:** Turn the model into an actual usable agent |
| **Time:** 30 minutes |
| **Cost:** $0 |
|
|
| **What the app contains:** |
| - Gradio chat interface |
| - Agent mode toggle (on/off) |
| - Tool registry with 7 built-in tools |
| - ReAct loop (think → act → observe → repeat) |
| - Tool execution log |
| - Safety filters (block dangerous commands) |
|
|
| ### Step 5.2: Test Agent Locally |
| **What:** Run the app and test with real user queries |
| **Why:** Make sure the whole system works end-to-end |
| **Time:** 15 minutes |
| **Cost:** $0 |
|
|
| ### Step 5.3: Deploy to HF Space |
| **What:** Upload app to a Gradio Space |
| **Why:** Share with the world! |
| **Time:** 15 minutes |
| **Cost:** $0 (Spaces free tier) |
|
|
| --- |
|
|
| ## Phase 6: Documentation & Publication (30 minutes) |
|
|
| ### Step 6.1: Update Model README |
| **What:** Write a compelling README for the model card |
| **Why:** Model cards are how people discover and understand your model |
| **Time:** 15 minutes |
| **Cost:** $0 |
|
|
| **What to include:** |
| - What the model does |
| - How it was trained |
| - How to use it |
| - Benchmarks/results |
| - Limitations |
| - Citation info |
|
|
| ### Step 6.2: Create Dataset Card |
| **What:** Document the training dataset |
| **Why:** Transparency is valued in the ML community |
| **Time:** 10 minutes |
| **Cost:** $0 |
|
|
| ### Step 6.3: Share Results |
| **What:** Post on social media, share with community |
| **Why:** Get feedback, attract collaborators |
| **Time:** 5 minutes |
| **Cost:** $0 |
|
|
| --- |
|
|
| ## 📅 Timeline Summary |
|
|
| | Phase | Steps | Time | Cost | Cumulative | |
| |-------|-------|------|------|------------| |
| | 1. Setup | 1.1-1.3 | 15 min | $0 | 15 min / $0 | |
| | 2. Script | 2.1-2.3 | 30 min | $0 | 45 min / $0 | |
| | 3. Training | 3.1-3.3 | 2-3 hrs | ~$1.50 | 3-4 hrs / $1.50 | |
| | 4. Testing | 4.1-4.3 | 30 min | $0 | 3.5-4.5 hrs / $1.50 | |
| | 5. App | 5.1-5.3 | 1 hr | $0 | 4.5-5.5 hrs / $1.50 | |
| | 6. Publish | 6.1-6.3 | 30 min | $0 | 5-6 hrs / $1.50 | |
|
|
| **Total time:** ~5-6 hours of active work |
| **Total cost:** ~$1.50 (training only) |
| **Total budget used:** ~15% of $10 budget ✅ |
|
|
| --- |
|
|
| ## 🎯 Decision Points |
|
|
| At each phase, we'll make decisions based on results: |
|
|
| ### After Phase 3 (Training): |
| **If training loss < 1.5 and eval loss < 1.8:** ✅ Proceed to testing |
| **If training loss > 2.0:** ⚠️ Consider more epochs or higher LR |
| **If eval loss >> train loss:** ❌ Overfitting — need more data or lower rank |
| **If model didn't push to Hub:** ❌ Stop and fix push_to_hub configuration |
|
|
| ### After Phase 4 (Testing): |
| **If model generates tool calls correctly:** ✅ Proceed to app |
| **If model generates text but not tool calls:** ⚠️ Need more MCP-specific training data |
| **If model hallucinates tools:** ⚠️ Need more diverse tool schemas in data |
| **If model refuses everything:** ⚠️ Too much safety data — need balance |
|
|
| ### After Phase 5 (App): |
| **If app works end-to-end:** ✅ Publish and celebrate! |
| **If tools fail to execute:** ⚠️ Fix tool implementations |
| **If model runs out of context:** ⚠️ Reduce max_iterations or use sliding window |
| |
| --- |
| |
| ## 💡 What You'll Learn During Execution |
| |
| ### During Phase 1: |
| - How to set up a GPU environment |
| - How to validate data formats |
| - How model tokenizers work |
| |
| ### During Phase 2: |
| - How to write production training scripts |
| - How LoRA configuration works |
| - How SFTConfig parameters affect training |
| |
| ### During Phase 3: |
| - How to submit jobs to cloud GPUs |
| - How to monitor training in real-time |
| - How to read loss curves |
| - How Trackio dashboards work |
| |
| ### During Phase 4: |
| - How to load fine-tuned models |
| - How to test models systematically |
| - How to identify model weaknesses |
| |
| ### During Phase 5: |
| - How to build agent applications |
| - How the ReAct pattern works in practice |
| - How tool registries function |
| - How to deploy Gradio apps |
| |
| ### During Phase 6: |
| - How to write effective model cards |
| - How to share research with the community |
| |
| --- |
| |
| ## 🚨 Contingency Plans |
| |
| ### If Training Fails (OOM Error) |
| **Symptom:** "CUDA out of memory" error |
| **Fix:** |
| 1. Reduce batch_size from 4 to 2 (keep accumulation at 4 → effective batch = 8) |
| 2. Reduce max_seq_length from 2048 to 1024 |
| 3. If still fails, use gradient checkpointing (already enabled) |
| 4. Last resort: upgrade to a10g-small (24GB VRAM, ~$1.20/hr) |
|
|
| ### If Training Is Too Slow |
| **Symptom:** Loss barely moving after 1 hour |
| **Fix:** |
| 1. Check learning rate — might be too low |
| 2. Increase warmup ratio from 0.1 to 0.2 |
| 3. Reduce gradient accumulation from 4 to 2 (faster but less stable) |
|
|
| ### If Model Doesn't Generate Tool Calls |
| **Symptom:** Model answers questions normally but doesn't use tools |
| **Fix:** |
| 1. Add more MCP-specific training data |
| 2. Adjust system prompt to emphasize tool use |
| 3. Use higher temperature (0.9) to encourage creativity |
| 4. Add few-shot examples in the system prompt |
|
|
| ### If Push to Hub Fails |
| **Symptom:** Model trained but not on Hub |
| **Fix:** |
| 1. Check HF token has write permissions |
| 2. Manually upload: `trainer.push_to_hub()` after training |
| 3. Save locally first: `trainer.save_model("./local-save")` |
|
|
| --- |
|
|
| ## 🎉 Success Criteria |
|
|
| We'll consider this project a success when: |
|
|
| - ✅ Model trains without errors (loss < 1.5) |
| - ✅ Model pushed to Hub successfully |
| - ✅ Model generates structured tool calls on test prompts |
| - ✅ Agent app runs locally with tool execution |
| - ✅ App deployed to HF Space |
| - ✅ Total cost under $10 (target: $1.50) |
|
|
| --- |
|
|
| ## 🚀 Ready? |
|
|
| When you've read all the files and feel confident, just say: |
|
|
| > **"START"** |
|
|
| And we'll begin with Phase 1. |
|
|
| --- |
|
|
| *Learning ML by building real things — one step at a time.* |
|
|
