07 — Complete Tool Research: From Basic to "WOW"
🎯 Why We Did This Research
You said: "The tooling you showed me is very very basic. We need something like Manus but under budget and under size for a gaming PC."
We did deep R&D. Here's what we found.
🔬 What We Discovered: smolagents
HUGE finding: HuggingFace has a library called smolagents that's DESIGNED for building agents with small models. It changes everything about our architecture.
Why smolagents Is Perfect For Us
| Feature | What It Means For Us |
|---|---|
| CodeAgent | Model writes PYTHON CODE instead of JSON tool calls — much easier for a 1.7B model! |
| add_base_tools=True | Free built-in tools: DuckDuckGo search, Python interpreter, audio transcriber |
| Built-in browser agent | Real browser automation with Selenium + Helium |
| Multi-agent support | Multiple specialized agents that collaborate (like Manus!) |
| GradioUI | One-line web interface: GradioUI(agent).launch() |
| TransformersModel | Use our local Qwen3-1.7B model directly |
| Memory management | Agent remembers past interactions |
| Secure execution | Can use E2B sandbox or Docker for code safety |
| Push to Hub | agent.push_to_hub("username/agent") — share with the world |
The Key Insight: CodeAgent vs ToolCallingAgent
smolagents has two types of agents:
ToolCallingAgent (What We Were Planning)
# Model generates JSON like this:
{"tool": "search", "arguments": {"query": "cats"}}
- ❌ Needs to understand complex JSON schemas
- ❌ Limited to predefined tools
- ❌ Harder for small models (1.7B) to get right
CodeAgent (What We SHOULD Use)
# Model generates Python like this:
search_result = search("cats")
print(search_result)
- ✅ Model already knows Python (trained on code!)
- ✅ Can combine tools with loops, if/else, math
- ✅ More expressive — one "tool call" can do complex logic
- ✅ Easier for small models to generate valid Python than valid JSON
- ✅ No need to train model on tool schemas!
THIS IS HUGE: With CodeAgent, our Qwen3-1.7B model doesn't need to be trained on tool-calling at all! It just needs to know how to write Python code, which it already does! The training becomes about teaching it to solve problems by writing Python scripts.
🏗️ Revised Architecture: The "Real" Mini-Manus
Instead of manually building loops, we use smolagents:
┌─────────────────────────────────────────────┐
│ smolagents Framework │
│ │
│ ┌───────────────────────────────────────┐ │
│ │ Manager Agent (Qwen3-1.7B) │ │
│ │ "Break this task into subtasks" │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌─────────┐ │ │
│ │ │ WebAgent │ │ CodeAgent│ │Research │ │ │
│ │ │ │ │ │ │ Agent │ │ │
│ │ │ Browser │ │ Python │ │ Search │ │ │
│ │ │ Helium │ │ Executor │ │ + Crawl│ │ │
│ │ │ │ │ │ │ │ │ │
│ │ └────┬─────┘ └────┬─────┘ └────┬────┘ │ │
│ │ └────────────┴────────────┘ │ │
│ │ │ │ │
│ │ ┌──────────┴──────────┐ │ │
│ │ │ Results Combined │ │ │
│ │ └──────────┬──────────┘ │ │
│ │ │ │ │
│ │ Final Answer │ │
│ └───────────────────────────────────────┘ │
│ │
│ Built-in Tools (add_base_tools=True): │
│ • DuckDuckGo Web Search │
│ • Python Code Interpreter │
│ • Audio Transcription (Whisper) │
│ │
│ Custom Tools We Add: │
│ • Browser Automation (Selenium/Helium) │
│ • File System Operations │
│ • GitHub Repository Reader │
│ • Image Generation (local models) │
│ • Data Analysis (pandas, charts) │
│ • PDF/DOCX Processing │
│ • Email/Calendar (local integration) │
│ │
└─────────────────────────────────────────────┘
🧰 Complete Tool List: From "Meh" to "WOW"
TIER 0: Free Built-in (smolagents add_base_tools=True)
These come FREE with smolagents. Just set add_base_tools=True.
| Tool | What It Does | Wow | Cost | VRAM |
|---|---|---|---|---|
| DuckDuckGo Search | Search the web, get results | 5/10 | $0 | 0GB |
| Python Interpreter | Execute Python code safely | 6/10 | $0 | 0GB |
| Audio Transcriber | Convert speech to text (Whisper) | 5/10 | $0 | 0GB* |
*Whisper runs on CPU or tiny GPU — negligible VRAM.
TIER 1: Essential WOW Tools (Low Effort, High Impact)
These are the FIRST tools to add after the basics.
1. Browser Automation (Helium + Selenium) ⭐⭐⭐⭐⭐
What it does: The agent can literally control a web browser — click buttons, fill forms, scroll pages, extract data.
Demo scenario:
User: "Find the cheapest flight from NYC to London next week"
Agent:
1. Opens Google Flights
2. Enters departure (NYC)
3. Enters destination (London)
4. Sets dates (next week)
5. Clicks search
6. Extracts prices
7. Returns: "Cheapest: $450 on Delta, departing Nov 15"
How to implement:
# pip install selenium helium
from selenium import webdriver
from helium import start_chrome, click, write, press, scroll_down
@tool
def browse_website(url: str, task: str) -> str:
"""Open a website and perform actions to complete a task."""
driver = start_chrome(url, headless=True)
# Agent writes Python code using this tool
# Example: click("Search"), write("flights"), press(ENTER)
# Then extracts text from the page
return page_text
Requirements: Chrome/Chromium installed, ~500MB RAM for browser Cost: $0 Wow factor: 10/10 — Users see the agent BROWSING THE WEB
2. File System Manager ⭐⭐⭐⭐
What it does: Read, write, edit, organize files. Move, copy, delete, search.
Demo scenario:
User: "Organize all my downloads — put PDFs in Documents/PDFs, images in Pictures"
Agent:
1. Lists downloads folder
2. Identifies file types
3. Creates destination folders
4. Moves files by type
5. Returns: "Organized 47 files: 12 PDFs, 23 images, 5 videos, 7 other"
How to implement: Python os, shutil, pathlib — built-in!
Requirements: File system access (local) Cost: $0 Wow factor: 7/10 — Useful but expected
3. GitHub Repository Analyzer ⭐⭐⭐⭐⭐
What it does: Clone repos, analyze code structure, summarize what a project does, find bugs.
Demo scenario:
User: "What does this repo do? https://github.com/torvalds/linux"
Agent:
1. git clone the repo
2. Reads README.md
3. Lists top-level directories
4. Analyzes key files (Makefile, main.c)
5. Returns: "This is the Linux kernel source code.
It contains the core operating system: process scheduler,
memory management, device drivers, file systems..."
How to implement: git CLI + Python file reading
Requirements: git installed, ~500MB for repo storage Cost: $0 Wow factor: 9/10 — Instant code understanding
4. Data Analyst (Pandas + Charts) ⭐⭐⭐⭐⭐
What it does: Load CSVs, Excel files, JSON data. Clean, analyze, visualize with charts.
Demo scenario:
User: "Analyze this sales CSV and tell me trends"
Agent:
1. Reads sales_data.csv
2. Runs pandas analysis
3. Generates charts (matplotlib/seaborn)
4. Returns: "Sales increased 23% in Q3. Top product: Widget Pro ($45K revenue)."
+ shows chart image
How to implement:
# pip install pandas matplotlib seaborn openpyxl
import pandas as pd
import matplotlib.pyplot as plt
@tool
def analyze_data(file_path: str, question: str) -> str:
"""Load data file and answer questions about it."""
df = pd.read_csv(file_path) # or read_excel, read_json
# Agent writes Python code to analyze
# Generates charts, saves as images
return analysis_result + chart_image_path
Requirements: Python libraries, ~200MB Cost: $0 Wow factor: 9/10 — Professional data analysis in seconds
TIER 2: Advanced WOW Tools (Medium Effort, High Impact)
5. Image Generator (Local Stable Diffusion) ⭐⭐⭐⭐⭐
What it does: Generate images from text descriptions using local AI models.
Demo scenario:
User: "Create a logo for my coffee shop 'Bean There'"
Agent:
1. Generates prompt: "professional coffee shop logo,
warm colors, coffee bean illustration, modern minimalist"
2. Runs local image generation
3. Returns: Generated logo image
How to implement:
# pip install diffusers transformers accelerate
from diffusers import StableDiffusionPipeline
import torch
# Load a small model (2GB)
pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16,
).to("cuda")
@tool
def generate_image(prompt: str, output_path: str = "output.png") -> str:
"""Generate an image from a text description."""
image = pipe(prompt, num_inference_steps=20).images[0]
image.save(output_path)
return output_path
Requirements: 4-6GB VRAM (can run on CPU but slow) Cost: $0 (model weights ~4GB download once) Wow factor: 10/10 — "You can GENERATE IMAGES?!"
Alternative for lower VRAM: Use FLUX-schnell or SDXL-Turbo (faster, smaller)
6. PDF/DOCX Document Processor ⭐⭐⭐⭐
What it does: Read PDFs, Word docs, extract text, summarize, answer questions about documents.
Demo scenario:
User: "Summarize this 50-page research paper for me"
Agent:
1. Reads PDF
2. Extracts text
3. Identifies sections (abstract, methods, results)
4. Summarizes each section
5. Returns: 1-page summary with key findings
How to implement:
# pip install PyPDF2 python-docx
import PyPDF2
from docx import Document
@tool
def read_document(file_path: str) -> str:
"""Read a PDF or Word document and return its text content."""
if file_path.endswith('.pdf'):
reader = PyPDF2.PdfReader(file_path)
return "\n".join(page.extract_text() for page in reader.pages)
elif file_path.endswith('.docx'):
doc = Document(file_path)
return "\n".join(p.text for p in doc.paragraphs)
Requirements: Python libraries, ~100MB Cost: $0 Wow factor: 8/10 — "It can read my documents!"
7. Code Repository Editor (Diff/Patch) ⭐⭐⭐⭐⭐
What it does: Not just read code, but EDIT it. Apply patches, refactor, fix bugs.
Demo scenario:
User: "Fix the bug in my app where it crashes on empty input"
Agent:
1. Reads the code file
2. Identifies the bug
3. Generates a fix
4. Applies the patch
5. Tests the fix
6. Returns: "Fixed! Added input validation on line 42."
How to implement: Python difflib + file writing
Requirements: Python standard library Cost: $0 Wow factor: 10/10 — "It fixed my code automatically!"
TIER 3: Super Advanced Tools (Higher Effort, Maximum WOW)
8. Local LLM-Powered Knowledge Base (RAG) ⭐⭐⭐⭐⭐
What it does: Index all your documents, notes, emails. Ask questions and get answers based on YOUR data.
Demo scenario:
User: "What did I decide about the marketing budget in last month's meeting?"
Agent:
1. Searches indexed documents
2. Finds meeting notes from March
3. Extracts relevant passage
4. Returns: "In the March 15 meeting, you decided to allocate
$5K to social media ads and $3K to email campaigns."
How to implement:
# pip install chromadb sentence-transformers
import chromadb
from sentence_transformers import SentenceTransformer
# Small embedding model (500MB)
embedder = SentenceTransformer('all-MiniLM-L6-v2')
client = chromadb.Client()
collection = client.create_collection("my_knowledge")
@tool
def index_documents(folder_path: str) -> str:
"""Index all documents in a folder for semantic search."""
# Read all files, chunk them, embed, store in Chroma
return f"Indexed {num_docs} documents"
@tool
def query_knowledge(question: str) -> str:
"""Ask a question about your indexed documents."""
results = collection.query(query_texts=[question], n_results=5)
return format_results(results)
Requirements: ~1GB for embedding model + storage Cost: $0 Wow factor: 10/10 — "It remembers everything I ever wrote!"
9. Screen Capture + Visual Understanding ⭐⭐⭐⭐⭐
What it does: Take screenshots, analyze what's on screen, help with UI tasks.
Demo scenario:
User: "Help me fill out this form — here's a screenshot"
Agent:
1. Takes screenshot
2. Analyzes image with vision model
3. Identifies form fields
4. Guides user: "Click the 'Name' field and type your name..."
5. Can even auto-fill if given data
How to implement:
# pip install Pillow transformers
from PIL import ImageGrab
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq
# Small vision model (4GB)
processor = AutoProcessor.from_pretrained("microsoft/git-base")
model = AutoModelForVision2Seq.from_pretrained("microsoft/git-base")
@tool
def analyze_screenshot(question: str = "What do you see?") -> str:
"""Take a screenshot and answer questions about it."""
screenshot = ImageGrab.grab()
inputs = processor(images=screenshot, text=question, return_tensors="pt")
outputs = model.generate(**inputs)
return processor.batch_decode(outputs, skip_special_tokens=True)[0]
Requirements: 4-6GB VRAM for vision model Cost: $0 Wow factor: 10/10 — "It can SEE my screen?!"
Alternative: Use Qwen2-VL-2B (multimodal, smaller) or don't use vision and just do OCR with Tesseract ($0 VRAM)
10. Video Processing (FFmpeg) ⭐⭐⭐⭐
What it does: Edit videos, extract clips, convert formats, add subtitles.
Demo scenario:
User: "Extract the best 30-second clip from this 10-minute video"
Agent:
1. Analyzes video frames
2. Identifies highlights (scene changes, audio peaks)
3. Extracts best segment
4. Returns: "Best clip: 3:45-4:15 — contains the product reveal"
How to implement:
# Requires ffmpeg installed
import subprocess
@tool
def process_video(input_path: str, operation: str, output_path: str) -> str:
"""Process a video file using ffmpeg."""
# operation: "trim", "extract_audio", "compress", "add_subtitles"
subprocess.run(["ffmpeg", "-i", input_path, ...])
return output_path
Requirements: ffmpeg installed (~100MB) Cost: $0 Wow factor: 8/10 — Useful for content creators
11. Email/Draft Composer ⭐⭐⭐⭐
What it does: Draft emails, letters, reports in professional formats.
Demo scenario:
User: "Draft a professional email to my boss asking for time off"
Agent:
1. Generates professional email
2. Saves as .eml or .docx
3. Returns: "Draft saved to drafts/time_off_request.docx"
How to implement: Python email module + python-docx
Requirements: Standard libraries Cost: $0 Wow factor: 7/10 — Practical but common
12. Presentation Generator ⭐⭐⭐⭐⭐
What it does: Create PowerPoint/Google Slides presentations from topics.
Demo scenario:
User: "Make a 10-slide presentation about AI trends in 2025"
Agent:
1. Researches topic (web search)
2. Structures 10 slides
3. Generates content for each
4. Creates PPTX file with formatting
5. Returns: "Presentation saved to AI_Trends_2025.pptx"
How to implement:
# pip install python-pptx
from pptx import Presentation
from pptx.util import Inches
@tool
def create_presentation(topic: str, num_slides: int, output_path: str) -> str:
"""Create a PowerPoint presentation on a given topic."""
prs = Presentation()
# Agent writes Python to add slides
# Can include research, charts, images
prs.save(output_path)
return output_path
Requirements: python-pptx library, ~50MB Cost: $0 Wow factor: 9/10 — "It made a whole presentation for me!"
📊 Tool Summary Matrix
| # | Tool | Wow | Difficulty | VRAM | Cost | Priority |
|---|---|---|---|---|---|---|
| 0 | DuckDuckGo Search | 5/10 | Trivial | 0GB | $0 | Built-in |
| 0 | Python Interpreter | 6/10 | Trivial | 0GB | $0 | Built-in |
| 1 | Browser Automation | 10/10 | Easy | 0GB* | $0 | FIRST |
| 2 | File System Manager | 7/10 | Trivial | 0GB | $0 | Essential |
| 3 | GitHub Repo Analyzer | 9/10 | Easy | 0GB | $0 | FIRST |
| 4 | Data Analyst (Pandas) | 9/10 | Easy | 0GB | $0 | FIRST |
| 5 | Image Generator (SD) | 10/10 | Medium | 4-6GB | $0 | Phase 2 |
| 6 | PDF/DOCX Processor | 8/10 | Easy | 0GB | $0 | Phase 2 |
| 7 | Code Editor (Diff) | 10/10 | Medium | 0GB | $0 | Phase 2 |
| 8 | Knowledge Base (RAG) | 10/10 | Medium | 1GB | $0 | Phase 2 |
| 9 | Screen Capture/Vision | 10/10 | Hard | 4-6GB | $0 | Phase 3 |
| 10 | Video Processing | 8/10 | Medium | 0GB | $0 | Phase 3 |
| 11 | Email Composer | 7/10 | Trivial | 0GB | $0 | Phase 3 |
| 12 | Presentation Generator | 9/10 | Medium | 0GB | $0 | Phase 2 |
*Browser uses system RAM, not GPU VRAM
💻 Gaming PC Requirements
Minimum Specs (Tier 1 + 2 tools)
| Component | Requirement | Why |
|---|---|---|
| GPU | 8GB VRAM | Qwen3-1.7B (4GB) + Image Gen (4GB) |
| RAM | 16GB | Browser + Python + file operations |
| Storage | 20GB free | Models, repos, generated files |
| OS | Windows 10/11 or Linux | Browser automation works on both |
Recommended Specs (All tiers)
| Component | Requirement | Why |
|---|---|---|
| GPU | 12GB+ VRAM | Qwen3 (4GB) + Image Gen (4GB) + Vision (4GB) |
| RAM | 32GB | Multiple tools running simultaneously |
| Storage | 50GB free | All models + generated content |
| CPU | Any modern CPU | Most tools are GPU-light |
Budget GPU Options
| GPU | VRAM | Price (Used) | Can Run |
|---|---|---|---|
| GTX 1660 Super | 6GB | $80-120 | Tiers 0-2 |
| RTX 3060 | 12GB | $200-280 | All tiers |
| RTX 4060 | 8GB | $280-320 | Tiers 0-2 |
| RTX 4060 Ti | 16GB | $380-450 | All tiers + future proof |
🎯 Recommended Implementation Phases
Phase 1: "Holy Crap It Works!" (Week 1)
Goal: Get basic agent running with web browsing and file operations.
Tools:
- ✅ DuckDuckGo Search (built-in)
- ✅ Python Interpreter (built-in)
- ✅ Browser Automation (Helium/Selenium)
- ✅ File System Manager
- ✅ GitHub Repo Analyzer
What the user sees:
User: "Find the top trending repo on GitHub and tell me what it does"
Agent: (opens browser, navigates, reads, returns summary)
VRAM needed: 4GB (just Qwen3-1.7B) Cost: $0 Time to build: 2-3 hours
Phase 2: "This Is Actually Useful" (Week 2)
Goal: Add data analysis, document processing, presentations.
New tools:
- ✅ Data Analyst (Pandas + charts)
- ✅ PDF/DOCX Processor
- ✅ Code Editor (diff/patch)
- ✅ Knowledge Base (RAG with Chroma)
- ✅ Presentation Generator
What the user sees:
User: "Analyze my sales CSV and make a presentation about Q3 trends"
Agent: (analyzes data, generates charts, creates 10-slide PPTX)
VRAM needed: 4GB (still just Qwen3) Cost: $0 Time to build: 4-6 hours
Phase 3: "This Is INSANE" (Week 3-4)
Goal: Add image generation, vision, video processing.
New tools:
- ✅ Image Generator (Stable Diffusion)
- ✅ Screen Capture + Vision
- ✅ Video Processing
- ✅ Email/Calendar integration
What the user sees:
User: "Create a logo for my coffee shop and make a promo video"
Agent: (generates logo image + creates video with music and text)
VRAM needed: 8-12GB Cost: $0 Time to build: 6-10 hours
🏆 Comparison: Mini-Manus vs Real Manus
| Capability | Manus | Mini-Manus (Our Build) | Gap |
|---|---|---|---|
| Web browsing | ✅ Real browser, 50+ parallel | ✅ Real browser, sequential | Smaller scale |
| File operations | ✅ Full VM access | ✅ Local file system | Same |
| Code execution | ✅ Cloud sandbox | ✅ Local Python + E2B/Docker | Same |
| Data analysis | ✅ Built-in | ✅ Pandas + charts | Same |
| Image generation | ✅ Yes | ✅ Local SD | Same |
| Document processing | ✅ Yes | ✅ PDF/DOCX | Same |
| Presentation creation | ✅ Yes | ✅ python-pptx | Same |
| Multi-agent | ✅ 3 specialized agents | ✅ smolagents multi-agent | Simpler |
| Persistent memory | ✅ Cloud VM persists | ✅ Chroma vector DB | Local only |
| Vision/Screenshots | ✅ Yes | ✅ Optional | Same |
| Video processing | ✅ Yes | ✅ FFmpeg | Same |
| Asynchronous | ✅ Runs while you sleep | ❌ Real-time only | Big gap |
| Parallel execution | ✅ 50+ agents | ❌ Sequential | Big gap |
| Cloud deployment | ✅ Hosted SaaS | ❌ Local/Gaming PC | Hosting gap |
| Cost | $$$/month | $0/month | We win! |
| Privacy | ❌ Cloud processes data | ✅ Everything local | We win! |
| Customizability | ❌ Closed source | ✅ Fully open | We win! |
Verdict: We get ~70% of Manus's capabilities for $0/month on a gaming PC. The main gaps are parallel execution and async/cloud hosting.
🔑 The smolagents CodeAgent Pattern
This is how we'll actually implement tools. Instead of JSON tool calls, our model writes Python code:
from smolagents import CodeAgent, TransformersModel
# Load our fine-tuned Qwen3-1.7B
model = TransformersModel("muhammadtlha944/MCP-Agent-1.7B")
# Create agent with ALL our tools
agent = CodeAgent(
model=model,
tools=[
# Built-in (free)
WebSearchTool(),
PythonInterpreterTool(),
# Our custom tools
BrowserTool(), # Helium/Selenium
FileSystemTool(), # Read/write files
GitHubTool(), # Clone/analyze repos
DataAnalysisTool(), # Pandas + charts
ImageGeneratorTool(), # Stable Diffusion
DocumentTool(), # PDF/DOCX
KnowledgeBaseTool(), # Chroma RAG
PresentationTool(), # python-pptx
VideoTool(), # FFmpeg
],
add_base_tools=True,
additional_authorized_imports=[
'pandas', 'numpy', 'matplotlib', 'requests',
'bs4', 'PIL', 'pytorch', 'diffusers'
],
)
# Run it!
agent.run("Find the cheapest flight from NYC to London next week")
# Agent will write Python code like:
# search_result = web_search("cheapest flight NYC to London next week")
# print(search_result)
# Then analyze and return answer
🎓 Key Insights
- CodeAgent > ToolCallingAgent for small models — Python is easier than JSON
- smolagents handles the hard parts — ReAct loop, memory, tool parsing, UI
- We don't need to train tool-calling — Qwen3 already knows Python!
- Most tools are FREE — Just Python libraries + system tools
- Tier 1 tools = 90% of wow factor — Browser + files + GitHub + data = impressive
- VRAM is the only real constraint — Image gen and vision need GPU
- Gaming PC is PERFECT — 8-12GB VRAM GPUs are cheap and powerful enough
🚀 What This Means for Our Project
The Training Changes
Original plan: Train model to generate JSON tool calls (MCP protocol) New plan: Train model to write Python code that solves problems
Why this is BETTER:
- Qwen3-1.7B is ALREADY trained on Python code (it's a code model!)
- We need MUCH less training data
- The model can combine tools creatively with loops, if/else
- No need to teach strict JSON schemas
- More natural for the model
New training focus:
- Problem-solving examples ("Given this task, write Python code")
- Multi-step reasoning ("First do A, then use result for B")
- Error handling ("If this fails, try that")
- Asking clarification ("I need more info about...")
The Architecture Changes
Original: Manual ReAct loop with JSON tool parsing New: smolagents CodeAgent with Python tool calls
Benefits:
- 10x less code to write
- Built-in error handling
- Built-in memory
- Built-in Gradio UI
- Built-in multi-agent support
- Community-tested framework
📋 Next Steps (When You Say START)
- Revisit training data — Shift from tool-calling to code-writing examples
- Fine-tune with new focus — Teach problem-solving via Python
- Build with smolagents — Use CodeAgent + TransformersModel
- Add tools incrementally — Phase 1 → Phase 2 → Phase 3
- Deploy to HF Space —
GradioUI(agent).launch()+agent.push_to_hub()
This research changes our project fundamentally — but for the better. We can build something MORE impressive with LESS work.