Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
Commit Β·
bea39a3
1
Parent(s): 0972775
system prompt and github tool desc. update
Browse files- agent/prompts/system_prompt.yaml +15 -14
- agent/tools/github_find_examples.py +129 -62
- agent/tools/github_list_repos.py +32 -19
- agent/tools/github_read_file.py +42 -17
- agent/tools/github_search_code.py +72 -36
agent/prompts/system_prompt.yaml
CHANGED
|
@@ -5,52 +5,53 @@ system_prompt: |
|
|
| 5 |
|
| 6 |
# Task Approach
|
| 7 |
|
| 8 |
-
**CRITICAL:
|
| 9 |
|
| 10 |
For ANY implementation task (training, fine-tuning, inference, data processing, etc.):
|
| 11 |
1. **FIRST**: Search HF documentation to find the recommended approach
|
| 12 |
- This is MANDATORY before writing any code or making implementation decisions
|
| 13 |
- Use `explore_hf_docs` to discover documentation structure for relevant libraries (e.g., "trl", "transformers", "diffusers")
|
|
|
|
| 14 |
- Use `fetch_hf_docs` to retrieve full content from specific documentation pages
|
| 15 |
-
- Use `search_hf_api_endpoints` to find API endpoints with usage examples
|
| 16 |
- Research what libraries to use, find code examples, understand best practices
|
| 17 |
-
- Skip ONLY for simple factual questions (e.g., "What is LoRA?")
|
| 18 |
|
| 19 |
-
2. **THEN**: Formulate a plan based on research findings. Pass todos to the
|
| 20 |
|
| 21 |
3. **FINALLY**: Implement using researched approaches
|
| 22 |
- Search for relevant models/datasets on HF Hub
|
|
|
|
| 23 |
- Use all available tools to complete the task
|
| 24 |
-
-
|
| 25 |
-
-
|
| 26 |
|
| 27 |
# Autonomy / Subordinate trade-off.
|
| 28 |
|
| 29 |
Your main goal is to achieve what the user asked. For this:
|
| 30 |
-
1.
|
| 31 |
|
| 32 |
However !! :
|
| 33 |
1. Don't surprise the user with costly, irreversible, or strange actions without asking.
|
| 34 |
-
2. Don't be shy to ask questions if needed.
|
| 35 |
3. Don't be overly talkative, explaining everything after a task ended.
|
| 36 |
|
| 37 |
|
| 38 |
# Conventions
|
| 39 |
|
| 40 |
- **ALWAYS search documentation BEFORE implementing** any ML workflow (training, inference, data processing, etc.) - This is non-negotiable
|
| 41 |
-
- Use `explore_hf_docs`, `fetch_hf_docs`, and `search_hf_api_endpoints` to research the correct approach
|
| 42 |
-
- Never assume you know the correct library, method, or approach - you must verify with documentation first
|
| 43 |
- Base your implementation on researched best practices, not general knowledge or assumptions
|
| 44 |
- Always search Hugging Face Hub for existing resources before suggesting custom implementations
|
| 45 |
- Keep in mind that a space is a repo, so you can create a space directly by uploading files that way. Repos should also be used to store files permanently : post-execution, files from jobs are not available.
|
| 46 |
- To run jobs, you must always pass the whole content of the file to execute. No files are available on server. Your local files and distant files are entirely seperate scopes.
|
| 47 |
- The HF_TOKEN is automatically loaded from the environment variables.
|
| 48 |
-
-
|
| 49 |
- When referencing models, datasets, or papers, include direct links from search results
|
| 50 |
-
- Before processing any dataset: inspect its actual structure first using the
|
| 51 |
-
- Follow ML best practices: proper train/val/test splits, reproducibility, evaluation metrics
|
| 52 |
- Unless absolutely necessary, don't ask user for action. This does not apply to follow-up questions you have.
|
| 53 |
-
- For training tasks, consider compute requirements and choose appropriate hardware.
|
| 54 |
- Never expose or log API keys, tokens, or secrets. Do not assume keys or secrets are available. Only Hugging Face private resources are available.
|
| 55 |
|
| 56 |
# Communication Style
|
|
|
|
| 5 |
|
| 6 |
# Task Approach
|
| 7 |
|
| 8 |
+
**CRITICAL: You always research first, then implement. You only make implementations that are guided by examples, best practices, or documentation.**
|
| 9 |
|
| 10 |
For ANY implementation task (training, fine-tuning, inference, data processing, etc.):
|
| 11 |
1. **FIRST**: Search HF documentation to find the recommended approach
|
| 12 |
- This is MANDATORY before writing any code or making implementation decisions
|
| 13 |
- Use `explore_hf_docs` to discover documentation structure for relevant libraries (e.g., "trl", "transformers", "diffusers")
|
| 14 |
+
- Use `github_find_examples` and `github_read_file` to discover best-practices on these libraries to reuse.
|
| 15 |
- Use `fetch_hf_docs` to retrieve full content from specific documentation pages
|
| 16 |
+
- Use `search_hf_api_endpoints` to find API endpoints (e.g. spaces, models, datasets, discussions, users, orgs, papers etc.) with usage examples and curl examples.
|
| 17 |
- Research what libraries to use, find code examples, understand best practices
|
| 18 |
+
- Skip ONLY for simple factual questions (e.g., "What is LoRA?").
|
| 19 |
|
| 20 |
+
2. **THEN**: Formulate a plan based on research findings. Pass todos to the `plan_tool`. Update as progress is made.
|
| 21 |
|
| 22 |
3. **FINALLY**: Implement using researched approaches
|
| 23 |
- Search for relevant models/datasets on HF Hub
|
| 24 |
+
- Always validate data structure and format before using it (libraries need specific formats, see documentation).
|
| 25 |
- Use all available tools to complete the task
|
| 26 |
+
- Always leverage existing implementations and resources before creating new ones
|
| 27 |
+
- Use multiple independent tools concurrently for efficiency
|
| 28 |
|
| 29 |
# Autonomy / Subordinate trade-off.
|
| 30 |
|
| 31 |
Your main goal is to achieve what the user asked. For this:
|
| 32 |
+
1. Research, then take action, follow-up, launch jobs. Ask for as little action from the user as possible. Do not ask them to do things you could do via a script or tool.
|
| 33 |
|
| 34 |
However !! :
|
| 35 |
1. Don't surprise the user with costly, irreversible, or strange actions without asking.
|
| 36 |
+
2. Don't be shy to ask clarifying questions if needed.
|
| 37 |
3. Don't be overly talkative, explaining everything after a task ended.
|
| 38 |
|
| 39 |
|
| 40 |
# Conventions
|
| 41 |
|
| 42 |
- **ALWAYS search documentation BEFORE implementing** any ML workflow (training, inference, data processing, etc.) - This is non-negotiable
|
| 43 |
+
- Use `explore_hf_docs`, `github_find_examples`, `fetch_hf_docs`, and `search_hf_api_endpoints` to research the correct approach
|
| 44 |
+
- Never assume you know the correct library, method, or approach - you must verify with documentation first. Documentation is the ultimate source of truth.
|
| 45 |
- Base your implementation on researched best practices, not general knowledge or assumptions
|
| 46 |
- Always search Hugging Face Hub for existing resources before suggesting custom implementations
|
| 47 |
- Keep in mind that a space is a repo, so you can create a space directly by uploading files that way. Repos should also be used to store files permanently : post-execution, files from jobs are not available.
|
| 48 |
- To run jobs, you must always pass the whole content of the file to execute. No files are available on server. Your local files and distant files are entirely seperate scopes.
|
| 49 |
- The HF_TOKEN is automatically loaded from the environment variables.
|
|
|
|
| 50 |
- When referencing models, datasets, or papers, include direct links from search results
|
| 51 |
+
- Before processing any dataset: inspect its actual structure first using the `hub_repo_details` tool. Never assume column names, datarow structure, or format: verify them beforehand.
|
| 52 |
+
- Follow ML best practices: proper train/val/test splits, reproducibility, evaluation metrics, pushing to hub.
|
| 53 |
- Unless absolutely necessary, don't ask user for action. This does not apply to follow-up questions you have.
|
| 54 |
+
- For training tasks, consider compute requirements and choose appropriate hardware based on this formula: approx_VRAM_needed = N_params Γ bytes_per_param Γ 1.5.
|
| 55 |
- Never expose or log API keys, tokens, or secrets. Do not assume keys or secrets are available. Only Hugging Face private resources are available.
|
| 56 |
|
| 57 |
# Communication Style
|
agent/tools/github_find_examples.py
CHANGED
|
@@ -12,25 +12,19 @@ from thefuzz import fuzz
|
|
| 12 |
|
| 13 |
from agent.tools.types import ToolResult
|
| 14 |
|
| 15 |
-
#
|
| 16 |
EXAMPLE_PATTERNS = [
|
| 17 |
-
|
|
|
|
| 18 |
"examples",
|
| 19 |
"example",
|
| 20 |
-
|
| 21 |
-
"
|
| 22 |
-
"
|
| 23 |
-
"demo",
|
| 24 |
# Tutorial/learning patterns
|
| 25 |
"tutorials",
|
| 26 |
"tutorial",
|
| 27 |
-
"guides",
|
| 28 |
-
"guide",
|
| 29 |
"quickstart",
|
| 30 |
-
"getting-started",
|
| 31 |
-
"getting_started",
|
| 32 |
-
"howto",
|
| 33 |
-
"how-to",
|
| 34 |
"walkthroughs",
|
| 35 |
"walkthrough",
|
| 36 |
# Cookbook/recipe patterns
|
|
@@ -38,28 +32,24 @@ EXAMPLE_PATTERNS = [
|
|
| 38 |
"cookbooks",
|
| 39 |
"recipes",
|
| 40 |
"recipe",
|
| 41 |
-
#
|
| 42 |
-
"
|
| 43 |
-
"
|
| 44 |
-
"
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
"
|
| 48 |
-
"
|
| 49 |
-
"
|
| 50 |
-
"
|
| 51 |
-
|
| 52 |
-
"
|
| 53 |
-
"
|
| 54 |
"use-cases",
|
| 55 |
"usecases",
|
| 56 |
"use_cases",
|
| 57 |
-
# Showcase/playground patterns
|
| 58 |
-
"showcase",
|
| 59 |
-
"playground",
|
| 60 |
"sandbox",
|
| 61 |
-
|
| 62 |
-
"scripts",
|
| 63 |
]
|
| 64 |
|
| 65 |
|
|
@@ -178,6 +168,45 @@ def _score_against_keyword(file_path: str, keyword: str) -> int:
|
|
| 178 |
return max(partial_score, token_score)
|
| 179 |
|
| 180 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
def _handle_repo_tree_errors(
|
| 182 |
all_files: List[Dict[str, Any]],
|
| 183 |
error: str,
|
|
@@ -308,23 +337,42 @@ def find_examples(
|
|
| 308 |
"totalResults": 0,
|
| 309 |
"resultsShared": 0,
|
| 310 |
}
|
|
|
|
|
|
|
|
|
|
| 311 |
else:
|
| 312 |
-
# No keyword:
|
| 313 |
-
scored_files = [
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
|
| 317 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
|
| 319 |
if not scored_files:
|
| 320 |
return {
|
| 321 |
-
"formatted": f"No example files found in {org}/{repo}
|
| 322 |
"totalResults": 0,
|
| 323 |
"resultsShared": 0,
|
| 324 |
}
|
| 325 |
|
| 326 |
-
|
| 327 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 328 |
|
| 329 |
# Limit results
|
| 330 |
results = scored_files[:max_results]
|
|
@@ -357,35 +405,54 @@ def find_examples(
|
|
| 357 |
GITHUB_FIND_EXAMPLES_TOOL_SPEC = {
|
| 358 |
"name": "github_find_examples",
|
| 359 |
"description": (
|
| 360 |
-
"
|
| 361 |
-
"
|
| 362 |
-
"
|
| 363 |
-
"
|
| 364 |
-
"
|
| 365 |
-
"
|
| 366 |
-
"
|
| 367 |
-
"
|
| 368 |
-
"
|
| 369 |
-
"
|
| 370 |
-
"
|
| 371 |
-
"
|
| 372 |
-
"
|
| 373 |
-
"
|
| 374 |
-
"
|
| 375 |
-
"
|
| 376 |
-
"
|
| 377 |
-
"
|
| 378 |
-
"
|
| 379 |
-
"
|
| 380 |
-
"
|
| 381 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 382 |
),
|
| 383 |
"parameters": {
|
| 384 |
"type": "object",
|
| 385 |
"properties": {
|
| 386 |
"keyword": {
|
| 387 |
"type": "string",
|
| 388 |
-
"description": "Keyword to fuzzy match against file paths (e.g., 'grpo', '
|
| 389 |
},
|
| 390 |
"repo": {
|
| 391 |
"type": "string",
|
|
|
|
| 12 |
|
| 13 |
from agent.tools.types import ToolResult
|
| 14 |
|
| 15 |
+
# In order of priority (lower index = higher priority for sorting)
|
| 16 |
EXAMPLE_PATTERNS = [
|
| 17 |
+
"scripts",
|
| 18 |
+
# General example patterns (catch-all, lower priority)
|
| 19 |
"examples",
|
| 20 |
"example",
|
| 21 |
+
# Notebook patterns
|
| 22 |
+
"notebooks",
|
| 23 |
+
"notebook",
|
|
|
|
| 24 |
# Tutorial/learning patterns
|
| 25 |
"tutorials",
|
| 26 |
"tutorial",
|
|
|
|
|
|
|
| 27 |
"quickstart",
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
"walkthroughs",
|
| 29 |
"walkthrough",
|
| 30 |
# Cookbook/recipe patterns
|
|
|
|
| 32 |
"cookbooks",
|
| 33 |
"recipes",
|
| 34 |
"recipe",
|
| 35 |
+
# Demo/sample patterns
|
| 36 |
+
"demos",
|
| 37 |
+
"demo",
|
| 38 |
+
"samples",
|
| 39 |
+
"sample",
|
| 40 |
+
# Other patterns
|
| 41 |
+
"guides",
|
| 42 |
+
"guide",
|
| 43 |
+
"getting-started",
|
| 44 |
+
"getting_started",
|
| 45 |
+
"playground",
|
| 46 |
+
"howto",
|
| 47 |
+
"how-to",
|
| 48 |
"use-cases",
|
| 49 |
"usecases",
|
| 50 |
"use_cases",
|
|
|
|
|
|
|
|
|
|
| 51 |
"sandbox",
|
| 52 |
+
"showcase",
|
|
|
|
| 53 |
]
|
| 54 |
|
| 55 |
|
|
|
|
| 168 |
return max(partial_score, token_score)
|
| 169 |
|
| 170 |
|
| 171 |
+
def _get_pattern_priority(file_path: str) -> tuple[int, int, int]:
|
| 172 |
+
"""
|
| 173 |
+
Get priority of a file path based on which example pattern directory it's in.
|
| 174 |
+
|
| 175 |
+
Returns: (in_examples_dir, pattern_priority, path_depth)
|
| 176 |
+
- in_examples_dir: 0 if in examples/ directory, 1 otherwise (lower is better)
|
| 177 |
+
- pattern_priority: Index in EXAMPLE_PATTERNS (lower is better), or 999 if no match
|
| 178 |
+
- path_depth: Number of path segments (lower is better)
|
| 179 |
+
|
| 180 |
+
Note: Prioritizes files in "examples/" directory first, then by most specific pattern match.
|
| 181 |
+
E.g., "examples/scripts/train.py" is better than "scripts/util.py"
|
| 182 |
+
"""
|
| 183 |
+
path_lower = file_path.lower()
|
| 184 |
+
path_parts = path_lower.split("/")
|
| 185 |
+
|
| 186 |
+
# Check if file is in examples/ directory (highest priority)
|
| 187 |
+
in_examples_dir = 0 if (path_parts[0] in ["examples", "example"]) else 1
|
| 188 |
+
|
| 189 |
+
# Find ALL matching patterns and use the best (lowest index) one
|
| 190 |
+
# But prefer deeper matches (more specific) over shallow ones
|
| 191 |
+
best_priority = 999
|
| 192 |
+
best_depth_at_match = -1
|
| 193 |
+
|
| 194 |
+
for i, pattern in enumerate(EXAMPLE_PATTERNS):
|
| 195 |
+
# Check if pattern appears as a directory component in the path
|
| 196 |
+
if pattern in path_parts:
|
| 197 |
+
# Find the depth where this pattern appears (rightmost occurrence)
|
| 198 |
+
depth = len(path_parts) - 1 - path_parts[::-1].index(pattern)
|
| 199 |
+
|
| 200 |
+
# Prefer deeper matches, or better priority if at same depth
|
| 201 |
+
if depth > best_depth_at_match or (
|
| 202 |
+
depth == best_depth_at_match and i < best_priority
|
| 203 |
+
):
|
| 204 |
+
best_priority = i
|
| 205 |
+
best_depth_at_match = depth
|
| 206 |
+
|
| 207 |
+
return (in_examples_dir, best_priority, len(path_parts))
|
| 208 |
+
|
| 209 |
+
|
| 210 |
def _handle_repo_tree_errors(
|
| 211 |
all_files: List[Dict[str, Any]],
|
| 212 |
error: str,
|
|
|
|
| 337 |
"totalResults": 0,
|
| 338 |
"resultsShared": 0,
|
| 339 |
}
|
| 340 |
+
|
| 341 |
+
# Sort by keyword score (descending) for best matches first
|
| 342 |
+
scored_files.sort(key=lambda x: x["score"], reverse=True)
|
| 343 |
else:
|
| 344 |
+
# No keyword: prioritize by pattern directory, then path depth
|
| 345 |
+
scored_files = []
|
| 346 |
+
for file in example_files:
|
| 347 |
+
in_examples_dir, pattern_priority, path_depth = _get_pattern_priority(
|
| 348 |
+
file["path"]
|
| 349 |
+
)
|
| 350 |
+
scored_files.append(
|
| 351 |
+
{
|
| 352 |
+
**file,
|
| 353 |
+
"score": file["example_score"],
|
| 354 |
+
"in_examples_dir": in_examples_dir,
|
| 355 |
+
"pattern_priority": pattern_priority,
|
| 356 |
+
"path_depth": path_depth,
|
| 357 |
+
}
|
| 358 |
+
)
|
| 359 |
|
| 360 |
if not scored_files:
|
| 361 |
return {
|
| 362 |
+
"formatted": f"No example files found in {org}/{repo}.",
|
| 363 |
"totalResults": 0,
|
| 364 |
"resultsShared": 0,
|
| 365 |
}
|
| 366 |
|
| 367 |
+
# Sort by: 1) files in examples/ dir first, 2) pattern priority (scripts > datasets > etc), 3) path depth, 4) path name
|
| 368 |
+
scored_files.sort(
|
| 369 |
+
key=lambda x: (
|
| 370 |
+
x["in_examples_dir"],
|
| 371 |
+
x["pattern_priority"],
|
| 372 |
+
x["path_depth"],
|
| 373 |
+
x["path"],
|
| 374 |
+
)
|
| 375 |
+
)
|
| 376 |
|
| 377 |
# Limit results
|
| 378 |
results = scored_files[:max_results]
|
|
|
|
| 405 |
GITHUB_FIND_EXAMPLES_TOOL_SPEC = {
|
| 406 |
"name": "github_find_examples",
|
| 407 |
"description": (
|
| 408 |
+
"Discover best practices, reusable scripts, tutorials, and demos for usinga specific library or framework. This is an important step before implementing anything ML related.",
|
| 409 |
+
"Use together with github_read_file tool.\n\n"
|
| 410 |
+
"## When to use this tool\n\n"
|
| 411 |
+
"- ALWAYS before implementing any training/inference/benchmarking or other ML related code or answering how-to question.\n"
|
| 412 |
+
"- When exploring a new repository and need to understand how to use it\n"
|
| 413 |
+
"## How it works\n\n"
|
| 414 |
+
"1. Fetches all (examples, tutorials, demos, notebooks, scripts, etc.) from the repository\n"
|
| 415 |
+
"2. If keyword provided, scores found files against the keyword using fuzzy matching\n"
|
| 416 |
+
"3. Returns best matches sorted by relevance score\n"
|
| 417 |
+
"## Examples\n\n"
|
| 418 |
+
"<example>\n"
|
| 419 |
+
"// ML Workflow Step: Find GRPO/SFT/DPO/RLOO etc training examples\n"
|
| 420 |
+
"// Task: Starting GRPO fine-tuning project, need reference implementations\n"
|
| 421 |
+
"{\n"
|
| 422 |
+
" keyword: 'grpo',\n"
|
| 423 |
+
" repo: 'trl',\n"
|
| 424 |
+
" org: 'huggingface'\n"
|
| 425 |
+
"}\n"
|
| 426 |
+
"// Returns: examples/scripts/grpo_agent.py, examples/scripts/grpo_vlm.py\n"
|
| 427 |
+
"// Next step: Use github_read_file to study the implementation\n"
|
| 428 |
+
"</example>\n\n"
|
| 429 |
+
"<example>\n"
|
| 430 |
+
"// ML Workflow Step: Discover all training examples in TRL\n"
|
| 431 |
+
"// Task: Exploring available training methods before choosing approach\n"
|
| 432 |
+
"{\n"
|
| 433 |
+
" repo: 'trl',\n"
|
| 434 |
+
" org: 'huggingface',\n"
|
| 435 |
+
" max_results: 20\n"
|
| 436 |
+
"}\n"
|
| 437 |
+
"// Lists all example scripts: PPO, DPO, GRPO, reward modeling, etc.\n"
|
| 438 |
+
"</example>\n\n"
|
| 439 |
+
"<example>\n"
|
| 440 |
+
"// ML Workflow Step: Find LoRA fine-tuning examples\n"
|
| 441 |
+
"// Task: Learning parameter-efficient fine-tuning with PEFT\n"
|
| 442 |
+
"{\n"
|
| 443 |
+
" keyword: 'lora',\n"
|
| 444 |
+
" repo: 'peft',\n"
|
| 445 |
+
" org: 'huggingface'\n"
|
| 446 |
+
"}\n"
|
| 447 |
+
"// Discovers LoRA configuration and training examples\n"
|
| 448 |
+
"</example>",
|
| 449 |
),
|
| 450 |
"parameters": {
|
| 451 |
"type": "object",
|
| 452 |
"properties": {
|
| 453 |
"keyword": {
|
| 454 |
"type": "string",
|
| 455 |
+
"description": "Keyword to fuzzy match against file paths (e.g., 'grpo', 'sft').",
|
| 456 |
},
|
| 457 |
"repo": {
|
| 458 |
"type": "string",
|
agent/tools/github_list_repos.py
CHANGED
|
@@ -200,26 +200,39 @@ def list_repos(
|
|
| 200 |
|
| 201 |
# Tool specification
|
| 202 |
GITHUB_LIST_REPOS_TOOL_SPEC = {
|
| 203 |
-
"name": "
|
| 204 |
"description": (
|
| 205 |
-
"List and
|
| 206 |
-
"
|
| 207 |
-
"
|
| 208 |
-
"
|
| 209 |
-
"-
|
| 210 |
-
"-
|
| 211 |
-
"-
|
| 212 |
-
"
|
| 213 |
-
"
|
| 214 |
-
"
|
| 215 |
-
"
|
| 216 |
-
"
|
| 217 |
-
"
|
| 218 |
-
"
|
| 219 |
-
"
|
| 220 |
-
"
|
| 221 |
-
"
|
| 222 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
),
|
| 224 |
"parameters": {
|
| 225 |
"type": "object",
|
|
|
|
| 200 |
|
| 201 |
# Tool specification
|
| 202 |
GITHUB_LIST_REPOS_TOOL_SPEC = {
|
| 203 |
+
"name": "github_list_repos",
|
| 204 |
"description": (
|
| 205 |
+
"List and discover repositories for any GitHub user or organization with flexible sorting.\n\n"
|
| 206 |
+
"Returns comprehensive repository information including stars, forks, language, topics, and direct URLs. "
|
| 207 |
+
"Sorts by stars, forks, update date, or creation date.\n\n"
|
| 208 |
+
"## When to use this tool\n\n"
|
| 209 |
+
"- When you need to find libraries to use in your implementation, or to explore what repositories exist for a task.\n"
|
| 210 |
+
"- When debugging an error to looking up if others are having the same issues in repositories."
|
| 211 |
+
"- When finding the most popular or active projects for a user or org\n"
|
| 212 |
+
"## Examples\n\n"
|
| 213 |
+
"<example>\n"
|
| 214 |
+
"// ML Workflow Step: Discover HF libraries for RLHF/alignment\n"
|
| 215 |
+
"// Use case: Find the right library for training with human feedback\n"
|
| 216 |
+
"{\n"
|
| 217 |
+
" owner: 'huggingface',\n"
|
| 218 |
+
" owner_type: 'org',\n"
|
| 219 |
+
" sort: 'stars',\n"
|
| 220 |
+
" limit: 10\n"
|
| 221 |
+
"}\n"
|
| 222 |
+
"// Returns: transformers, trl, peft, accelerate, diffusers...\n"
|
| 223 |
+
"</example>\n\n"
|
| 224 |
+
"<example>\n"
|
| 225 |
+
"// ML Workflow Step: Check for recently updated HF repos\n"
|
| 226 |
+
"// Use case: Find actively maintained libraries with latest features\n"
|
| 227 |
+
"{\n"
|
| 228 |
+
" owner: 'huggingface',\n"
|
| 229 |
+
" owner_type: 'org',\n"
|
| 230 |
+
" sort: 'updated',\n"
|
| 231 |
+
" order: 'desc',\n"
|
| 232 |
+
" limit: 15\n"
|
| 233 |
+
"}\n"
|
| 234 |
+
"// Helps identify which repos have recent improvements/fixes\n"
|
| 235 |
+
"</example>"
|
| 236 |
),
|
| 237 |
"parameters": {
|
| 238 |
"type": "object",
|
agent/tools/github_read_file.py
CHANGED
|
@@ -248,24 +248,49 @@ def read_file(
|
|
| 248 |
|
| 249 |
# Tool specification
|
| 250 |
GITHUB_READ_FILE_TOOL_SPEC = {
|
| 251 |
-
"name": "
|
| 252 |
"description": (
|
| 253 |
-
"Read file contents from any GitHub repository with
|
| 254 |
-
"
|
| 255 |
-
"
|
| 256 |
-
"-
|
| 257 |
-
"-
|
| 258 |
-
"-
|
| 259 |
-
"
|
| 260 |
-
"
|
| 261 |
-
"
|
| 262 |
-
"
|
| 263 |
-
"
|
| 264 |
-
"
|
| 265 |
-
"
|
| 266 |
-
"
|
| 267 |
-
"
|
| 268 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 269 |
),
|
| 270 |
"parameters": {
|
| 271 |
"type": "object",
|
|
|
|
| 248 |
|
| 249 |
# Tool specification
|
| 250 |
GITHUB_READ_FILE_TOOL_SPEC = {
|
| 251 |
+
"name": "github_read_file",
|
| 252 |
"description": (
|
| 253 |
+
"Read file contents from any GitHub repository with line range support.\n\n"
|
| 254 |
+
"Fetches exact file contents in the given line range (default 300 lines, use line_start/line_end adjust). \n\n"
|
| 255 |
+
"## When to use this tool\n\n"
|
| 256 |
+
"- When reading example code, implementations, or documentation on a specific github file\n"
|
| 257 |
+
"- When you found a file via github_list_repos, or github_find_examples and need its contents\n"
|
| 258 |
+
"- When investigating specific code sections with line ranges\n"
|
| 259 |
+
"- When reading from specific branches, tags, or commits\n"
|
| 260 |
+
"## When NOT to use this tool\n\n"
|
| 261 |
+
"- When you don't know the exact file path beforehand (use github_search_code or github_find_examples first)\n\n"
|
| 262 |
+
"## Examples\n\n"
|
| 263 |
+
"<example>\n"
|
| 264 |
+
"// ML Workflow Step: Reading example code from for GRPO training with TRL\n"
|
| 265 |
+
"// Use case: Read trainer class to understand API and methods\n"
|
| 266 |
+
"{\n"
|
| 267 |
+
" repo: 'huggingface/trl',\n"
|
| 268 |
+
" path: 'trl/trainer/grpo_trainer.py',\n"
|
| 269 |
+
" line_start: 1,\n"
|
| 270 |
+
" line_end: 200\n"
|
| 271 |
+
"}\n"
|
| 272 |
+
"// Read class definition and constructor to understand parameters\n"
|
| 273 |
+
"</example>\n\n"
|
| 274 |
+
"<example>\n"
|
| 275 |
+
"// ML Workflow Step: Study complete training script\n"
|
| 276 |
+
"// Use case: Learn end-to-end VLM fine-tuning with GRPO\n"
|
| 277 |
+
"{\n"
|
| 278 |
+
" repo: 'huggingface/trl',\n"
|
| 279 |
+
" path: 'examples/scripts/grpo_vlm.py'\n"
|
| 280 |
+
"}\n"
|
| 281 |
+
"// Returns first 300 lines of the file\n"
|
| 282 |
+
"</example>\n\n"
|
| 283 |
+
"<example>\n"
|
| 284 |
+
"// ML Workflow Step: Check configuration patterns\n"
|
| 285 |
+
"// Use case: Learn how to structure training configs\n"
|
| 286 |
+
"{\n"
|
| 287 |
+
" repo: 'huggingface/transformers',\n"
|
| 288 |
+
" path: 'examples/pytorch/language-modeling/run_clm.py',\n"
|
| 289 |
+
" line_start: 50,\n"
|
| 290 |
+
" line_end: 150\n"
|
| 291 |
+
"}\n"
|
| 292 |
+
"// Read argument parsing and config setup section\n"
|
| 293 |
+
"</example>"
|
| 294 |
),
|
| 295 |
"parameters": {
|
| 296 |
"type": "object",
|
agent/tools/github_search_code.py
CHANGED
|
@@ -332,46 +332,82 @@ def search_code(
|
|
| 332 |
|
| 333 |
# Tool specification
|
| 334 |
GITHUB_SEARCH_CODE_TOOL_SPEC = {
|
| 335 |
-
"name": "
|
| 336 |
"description": (
|
| 337 |
-
"Search for code patterns across GitHub with intelligent pattern matching.\n\n"
|
| 338 |
-
"
|
| 339 |
-
"
|
| 340 |
-
"-
|
| 341 |
-
"
|
| 342 |
-
"-
|
| 343 |
-
"-
|
| 344 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 345 |
"- **Extension**: `'*.py'` or `'**/*.py'` β All Python files\n"
|
| 346 |
-
"- **Directory**: `'src/**/*.js'` β JavaScript files in src/ (client-filtered)\n"
|
| 347 |
"- **Pattern**: `'test_*.py'` β Files matching pattern (client-filtered)\n"
|
| 348 |
"- **Exact path**: `'README.md'` β Specific file\n\n"
|
| 349 |
-
"## How
|
| 350 |
-
"1.
|
| 351 |
-
"2.
|
| 352 |
-
"3.
|
| 353 |
-
"
|
| 354 |
-
"
|
| 355 |
-
"
|
| 356 |
-
"
|
| 357 |
-
"
|
| 358 |
-
"
|
| 359 |
-
"
|
| 360 |
-
"
|
| 361 |
-
"
|
| 362 |
-
"
|
| 363 |
-
"
|
| 364 |
-
"
|
| 365 |
-
"
|
| 366 |
-
"
|
| 367 |
-
"
|
| 368 |
-
"
|
| 369 |
-
"
|
| 370 |
-
"
|
| 371 |
-
"
|
| 372 |
-
"
|
| 373 |
-
"
|
| 374 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 375 |
),
|
| 376 |
"parameters": {
|
| 377 |
"type": "object",
|
|
|
|
| 332 |
|
| 333 |
# Tool specification
|
| 334 |
GITHUB_SEARCH_CODE_TOOL_SPEC = {
|
| 335 |
+
"name": "github_search_code",
|
| 336 |
"description": (
|
| 337 |
+
"Search for code patterns across GitHub repositories with intelligent pattern matching.\n\n"
|
| 338 |
+
"Searches for specific code patterns, functions, classes, or implementations across GitHub. "
|
| 339 |
+
"Intelligently maps patterns to GitHub's Code Search API for efficient server-side filtering, "
|
| 340 |
+
"with automatic client-side filtering for complex patterns. Returns code snippets with context.\n\n"
|
| 341 |
+
"## When to use this tool\n\n"
|
| 342 |
+
"- When searching for specific code patterns, functions, or classes across repositories\n"
|
| 343 |
+
"- When looking for implementation examples of specific methods or APIs\n"
|
| 344 |
+
"- When you need to find where specific code exists across multiple files or repos\n"
|
| 345 |
+
"- When investigating how a feature is implemented in different repositories\n"
|
| 346 |
+
"- When searching for TODO comments, specific patterns, or code structures\n"
|
| 347 |
+
"- Use this for searching actual implementation code (not examples - use github_find_examples for those)\n\n"
|
| 348 |
+
"## When NOT to use this tool\n\n"
|
| 349 |
+
"- When looking for example files or tutorials (use github_find_examples instead)\n"
|
| 350 |
+
"- When you already know the exact file path (use github_read_file directly)\n"
|
| 351 |
+
"- When you need to list repositories (use github_list_repos instead)\n\n"
|
| 352 |
+
"## Repository Patterns\n\n"
|
| 353 |
+
"- **Exact repo**: `'huggingface/trl'` β Searches only that repository\n"
|
| 354 |
+
"- **Organization**: `'huggingface'` or `'huggingface/*'` β All repos in organization\n"
|
| 355 |
+
"- **All GitHub**: `'*/*'` or omit repo_pattern β Searches across all GitHub\n"
|
| 356 |
+
"- **Wildcards**: `'huggingface/trl*'` β Automatic client-side filtering for complex patterns\n\n"
|
| 357 |
+
"## Path Patterns\n\n"
|
| 358 |
"- **Extension**: `'*.py'` or `'**/*.py'` β All Python files\n"
|
| 359 |
+
"- **Directory**: `'src/**/*.js'` β JavaScript files in src/ directory (client-filtered)\n"
|
| 360 |
"- **Pattern**: `'test_*.py'` β Files matching pattern (client-filtered)\n"
|
| 361 |
"- **Exact path**: `'README.md'` β Specific file\n\n"
|
| 362 |
+
"## How it works\n\n"
|
| 363 |
+
"1. Parses repository and path patterns\n"
|
| 364 |
+
"2. Converts to GitHub API filters when possible (server-side, fast)\n"
|
| 365 |
+
"3. Falls back to client-side filtering for complex patterns\n"
|
| 366 |
+
"4. Returns code snippets with line numbers, URLs, and file refs\n"
|
| 367 |
+
"5. Results can be used directly with github_read_file tool\n\n"
|
| 368 |
+
"## Examples\n\n"
|
| 369 |
+
"<example>\n"
|
| 370 |
+
"// ML Workflow Step: Find how AutoModelForCausalLM is used\n"
|
| 371 |
+
"// Use case: Learning best practices for loading LLMs in TRL\n"
|
| 372 |
+
"{\n"
|
| 373 |
+
" query: 'AutoModelForCausalLM.from_pretrained',\n"
|
| 374 |
+
" repo_pattern: 'huggingface/trl',\n"
|
| 375 |
+
" path_pattern: '*.py'\n"
|
| 376 |
+
"}\n"
|
| 377 |
+
"// Finds all model loading patterns with quantization, device_map, etc.\n"
|
| 378 |
+
"</example>\n\n"
|
| 379 |
+
"<example>\n"
|
| 380 |
+
"// ML Workflow Step: Discover TrainingArguments configurations\n"
|
| 381 |
+
"// Use case: Setting up training hyperparameters correctly\n"
|
| 382 |
+
"{\n"
|
| 383 |
+
" query: 'TrainingArguments',\n"
|
| 384 |
+
" repo_pattern: 'huggingface/transformers',\n"
|
| 385 |
+
" path_pattern: 'examples/**/*.py',\n"
|
| 386 |
+
" max_results: 10\n"
|
| 387 |
+
"}\n"
|
| 388 |
+
"// Shows various TrainingArguments setups across different tasks\n"
|
| 389 |
+
"</example>\n\n"
|
| 390 |
+
"<example>\n"
|
| 391 |
+
"// ML Workflow Step: Find dataset preprocessing patterns\n"
|
| 392 |
+
"// Use case: Learning how to prepare data for instruction tuning\n"
|
| 393 |
+
"{\n"
|
| 394 |
+
" query: 'map(tokenize',\n"
|
| 395 |
+
" repo_pattern: 'huggingface',\n"
|
| 396 |
+
" path_pattern: '*.py'\n"
|
| 397 |
+
"}\n"
|
| 398 |
+
"// Discovers tokenization and dataset mapping patterns\n"
|
| 399 |
+
"</example>\n\n"
|
| 400 |
+
"<example>\n"
|
| 401 |
+
"// ML Workflow Step: Find all Trainer class implementations\n"
|
| 402 |
+
"// Use case: Understanding available trainer variants for different tasks\n"
|
| 403 |
+
"{\n"
|
| 404 |
+
" query: 'class \\\\w+Trainer\\\\(',\n"
|
| 405 |
+
" repo_pattern: 'huggingface/trl',\n"
|
| 406 |
+
" path_pattern: 'trl/trainer/**/*.py',\n"
|
| 407 |
+
" regex: true\n"
|
| 408 |
+
"}\n"
|
| 409 |
+
"// Lists: GRPOTrainer, DPOTrainer, PPOTrainer, RewardTrainer, etc.\n"
|
| 410 |
+
"</example>"
|
| 411 |
),
|
| 412 |
"parameters": {
|
| 413 |
"type": "object",
|