Merge branch 'main' into dataset_tool_improved

.gitignore

@@ -16,4 +16,5 @@ wheels/
 /logs
 hf-agent-leaderboard/
 .cursor/
-session_logs/
+session_logs/
+skills/

agent/MCP_INTEGRATION.md

@@ -1,205 +0,0 @@
-# MCP Integration for HF Agent
-
-This agent now supports the Model Context Protocol (MCP), allowing it to connect to and use tools from MCP servers.
-
-## Overview
-
-The MCP integration allows the agent to:
-- Connect to multiple MCP servers simultaneously
-- Automatically discover and use tools from connected servers
-- Execute tool calls through the MCP protocol
-- Seamlessly integrate MCP tools with the agent's existing tool system
-
-## Architecture
-
-The integration consists of several components:
-
-1. **MCPClient** (`agent/core/mcp_client.py`): Manages connections to MCP servers
-2. **ToolExecutor** (`agent/core/executor.py`): Executes both MCP and local tools
-3. **Config** (`agent/config.py`): Stores MCP server configurations
-4. **Session** (`agent/core/session.py`): Initializes MCP connections and manages lifecycle
-
-## Configuration
-
-To use MCP servers with your agent, add them to your configuration file:
-
-```json
-{
-  "model_name": "anthropic/claude-sonnet-4-5-20250929",
-  "tools": [],
-  "system_prompt_path": "",
-  "mcp_servers": [
-    {
-      "name": "weather",
-      "command": "python",
-      "args": ["path/to/weather_server.py"],
-      "env": null
-    },
-    {
-      "name": "filesystem",
-      "command": "node",
-      "args": ["path/to/filesystem_server.js"],
-      "env": {
-        "ALLOWED_PATHS": "/home/user/documents"
-      }
-    }
-  ]
-}
-```
-
-### Configuration Fields
-
-- `name`: Unique identifier for the MCP server
-- `command`: Command to execute the server (`python`, `node`, etc.)
-- `args`: Arguments to pass to the command (path to server script)
-- `env`: (Optional) Environment variables for the server process
-
-## Usage
-
-### Basic Usage
-
-```python
-import asyncio
-from agent.config import Config, load_config
-from agent.core.agent_loop import submission_loop
-
-async def main():
-    # Load config with MCP servers
-    config = load_config("config.json")
-
-    # Create queues
-    submission_queue = asyncio.Queue()
-    event_queue = asyncio.Queue()
-
-    # Start agent loop (MCP connections initialized automatically)
-    await submission_loop(submission_queue, event_queue, config)
-
-if __name__ == "__main__":
-    asyncio.run(main())
-```
-
-### Programmatic Configuration
-
-```python
-from agent.config import Config, MCPServerConfig
-
-config = Config(
-    model_name="anthropic/claude-sonnet-4-5-20250929",
-    tools=[],
-    system_prompt_path="",
-    mcp_servers=[
-        MCPServerConfig(
-            name="weather",
-            command="python",
-            args=["weather_server.py"],
-            env=None
-        )
-    ]
-)
-```
-
-## How It Works
-
-1. **Initialization**: When the agent loop starts, it calls `session.initialize_mcp()`
-2. **Connection**: The session connects to all configured MCP servers
-3. **Tool Discovery**: Tools from all servers are discovered and added to the agent's tool list
-4. **Tool Naming**: MCP tools are prefixed with their server name (e.g., `weather__get_forecast`)
-5. **Execution**: When the LLM calls a tool, the ToolExecutor routes it to the appropriate MCP server
-6. **Cleanup**: When the agent shuts down, all MCP connections are cleaned up properly
-
-## Tool Naming Convention
-
-MCP tools are automatically prefixed with their server name to avoid conflicts:
-
-- Original tool: `get_forecast`
-- MCP tool name: `weather__get_forecast`
-
-This ensures that tools from different servers don't conflict, even if they have the same name.
-
-## Example: Creating a Simple MCP Server
-
-Here's a minimal example of an MCP server (save as `calculator_server.py`):
-
-```python
-import asyncio
-from mcp.server import Server, stdio_server
-from mcp.types import Tool, TextContent
-
-app = Server("calculator")
-
-@app.list_tools()
-async def list_tools() -> list[Tool]:
-    return [
-        Tool(
-            name="add",
-            description="Add two numbers",
-            inputSchema={
-                "type": "object",
-                "properties": {
-                    "a": {"type": "number"},
-                    "b": {"type": "number"}
-                },
-                "required": ["a", "b"]
-            }
-        )
-    ]
-
-@app.call_tool()
-async def call_tool(name: str, arguments: dict) -> list[TextContent]:
-    if name == "add":
-        result = arguments["a"] + arguments["b"]
-        return [TextContent(type="text", text=str(result))]
-
-    raise ValueError(f"Unknown tool: {name}")
-
-async def main():
-    async with stdio_server() as (read_stream, write_stream):
-        await app.run(read_stream, write_stream, app.create_initialization_options())
-
-if __name__ == "__main__":
-    asyncio.run(main())
-```
-
-## Troubleshooting
-
-### Server Connection Issues
-
-If you see errors connecting to an MCP server:
-
-1. Check that the server script path is correct
-2. Ensure the command (`python`, `node`) is in your PATH
-3. Verify the server script is executable
-4. Check server logs for initialization errors
-
-### Tool Not Found
-
-If the agent can't find an MCP tool:
-
-1. Verify the server is connected (check startup logs)
-2. Check tool naming (should be `servername__toolname`)
-3. Ensure the server properly implements `list_tools()`
-
-### Performance Considerations
-
-- MCP server initialization happens once at startup
-- Tool calls are asynchronous and don't block the agent
-- Multiple servers can be used simultaneously
-- Consider using local tools for high-frequency operations
-
-## Best Practices
-
-1. **Unique Server Names**: Give each MCP server a unique, descriptive name
-2. **Error Handling**: MCP connection failures are logged but don't crash the agent
-3. **Resource Cleanup**: Always let the agent shut down gracefully to cleanup connections
-4. **Testing**: Test MCP servers independently before integrating them
-5. **Security**: Be cautious with file system and network access in MCP servers
-
-## Future Enhancements
-
-Potential improvements to consider:
-
-- Dynamic server addition/removal during runtime
-- Server health monitoring and auto-reconnection
-- Tool caching and performance optimization
-- Support for MCP resources and prompts
-- Rate limiting and timeout configuration

agent/codex_agent_demo.py

@@ -1,470 +0,0 @@
-"""
-Minimum Viable Implementation of Codex Agent Loop in Python
-
-This demonstrates the core architecture patterns from codex-rs:
-- Async submission loop (like submission_loop in codex.rs)
-- Context manager for conversation history
-- Channel-based communication (submissions in, events out)
-- Handler pattern for operations
-"""
-
-import asyncio
-from dataclasses import dataclass, field
-from datetime import datetime
-from enum import Enum
-from typing import Any, Dict, List, Optional
-
-# ============================================================================
-# PROTOCOL TYPES (ResponseItem equivalents)
-# ============================================================================
-
-
-class MessageRole(Enum):
-    SYSTEM = "system"
-    USER = "user"
-    ASSISTANT = "assistant"
-
-
-@dataclass
-class Message:
-    role: MessageRole
-    content: str
-    timestamp: datetime = field(default_factory=datetime.now)
-
-
-@dataclass
-class ToolCall:
-    call_id: str
-    tool_name: str
-    arguments: Dict[str, Any]
-
-
-@dataclass
-class ToolOutput:
-    call_id: str
-    content: str
-    success: bool = True
-
-
-# ============================================================================
-# CONTEXT MANAGER (like context_manager/history.rs)
-# ============================================================================
-
-
-class ContextManager:
-    """
-    Manages conversation history with normalization and truncation.
-    Based on codex-rs/core/src/context_manager/history.rs
-    """
-
-    def __init__(self, max_history_length: int = 1000):
-        self.items: List[Any] = []  # Oldest → Newest
-        self.token_count: int = 0
-        self.max_history_length = max_history_length
-
-    def record_items(self, items: List[Any]) -> None:
-        """Record new items to history (like record_items in history.rs:41)"""
-        for item in items:
-            # Filter and process items
-            if self._is_api_message(item):
-                processed = self._process_item(item)
-                self.items.append(processed)
-
-    def _is_api_message(self, item: Any) -> bool:
-        """Filter out system messages (like is_api_message in history.rs:157)"""
-        if isinstance(item, Message):
-            return item.role != MessageRole.SYSTEM
-        return isinstance(item, (ToolCall, ToolOutput))
-
-    def _process_item(self, item: Any) -> Any:
-        """Process item before adding (like process_item in history.rs:119)"""
-        # Truncate long outputs
-        if isinstance(item, ToolOutput):
-            if len(item.content) > 2000:
-                item.content = item.content[:2000] + "...[truncated]"
-        return item
-
-    def get_history_for_prompt(self) -> List[Any]:
-        """
-        Get normalized history ready for model
-        (like get_history_for_prompt in history.rs:65)
-        """
-        self._normalize_history()
-        return self.items.copy()
-
-    def _normalize_history(self) -> None:
-        """
-        Enforce invariants (like normalize_history in history.rs:102):
-        1. Every tool call has corresponding output
-        2. Every output has corresponding call
-        """
-        # Build mapping of call_id → call
-        calls = {}
-        outputs = {}
-
-        for item in self.items:
-            if isinstance(item, ToolCall):
-                calls[item.call_id] = item
-            elif isinstance(item, ToolOutput):
-                outputs[item.call_id] = item
-
-        # Remove orphan outputs (no matching call)
-        self.items = [
-            item
-            for item in self.items
-            if not isinstance(item, ToolOutput) or item.call_id in calls
-        ]
-
-        # Add missing outputs for calls (create synthetic outputs)
-        for call_id, call in calls.items():
-            if call_id not in outputs:
-                self.items.append(
-                    ToolOutput(
-                        call_id=call_id, content="[No output recorded]", success=False
-                    )
-                )
-
-    def remove_first_item(self) -> None:
-        """Remove oldest item for compaction (like remove_first_item in history.rs:71)"""
-        if self.items:
-            removed = self.items.pop(0)
-            # Also remove corresponding pair if needed
-            if isinstance(removed, ToolCall):
-                self.items = [
-                    item
-                    for item in self.items
-                    if not (
-                        isinstance(item, ToolOutput) and item.call_id == removed.call_id
-                    )
-                ]
-            elif isinstance(removed, ToolOutput):
-                self.items = [
-                    item
-                    for item in self.items
-                    if not (
-                        isinstance(item, ToolCall) and item.call_id == removed.call_id
-                    )
-                ]
-
-    def compact(self, target_size: int) -> None:
-        """Remove old items until we're under target size"""
-        while len(self.items) > target_size:
-            self.remove_first_item()
-
-
-# ============================================================================
-# OPERATIONS (like Op enum in codex.rs)
-# ============================================================================
-
-
-class OpType(Enum):
-    USER_INPUT = "user_input"
-    EXEC_APPROVAL = "exec_approval"
-    INTERRUPT = "interrupt"
-    UNDO = "undo"
-    COMPACT = "compact"
-    SHUTDOWN = "shutdown"
-
-
-@dataclass
-class Operation:
-    op_type: OpType
-    data: Optional[Dict[str, Any]] = None
-
-
-@dataclass
-class Submission:
-    id: str
-    operation: Operation
-
-
-# ============================================================================
-# EVENTS (like Event in codex-rs)
-# ============================================================================
-
-
-@dataclass
-class Event:
-    event_type: str
-    data: Optional[Dict[str, Any]] = None
-
-
-# ============================================================================
-# SESSION STATE (like Session in codex.rs)
-# ============================================================================
-
-
-class Session:
-    """
-    Maintains agent session state
-    Similar to Session in codex-rs/core/src/codex.rs
-    """
-
-    def __init__(self, event_queue: asyncio.Queue):
-        self.context_manager = ContextManager(tool_specs=[])
-        self.event_queue = event_queue
-        self.is_running = True
-        self.current_task: Optional[asyncio.Task] = None
-
-    async def send_event(self, event: Event) -> None:
-        """Send event back to client"""
-        await self.event_queue.put(event)
-
-    def interrupt(self) -> None:
-        """Interrupt current running task"""
-        if self.current_task and not self.current_task.done():
-            self.current_task.cancel()
-
-
-# ============================================================================
-# OPERATION HANDLERS (like handlers module in codex.rs:1343)
-# ============================================================================
-
-
-class Handlers:
-    """Handler functions for each operation type"""
-
-    @staticmethod
-    async def user_input(session: Session, text: str) -> None:
-        """Handle user input (like user_input_or_turn in codex.rs:1291)"""
-        # Add user message to history
-        user_msg = Message(role=MessageRole.USER, content=text)
-        session.context_manager.record_items([user_msg])
-
-        # Send event that we're processing
-        await session.send_event(
-            Event(event_type="processing", data={"message": "Processing user input"})
-        )
-
-        # Simulate agent processing
-        await asyncio.sleep(0.1)
-
-        # Generate mock assistant response
-        assistant_msg = Message(
-            role=MessageRole.ASSISTANT, content=f"I received: {text}"
-        )
-        session.context_manager.record_items([assistant_msg])
-
-        # Simulate tool call
-        tool_call = ToolCall(
-            call_id="call_123", tool_name="bash", arguments={"command": "echo 'hello'"}
-        )
-        session.context_manager.record_items([tool_call])
-
-        # Simulate tool execution
-        await asyncio.sleep(0.1)
-
-        tool_output = ToolOutput(call_id="call_123", content="hello\n", success=True)
-        session.context_manager.record_items([tool_output])
-
-        # Send completion event
-        await session.send_event(
-            Event(
-                event_type="turn_complete",
-                data={"history_size": len(session.context_manager.items)},
-            )
-        )
-
-    @staticmethod
-    async def interrupt(session: Session) -> None:
-        """Handle interrupt (like interrupt in codex.rs:1266)"""
-        session.interrupt()
-        await session.send_event(Event(event_type="interrupted"))
-
-    @staticmethod
-    async def compact(session: Session) -> None:
-        """Handle compact (like compact in codex.rs:1317)"""
-        old_size = len(session.context_manager.items)
-        session.context_manager.compact(target_size=10)
-        new_size = len(session.context_manager.items)
-
-        await session.send_event(
-            Event(
-                event_type="compacted",
-                data={"removed": old_size - new_size, "remaining": new_size},
-            )
-        )
-
-    @staticmethod
-    async def undo(session: Session) -> None:
-        """Handle undo (like undo in codex.rs:1314)"""
-        # Remove last user turn and all following items
-        # Simplified: just remove last 2 items
-        for _ in range(min(2, len(session.context_manager.items))):
-            session.context_manager.items.pop()
-
-        await session.send_event(Event(event_type="undo_complete"))
-
-    @staticmethod
-    async def shutdown(session: Session) -> bool:
-        """Handle shutdown (like shutdown in codex.rs:1329)"""
-        session.is_running = False
-        await session.send_event(Event(event_type="shutdown"))
-        return True
-
-
-# ============================================================================
-# MAIN AGENT LOOP (like submission_loop in codex.rs:1259)
-# ============================================================================
-
-
-async def submission_loop(
-    submission_queue: asyncio.Queue, event_queue: asyncio.Queue
-) -> None:
-    """
-    Main agent loop - processes submissions and dispatches to handlers.
-    This is the core of the agent (like submission_loop in codex.rs:1259-1340)
-    """
-    session = Session(event_queue)
-
-    print("🤖 Agent loop started")
-
-    # Main processing loop
-    while session.is_running:
-        try:
-            # Wait for next submission (like rx_sub.recv() in codex.rs:1262)
-            submission = await submission_queue.get()
-
-            print(f"📨 Received: {submission.operation.op_type.value}")
-
-            # Dispatch to handler based on operation type
-            # (like match in codex.rs:1264-1337)
-            op = submission.operation
-
-            if op.op_type == OpType.USER_INPUT:
-                text = op.data.get("text", "") if op.data else ""
-                await Handlers.user_input(session, text)
-
-            elif op.op_type == OpType.INTERRUPT:
-                await Handlers.interrupt(session)
-
-            elif op.op_type == OpType.COMPACT:
-                await Handlers.compact(session)
-
-            elif op.op_type == OpType.UNDO:
-                await Handlers.undo(session)
-
-            elif op.op_type == OpType.SHUTDOWN:
-                if await Handlers.shutdown(session):
-                    break
-
-            else:
-                print(f"⚠️  Unknown operation: {op.op_type}")
-
-        except asyncio.CancelledError:
-            break
-        except Exception as e:
-            print(f"❌ Error in agent loop: {e}")
-            await session.send_event(Event(event_type="error", data={"error": str(e)}))
-
-    print("🛑 Agent loop exited")
-
-
-# ============================================================================
-# CODEX INTERFACE (like Codex struct in codex.rs:154)
-# ============================================================================
-
-
-class Codex:
-    """
-    Main interface to the agent (like Codex in codex.rs:154-246)
-    Provides submit() and next_event() methods
-    """
-
-    def __init__(self):
-        self.submission_queue = asyncio.Queue()
-        self.event_queue = asyncio.Queue()
-        self.agent_task: Optional[asyncio.Task] = None
-        self.submission_counter = 0
-
-    async def spawn(self) -> None:
-        """Spawn the agent loop (like Codex::spawn in codex.rs:156)"""
-        self.agent_task = asyncio.create_task(
-            submission_loop(self.submission_queue, self.event_queue)
-        )
-
-    async def submit(self, operation: Operation) -> str:
-        """Submit operation to agent (like Codex::submit in codex.rs:218)"""
-        self.submission_counter += 1
-        submission = Submission(
-            id=f"sub_{self.submission_counter}", operation=operation
-        )
-        await self.submission_queue.put(submission)
-        return submission.id
-
-    async def next_event(self) -> Optional[Event]:
-        """Get next event from agent (like Codex::next_event in codex.rs:238)"""
-        try:
-            return await asyncio.wait_for(self.event_queue.get(), timeout=1.0)
-        except asyncio.TimeoutError:
-            return None
-
-    async def shutdown(self) -> None:
-        """Shutdown the agent"""
-        await self.submit(Operation(op_type=OpType.SHUTDOWN))
-        if self.agent_task:
-            await self.agent_task
-
-
-# ============================================================================
-# DEMO / EXAMPLE USAGE
-# ============================================================================
-
-
-async def main():
-    """Demo of the agent system"""
-    print("=" * 60)
-    print("Codex Agent Loop Demo (Python MVP)")
-    print("=" * 60)
-
-    # Create and spawn agent
-    codex = Codex()
-    await codex.spawn()
-
-    # Submit some operations
-    print("\n1️⃣  Submitting user input...")
-    await codex.submit(
-        Operation(op_type=OpType.USER_INPUT, data={"text": "Hello, agent!"})
-    )
-
-    # Receive events
-    for _ in range(3):
-        event = await codex.next_event()
-        if event:
-            print(f"   ✅ Event: {event.event_type} - {event.data}")
-
-    print("\n2️⃣  Submitting another input...")
-    await codex.submit(
-        Operation(op_type=OpType.USER_INPUT, data={"text": "What's the weather?"})
-    )
-
-    for _ in range(3):
-        event = await codex.next_event()
-        if event:
-            print(f"   ✅ Event: {event.event_type} - {event.data}")
-
-    print("\n3️⃣  Compacting history...")
-    await codex.submit(Operation(op_type=OpType.COMPACT))
-
-    event = await codex.next_event()
-    if event:
-        print(f"   ✅ Event: {event.event_type} - {event.data}")
-
-    print("\n4️⃣  Undoing last turn...")
-    await codex.submit(Operation(op_type=OpType.UNDO))
-
-    event = await codex.next_event()
-    if event:
-        print(f"   ✅ Event: {event.event_type}")
-
-    # Shutdown
-    print("\n5️⃣  Shutting down...")
-    await codex.shutdown()
-
-    print("\n" + "=" * 60)
-    print("Demo complete!")
-    print("=" * 60)
-
-
-if __name__ == "__main__":
-    asyncio.run(main())

agent/context_manager/manager.py

@@ -8,6 +8,7 @@ from pathlib import Path
 from typing import Any
 
 import yaml
+from huggingface_hub import HfApi
 from jinja2 import Template
 from litellm import Message, acompletion
 
@@ -24,7 +25,8 @@ class ContextManager:
         prompt_file_suffix: str = "system_prompt_v2.yaml",
     ):
         self.system_prompt = self._load_system_prompt(
-            tool_specs or [], prompt_file_suffix="system_prompt_v2.yaml"
+            tool_specs or [],
+            prompt_file_suffix="system_prompt_v2.yaml",
         )
         self.max_context = max_context
         self.compact_size = int(max_context * compact_size)
@@ -58,6 +60,7 @@ class ContextManager:
             current_date=current_date,
             current_time=current_time,
             current_timezone=current_timezone,
+            hf_user_info=HfApi().whoami().get("name"),
         )
 
     def add_message(self, message: Message, token_count: int = None) -> None:

agent/core/agent_loop.py

@@ -76,7 +76,19 @@ def _needs_approval(tool_name: str, tool_args: dict, config: Config | None = Non
         # Other operations (create_repo, etc.) always require approval
         if operation in ["create_repo"]:
             return True
-    
+
+    # hf_repo_files: upload (can overwrite) and delete require approval
+    if tool_name == "hf_repo_files":
+        operation = tool_args.get("operation", "")
+        if operation in ["upload", "delete"]:
+            return True
+
+    # hf_repo_git: destructive operations require approval
+    if tool_name == "hf_repo_git":
+        operation = tool_args.get("operation", "")
+        if operation in ["delete_branch", "delete_tag", "merge_pr", "create_repo", "update_repo"]:
+            return True
+
     return False
 
 

agent/core/tools.py

@@ -35,22 +35,29 @@ from agent.tools.github_read_file import (
     GITHUB_READ_FILE_TOOL_SPEC,
     github_read_file_handler,
 )
+from agent.tools.hf_repo_files_tool import (
+    HF_REPO_FILES_TOOL_SPEC,
+    hf_repo_files_handler,
+)
+from agent.tools.hf_repo_git_tool import (
+    HF_REPO_GIT_TOOL_SPEC,
+    hf_repo_git_handler,
+)
 from agent.tools.jobs_tool import HF_JOBS_TOOL_SPEC, hf_jobs_handler
 from agent.tools.plan_tool import PLAN_TOOL_SPEC, plan_tool_handler
-from agent.tools.private_hf_repo_tools import (
-    PRIVATE_HF_REPO_TOOL_SPEC,
-    private_hf_repo_handler,
-)
 
-# NOTE: Utils tool disabled - date/time now loaded into system prompt at initialization
-# from agent.tools.utils_tools import UTILS_TOOL_SPEC, utils_handler
+# NOTE: Private HF repo tool disabled - replaced by hf_repo_files and hf_repo_git
+# from agent.tools.private_hf_repo_tools import (
+#     PRIVATE_HF_REPO_TOOL_SPEC,
+#     private_hf_repo_handler,
+# )
 
 # Suppress aiohttp deprecation warning
 warnings.filterwarnings(
     "ignore", category=DeprecationWarning, module="aiohttp.connector"
 )
 
-NOT_ALLOWED_TOOL_NAMES = ["hf_jobs", "hf_doc_search", "hf_doc_fetch"]
+NOT_ALLOWED_TOOL_NAMES = ["hf_jobs", "hf_doc_search", "hf_doc_fetch", "hf_whoami"]
 
 
 def convert_mcp_content_to_string(content: list) -> str:
@@ -281,20 +288,19 @@ def create_builtin_tools() -> list[ToolSpec]:
             parameters=HF_JOBS_TOOL_SPEC["parameters"],
             handler=hf_jobs_handler,
         ),
+        # HF Repo management tools
         ToolSpec(
-            name=PRIVATE_HF_REPO_TOOL_SPEC["name"],
-            description=PRIVATE_HF_REPO_TOOL_SPEC["description"],
-            parameters=PRIVATE_HF_REPO_TOOL_SPEC["parameters"],
-            handler=private_hf_repo_handler,
+            name=HF_REPO_FILES_TOOL_SPEC["name"],
+            description=HF_REPO_FILES_TOOL_SPEC["description"],
+            parameters=HF_REPO_FILES_TOOL_SPEC["parameters"],
+            handler=hf_repo_files_handler,
+        ),
+        ToolSpec(
+            name=HF_REPO_GIT_TOOL_SPEC["name"],
+            description=HF_REPO_GIT_TOOL_SPEC["description"],
+            parameters=HF_REPO_GIT_TOOL_SPEC["parameters"],
+            handler=hf_repo_git_handler,
         ),
-        # NOTE: Utils tool disabled - date/time now loaded into system prompt at initialization (less tool calls=more reliablity)
-        # ToolSpec(
-        #     name=UTILS_TOOL_SPEC["name"],
-        #     description=UTILS_TOOL_SPEC["description"],
-        #     parameters=UTILS_TOOL_SPEC["parameters"],
-        #     handler=utils_handler,
-        # ),
-        # GitHub tools
         # NOTE: Github search code tool disabled - a bit buggy
         # ToolSpec(
         #     name=GITHUB_SEARCH_CODE_TOOL_SPEC["name"],

agent/main.py

@@ -287,6 +287,95 @@ async def event_listener(
                                     if len(all_lines) > 5:
                                         print("...")
 
+                    elif tool_name == "hf_repo_files":
+                        # Handle repo files operations (upload, delete)
+                        repo_id = arguments.get("repo_id", "")
+                        repo_type = arguments.get("repo_type", "model")
+                        revision = arguments.get("revision", "main")
+
+                        # Build repo URL
+                        if repo_type == "model":
+                            repo_url = f"https://huggingface.co/{repo_id}"
+                        else:
+                            repo_url = f"https://huggingface.co/{repo_type}s/{repo_id}"
+
+                        print(f"Repository: {repo_id}")
+                        print(f"Type: {repo_type}")
+                        print(f"Branch: {revision}")
+                        print(f"URL: {repo_url}")
+
+                        if operation == "upload":
+                            path = arguments.get("path", "")
+                            content = arguments.get("content", "")
+                            create_pr = arguments.get("create_pr", False)
+
+                            print(f"File: {path}")
+                            if create_pr:
+                                print("Mode: Create PR")
+
+                            if isinstance(content, str):
+                                all_lines = content.split("\n")
+                                line_count = len(all_lines)
+                                size_bytes = len(content.encode("utf-8"))
+                                size_kb = size_bytes / 1024
+
+                                print(f"Lines: {line_count}")
+                                if size_kb < 1024:
+                                    print(f"Size: {size_kb:.2f} KB")
+                                else:
+                                    print(f"Size: {size_kb / 1024:.2f} MB")
+
+                                # Show full content
+                                print(f"Content:\n{content}")
+
+                        elif operation == "delete":
+                            patterns = arguments.get("patterns", [])
+                            if isinstance(patterns, str):
+                                patterns = [patterns]
+                            print(f"Patterns to delete: {', '.join(patterns)}")
+
+                    elif tool_name == "hf_repo_git":
+                        # Handle git operations (branches, tags, PRs, repo management)
+                        repo_id = arguments.get("repo_id", "")
+                        repo_type = arguments.get("repo_type", "model")
+
+                        # Build repo URL
+                        if repo_type == "model":
+                            repo_url = f"https://huggingface.co/{repo_id}"
+                        else:
+                            repo_url = f"https://huggingface.co/{repo_type}s/{repo_id}"
+
+                        print(f"Repository: {repo_id}")
+                        print(f"Type: {repo_type}")
+                        print(f"URL: {repo_url}")
+
+                        if operation == "delete_branch":
+                            branch = arguments.get("branch", "")
+                            print(f"Branch to delete: {branch}")
+
+                        elif operation == "delete_tag":
+                            tag = arguments.get("tag", "")
+                            print(f"Tag to delete: {tag}")
+
+                        elif operation == "merge_pr":
+                            pr_num = arguments.get("pr_num", "")
+                            print(f"PR to merge: #{pr_num}")
+
+                        elif operation == "create_repo":
+                            private = arguments.get("private", False)
+                            space_sdk = arguments.get("space_sdk")
+                            print(f"Private: {private}")
+                            if space_sdk:
+                                print(f"Space SDK: {space_sdk}")
+
+                        elif operation == "update_repo":
+                            private = arguments.get("private")
+                            gated = arguments.get("gated")
+                            if private is not None:
+                                print(f"Private: {private}")
+                            if gated is not None:
+                                print(f"Gated: {gated}")
+
                     # Get user decision for this item
                     response = await prompt_session.prompt_async(
                         f"Approve item {i}? (y=yes, yolo=approve all, n=no, or provide feedback): "

agent/prompts/system_prompt_v2.yaml

@@ -2,6 +2,7 @@ system_prompt: |
   You are Hugging Face Agent, a skilled AI assistant for machine learning engineering with deep expertise in the Hugging Face ecosystem. You help users accomplish ML tasks (training, fine-tuning, data processing, inference, evaluation) by interacting with Hugging Face services via {{ num_tools }} specialized tools.
 
   _Current Time: **{{ current_date }} {{ current_time }} ({{ current_timezone }})**_
+  {% if hf_user_info %}_AUTHENTICATED ON HF AS: **{{ hf_user_info }}**_{% endif %}
 
   # Core Mission & Behavior
 
@@ -330,11 +331,6 @@ system_prompt: |
   - Check model size, architecture, requirements
   - Verify dataset columns, splits, size
 
-  **hf_whoami:**
-  - Check authentication status
-  - Verify token has correct permissions
-  - Use before operations requiring write access
-
   ## Execution & Storage Tools
 
   **hf_jobs:**
@@ -456,8 +452,6 @@ system_prompt: |
       hub_model_id="username/model-name",  # ← Must be set
       # ...
   )
-
-  # Verify token: hf_whoami()
   ```
 
   ### Dataset Format Mismatch

agent/tools/docs_tools.py

@@ -1,289 +1,474 @@
 """
-Documentation search tools for the HF Agent
-Tools for exploring and fetching HuggingFace documentation and API specifications
+Documentation search tools for exploring HuggingFace and Gradio documentation.
 """
 
 import asyncio
+import json
 import os
 from typing import Any
 
 import httpx
 from bs4 import BeautifulSoup
+from whoosh.analysis import StemmingAnalyzer
+from whoosh.fields import ID, TEXT, Schema
+from whoosh.filedb.filestore import RamStorage
+from whoosh.qparser import MultifieldParser, OrGroup
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+
+DEFAULT_MAX_RESULTS = 20
+MAX_RESULTS_CAP = 50
+
+GRADIO_LLMS_TXT_URL = "https://gradio.app/llms.txt"
+GRADIO_SEARCH_URL = "https://playground-worker.pages.dev/api/prompt"
+
+COMPOSITE_ENDPOINTS: dict[str, list[str]] = {
+    "optimum": [
+        "optimum",
+        "optimum-habana",
+        "optimum-neuron",
+        "optimum-intel",
+        "optimum-executorch",
+        "optimum-tpu",
+    ],
+    "courses": [
+        "llm-course",
+        "robotics-course",
+        "mcp-course",
+        "smol-course",
+        "agents-course",
+        "deep-rl-course",
+        "computer-vision-course",
+        "audio-course",
+        "ml-games-course",
+        "diffusion-course",
+        "ml-for-3d-course",
+        "cookbook",
+    ],
+}
 
-# Cache for OpenAPI spec to avoid repeated fetches
-_openapi_spec_cache: dict[str, Any] | None = None
+# ---------------------------------------------------------------------------
+# Caches
+# ---------------------------------------------------------------------------
 
+_docs_cache: dict[str, list[dict[str, str]]] = {}
+_index_cache: dict[str, tuple[Any, MultifieldParser]] = {}
+_cache_lock = asyncio.Lock()
+_openapi_cache: dict[str, Any] | None = None
 
-async def _fetch_html_page(hf_token: str, endpoint: str) -> str:
-    """Fetch the HTML page for a given endpoint"""
-    base_url = "https://huggingface.co/docs"
-    url = f"{base_url}/{endpoint}"
-    headers = {"Authorization": f"Bearer {hf_token}"}
+# ---------------------------------------------------------------------------
+# Gradio Documentation
+# ---------------------------------------------------------------------------
 
+
+async def _fetch_gradio_docs(query: str | None = None) -> str:
+    """
+    Fetch Gradio documentation.
+    Without query: Get full documentation from llms.txt
+    With query: Run embedding search on guides/demos for relevant content
+    """
     async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
-        response = await client.get(url, headers=headers)
-        response.raise_for_status()
+        if not query:
+            resp = await client.get(GRADIO_LLMS_TXT_URL)
+            resp.raise_for_status()
+            return resp.text
+
+        resp = await client.post(
+            GRADIO_SEARCH_URL,
+            headers={
+                "Content-Type": "application/json",
+                "Origin": "https://gradio-docs-mcp.up.railway.app",
+            },
+            json={
+                "prompt_to_embed": query,
+                "SYSTEM_PROMPT": "$INSERT_GUIDES_DOCS_DEMOS",
+                "FALLBACK_PROMPT": "No results found",
+            },
+        )
+        resp.raise_for_status()
+        return resp.json().get("SYS_PROMPT", "No results found")
 
-    return response.text
 
+# ---------------------------------------------------------------------------
+# HF Documentation - Fetching
+# ---------------------------------------------------------------------------
 
-def _parse_sidebar_navigation(html_content: str) -> list[dict[str, str]]:
-    """Parse the sidebar navigation and extract all links"""
-    soup = BeautifulSoup(html_content, "html.parser")
-    sidebar = soup.find("nav", class_=lambda x: x and "flex-auto" in x)
 
-    if not sidebar:
-        raise ValueError("Could not find navigation sidebar")
+async def _fetch_endpoint_docs(hf_token: str, endpoint: str) -> list[dict[str, str]]:
+    """Fetch all docs for an endpoint by parsing sidebar and fetching each page."""
+    url = f"https://huggingface.co/docs/{endpoint}"
+    headers = {"Authorization": f"Bearer {hf_token}"}
 
-    links = sidebar.find_all("a", href=True)
-    nav_data = []
+    async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
+        resp = await client.get(url, headers=headers)
+        resp.raise_for_status()
+
+        soup = BeautifulSoup(resp.text, "html.parser")
+        sidebar = soup.find("nav", class_=lambda x: x and "flex-auto" in x)
+        if not sidebar:
+            raise ValueError(f"Could not find navigation sidebar for '{endpoint}'")
+
+        nav_items = []
+        for link in sidebar.find_all("a", href=True):
+            href = link["href"]
+            page_url = f"https://huggingface.co{href}" if href.startswith("/") else href
+            nav_items.append({"title": link.get_text(strip=True), "url": page_url})
+
+        if not nav_items:
+            raise ValueError(f"No navigation links found for '{endpoint}'")
+
+        async def fetch_page(item: dict[str, str]) -> dict[str, str]:
+            md_url = f"{item['url']}.md"
+            try:
+                r = await client.get(md_url, headers=headers)
+                r.raise_for_status()
+                content = r.text.strip()
+                glimpse = content[:200] + "..." if len(content) > 200 else content
+            except Exception as e:
+                content, glimpse = "", f"[Could not fetch: {str(e)[:50]}]"
+            return {
+                "title": item["title"],
+                "url": item["url"],
+                "md_url": md_url,
+                "glimpse": glimpse,
+                "content": content,
+                "section": endpoint,
+            }
+
+        return list(await asyncio.gather(*[fetch_page(item) for item in nav_items]))
+
+
+async def _get_docs(hf_token: str, endpoint: str) -> list[dict[str, str]]:
+    """Get docs for endpoint with caching. Expands composite endpoints."""
+    async with _cache_lock:
+        if endpoint in _docs_cache:
+            return _docs_cache[endpoint]
+
+    sub_endpoints = COMPOSITE_ENDPOINTS.get(endpoint, [endpoint])
+    all_docs: list[dict[str, str]] = []
+
+    for sub in sub_endpoints:
+        async with _cache_lock:
+            if sub in _docs_cache:
+                all_docs.extend(_docs_cache[sub])
+                continue
 
-    for link in links:
-        title = link.get_text(strip=True)
-        href = link["href"]
+        docs = await _fetch_endpoint_docs(hf_token, sub)
+        async with _cache_lock:
+            _docs_cache[sub] = docs
+        all_docs.extend(docs)
+
+    async with _cache_lock:
+        _docs_cache[endpoint] = all_docs
+    return all_docs
+
+
+# ---------------------------------------------------------------------------
+# HF Documentation - Search
+# ---------------------------------------------------------------------------
+
+
+async def _build_search_index(
+    endpoint: str, docs: list[dict[str, str]]
+) -> tuple[Any, MultifieldParser]:
+    """Build or retrieve cached Whoosh search index."""
+    async with _cache_lock:
+        if endpoint in _index_cache:
+            return _index_cache[endpoint]
+
+    analyzer = StemmingAnalyzer()
+    schema = Schema(
+        title=TEXT(stored=True, analyzer=analyzer),
+        url=ID(stored=True, unique=True),
+        md_url=ID(stored=True),
+        section=ID(stored=True),
+        glimpse=TEXT(stored=True, analyzer=analyzer),
+        content=TEXT(stored=False, analyzer=analyzer),
+    )
+    storage = RamStorage()
+    index = storage.create_index(schema)
+    writer = index.writer()
+    for doc in docs:
+        writer.add_document(
+            title=doc.get("title", ""),
+            url=doc.get("url", ""),
+            md_url=doc.get("md_url", ""),
+            section=doc.get("section", endpoint),
+            glimpse=doc.get("glimpse", ""),
+            content=doc.get("content", ""),
+        )
+    writer.commit()
 
-        # Make URL absolute
-        page_url = f"https://huggingface.co{href}" if href.startswith("/") else href
-        nav_data.append({"title": title, "url": page_url})
+    parser = MultifieldParser(
+        ["title", "content"],
+        schema=schema,
+        fieldboosts={"title": 2.0, "content": 1.0},
+        group=OrGroup,
+    )
 
-    return nav_data
+    async with _cache_lock:
+        _index_cache[endpoint] = (index, parser)
+    return index, parser
 
 
-async def _fetch_single_glimpse(
-    client: httpx.AsyncClient, hf_token: str, item: dict[str, str]
-) -> dict[str, str]:
-    """Fetch a glimpse (first 300 chars) for a single page"""
-    md_url = f"{item['url']}.md"
-    headers = {"Authorization": f"Bearer {hf_token}"}
+async def _search_docs(
+    endpoint: str, docs: list[dict[str, str]], query: str, limit: int
+) -> tuple[list[dict[str, Any]], str | None]:
+    """Search docs using Whoosh. Returns (results, fallback_message)."""
+    index, parser = await _build_search_index(endpoint, docs)
 
     try:
-        response = await client.get(md_url, headers=headers)
-        response.raise_for_status()
-
-        content = response.text
-        glimpse = content[:300].strip()
-        if len(content) > 300:
-            glimpse += "..."
-
-        return {
-            "title": item["title"],
-            "url": item["url"],
-            "md_url": md_url,
-            "glimpse": glimpse,
-        }
-    except Exception as e:
-        return {
-            "title": item["title"],
-            "url": item["url"],
-            "md_url": md_url,
-            "glimpse": f"[Could not fetch glimpse: {str(e)[:50]}]",
-        }
-
-
-async def _fetch_all_glimpses(
-    hf_token: str, nav_data: list[dict[str, str]]
-) -> list[dict[str, str]]:
-    """Fetch glimpses for all pages in parallel"""
-    async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
-        result_items = await asyncio.gather(
-            *[_fetch_single_glimpse(client, hf_token, item) for item in nav_data]
-        )
+        query_obj = parser.parse(query)
+    except Exception:
+        return [], "Query contained unsupported syntax; showing default ordering."
+
+    with index.searcher() as searcher:
+        results = searcher.search(query_obj, limit=limit)
+        matches = [
+            {
+                "title": hit["title"],
+                "url": hit["url"],
+                "md_url": hit.get("md_url", ""),
+                "section": hit.get("section", endpoint),
+                "glimpse": hit["glimpse"],
+                "score": round(hit.score, 2),
+            }
+            for hit in results
+        ]
+
+    if not matches:
+        return [], "No strong matches found; showing default ordering."
+    return matches, None
+
+
+# ---------------------------------------------------------------------------
+# HF Documentation - Formatting
+# ---------------------------------------------------------------------------
+
+
+def _format_results(
+    endpoint: str,
+    items: list[dict[str, Any]],
+    total: int,
+    query: str | None = None,
+    note: str | None = None,
+) -> str:
+    """Format search results as readable text."""
+    base_url = f"https://huggingface.co/docs/{endpoint}"
+    out = f"Documentation structure for: {base_url}\n\n"
 
-    return list(result_items)
+    if query:
+        out += f"Query: '{query}' → showing {len(items)} result(s) out of {total} pages"
+        if note:
+            out += f" ({note})"
+        out += "\n\n"
+    else:
+        out += f"Found {len(items)} page(s) (total available: {total}).\n"
+        if note:
+            out += f"({note})\n"
+        out += "\n"
 
+    for i, item in enumerate(items, 1):
+        out += f"{i}. **{item['title']}**\n"
+        out += f"   URL: {item['url']}\n"
+        out += f"   Section: {item.get('section', endpoint)}\n"
+        if query and "score" in item:
+            out += f"   Relevance score: {item['score']:.2f}\n"
+        out += f"   Glimpse: {item['glimpse']}\n\n"
+
+    return out
 
-def _format_exploration_results(
-    endpoint: str, result_items: list[dict[str, str]]
-) -> str:
-    """Format the exploration results as a readable string"""
-    base_url = "https://huggingface.co/docs"
-    url = f"{base_url}/{endpoint}"
-    result = f"Documentation structure for: {url}\n\n"
-    result += f"Found {len(result_items)} pages:\n\n"
 
-    for i, item in enumerate(result_items, 1):
-        result += f"{i}. **{item['title']}**\n"
-        result += f"   URL: {item['url']}\n"
-        result += f"   Glimpse: {item['glimpse']}\n\n"
+# ---------------------------------------------------------------------------
+# Handlers
+# ---------------------------------------------------------------------------
 
-    return result
 
+async def explore_hf_docs_handler(arguments: dict[str, Any]) -> tuple[str, bool]:
+    """Explore documentation structure with optional search query."""
+    endpoint = arguments.get("endpoint", "").lstrip("/")
+    query = arguments.get("query")
+    max_results = arguments.get("max_results")
 
-async def explore_hf_docs(hf_token: str, endpoint: str) -> str:
-    """Main function to explore documentation structure"""
-    # Fetch HTML page
-    html_content = await _fetch_html_page(hf_token, endpoint)
+    if not endpoint:
+        return "Error: No endpoint provided", False
 
-    # Parse navigation
-    nav_data = _parse_sidebar_navigation(html_content)
+    # Gradio uses its own API
+    if endpoint.lower() == "gradio":
+        try:
+            clean_query = (
+                query.strip() if isinstance(query, str) and query.strip() else None
+            )
+            content = await _fetch_gradio_docs(clean_query)
+            header = "# Gradio Documentation\n\n"
+            if clean_query:
+                header += f"Query: '{clean_query}'\n\n"
+            header += "Source: https://gradio.app/docs\n\n---\n\n"
+            return header + content, True
+        except httpx.HTTPStatusError as e:
+            return f"HTTP error fetching Gradio docs: {e.response.status_code}", False
+        except httpx.RequestError as e:
+            return f"Request error fetching Gradio docs: {str(e)}", False
+        except Exception as e:
+            return f"Error fetching Gradio docs: {str(e)}", False
+
+    # HF docs
+    hf_token = os.environ.get("HF_TOKEN")
+    if not hf_token:
+        return "Error: HF_TOKEN environment variable not set", False
 
-    if not nav_data:
-        raise ValueError(f"No navigation links found for endpoint '{endpoint}'")
+    try:
+        max_results_int = int(max_results) if max_results is not None else None
+    except (TypeError, ValueError):
+        return "Error: max_results must be an integer", False
 
-    # Fetch all glimpses in parallel
-    result_items = await _fetch_all_glimpses(hf_token, nav_data)
+    if max_results_int is not None and max_results_int <= 0:
+        return "Error: max_results must be greater than zero", False
 
-    # Format results
-    result = _format_exploration_results(endpoint, result_items)
+    try:
+        docs = await _get_docs(hf_token, endpoint)
+        total = len(docs)
+
+        # Determine limit
+        if max_results_int is None:
+            limit = DEFAULT_MAX_RESULTS
+            limit_note = f"Showing top {DEFAULT_MAX_RESULTS} results (set max_results to adjust)."
+        elif max_results_int > MAX_RESULTS_CAP:
+            limit = MAX_RESULTS_CAP
+            limit_note = f"Requested {max_results_int} but showing top {MAX_RESULTS_CAP} (maximum)."
+        else:
+            limit = max_results_int
+            limit_note = None
+
+        # Search or paginate
+        clean_query = (
+            query.strip() if isinstance(query, str) and query.strip() else None
+        )
+        fallback_msg = None
 
-    return result
+        if clean_query:
+            results, fallback_msg = await _search_docs(
+                endpoint, docs, clean_query, limit
+            )
+            if not results:
+                results = docs[:limit]
+        else:
+            results = docs[:limit]
 
+        # Combine notes
+        notes = []
+        if fallback_msg:
+            notes.append(fallback_msg)
+        if limit_note:
+            notes.append(limit_note)
+        note = "; ".join(notes) if notes else None
 
-async def explore_hf_docs_handler(arguments: dict[str, Any]) -> tuple[str, bool]:
-    """
-    Explore the documentation structure for a given endpoint by parsing the sidebar navigation
+        return _format_results(endpoint, results, total, clean_query, note), True
 
-    Args:
-        arguments: Dictionary with 'endpoint' parameter (e.g., 'trl', 'transformers', etc.)
+    except httpx.HTTPStatusError as e:
+        return f"HTTP error: {e.response.status_code} - {e.response.text[:200]}", False
+    except httpx.RequestError as e:
+        return f"Request error: {str(e)}", False
+    except ValueError as e:
+        return f"Error: {str(e)}", False
+    except Exception as e:
+        return f"Unexpected error: {str(e)}", False
 
-    Returns:
-        Tuple of (structured_navigation_with_glimpses, success)
-    """
-    endpoint = arguments.get("endpoint", "")
 
-    if not endpoint:
-        return "Error: No endpoint provided", False
+async def hf_docs_fetch_handler(arguments: dict[str, Any]) -> tuple[str, bool]:
+    """Fetch full markdown content of a documentation page."""
+    url = arguments.get("url", "")
+    if not url:
+        return "Error: No URL provided", False
 
-    # Get HF token from environment
     hf_token = os.environ.get("HF_TOKEN")
-
     if not hf_token:
         return "Error: HF_TOKEN environment variable not set", False
 
-    endpoint = endpoint.lstrip("/")
+    if not url.endswith(".md"):
+        url = f"{url}.md"
 
     try:
-        result = await explore_hf_docs(hf_token, endpoint)
-        return result, True
-
+        async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
+            resp = await client.get(
+                url, headers={"Authorization": f"Bearer {hf_token}"}
+            )
+            resp.raise_for_status()
+        return f"Documentation from: {url}\n\n{resp.text}", True
     except httpx.HTTPStatusError as e:
         return (
-            f"HTTP error: {e.response.status_code} - {e.response.text[:200]}",
+            f"HTTP error fetching {url}: {e.response.status_code} - {e.response.text[:200]}",
             False,
         )
     except httpx.RequestError as e:
-        return f"Request error: {str(e)}", False
-    except ValueError as e:
-        return f"Error: {str(e)}", False
+        return f"Request error fetching {url}: {str(e)}", False
     except Exception as e:
-        return f"Unexpected error: {str(e)}", False
+        return f"Error fetching documentation: {str(e)}", False
 
 
-async def _fetch_openapi_spec() -> dict[str, Any]:
-    """Fetch and cache the HuggingFace OpenAPI specification"""
-    global _openapi_spec_cache
+# ---------------------------------------------------------------------------
+# OpenAPI Search
+# ---------------------------------------------------------------------------
 
-    if _openapi_spec_cache is not None:
-        return _openapi_spec_cache
 
-    url = "https://huggingface.co/.well-known/openapi.json"
+async def _fetch_openapi_spec() -> dict[str, Any]:
+    """Fetch and cache HuggingFace OpenAPI specification."""
+    global _openapi_cache
+    if _openapi_cache is not None:
+        return _openapi_cache
 
     async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
-        response = await client.get(url)
-        response.raise_for_status()
+        resp = await client.get("https://huggingface.co/.well-known/openapi.json")
+        resp.raise_for_status()
 
-    spec = response.json()
-    _openapi_spec_cache = spec
-
-    return spec
+    _openapi_cache = resp.json()
+    return _openapi_cache
 
 
 def _extract_all_tags(spec: dict[str, Any]) -> list[str]:
-    """Extract all unique tags from the OpenAPI spec"""
+    """Extract all unique tags from OpenAPI spec."""
     tags = set()
-
-    # Get tags from the tags section
     for tag_obj in spec.get("tags", []):
         if "name" in tag_obj:
             tags.add(tag_obj["name"])
-
-    # Also get tags from paths (in case some aren't in the tags section)
-    for path, path_item in spec.get("paths", {}).items():
-        for method, operation in path_item.items():
+    for path_item in spec.get("paths", {}).values():
+        for method, op in path_item.items():
             if method in ["get", "post", "put", "delete", "patch", "head", "options"]:
-                for tag in operation.get("tags", []):
+                for tag in op.get("tags", []):
                     tags.add(tag)
-
-    return sorted(list(tags))
-
-
-def _search_openapi_by_tag(spec: dict[str, Any], tag: str) -> list[dict[str, Any]]:
-    """Search for API endpoints with a specific tag"""
-    results = []
-    paths = spec.get("paths", {})
-    servers = spec.get("servers", [])
-    base_url = (
-        servers[0].get("url", "https://huggingface.co")
-        if servers
-        else "https://huggingface.co"
-    )
-
-    for path, path_item in paths.items():
-        for method, operation in path_item.items():
-            if method not in [
-                "get",
-                "post",
-                "put",
-                "delete",
-                "patch",
-                "head",
-                "options",
-            ]:
-                continue
-
-            operation_tags = operation.get("tags", [])
-            if tag in operation_tags:
-                # Extract parameters
-                parameters = operation.get("parameters", [])
-                request_body = operation.get("requestBody", {})
-                responses = operation.get("responses", {})
-
-                results.append(
-                    {
-                        "path": path,
-                        "method": method.upper(),
-                        "operationId": operation.get("operationId", ""),
-                        "summary": operation.get("summary", ""),
-                        "description": operation.get("description", ""),
-                        "parameters": parameters,
-                        "request_body": request_body,
-                        "responses": responses,
-                        "base_url": base_url,
-                    }
-                )
-
-    return results
+    return sorted(tags)
 
 
 def _generate_curl_example(endpoint: dict[str, Any]) -> str:
-    """Generate a curl command example for an endpoint"""
+    """Generate curl command example for an endpoint."""
     method = endpoint["method"]
     path = endpoint["path"]
     base_url = endpoint["base_url"]
 
-    # Build the full URL with example path parameters
+    # Build URL with path parameters
     full_path = path
     for param in endpoint.get("parameters", []):
         if param.get("in") == "path" and param.get("required"):
-            param_name = param["name"]
+            name = param["name"]
             example = param.get(
-                "example", param.get("schema", {}).get("example", f"<{param_name}>")
+                "example", param.get("schema", {}).get("example", f"<{name}>")
             )
-            full_path = full_path.replace(f"{{{param_name}}}", str(example))
+            full_path = full_path.replace(f"{{{name}}}", str(example))
 
     curl = f"curl -X {method} \\\n  '{base_url}{full_path}'"
 
-    # Add query parameters if any
+    # Add query parameters
     query_params = [p for p in endpoint.get("parameters", []) if p.get("in") == "query"]
     if query_params and query_params[0].get("required"):
         param = query_params[0]
         example = param.get("example", param.get("schema", {}).get("example", "value"))
         curl += f"?{param['name']}={example}"
 
-    # Add headers
     curl += " \\\n  -H 'Authorization: Bearer $HF_TOKEN'"
 
-    # Add request body if applicable
+    # Add request body
     if method in ["POST", "PUT", "PATCH"] and endpoint.get("request_body"):
         content = endpoint["request_body"].get("content", {})
         if "application/json" in content:
@@ -291,8 +476,6 @@ def _generate_curl_example(endpoint: dict[str, Any]) -> str:
             schema = content["application/json"].get("schema", {})
             example = schema.get("example", "{}")
             if isinstance(example, dict):
-                import json
-
                 example = json.dumps(example, indent=2)
             curl += f" \\\n  -d '{example}'"
 
@@ -300,72 +483,50 @@ def _generate_curl_example(endpoint: dict[str, Any]) -> str:
 
 
 def _format_parameters(parameters: list[dict[str, Any]]) -> str:
-    """Format parameter information from OpenAPI spec"""
+    """Format parameter information from OpenAPI spec."""
     if not parameters:
         return ""
 
-    # Group parameters by type
     path_params = [p for p in parameters if p.get("in") == "path"]
     query_params = [p for p in parameters if p.get("in") == "query"]
     header_params = [p for p in parameters if p.get("in") == "header"]
 
     output = []
 
-    if path_params:
-        output.append("**Path Parameters:**")
-        for param in path_params:
-            name = param.get("name", "")
-            required = " (required)" if param.get("required") else " (optional)"
-            description = param.get("description", "")
-            param_type = param.get("schema", {}).get("type", "string")
-            example = param.get("example") or param.get("schema", {}).get("example", "")
-
-            output.append(f"- `{name}` ({param_type}){required}: {description}")
-            if example:
-                output.append(f"  Example: `{example}`")
-
-    if query_params:
+    for label, params in [
+        ("Path Parameters", path_params),
+        ("Query Parameters", query_params),
+        ("Header Parameters", header_params),
+    ]:
+        if not params:
+            continue
         if output:
             output.append("")
-        output.append("**Query Parameters:**")
-        for param in query_params:
-            name = param.get("name", "")
-            required = " (required)" if param.get("required") else " (optional)"
-            description = param.get("description", "")
-            param_type = param.get("schema", {}).get("type", "string")
-            example = param.get("example") or param.get("schema", {}).get("example", "")
-
-            output.append(f"- `{name}` ({param_type}){required}: {description}")
+        output.append(f"**{label}:**")
+        for p in params:
+            name = p.get("name", "")
+            required = " (required)" if p.get("required") else " (optional)"
+            desc = p.get("description", "")
+            ptype = p.get("schema", {}).get("type", "string")
+            example = p.get("example") or p.get("schema", {}).get("example", "")
+
+            output.append(f"- `{name}` ({ptype}){required}: {desc}")
             if example:
                 output.append(f"  Example: `{example}`")
 
-    if header_params:
-        if output:
-            output.append("")
-        output.append("**Header Parameters:**")
-        for param in header_params:
-            name = param.get("name", "")
-            required = " (required)" if param.get("required") else " (optional)"
-            description = param.get("description", "")
-
-            output.append(f"- `{name}`{required}: {description}")
-
     return "\n".join(output)
 
 
 def _format_response_info(responses: dict[str, Any]) -> str:
-    """Format response information from OpenAPI spec"""
+    """Format response information from OpenAPI spec."""
     if not responses:
         return "No response information available"
 
     output = []
-    for status_code, response_obj in list(responses.items())[
-        :3
-    ]:  # Show first 3 status codes
-        desc = response_obj.get("description", "")
-        output.append(f"- **{status_code}**: {desc}")
-
-        content = response_obj.get("content", {})
+    for status, resp_obj in list(responses.items())[:3]:
+        desc = resp_obj.get("description", "")
+        output.append(f"- **{status}**: {desc}")
+        content = resp_obj.get("content", {})
         if "application/json" in content:
             schema = content["application/json"].get("schema", {})
             if "type" in schema:
@@ -375,72 +536,87 @@ def _format_response_info(responses: dict[str, Any]) -> str:
 
 
 def _format_openapi_results(results: list[dict[str, Any]], tag: str) -> str:
-    """Format OpenAPI search results as markdown with curl examples"""
+    """Format OpenAPI search results with curl examples."""
     if not results:
         return f"No API endpoints found with tag '{tag}'"
 
-    output = f"# API Endpoints for tag: `{tag}`\n\n"
-    output += f"Found {len(results)} endpoint(s)\n\n"
-    output += "---\n\n"
+    out = f"# API Endpoints for tag: `{tag}`\n\n"
+    out += f"Found {len(results)} endpoint(s)\n\n---\n\n"
 
-    for i, endpoint in enumerate(results, 1):
-        output += f"## {i}. {endpoint['method']} {endpoint['path']}\n\n"
+    for i, ep in enumerate(results, 1):
+        out += f"## {i}. {ep['method']} {ep['path']}\n\n"
 
-        if endpoint["summary"]:
-            output += f"**Summary:** {endpoint['summary']}\n\n"
+        if ep["summary"]:
+            out += f"**Summary:** {ep['summary']}\n\n"
 
-        if endpoint["description"]:
-            desc = endpoint["description"][:300]
-            if len(endpoint["description"]) > 300:
+        if ep["description"]:
+            desc = ep["description"][:300]
+            if len(ep["description"]) > 300:
                 desc += "..."
-            output += f"**Description:** {desc}\n\n"
+            out += f"**Description:** {desc}\n\n"
 
-        # Parameters
-        params_info = _format_parameters(endpoint.get("parameters", []))
+        params_info = _format_parameters(ep.get("parameters", []))
         if params_info:
-            output += params_info + "\n\n"
-
-        # Curl example
-        output += "**Usage:**\n```bash\n"
-        output += _generate_curl_example(endpoint)
-        output += "\n```\n\n"
+            out += params_info + "\n\n"
 
-        # Response info
-        output += "**Returns:**\n"
-        output += _format_response_info(endpoint["responses"])
-        output += "\n\n"
+        out += "**Usage:**\n```bash\n"
+        out += _generate_curl_example(ep)
+        out += "\n```\n\n"
 
-        output += "---\n\n"
+        out += "**Returns:**\n"
+        out += _format_response_info(ep["responses"])
+        out += "\n\n---\n\n"
 
-    return output
+    return out
 
 
 async def search_openapi_handler(arguments: dict[str, Any]) -> tuple[str, bool]:
-    """
-    Search the HuggingFace OpenAPI specification by tag
-
-    Args:
-        arguments: Dictionary with 'tag' parameter
-
-    Returns:
-        Tuple of (search_results, success)
-    """
+    """Search HuggingFace OpenAPI specification by tag."""
     tag = arguments.get("tag", "")
-
     if not tag:
         return "Error: No tag provided", False
 
     try:
-        # Fetch OpenAPI spec (cached after first fetch)
         spec = await _fetch_openapi_spec()
+        paths = spec.get("paths", {})
+        servers = spec.get("servers", [])
+        base_url = (
+            servers[0].get("url", "https://huggingface.co")
+            if servers
+            else "https://huggingface.co"
+        )
 
-        # Search for endpoints with this tag
-        results = _search_openapi_by_tag(spec, tag)
+        results = []
+        for path, path_item in paths.items():
+            for method, op in path_item.items():
+                if method not in [
+                    "get",
+                    "post",
+                    "put",
+                    "delete",
+                    "patch",
+                    "head",
+                    "options",
+                ]:
+                    continue
+                if tag not in op.get("tags", []):
+                    continue
 
-        # Format results
-        formatted = _format_openapi_results(results, tag)
+                results.append(
+                    {
+                        "path": path,
+                        "method": method.upper(),
+                        "operationId": op.get("operationId", ""),
+                        "summary": op.get("summary", ""),
+                        "description": op.get("description", ""),
+                        "parameters": op.get("parameters", []),
+                        "request_body": op.get("requestBody", {}),
+                        "responses": op.get("responses", {}),
+                        "base_url": base_url,
+                    }
+                )
 
-        return formatted, True
+        return _format_openapi_results(results, tag), True
 
     except httpx.HTTPStatusError as e:
         return f"HTTP error fetching OpenAPI spec: {e.response.status_code}", False
@@ -450,66 +626,86 @@ async def search_openapi_handler(arguments: dict[str, Any]) -> tuple[str, bool]:
         return f"Error searching OpenAPI spec: {str(e)}", False
 
 
-async def hf_docs_fetch_handler(arguments: dict[str, Any]) -> tuple[str, bool]:
-    """
-    Fetch full documentation content from a specific HF docs page
-
-    Args:
-        arguments: Dictionary with 'url' parameter (full URL to the doc page)
-
-    Returns:
-        Tuple of (full_markdown_content, success)
-    """
-    url = arguments.get("url", "")
-
-    if not url:
-        return "Error: No URL provided", False
-
-    # Get HF token from environment
-    hf_token = os.environ.get("HF_TOKEN")
-
-    if not hf_token:
-        return (
-            "Error: HF_TOKEN environment variable not set",
-            False,
-        )
-
-    # Add .md extension if not already present
-    if not url.endswith(".md"):
-        url = f"{url}.md"
-
-    try:
-        # Make request with auth
-        headers = {"Authorization": f"Bearer {hf_token}"}
-
-        async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
-            response = await client.get(url, headers=headers)
-            response.raise_for_status()
-
-        content = response.text
-
-        # Return the markdown content directly
-        result = f"Documentation from: {url}\n\n{content}"
-
-        return result, True
+async def _get_api_search_tool_spec() -> dict[str, Any]:
+    """Generate OpenAPI tool spec with tags populated at runtime."""
+    spec = await _fetch_openapi_spec()
+    tags = _extract_all_tags(spec)
 
-    except httpx.HTTPStatusError as e:
-        return (
-            f"HTTP error fetching {url}: {e.response.status_code} - {e.response.text[:200]}",
-            False,
-        )
-    except httpx.RequestError as e:
-        return f"Request error fetching {url}: {str(e)}", False
-    except Exception as e:
-        return f"Error fetching documentation: {str(e)}", False
+    return {
+        "name": "search_hf_api_endpoints",
+        "description": (
+            "Search HuggingFace OpenAPI specification by tag to find API endpoints with curl examples. "
+            "**Use when:** (1) Need to interact with HF Hub API directly, (2) Building scripts for repo operations, "
+            "(3) Need authentication patterns, (4) Understanding API parameters and responses, "
+            "(5) Need curl examples for HTTP requests. "
+            "Returns: Endpoint paths, methods, parameters, curl examples with authentication, and response schemas. "
+            "Tags group related operations: repos, models, datasets, inference, spaces, etc."
+        ),
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "tag": {
+                    "type": "string",
+                    "enum": tags,
+                    "description": "The API tag to search for. Each tag groups related API endpoints.",
+                },
+            },
+            "required": ["tag"],
+        },
+    }
 
 
-# Tool specifications for documentation search
+# ---------------------------------------------------------------------------
+# Tool Specifications
+# ---------------------------------------------------------------------------
+
+DOC_ENDPOINTS = [
+    "hub",
+    "transformers",
+    "diffusers",
+    "datasets",
+    "gradio",
+    "trackio",
+    "smolagents",
+    "huggingface_hub",
+    "huggingface.js",
+    "transformers.js",
+    "inference-providers",
+    "inference-endpoints",
+    "peft",
+    "accelerate",
+    "optimum",
+    "tokenizers",
+    "courses",
+    "evaluate",
+    "tasks",
+    "dataset-viewer",
+    "trl",
+    "simulate",
+    "sagemaker",
+    "timm",
+    "safetensors",
+    "tgi",
+    "setfit",
+    "lerobot",
+    "autotrain",
+    "tei",
+    "bitsandbytes",
+    "sentence_transformers",
+    "chat-ui",
+    "leaderboards",
+    "lighteval",
+    "argilla",
+    "distilabel",
+    "microsoft-azure",
+    "kernels",
+    "google-cloud",
+]
 
 EXPLORE_HF_DOCS_TOOL_SPEC = {
     "name": "explore_hf_docs",
     "description": (
-        "Explore Hugging Face documentation structure and discover available pages with 300-character previews. "
+        "Explore Hugging Face documentation structure and discover available pages with 200-character previews. "
         "⚠️ MANDATORY: ALWAYS use this BEFORE implementing any ML task (training, fine-tuning, data processing, inference). "
         "Your training data may be outdated - current documentation is the source of truth. "
         "**Use when:** (1) Starting any implementation task, (2) User asks 'how to' questions, "
@@ -519,77 +715,22 @@ EXPLORE_HF_DOCS_TOOL_SPEC = {
         "Returns: Sidebar navigation with titles, URLs, and glimpses of all pages in the selected documentation. "
         "**Then:** Use fetch_hf_docs with specific URLs from results to get full content. "
         "**Critical for reliability:** Never implement based on internal knowledge without checking current docs first - APIs change frequently."
+        " By default returns the top 20 results; set max_results (max 50) to adjust."
     ),
     "parameters": {
         "type": "object",
         "properties": {
             "endpoint": {
                 "type": "string",
-                "enum": [
-                    "hub",
-                    "transformers",
-                    "diffusers",
-                    "datasets",
-                    "gradio",
-                    "trackio",
-                    "smolagents",
-                    "huggingface_hub",
-                    "huggingface.js",
-                    "transformers.js",
-                    "inference-providers",
-                    "inference-endpoints",
-                    "peft",
-                    "accelerate",
-                    "optimum",
-                    "optimum-habana",
-                    "optimum-neuron",
-                    "optimum-intel",
-                    "optimum-executorch",
-                    "optimum-tpu",
-                    "tokenizers",
-                    "llm-course",
-                    "robotics-course",
-                    "mcp-course",
-                    "smol-course",
-                    "agents-course",
-                    "deep-rl-course",
-                    "computer-vision-course",
-                    "evaluate",
-                    "tasks",
-                    "dataset-viewer",
-                    "trl",
-                    "simulate",
-                    "sagemaker",
-                    "timm",
-                    "safetensors",
-                    "tgi",
-                    "setfit",
-                    "audio-course",
-                    "lerobot",
-                    "autotrain",
-                    "tei",
-                    "bitsandbytes",
-                    "cookbook",
-                    "sentence_transformers",
-                    "ml-games-course",
-                    "diffusion-course",
-                    "ml-for-3d-course",
-                    "chat-ui",
-                    "leaderboards",
-                    "lighteval",
-                    "argilla",
-                    "distilabel",
-                    "microsoft-azure",
-                    "kernels",
-                    "google-cloud",
-                ],
+                "enum": DOC_ENDPOINTS,
                 "description": (
                     "The documentation endpoint to explore. Each endpoint corresponds to a major section of the Hugging Face documentation:\n\n"
+                    "• courses — All Hugging Face courses (LLM, robotics, MCP, smol (llm training), agents, deep RL, computer vision, games, diffusion, 3D, audio) and the cookbook recipes. Probably the best place for examples.\n"
                     "• hub — Find answers to questions about models/datasets/spaces, auth, versioning, metadata.\n"
                     "• transformers — Core model library: architectures, configs, tokenizers, training & inference APIs.\n"
                     "• diffusers — Diffusion pipelines, schedulers, fine-tuning, training, and deployment patterns.\n"
                     "• datasets — Dataset loading, streaming, processing, Arrow format, Hub integration.\n"
-                    "• gradio — UI components and demos for interacting with ML models.\n"
+                    "• gradio — UI components and demos for ML models. Uses Gradio's native API: without query returns full docs (llms.txt), with query uses embedding search for precise results.\n"
                     "• trackio — Experiment tracking, metrics logging, and run comparison.\n"
                     "• smolagents — Lightweight agent abstractions and tool-using patterns.\n"
                     "• huggingface_hub — Python client for Hub operations (auth, upload/download, repo management).\n"
@@ -599,20 +740,8 @@ EXPLORE_HF_DOCS_TOOL_SPEC = {
                     "• inference-endpoints — Managed, scalable model deployments on HF infrastructure.\n"
                     "• peft — Parameter-efficient fine-tuning methods (LoRA, adapters, etc.).\n"
                     "• accelerate — Hardware-agnostic, distributed and mixed-precision training orchestration.\n"
-                    "• optimum — Hardware-aware optimization and model export tooling.\n"
-                    "• optimum-habana — Training and inference on Habana Gaudi accelerators.\n"
-                    "• optimum-neuron — Optimization workflows for AWS Inferentia/Trainium.\n"
-                    "• optimum-intel — Intel CPU/GPU optimizations (OpenVINO, IPEX).\n"
-                    "• optimum-executorch — Exporting models to ExecuTorch for edge/mobile.\n"
-                    "• optimum-tpu — TPU-specific training and optimization paths.\n"
+                    "• optimum — Hardware-aware optimization and model export tooling, including Habana, Neuron, Intel, ExecuTorch, and TPU variants.\n"
                     "• tokenizers — Fast tokenizer internals, training, and low-level APIs.\n"
-                    "• llm-course — End-to-end LLM concepts, training, and deployment.\n"
-                    "• robotics-course — Learning-based robotics foundations.\n"
-                    "• mcp-course — Model Context Protocol concepts and usage.\n"
-                    "• smol-course — Small-model and efficiency-focused workflows.\n"
-                    "• agents-course — Tool-using, planning, and multi-step agent design.\n"
-                    "• deep-rl-course — Deep reinforcement learning foundations.\n"
-                    "• computer-vision-course — Vision models, datasets, and pipelines.\n"
                     "• evaluate — Metrics, evaluation workflows, and training-loop integration.\n"
                     "• tasks — Canonical task definitions and model categorization.\n"
                     "• dataset-viewer — Dataset preview, streaming views, and viewer internals.\n"
@@ -623,16 +752,11 @@ EXPLORE_HF_DOCS_TOOL_SPEC = {
                     "• safetensors — Safe, fast tensor serialization format.\n"
                     "• tgi — High-throughput text generation server for LLMs.\n"
                     "• setfit — Few-shot text classification via sentence embeddings.\n"
-                    "• audio-course — Speech and audio models, datasets, and tasks.\n"
                     "• lerobot — Robotics datasets, policies, and learning workflows.\n"
                     "• autotrain — No/low-code model training on Hugging Face.\n"
                     "• tei — Optimized inference server for embedding workloads.\n"
                     "• bitsandbytes — Quantization and memory-efficient optimizers.\n"
-                    "• cookbook — Practical, task-oriented recipes across the ecosystem.\n"
                     "• sentence_transformers — Embedding models, training recipes, similarity/search workflows.\n"
-                    "• ml-games-course — Game-based ML and reinforcement learning experiments.\n"
-                    "• diffusion-course — Diffusion model theory and hands-on practice.\n"
-                    "• ml-for-3d-course — 3D representations, models, and learning techniques.\n"
                     "• chat-ui — Reference chat interfaces for LLM deployment.\n"
                     "• leaderboards — Evaluation leaderboards and submission mechanics.\n"
                     "• lighteval — Lightweight, reproducible LLM evaluation framework.\n"
@@ -643,6 +767,19 @@ EXPLORE_HF_DOCS_TOOL_SPEC = {
                     "• google-cloud — GCP deployment and serving workflows.\n"
                 ),
             },
+            "query": {
+                "type": "string",
+                "description": (
+                    "Optional keyword query to rank and filter documentation pages. "
+                    "For Gradio, use concise queries like 'how to use the image component' or 'audio component demo'."
+                ),
+            },
+            "max_results": {
+                "type": "integer",
+                "description": "Max results (default 20, max 50). Ignored for Gradio.",
+                "minimum": 1,
+                "maximum": 50,
+            },
         },
         "required": ["endpoint"],
     },
@@ -677,40 +814,3 @@ HF_DOCS_FETCH_TOOL_SPEC = {
         "required": ["url"],
     },
 }
-
-
-async def _get_api_search_tool_spec() -> dict[str, Any]:
-    """
-    Dynamically generate the OpenAPI tool spec with tag enum populated at runtime
-    This must be called async to fetch the OpenAPI spec and extract tags
-    """
-    spec = await _fetch_openapi_spec()
-    tags = _extract_all_tags(spec)
-
-    return {
-        "name": "search_hf_api_endpoints",
-        "description": (
-            "Search HuggingFace OpenAPI specification by tag to find API endpoints with curl examples. "
-            "**Use when:** (1) Need to interact with HF Hub API directly, (2) Building scripts for repo operations, "
-            "(3) Need authentication patterns, (4) Understanding API parameters and responses, "
-            "(5) Need curl examples for HTTP requests. "
-            "Returns: Endpoint paths, methods, parameters, curl examples with authentication, and response schemas. "
-            "**Pattern:** search_hf_api_endpoints (find endpoint) → use curl pattern in implementation. "
-            "Tags group related operations: repos, models, datasets, inference, spaces, etc. "
-            "**Note:** Each result includes curl example with $HF_TOKEN placeholder for authentication. "
-            "**For tool building:** This provides the API foundation for creating Hub interaction scripts."
-        ),
-        "parameters": {
-            "type": "object",
-            "properties": {
-                "tag": {
-                    "type": "string",
-                    "enum": tags,
-                    "description": (
-                        "The API tag to search for. Each tag groups related API endpoints. "
-                    ),
-                },
-            },
-            "required": ["tag"],
-        },
-    }

agent/tools/hf_repo_files_tool.py

@@ -0,0 +1,322 @@
+"""
+HF Repo Files Tool - File operations on Hugging Face repositories
+
+Operations: list, read, upload, delete
+"""
+
+import asyncio
+from typing import Any, Dict, Literal, Optional
+
+from huggingface_hub import HfApi, hf_hub_download
+from huggingface_hub.utils import EntryNotFoundError, RepositoryNotFoundError
+
+from agent.tools.types import ToolResult
+
+OperationType = Literal["list", "read", "upload", "delete"]
+
+
+async def _async_call(func, *args, **kwargs):
+    """Wrap synchronous HfApi calls for async context."""
+    return await asyncio.to_thread(func, *args, **kwargs)
+
+
+def _build_repo_url(repo_id: str, repo_type: str = "model") -> str:
+    """Build the Hub URL for a repository."""
+    if repo_type == "model":
+        return f"https://huggingface.co/{repo_id}"
+    return f"https://huggingface.co/{repo_type}s/{repo_id}"
+
+
+def _format_size(size_bytes: int) -> str:
+    """Format file size in human-readable form."""
+    for unit in ["B", "KB", "MB", "GB", "TB"]:
+        if size_bytes < 1024:
+            return f"{size_bytes:.1f}{unit}"
+        size_bytes /= 1024
+    return f"{size_bytes:.1f}PB"
+
+
+class HfRepoFilesTool:
+    """Tool for file operations on HF repos."""
+
+    def __init__(self, hf_token: Optional[str] = None):
+        self.api = HfApi(token=hf_token)
+
+    async def execute(self, args: Dict[str, Any]) -> ToolResult:
+        """Execute the specified operation."""
+        operation = args.get("operation")
+
+        if not operation:
+            return self._help()
+
+        try:
+            handlers = {
+                "list": self._list,
+                "read": self._read,
+                "upload": self._upload,
+                "delete": self._delete,
+            }
+
+            handler = handlers.get(operation)
+            if handler:
+                return await handler(args)
+            else:
+                return self._error(f"Unknown operation: {operation}. Valid: list, read, upload, delete")
+
+        except RepositoryNotFoundError:
+            return self._error(f"Repository not found: {args.get('repo_id')}")
+        except EntryNotFoundError:
+            return self._error(f"File not found: {args.get('path')}")
+        except Exception as e:
+            return self._error(f"Error: {str(e)}")
+
+    def _help(self) -> ToolResult:
+        """Show usage instructions."""
+        return {
+            "formatted": """**hf_repo_files** - File operations on HF repos
+
+**Operations:**
+- `list` - List files: `{"operation": "list", "repo_id": "gpt2"}`
+- `read` - Read file: `{"operation": "read", "repo_id": "gpt2", "path": "config.json"}`
+- `upload` - Upload: `{"operation": "upload", "repo_id": "my-model", "path": "README.md", "content": "..."}`
+- `delete` - Delete: `{"operation": "delete", "repo_id": "my-model", "patterns": ["*.tmp"]}`
+
+**Common params:** repo_id (required), repo_type (model/dataset/space), revision (default: main)""",
+            "totalResults": 1,
+            "resultsShared": 1,
+        }
+
+    async def _list(self, args: Dict[str, Any]) -> ToolResult:
+        """List files in a repository."""
+        repo_id = args.get("repo_id")
+        if not repo_id:
+            return self._error("repo_id is required")
+
+        repo_type = args.get("repo_type", "model")
+        revision = args.get("revision", "main")
+        path = args.get("path", "")
+
+        items = list(await _async_call(
+            self.api.list_repo_tree,
+            repo_id=repo_id,
+            repo_type=repo_type,
+            revision=revision,
+            path_in_repo=path,
+            recursive=True,
+        ))
+
+        if not items:
+            return {"formatted": f"No files in {repo_id}", "totalResults": 0, "resultsShared": 0}
+
+        lines = []
+        total_size = 0
+        for item in sorted(items, key=lambda x: x.path):
+            if hasattr(item, "size") and item.size:
+                total_size += item.size
+                lines.append(f"{item.path} ({_format_size(item.size)})")
+            else:
+                lines.append(f"{item.path}/")
+
+        url = _build_repo_url(repo_id, repo_type)
+        response = f"**{repo_id}** ({len(items)} files, {_format_size(total_size)})\n{url}/tree/{revision}\n\n" + "\n".join(lines)
+
+        return {"formatted": response, "totalResults": len(items), "resultsShared": len(items)}
+
+    async def _read(self, args: Dict[str, Any]) -> ToolResult:
+        """Read file content from a repository."""
+        repo_id = args.get("repo_id")
+        path = args.get("path")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not path:
+            return self._error("path is required")
+
+        repo_type = args.get("repo_type", "model")
+        revision = args.get("revision", "main")
+        max_chars = args.get("max_chars", 50000)
+
+        file_path = await _async_call(
+            hf_hub_download,
+            repo_id=repo_id,
+            filename=path,
+            repo_type=repo_type,
+            revision=revision,
+            token=self.api.token,
+        )
+
+        try:
+            with open(file_path, "r", encoding="utf-8") as f:
+                content = f.read()
+
+            truncated = len(content) > max_chars
+            if truncated:
+                content = content[:max_chars]
+
+            url = f"{_build_repo_url(repo_id, repo_type)}/blob/{revision}/{path}"
+            response = f"**{path}**{' (truncated)' if truncated else ''}\n{url}\n\n```\n{content}\n```"
+
+            return {"formatted": response, "totalResults": 1, "resultsShared": 1}
+
+        except UnicodeDecodeError:
+            import os
+            size = os.path.getsize(file_path)
+            return {"formatted": f"Binary file ({_format_size(size)})", "totalResults": 1, "resultsShared": 1}
+
+    async def _upload(self, args: Dict[str, Any]) -> ToolResult:
+        """Upload content to a repository."""
+        repo_id = args.get("repo_id")
+        path = args.get("path")
+        content = args.get("content")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not path:
+            return self._error("path is required")
+        if content is None:
+            return self._error("content is required")
+
+        repo_type = args.get("repo_type", "model")
+        revision = args.get("revision", "main")
+        create_pr = args.get("create_pr", False)
+        commit_message = args.get("commit_message", f"Upload {path}")
+
+        file_bytes = content.encode("utf-8") if isinstance(content, str) else content
+
+        result = await _async_call(
+            self.api.upload_file,
+            path_or_fileobj=file_bytes,
+            path_in_repo=path,
+            repo_id=repo_id,
+            repo_type=repo_type,
+            revision=revision,
+            commit_message=commit_message,
+            create_pr=create_pr,
+        )
+
+        url = _build_repo_url(repo_id, repo_type)
+        if create_pr and hasattr(result, "pr_url"):
+            response = f"**Uploaded as PR**\n{result.pr_url}"
+        else:
+            response = f"**Uploaded:** {path}\n{url}/blob/{revision}/{path}"
+
+        return {"formatted": response, "totalResults": 1, "resultsShared": 1}
+
+    async def _delete(self, args: Dict[str, Any]) -> ToolResult:
+        """Delete files from a repository."""
+        repo_id = args.get("repo_id")
+        patterns = args.get("patterns")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not patterns:
+            return self._error("patterns is required (list of paths/wildcards)")
+
+        if isinstance(patterns, str):
+            patterns = [patterns]
+
+        repo_type = args.get("repo_type", "model")
+        revision = args.get("revision", "main")
+        create_pr = args.get("create_pr", False)
+        commit_message = args.get("commit_message", f"Delete {', '.join(patterns)}")
+
+        await _async_call(
+            self.api.delete_files,
+            repo_id=repo_id,
+            delete_patterns=patterns,
+            repo_type=repo_type,
+            revision=revision,
+            commit_message=commit_message,
+            create_pr=create_pr,
+        )
+
+        response = f"**Deleted:** {', '.join(patterns)} from {repo_id}"
+        return {"formatted": response, "totalResults": 1, "resultsShared": 1}
+
+    def _error(self, message: str) -> ToolResult:
+        """Return an error result."""
+        return {"formatted": message, "totalResults": 0, "resultsShared": 0, "isError": True}
+
+
+# Tool specification
+HF_REPO_FILES_TOOL_SPEC = {
+    "name": "hf_repo_files",
+    "description": (
+        "Read and write files in HF repos (models/datasets/spaces).\n\n"
+        "## Operations\n"
+        "- **list**: List files with sizes and structure\n"
+        "- **read**: Read file content (text files only)\n"
+        "- **upload**: Upload content to repo (can create PR)\n"
+        "- **delete**: Delete files/folders (supports wildcards like *.tmp)\n\n"
+        "## Use when\n"
+        "- Need to see what files exist in a repo\n"
+        "- Want to read config.json, README.md, or other text files\n"
+        "- Uploading training scripts, configs, or results to a repo\n"
+        "- Cleaning up temporary files from a repo\n\n"
+        "## Examples\n"
+        '{"operation": "list", "repo_id": "meta-llama/Llama-2-7b"}\n'
+        '{"operation": "read", "repo_id": "gpt2", "path": "config.json"}\n'
+        '{"operation": "upload", "repo_id": "my-model", "path": "README.md", "content": "# My Model"}\n'
+        '{"operation": "upload", "repo_id": "org/model", "path": "fix.py", "content": "...", "create_pr": true}\n'
+        '{"operation": "delete", "repo_id": "my-model", "patterns": ["*.tmp", "logs/"]}\n\n'
+        "## Notes\n"
+        "- For binary files (safetensors, bin), use list to see them but can't read content\n"
+        "- upload/delete require approval (can overwrite/destroy data)\n"
+        "- Use create_pr=true to propose changes instead of direct commit\n"
+    ),
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "operation": {
+                "type": "string",
+                "enum": ["list", "read", "upload", "delete"],
+                "description": "Operation: list, read, upload, delete",
+            },
+            "repo_id": {
+                "type": "string",
+                "description": "Repository ID (e.g., 'username/repo-name')",
+            },
+            "repo_type": {
+                "type": "string",
+                "enum": ["model", "dataset", "space"],
+                "description": "Repository type (default: model)",
+            },
+            "revision": {
+                "type": "string",
+                "description": "Branch/tag/commit (default: main)",
+            },
+            "path": {
+                "type": "string",
+                "description": "File path for read/upload",
+            },
+            "content": {
+                "type": "string",
+                "description": "File content for upload",
+            },
+            "patterns": {
+                "type": "array",
+                "items": {"type": "string"},
+                "description": "Patterns to delete (e.g., ['*.tmp', 'logs/'])",
+            },
+            "create_pr": {
+                "type": "boolean",
+                "description": "Create PR instead of direct commit",
+            },
+            "commit_message": {
+                "type": "string",
+                "description": "Custom commit message",
+            },
+        },
+        "required": ["operation"],
+    },
+}
+
+
+async def hf_repo_files_handler(arguments: Dict[str, Any]) -> tuple[str, bool]:
+    """Handler for agent tool router."""
+    try:
+        tool = HfRepoFilesTool()
+        result = await tool.execute(arguments)
+        return result["formatted"], not result.get("isError", False)
+    except Exception as e:
+        return f"Error: {str(e)}", False

agent/tools/hf_repo_git_tool.py

@@ -0,0 +1,663 @@
+"""
+HF Repo Git Tool - Git-like operations on Hugging Face repositories
+
+Operations: branches, tags, PRs, repo management
+"""
+
+import asyncio
+from typing import Any, Dict, Literal, Optional
+
+from huggingface_hub import HfApi
+from huggingface_hub.utils import RepositoryNotFoundError
+
+from agent.tools.types import ToolResult
+
+OperationType = Literal[
+    "create_branch", "delete_branch",
+    "create_tag", "delete_tag",
+    "list_refs",
+    "create_pr", "list_prs", "get_pr", "merge_pr", "close_pr", "comment_pr", "change_pr_status",
+    "create_repo", "update_repo",
+]
+
+
+async def _async_call(func, *args, **kwargs):
+    """Wrap synchronous HfApi calls for async context."""
+    return await asyncio.to_thread(func, *args, **kwargs)
+
+
+def _build_repo_url(repo_id: str, repo_type: str = "model") -> str:
+    """Build the Hub URL for a repository."""
+    if repo_type == "model":
+        return f"https://huggingface.co/{repo_id}"
+    return f"https://huggingface.co/{repo_type}s/{repo_id}"
+
+
+class HfRepoGitTool:
+    """Tool for git-like operations on HF repos."""
+
+    def __init__(self, hf_token: Optional[str] = None):
+        self.api = HfApi(token=hf_token)
+
+    async def execute(self, args: Dict[str, Any]) -> ToolResult:
+        """Execute the specified operation."""
+        operation = args.get("operation")
+
+        if not operation:
+            return self._help()
+
+        try:
+            handlers = {
+                "create_branch": self._create_branch,
+                "delete_branch": self._delete_branch,
+                "create_tag": self._create_tag,
+                "delete_tag": self._delete_tag,
+                "list_refs": self._list_refs,
+                "create_pr": self._create_pr,
+                "list_prs": self._list_prs,
+                "get_pr": self._get_pr,
+                "merge_pr": self._merge_pr,
+                "close_pr": self._close_pr,
+                "comment_pr": self._comment_pr,
+                "change_pr_status": self._change_pr_status,
+                "create_repo": self._create_repo,
+                "update_repo": self._update_repo,
+            }
+
+            handler = handlers.get(operation)
+            if handler:
+                return await handler(args)
+            else:
+                ops = ", ".join(handlers.keys())
+                return self._error(f"Unknown operation: {operation}. Valid: {ops}")
+
+        except RepositoryNotFoundError:
+            return self._error(f"Repository not found: {args.get('repo_id')}")
+        except Exception as e:
+            return self._error(f"Error: {str(e)}")
+
+    def _help(self) -> ToolResult:
+        """Show usage instructions."""
+        return {
+            "formatted": """**hf_repo_git** - Git-like operations on HF repos
+
+**Branch/Tag:**
+- `create_branch`: `{"operation": "create_branch", "repo_id": "...", "branch": "dev"}`
+- `delete_branch`: `{"operation": "delete_branch", "repo_id": "...", "branch": "dev"}`
+- `create_tag`: `{"operation": "create_tag", "repo_id": "...", "tag": "v1.0"}`
+- `delete_tag`: `{"operation": "delete_tag", "repo_id": "...", "tag": "v1.0"}`
+- `list_refs`: `{"operation": "list_refs", "repo_id": "..."}`
+
+**PRs:**
+- `create_pr`: `{"operation": "create_pr", "repo_id": "...", "title": "..."}` (creates draft PR)
+- `list_prs`: `{"operation": "list_prs", "repo_id": "..."}` (shows status: draft/open/merged/closed)
+- `get_pr`: `{"operation": "get_pr", "repo_id": "...", "pr_num": 1}` (shows status)
+- `change_pr_status`: `{"operation": "change_pr_status", "repo_id": "...", "pr_num": 1, "new_status": "open"}` (change draft to open)
+- `merge_pr`: `{"operation": "merge_pr", "repo_id": "...", "pr_num": 1}`
+- `close_pr`: `{"operation": "close_pr", "repo_id": "...", "pr_num": 1}`
+- `comment_pr`: `{"operation": "comment_pr", "repo_id": "...", "pr_num": 1, "comment": "..."}`
+
+**Repo:**
+- `create_repo`: `{"operation": "create_repo", "repo_id": "my-model", "private": true}`
+- `update_repo`: `{"operation": "update_repo", "repo_id": "...", "private": false}`""",
+            "totalResults": 1,
+            "resultsShared": 1,
+        }
+
+    # =========================================================================
+    # BRANCH OPERATIONS
+    # =========================================================================
+
+    async def _create_branch(self, args: Dict[str, Any]) -> ToolResult:
+        """Create a new branch."""
+        repo_id = args.get("repo_id")
+        branch = args.get("branch")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not branch:
+            return self._error("branch is required")
+
+        repo_type = args.get("repo_type", "model")
+        from_rev = args.get("from_rev", "main")
+
+        await _async_call(
+            self.api.create_branch,
+            repo_id=repo_id,
+            branch=branch,
+            revision=from_rev,
+            repo_type=repo_type,
+            exist_ok=args.get("exist_ok", False),
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/tree/{branch}"
+        return {"formatted": f"**Branch created:** {branch}\n{url}", "totalResults": 1, "resultsShared": 1}
+
+    async def _delete_branch(self, args: Dict[str, Any]) -> ToolResult:
+        """Delete a branch."""
+        repo_id = args.get("repo_id")
+        branch = args.get("branch")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not branch:
+            return self._error("branch is required")
+
+        repo_type = args.get("repo_type", "model")
+
+        await _async_call(
+            self.api.delete_branch,
+            repo_id=repo_id,
+            branch=branch,
+            repo_type=repo_type,
+        )
+
+        return {"formatted": f"**Branch deleted:** {branch}", "totalResults": 1, "resultsShared": 1}
+
+    # =========================================================================
+    # TAG OPERATIONS
+    # =========================================================================
+
+    async def _create_tag(self, args: Dict[str, Any]) -> ToolResult:
+        """Create a tag."""
+        repo_id = args.get("repo_id")
+        tag = args.get("tag")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not tag:
+            return self._error("tag is required")
+
+        repo_type = args.get("repo_type", "model")
+        revision = args.get("revision", "main")
+        tag_message = args.get("tag_message", "")
+
+        await _async_call(
+            self.api.create_tag,
+            repo_id=repo_id,
+            tag=tag,
+            revision=revision,
+            tag_message=tag_message,
+            repo_type=repo_type,
+            exist_ok=args.get("exist_ok", False),
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/tree/{tag}"
+        return {"formatted": f"**Tag created:** {tag}\n{url}", "totalResults": 1, "resultsShared": 1}
+
+    async def _delete_tag(self, args: Dict[str, Any]) -> ToolResult:
+        """Delete a tag."""
+        repo_id = args.get("repo_id")
+        tag = args.get("tag")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not tag:
+            return self._error("tag is required")
+
+        repo_type = args.get("repo_type", "model")
+
+        await _async_call(
+            self.api.delete_tag,
+            repo_id=repo_id,
+            tag=tag,
+            repo_type=repo_type,
+        )
+
+        return {"formatted": f"**Tag deleted:** {tag}", "totalResults": 1, "resultsShared": 1}
+
+    # =========================================================================
+    # LIST REFS
+    # =========================================================================
+
+    async def _list_refs(self, args: Dict[str, Any]) -> ToolResult:
+        """List branches and tags."""
+        repo_id = args.get("repo_id")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+
+        repo_type = args.get("repo_type", "model")
+
+        refs = await _async_call(
+            self.api.list_repo_refs,
+            repo_id=repo_id,
+            repo_type=repo_type,
+        )
+
+        branches = [b.name for b in refs.branches] if refs.branches else []
+        tags = [t.name for t in refs.tags] if hasattr(refs, 'tags') and refs.tags else []
+
+        url = _build_repo_url(repo_id, repo_type)
+        lines = [f"**{repo_id}**", url, ""]
+
+        if branches:
+            lines.append(f"**Branches ({len(branches)}):** " + ", ".join(branches))
+        else:
+            lines.append("**Branches:** none")
+
+        if tags:
+            lines.append(f"**Tags ({len(tags)}):** " + ", ".join(tags))
+        else:
+            lines.append("**Tags:** none")
+
+        return {"formatted": "\n".join(lines), "totalResults": len(branches) + len(tags), "resultsShared": len(branches) + len(tags)}
+
+    # =========================================================================
+    # PR OPERATIONS
+    # =========================================================================
+
+    async def _create_pr(self, args: Dict[str, Any]) -> ToolResult:
+        """Create a pull request."""
+        repo_id = args.get("repo_id")
+        title = args.get("title")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not title:
+            return self._error("title is required")
+
+        repo_type = args.get("repo_type", "model")
+        description = args.get("description", "")
+
+        result = await _async_call(
+            self.api.create_pull_request,
+            repo_id=repo_id,
+            title=title,
+            description=description,
+            repo_type=repo_type,
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/discussions/{result.num}"
+        return {
+            "formatted": f"**Draft PR #{result.num} created:** {title}\n{url}\n\nAdd commits via upload with revision=\"refs/pr/{result.num}\"",
+            "totalResults": 1,
+            "resultsShared": 1,
+        }
+
+    async def _list_prs(self, args: Dict[str, Any]) -> ToolResult:
+        """List PRs and discussions."""
+        repo_id = args.get("repo_id")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+
+        repo_type = args.get("repo_type", "model")
+        status = args.get("status", "all")  # open, closed, all
+
+        discussions = list(self.api.get_repo_discussions(
+            repo_id=repo_id,
+            repo_type=repo_type,
+            discussion_status=status if status != "all" else None,
+        ))
+
+        if not discussions:
+            return {"formatted": f"No discussions in {repo_id}", "totalResults": 0, "resultsShared": 0}
+
+        url = _build_repo_url(repo_id, repo_type)
+        lines = [f"**{repo_id}** - {len(discussions)} discussions", f"{url}/discussions", ""]
+
+        for d in discussions[:20]:
+            if d.status == "draft":
+                status_label = "[DRAFT]"
+            elif d.status == "open":
+                status_label = "[OPEN]"
+            elif d.status == "merged":
+                status_label = "[MERGED]"
+            else:
+                status_label = "[CLOSED]"
+            type_label = "PR" if d.is_pull_request else "D"
+            lines.append(f"{status_label} #{d.num} [{type_label}] {d.title}")
+
+        return {"formatted": "\n".join(lines), "totalResults": len(discussions), "resultsShared": min(20, len(discussions))}
+
+    async def _get_pr(self, args: Dict[str, Any]) -> ToolResult:
+        """Get PR details."""
+        repo_id = args.get("repo_id")
+        pr_num = args.get("pr_num")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not pr_num:
+            return self._error("pr_num is required")
+
+        repo_type = args.get("repo_type", "model")
+
+        pr = await _async_call(
+            self.api.get_discussion_details,
+            repo_id=repo_id,
+            discussion_num=int(pr_num),
+            repo_type=repo_type,
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/discussions/{pr_num}"
+        status_map = {
+            "draft": "Draft",
+            "open": "Open",
+            "merged": "Merged",
+            "closed": "Closed"
+        }
+        status = status_map.get(pr.status, pr.status.capitalize())
+        type_label = "Pull Request" if pr.is_pull_request else "Discussion"
+
+        lines = [
+            f"**{type_label} #{pr_num}:** {pr.title}",
+            f"**Status:** {status}",
+            f"**Author:** {pr.author}",
+            url,
+        ]
+
+        if pr.is_pull_request:
+            if pr.status == "draft":
+                lines.append(f"\nTo add commits: upload with revision=\"refs/pr/{pr_num}\"")
+            elif pr.status == "open":
+                lines.append(f"\nTo add commits: upload with revision=\"refs/pr/{pr_num}\"")
+
+        return {"formatted": "\n".join(lines), "totalResults": 1, "resultsShared": 1}
+
+    async def _merge_pr(self, args: Dict[str, Any]) -> ToolResult:
+        """Merge a pull request."""
+        repo_id = args.get("repo_id")
+        pr_num = args.get("pr_num")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not pr_num:
+            return self._error("pr_num is required")
+
+        repo_type = args.get("repo_type", "model")
+        comment = args.get("comment", "")
+
+        await _async_call(
+            self.api.merge_pull_request,
+            repo_id=repo_id,
+            discussion_num=int(pr_num),
+            comment=comment,
+            repo_type=repo_type,
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/discussions/{pr_num}"
+        return {"formatted": f"**PR #{pr_num} merged**\n{url}", "totalResults": 1, "resultsShared": 1}
+
+    async def _close_pr(self, args: Dict[str, Any]) -> ToolResult:
+        """Close a PR/discussion."""
+        repo_id = args.get("repo_id")
+        pr_num = args.get("pr_num")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not pr_num:
+            return self._error("pr_num is required")
+
+        repo_type = args.get("repo_type", "model")
+        comment = args.get("comment", "")
+
+        await _async_call(
+            self.api.change_discussion_status,
+            repo_id=repo_id,
+            discussion_num=int(pr_num),
+            new_status="closed",
+            comment=comment,
+            repo_type=repo_type,
+        )
+
+        return {"formatted": f"**Discussion #{pr_num} closed**", "totalResults": 1, "resultsShared": 1}
+
+    async def _comment_pr(self, args: Dict[str, Any]) -> ToolResult:
+        """Add a comment to a PR/discussion."""
+        repo_id = args.get("repo_id")
+        pr_num = args.get("pr_num")
+        comment = args.get("comment")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not pr_num:
+            return self._error("pr_num is required")
+        if not comment:
+            return self._error("comment is required")
+
+        repo_type = args.get("repo_type", "model")
+
+        await _async_call(
+            self.api.comment_discussion,
+            repo_id=repo_id,
+            discussion_num=int(pr_num),
+            comment=comment,
+            repo_type=repo_type,
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/discussions/{pr_num}"
+        return {"formatted": f"**Comment added to #{pr_num}**\n{url}", "totalResults": 1, "resultsShared": 1}
+
+    async def _change_pr_status(self, args: Dict[str, Any]) -> ToolResult:
+        """Change PR/discussion status (mainly to convert draft to open)."""
+        repo_id = args.get("repo_id")
+        pr_num = args.get("pr_num")
+        new_status = args.get("new_status")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+        if not pr_num:
+            return self._error("pr_num is required")
+        if not new_status:
+            return self._error("new_status is required (open or closed)")
+
+        repo_type = args.get("repo_type", "model")
+        comment = args.get("comment", "")
+
+        await _async_call(
+            self.api.change_discussion_status,
+            repo_id=repo_id,
+            discussion_num=int(pr_num),
+            new_status=new_status,
+            comment=comment,
+            repo_type=repo_type,
+        )
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/discussions/{pr_num}"
+        return {"formatted": f"**PR #{pr_num} status changed to {new_status}**\n{url}", "totalResults": 1, "resultsShared": 1}
+
+    # =========================================================================
+    # REPO MANAGEMENT
+    # =========================================================================
+
+    async def _create_repo(self, args: Dict[str, Any]) -> ToolResult:
+        """Create a new repository."""
+        repo_id = args.get("repo_id")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+
+        repo_type = args.get("repo_type", "model")
+        private = args.get("private", True)
+        space_sdk = args.get("space_sdk")
+
+        if repo_type == "space" and not space_sdk:
+            return self._error("space_sdk required for spaces (gradio/streamlit/docker/static)")
+
+        kwargs = {
+            "repo_id": repo_id,
+            "repo_type": repo_type,
+            "private": private,
+            "exist_ok": args.get("exist_ok", False),
+        }
+        if space_sdk:
+            kwargs["space_sdk"] = space_sdk
+
+        result = await _async_call(self.api.create_repo, **kwargs)
+
+        return {
+            "formatted": f"**Repository created:** {repo_id}\n**Private:** {private}\n{result}",
+            "totalResults": 1,
+            "resultsShared": 1,
+        }
+
+    async def _update_repo(self, args: Dict[str, Any]) -> ToolResult:
+        """Update repository settings."""
+        repo_id = args.get("repo_id")
+
+        if not repo_id:
+            return self._error("repo_id is required")
+
+        repo_type = args.get("repo_type", "model")
+        private = args.get("private")
+        gated = args.get("gated")
+
+        if private is None and gated is None:
+            return self._error("Specify private (bool) or gated ('auto'/'manual'/false)")
+
+        kwargs = {"repo_id": repo_id, "repo_type": repo_type}
+        if private is not None:
+            kwargs["private"] = private
+        if gated is not None:
+            kwargs["gated"] = gated
+
+        await _async_call(self.api.update_repo_settings, **kwargs)
+
+        changes = []
+        if private is not None:
+            changes.append(f"private={private}")
+        if gated is not None:
+            changes.append(f"gated={gated}")
+
+        url = f"{_build_repo_url(repo_id, repo_type)}/settings"
+        return {"formatted": f"**Settings updated:** {', '.join(changes)}\n{url}", "totalResults": 1, "resultsShared": 1}
+
+    def _error(self, message: str) -> ToolResult:
+        """Return an error result."""
+        return {"formatted": message, "totalResults": 0, "resultsShared": 0, "isError": True}
+
+
+# Tool specification
+HF_REPO_GIT_TOOL_SPEC = {
+    "name": "hf_repo_git",
+    "description": (
+        "Git-like operations on HF repos: branches, tags, PRs, and repo management.\n\n"
+        "## Operations\n"
+        "**Branches:** create_branch, delete_branch, list_refs\n"
+        "**Tags:** create_tag, delete_tag\n"
+        "**PRs:** create_pr, list_prs, get_pr, merge_pr, close_pr, comment_pr, change_pr_status\n"
+        "**Repo:** create_repo, update_repo\n\n"
+        "## Use when\n"
+        "- Creating feature branches for experiments\n"
+        "- Tagging model versions (v1.0, v2.0)\n"
+        "- Opening PRs to contribute to repos you don't own\n"
+        "- Reviewing and merging PRs on your repos\n"
+        "- Creating new model/dataset/space repos\n"
+        "- Changing repo visibility (public/private) or gated access\n\n"
+        "## Examples\n"
+        '{"operation": "list_refs", "repo_id": "my-model"}\n'
+        '{"operation": "create_branch", "repo_id": "my-model", "branch": "experiment-v2"}\n'
+        '{"operation": "create_tag", "repo_id": "my-model", "tag": "v1.0", "revision": "main"}\n'
+        '{"operation": "create_pr", "repo_id": "org/model", "title": "Fix tokenizer config"}\n'
+        '{"operation": "change_pr_status", "repo_id": "my-model", "pr_num": 1, "new_status": "open"}\n'
+        '{"operation": "merge_pr", "repo_id": "my-model", "pr_num": 3}\n'
+        '{"operation": "create_repo", "repo_id": "my-new-model", "private": true}\n'
+        '{"operation": "update_repo", "repo_id": "my-model", "gated": "auto"}\n\n'
+        "## PR Workflow\n"
+        "1. create_pr → creates draft PR (empty by default)\n"
+        "2. Upload files with revision='refs/pr/N' to add commits\n"
+        "3. change_pr_status with new_status='open' to publish (convert draft to open)\n"
+        "4. merge_pr when ready\n\n"
+        "## Notes\n"
+        "- PR status: draft (default), open, merged, closed\n"
+        "- delete_branch, delete_tag, merge_pr, create_repo, update_repo require approval\n"
+        "- For spaces, create_repo needs space_sdk (gradio/streamlit/docker/static)\n"
+        "- gated options: 'auto' (instant), 'manual' (review), false (open)\n"
+    ),
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "operation": {
+                "type": "string",
+                "enum": [
+                    "create_branch", "delete_branch",
+                    "create_tag", "delete_tag", "list_refs",
+                    "create_pr", "list_prs", "get_pr", "merge_pr", "close_pr", "comment_pr", "change_pr_status",
+                    "create_repo", "update_repo",
+                ],
+                "description": "Operation to execute",
+            },
+            "repo_id": {
+                "type": "string",
+                "description": "Repository ID (e.g., 'username/repo-name')",
+            },
+            "repo_type": {
+                "type": "string",
+                "enum": ["model", "dataset", "space"],
+                "description": "Repository type (default: model)",
+            },
+            "branch": {
+                "type": "string",
+                "description": "Branch name (create_branch, delete_branch)",
+            },
+            "from_rev": {
+                "type": "string",
+                "description": "Create branch from this revision (default: main)",
+            },
+            "tag": {
+                "type": "string",
+                "description": "Tag name (create_tag, delete_tag)",
+            },
+            "revision": {
+                "type": "string",
+                "description": "Revision for tag (default: main)",
+            },
+            "tag_message": {
+                "type": "string",
+                "description": "Tag description",
+            },
+            "title": {
+                "type": "string",
+                "description": "PR title (create_pr)",
+            },
+            "description": {
+                "type": "string",
+                "description": "PR description (create_pr)",
+            },
+            "pr_num": {
+                "type": "integer",
+                "description": "PR/discussion number",
+            },
+            "comment": {
+                "type": "string",
+                "description": "Comment text",
+            },
+            "status": {
+                "type": "string",
+                "enum": ["open", "closed", "all"],
+                "description": "Filter PRs by status (list_prs)",
+            },
+            "new_status": {
+                "type": "string",
+                "enum": ["open", "closed"],
+                "description": "New status for PR/discussion (change_pr_status)",
+            },
+            "private": {
+                "type": "boolean",
+                "description": "Make repo private (create_repo, update_repo)",
+            },
+            "gated": {
+                "type": "string",
+                "enum": ["auto", "manual", "false"],
+                "description": "Gated access setting (update_repo)",
+            },
+            "space_sdk": {
+                "type": "string",
+                "enum": ["gradio", "streamlit", "docker", "static"],
+                "description": "Space SDK (required for create_repo with space)",
+            },
+        },
+        "required": ["operation"],
+    },
+}
+
+
+async def hf_repo_git_handler(arguments: Dict[str, Any]) -> tuple[str, bool]:
+    """Handler for agent tool router."""
+    try:
+        tool = HfRepoGitTool()
+        result = await tool.execute(arguments)
+        return result["formatted"], not result.get("isError", False)
+    except Exception as e:
+        return f"Error: {str(e)}", False

agent/tools/utils_tools.py

@@ -1,203 +0,0 @@
-"""
-Utils Tools - General utility operations
-
-Provides system information like current date/time with timezone support.
-"""
-
-import zoneinfo
-from datetime import datetime
-from typing import Any, Dict, Literal
-
-from agent.tools.types import ToolResult
-
-# Operation names
-OperationType = Literal["get_datetime"]
-
-
-class UtilsTool:
-    """Tool for general utility operations."""
-
-    async def execute(self, params: Dict[str, Any]) -> ToolResult:
-        """Execute the specified utility operation."""
-        operation = params.get("operation")
-        args = params.get("args", {})
-
-        # If no operation provided, return usage instructions
-        if not operation:
-            return self._show_help()
-
-        # Normalize operation name
-        operation = operation.lower()
-
-        # Check if help is requested
-        if args.get("help"):
-            return self._show_operation_help(operation)
-
-        try:
-            # Route to appropriate handler
-            if operation == "get_datetime":
-                return await self._get_datetime(args)
-            else:
-                return {
-                    "formatted": f'Unknown operation: "{operation}"\n\n'
-                    "Available operations: get_datetime\n\n"
-                    "Call this tool with no operation for full usage instructions.",
-                    "totalResults": 0,
-                    "resultsShared": 0,
-                    "isError": True,
-                }
-
-        except Exception as e:
-            return {
-                "formatted": f"Error executing {operation}: {str(e)}",
-                "totalResults": 0,
-                "resultsShared": 0,
-                "isError": True,
-            }
-
-    def _show_help(self) -> ToolResult:
-        """Show usage instructions when tool is called with no arguments."""
-        usage_text = """# Utils Tool
-
-Utility operations for system information.
-
-## Available Commands
-
-- **get_datetime** - Get current date and time with timezone support
-
-## Examples
-
-### Get current date and time (Paris timezone by default)
-Call this tool with:
-```json
-{
-  "operation": "get_datetime",
-  "args": {}
-}
-```
-
-### Get current date and time in a specific timezone
-Call this tool with:
-```json
-{
-  "operation": "get_datetime",
-  "args": {
-    "timezone": "America/New_York"
-  }
-}
-```
-
-Common timezones: Europe/Paris, America/New_York, America/Los_Angeles, Asia/Tokyo, UTC
-
-## Tips
-
-- **Default timezone**: Paris (Europe/Paris)
-- **Date format**: dd-mm-yyyy
-- **Time format**: HH:MM:SS.mmm (24-hour format with milliseconds)
-- **Timezone names**: Use IANA timezone database names (e.g., "Europe/Paris", "UTC")
-"""
-        return {"formatted": usage_text, "totalResults": 1, "resultsShared": 1}
-
-    def _show_operation_help(self, operation: str) -> ToolResult:
-        """Show help for a specific operation."""
-        help_text = f"Help for operation: {operation}\n\nCall with appropriate arguments. Use the main help for examples."
-        return {"formatted": help_text, "totalResults": 1, "resultsShared": 1}
-
-    async def _get_datetime(self, args: Dict[str, Any]) -> ToolResult:
-        """Get current date and time with timezone support."""
-        timezone_name = args.get("timezone", "Europe/Paris")
-
-        try:
-            # Get timezone object
-            tz = zoneinfo.ZoneInfo(timezone_name)
-
-            # Get current datetime in specified timezone
-            now = datetime.now(tz)
-
-            # Format date as dd-mm-yyyy
-            date_str = now.strftime("%d-%m-%Y")
-
-            # Format time as HH:MM:SS.mmm
-            time_str = now.strftime("%H:%M:%S.%f")[
-                :-3
-            ]  # Remove last 3 digits to keep only milliseconds
-
-            # Get timezone abbreviation/offset
-            tz_offset = now.strftime("%z")
-            tz_name = now.strftime("%Z")
-
-            response = f"""✓ Current date and time
-
-**Date:** {date_str}
-**Time:** {time_str}
-**Timezone:** {timezone_name} ({tz_name}, UTC{tz_offset[:3]}:{tz_offset[3:]})
-
-**ISO Format:** {now.isoformat()}
-**Unix Timestamp:** {int(now.timestamp())}"""
-
-            return {"formatted": response, "totalResults": 1, "resultsShared": 1}
-
-        except zoneinfo.ZoneInfoNotFoundError:
-            return {
-                "formatted": f"Invalid timezone: {timezone_name}\n\n"
-                "Use IANA timezone database names like:\n"
-                "- Europe/Paris\n"
-                "- America/New_York\n"
-                "- Asia/Tokyo\n"
-                "- UTC\n\n"
-                "See: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones",
-                "totalResults": 0,
-                "resultsShared": 0,
-                "isError": True,
-            }
-        except Exception as e:
-            return {
-                "formatted": f"Failed to get date/time: {str(e)}",
-                "totalResults": 0,
-                "resultsShared": 0,
-                "isError": True,
-            }
-
-
-# Tool specification for agent registration
-UTILS_TOOL_SPEC = {
-    "name": "utils",
-    "description": (
-        "System utility operations - currently provides date/time with timezone support. "
-        "**Use when:** (1) Need current date for logging/timestamps, (2) User asks 'what time is it', "
-        "(3) Need timezone-aware datetime for scheduling/coordination, (4) Creating timestamped filenames. "
-        "**Operation:** get_datetime with optional timezone parameter (default: Europe/Paris). "
-        "Returns: Date (dd-mm-yyyy), time (HH:MM:SS.mmm), timezone info, ISO format, Unix timestamp. "
-        "**Pattern:** utils get_datetime → use timestamp in filename/log → upload to hf_private_repos. "
-        "Supports IANA timezone names: 'Europe/Paris', 'America/New_York', 'Asia/Tokyo', 'UTC'."
-    ),
-    "parameters": {
-        "type": "object",
-        "properties": {
-            "operation": {
-                "type": "string",
-                "enum": ["get_datetime"],
-                "description": "Operation to execute. Valid values: [get_datetime]",
-            },
-            "args": {
-                "type": "object",
-                "description": (
-                    "Operation-specific arguments as a JSON object. "
-                    "For get_datetime: timezone (string, optional, default: Europe/Paris). "
-                    "Use IANA timezone names like 'America/New_York', 'Asia/Tokyo', 'UTC'."
-                ),
-                "additionalProperties": True,
-            },
-        },
-    },
-}
-
-
-async def utils_handler(arguments: Dict[str, Any]) -> tuple[str, bool]:
-    """Handler for agent tool router."""
-    try:
-        tool = UtilsTool()
-        result = await tool.execute(arguments)
-        return result["formatted"], not result.get("isError", False)
-    except Exception as e:
-        return f"Error executing Utils tool: {str(e)}", False

agent/utils/__init__.py

@@ -1,7 +1,3 @@
 """
 Utility functions and helpers
 """
-
-from agent.utils.logging import setup_logger
-
-__all__ = ["setup_logger"]

agent/utils/logging.py

@@ -1,40 +0,0 @@
-"""
-Logging utilities
-"""
-
-import logging
-import sys
-from pathlib import Path
-from typing import Optional
-
-
-def setup_logger(
-    name: str = "hf_agent", level: int = logging.INFO, log_file: Optional[Path] = None
-) -> logging.Logger:
-    """Setup and configure logger"""
-
-    logger = logging.getLogger(name)
-    logger.setLevel(level)
-
-    # Remove existing handlers
-    logger.handlers = []
-
-    # Console handler
-    console_handler = logging.StreamHandler(sys.stdout)
-    console_handler.setLevel(level)
-    console_format = logging.Formatter(
-        "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S",
-    )
-    console_handler.setFormatter(console_format)
-    logger.addHandler(console_handler)
-
-    # File handler if log_file specified
-    if log_file:
-        log_file.parent.mkdir(parents=True, exist_ok=True)
-        file_handler = logging.FileHandler(log_file)
-        file_handler.setLevel(level)
-        file_handler.setFormatter(console_format)
-        logger.addHandler(file_handler)
-
-    return logger

agent/utils/terminal_display.py

@@ -94,13 +94,18 @@ def format_tool_call(tool_name: str, arguments: str) -> str:
 
 def format_tool_output(output: str, success: bool, truncate: bool = True) -> str:
     """Format tool output with color and optional truncation"""
+    original_length = len(output)
     if truncate:
         output = truncate_to_lines(output, max_lines=6)
 
     if success:
-        return f"{Colors.YELLOW}Tool output:{Colors.RESET}\n{output}"
+        return (
+            f"{Colors.YELLOW}Tool output ({original_length} tkns): {Colors.RESET}\n{output}"
+        )
     else:
-        return f"{Colors.RED}Tool output:{Colors.RESET}\n{output}"
+        return (
+            f"{Colors.RED}Tool output ({original_length} tokens): {Colors.RESET}\n{output}"
+        )
 
 
 def format_turn_complete() -> str:

pyproject.toml

@@ -24,6 +24,7 @@ agent = [
     "nbconvert>=7.16.6",
     "nbformat>=5.10.4",
     "datasets>=4.3.0",  # For session logging to HF datasets
+    "whoosh>=2.7.4",
 ]
 
 # Evaluation/benchmarking dependencies

test_dataset_tools.py

@@ -1,79 +0,0 @@
-"""
-Test script for unified dataset inspection tool
-"""
-
-import asyncio
-import sys
-from typing import TypedDict
-from unittest.mock import MagicMock
-
-
-# Mock the types module before importing dataset_tools
-class ToolResult(TypedDict, total=False):
-    formatted: str
-    totalResults: int
-    resultsShared: int
-    isError: bool
-
-
-mock_types = MagicMock()
-mock_types.ToolResult = ToolResult
-sys.modules["agent.tools.types"] = mock_types
-
-# Now import directly from the file
-sys.path.insert(0, "/Users/akseljoonas/Documents/hf-agent/agent/tools")
-from dataset_tools import hf_inspect_dataset_handler, inspect_dataset
-
-
-async def test_inspect_dataset():
-    """Test the unified inspect_dataset function"""
-    print("=" * 70)
-    print("Testing inspect_dataset()")
-    print("=" * 70)
-
-    # Test with akseljoonas/hf-agent-sessions as specified
-    print("\n→ inspect_dataset('akseljoonas/hf-agent-sessions'):")
-    result = await inspect_dataset("akseljoonas/hf-agent-sessions")
-    print(f"   isError: {result['isError']}")
-    print(f"   Output:\n{result['formatted']}")
-
-    print("\n" + "=" * 70)
-
-    # # Test with stanfordnlp/imdb
-    # print("\n→ inspect_dataset('stanfordnlp/imdb'):")
-    # result = await inspect_dataset("stanfordnlp/imdb")
-    # print(f"   isError: {result['isError']}")
-    # print(f"   Output:\n{result['formatted']}")
-
-    # print("\n" + "=" * 70)
-
-    # # Test with multi-config dataset
-    # print("\n→ inspect_dataset('nyu-mll/glue', config='mrpc'):")
-    # result = await inspect_dataset("nyu-mll/glue", config="mrpc")
-    # print(f"   isError: {result['isError']}")
-    # print(f"   Output:\n{result['formatted']}")
-
-
-async def test_handler():
-    """Test the handler (what the agent calls)"""
-    print("\n" + "=" * 70)
-    print("Testing hf_inspect_dataset_handler()")
-    print("=" * 70)
-
-    result, success = await hf_inspect_dataset_handler(
-        {
-            "dataset": "stanfordnlp/imdb",
-            "sample_rows": 2,
-        }
-    )
-    print("\n→ Handler result:")
-    print(f"   success: {success}")
-    print(f"   output:\n{result}")
-
-
-if __name__ == "__main__":
-    print("\nUnified Dataset Inspection Tool Test\n")
-    asyncio.run(test_inspect_dataset())
-    # asyncio.run(test_handler())
-    print("\n" + "=" * 70)
-    print("Done!")