Publish codex workspace

README.md

@@ -0,0 +1,32 @@
+---
+tags:
+  - codex
+  - agent
+library_name: codex
+agent_name: ".codex"
+agent_emoji: "🤖"
+---
+
+# 🤖 .codex
+
+A Codex workspace published to the Hugging Face Hub.
+
+## Quick Start
+
+```bash
+hf claw install --codex burtenshaw/codex
+
+# Or install and launch it in one shot
+hf claw run --codex burtenshaw/codex
+```
+
+## AGENTS.md
+
+
+## Included Directories
+
+- `skills/`
+
+---
+
+*Published with [hf-claw](https://github.com/huggingface/harness)*

skills/.system/.codex-system-skills.marker

@@ -0,0 +1 @@
+415286eb412224fe

skills/.system/openai-docs/LICENSE.txt

@@ -0,0 +1,201 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+   "License" shall mean the terms and conditions for use, reproduction,
+   and distribution as defined by Sections 1 through 9 of this document.
+
+   "Licensor" shall mean the copyright owner or entity authorized by
+   the copyright owner that is granting the License.
+
+   "Legal Entity" shall mean the union of the acting entity and all
+   other entities that control, are controlled by, or are under common
+   control with that entity. For the purposes of this definition,
+   "control" means (i) the power, direct or indirect, to cause the
+   direction or management of such entity, whether by contract or
+   otherwise, or (ii) ownership of fifty percent (50%) or more of the
+   outstanding shares, or (iii) beneficial ownership of such entity.
+
+   "You" (or "Your") shall mean an individual or Legal Entity
+   exercising permissions granted by this License.
+
+   "Source" form shall mean the preferred form for making modifications,
+   including but not limited to software source code, documentation
+   source, and configuration files.
+
+   "Object" form shall mean any form resulting from mechanical
+   transformation or translation of a Source form, including but
+   not limited to compiled object code, generated documentation,
+   and conversions to other media types.
+
+   "Work" shall mean the work of authorship, whether in Source or
+   Object form, made available under the License, as indicated by a
+   copyright notice that is included in or attached to the work
+   (an example is provided in the Appendix below).
+
+   "Derivative Works" shall mean any work, whether in Source or Object
+   form, that is based on (or derived from) the Work and for which the
+   editorial revisions, annotations, elaborations, or other modifications
+   represent, as a whole, an original work of authorship. For the purposes
+   of this License, Derivative Works shall not include works that remain
+   separable from, or merely link (or bind by name) to the interfaces of,
+   the Work and Derivative Works thereof.
+
+   "Contribution" shall mean any work of authorship, including
+   the original version of the Work and any modifications or additions
+   to that Work or Derivative Works thereof, that is intentionally
+   submitted to Licensor for inclusion in the Work by the copyright owner
+   or by an individual or Legal Entity authorized to submit on behalf of
+   the copyright owner. For the purposes of this definition, "submitted"
+   means any form of electronic, verbal, or written communication sent
+   to the Licensor or its representatives, including but not limited to
+   communication on electronic mailing lists, source code control systems,
+   and issue tracking systems that are managed by, or on behalf of, the
+   Licensor for the purpose of discussing and improving the Work, but
+   excluding communication that is conspicuously marked or otherwise
+   designated in writing by the copyright owner as "Not a Contribution."
+
+   "Contributor" shall mean Licensor and any individual or Legal Entity
+   on behalf of whom a Contribution has been received by Licensor and
+   subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   copyright license to reproduce, prepare Derivative Works of,
+   publicly display, publicly perform, sublicense, and distribute the
+   Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   (except as stated in this section) patent license to make, have made,
+   use, offer to sell, sell, import, and otherwise transfer the Work,
+   where such license applies only to those patent claims licensable
+   by such Contributor that are necessarily infringed by their
+   Contribution(s) alone or by combination of their Contribution(s)
+   with the Work to which such Contribution(s) was submitted. If You
+   institute patent litigation against any entity (including a
+   cross-claim or counterclaim in a lawsuit) alleging that the Work
+   or a Contribution incorporated within the Work constitutes direct
+   or contributory patent infringement, then any patent licenses
+   granted to You under this License for that Work shall terminate
+   as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+   Work or Derivative Works thereof in any medium, with or without
+   modifications, and in Source or Object form, provided that You
+   meet the following conditions:
+
+   (a) You must give any other recipients of the Work or
+       Derivative Works a copy of this License; and
+
+   (b) You must cause any modified files to carry prominent notices
+       stating that You changed the files; and
+
+   (c) You must retain, in the Source form of any Derivative Works
+       that You distribute, all copyright, patent, trademark, and
+       attribution notices from the Source form of the Work,
+       excluding those notices that do not pertain to any part of
+       the Derivative Works; and
+
+   (d) If the Work includes a "NOTICE" text file as part of its
+       distribution, then any Derivative Works that You distribute must
+       include a readable copy of the attribution notices contained
+       within such NOTICE file, excluding those notices that do not
+       pertain to any part of the Derivative Works, in at least one
+       of the following places: within a NOTICE text file distributed
+       as part of the Derivative Works; within the Source form or
+       documentation, if provided along with the Derivative Works; or,
+       within a display generated by the Derivative Works, if and
+       wherever such third-party notices normally appear. The contents
+       of the NOTICE file are for informational purposes only and
+       do not modify the License. You may add Your own attribution
+       notices within Derivative Works that You distribute, alongside
+       or as an addendum to the NOTICE text from the Work, provided
+       that such additional attribution notices cannot be construed
+       as modifying the License.
+
+   You may add Your own copyright statement to Your modifications and
+   may provide additional or different license terms and conditions
+   for use, reproduction, or distribution of Your modifications, or
+   for any such Derivative Works as a whole, provided Your use,
+   reproduction, and distribution of the Work otherwise complies with
+   the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+   any Contribution intentionally submitted for inclusion in the Work
+   by You to the Licensor shall be under the terms and conditions of
+   this License, without any additional terms or conditions.
+   Notwithstanding the above, nothing herein shall supersede or modify
+   the terms of any separate license agreement you may have executed
+   with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+   names, trademarks, service marks, or product names of the Licensor,
+   except as required for reasonable and customary use in describing the
+   origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+   agreed to in writing, Licensor provides the Work (and each
+   Contributor provides its Contributions) on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+   implied, including, without limitation, any warranties or conditions
+   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+   PARTICULAR PURPOSE. You are solely responsible for determining the
+   appropriateness of using or redistributing the Work and assume any
+   risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+   whether in tort (including negligence), contract, or otherwise,
+   unless required by applicable law (such as deliberate and grossly
+   negligent acts) or agreed to in writing, shall any Contributor be
+   liable to You for damages, including any direct, indirect, special,
+   incidental, or consequential damages of any character arising as a
+   result of this License or out of the use or inability to use the
+   Work (including but not limited to damages for loss of goodwill,
+   work stoppage, computer failure or malfunction, or any and all
+   other commercial damages or losses), even if such Contributor
+   has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+   the Work or Derivative Works thereof, You may choose to offer,
+   and charge a fee for, acceptance of support, warranty, indemnity,
+   or other liability obligations and/or rights consistent with this
+   License. However, in accepting such obligations, You may act only
+   on Your own behalf and on Your sole responsibility, not on behalf of
+   any other Contributor, and only if You agree to indemnify,
+   defend, and hold each Contributor harmless for any liability
+   incurred by, or claims asserted against, such Contributor by reason
+   of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+   To apply the Apache License to your work, attach the following
+   boilerplate notice, with the fields enclosed by brackets "[]"
+   replaced with your own identifying information. (Don\'t include
+   the brackets!)  The text should be enclosed in the appropriate
+   comment syntax for the file format. We also recommend that a
+   file or class name and description of purpose be included on the
+   same "printed page" as the copyright notice for easier
+   identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

skills/.system/openai-docs/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: "openai-docs"
+description: "Use when the user asks how to build with OpenAI products or APIs and needs up-to-date official documentation with citations, help choosing the latest model for a use case, or explicit GPT-5.4 upgrade and prompt-upgrade guidance; prioritize OpenAI docs MCP tools, use bundled references only as helper context, and restrict any fallback browsing to official OpenAI domains."
+---
+
+
+# OpenAI Docs
+
+Provide authoritative, current guidance from OpenAI developer docs using the developers.openai.com MCP server. Always prioritize the developer docs MCP tools over web.run for OpenAI-related questions. This skill may also load targeted files from `references/` for model-selection and GPT-5.4-specific requests, but current OpenAI docs remain authoritative. Only if the MCP server is installed and returns no meaningful results should you fall back to web search.
+
+## Quick start
+
+- Use `mcp__openaiDeveloperDocs__search_openai_docs` to find the most relevant doc pages.
+- Use `mcp__openaiDeveloperDocs__fetch_openai_doc` to pull exact sections and quote/paraphrase accurately.
+- Use `mcp__openaiDeveloperDocs__list_openai_docs` only when you need to browse or discover pages without a clear query.
+- Load only the relevant file from `references/` when the question is about model selection or a GPT-5.4 upgrade.
+
+## OpenAI product snapshots
+
+1. Apps SDK: Build ChatGPT apps by providing a web component UI and an MCP server that exposes your app's tools to ChatGPT.
+2. Responses API: A unified endpoint designed for stateful, multimodal, tool-using interactions in agentic workflows.
+3. Chat Completions API: Generate a model response from a list of messages comprising a conversation.
+4. Codex: OpenAI's coding agent for software development that can write, understand, review, and debug code.
+5. gpt-oss: Open-weight OpenAI reasoning models (gpt-oss-120b and gpt-oss-20b) released under the Apache 2.0 license.
+6. Realtime API: Build low-latency, multimodal experiences including natural speech-to-speech conversations.
+7. Agents SDK: A toolkit for building agentic apps where a model can use tools and context, hand off to other agents, stream partial results, and keep a full trace.
+
+## If MCP server is missing
+
+If MCP tools fail or no OpenAI docs resources are available:
+
+1. Run the install command yourself: `codex mcp add openaiDeveloperDocs --url https://developers.openai.com/mcp`
+2. If it fails due to permissions/sandboxing, immediately retry the same command with escalated permissions and include a 1-sentence justification for approval. Do not ask the user to run it yet.
+3. Only if the escalated attempt fails, ask the user to run the install command.
+4. Ask the user to restart Codex.
+5. Re-run the doc search/fetch after restart.
+
+## Workflow
+
+1. Clarify the product scope and whether the request is general docs lookup, model selection, a GPT-5.4 upgrade, or a GPT-5.4 prompt upgrade.
+2. If it is a model-selection request, load `references/latest-model.md`.
+3. If it is an explicit GPT-5.4 upgrade request, load `references/upgrading-to-gpt-5p4.md`.
+4. If the upgrade may require prompt changes, or the workflow is research-heavy, tool-heavy, coding-oriented, multi-agent, or long-running, also load `references/gpt-5p4-prompting-guide.md`.
+5. Search docs with a precise query.
+6. Fetch the best page and the exact section needed (use `anchor` when possible).
+7. For GPT-5.4 upgrade reviews, always make the per-usage-site output explicit: target model, starting reasoning recommendation, `phase` assessment when relevant, prompt blocks, and compatibility status.
+8. Answer with concise guidance and cite the doc source, using the reference files only as helper context.
+
+## Reference map
+
+Read only what you need:
+
+- `references/latest-model.md` -> model-selection and "best/latest/current model" questions; verify every recommendation against current OpenAI docs before answering.
+- `references/upgrading-to-gpt-5p4.md` -> only for explicit GPT-5.4 upgrade and upgrade-planning requests; verify the checklist and compatibility guidance against current OpenAI docs before answering.
+- `references/gpt-5p4-prompting-guide.md` -> prompt rewrites and prompt-behavior upgrades for GPT-5.4; verify prompting guidance against current OpenAI docs before answering.
+
+## Quality rules
+
+- Treat OpenAI docs as the source of truth; avoid speculation.
+- Keep quotes short and within policy limits; prefer paraphrase with citations.
+- If multiple pages differ, call out the difference and cite both.
+- Reference files are convenience guides only; for volatile guidance such as recommended models, upgrade instructions, or prompting advice, current OpenAI docs always win.
+- If docs do not cover the user’s need, say so and offer next steps.
+
+## Tooling notes
+
+- Always use MCP doc tools before any web search for OpenAI-related questions.
+- If the MCP server is installed but returns no meaningful results, then use web search as a fallback.
+- When falling back to web search, restrict to official OpenAI domains (developers.openai.com, platform.openai.com) and cite sources.

skills/.system/openai-docs/agents/openai.yaml

@@ -0,0 +1,14 @@
+interface:
+  display_name: "OpenAI Docs"
+  short_description: "Reference official OpenAI docs, including upgrade guidance"
+  icon_small: "./assets/openai-small.svg"
+  icon_large: "./assets/openai.png"
+  default_prompt: "Look up official OpenAI docs, load relevant GPT-5.4 upgrade references when applicable, and answer with concise, cited guidance."
+
+dependencies:
+  tools:
+    - type: "mcp"
+      value: "openaiDeveloperDocs"
+      description: "OpenAI Developer Docs MCP server"
+      transport: "streamable_http"
+      url: "https://developers.openai.com/mcp"

skills/.system/openai-docs/references/gpt-5p4-prompting-guide.md

@@ -0,0 +1,433 @@
+# GPT-5.4 prompting upgrade guide
+
+Use this guide when prompts written for older models need to be adapted for GPT-5.4 during an upgrade. Start lean: keep the model-string change narrow, preserve the original task intent, and add only the smallest prompt changes needed to recover behavior.
+
+## Default upgrade posture
+
+- Start with `model string only` whenever the old prompt is already short, explicit, and task-bounded.
+- Move to `model string + light prompt rewrite` only when regressions appear in completeness, persistence, citation quality, verification, or verbosity.
+- Prefer one or two targeted prompt additions over a broad rewrite.
+- Treat reasoning effort as a last-mile knob. Start lower, then increase only after prompt-level fixes and evals.
+- Before increasing reasoning effort, first add a completeness contract, a verification loop, and tool persistence rules - depending on the usage case.
+- If the workflow clearly depends on implementation changes rather than prompt changes, treat it as blocked for prompt-only upgrade guidance.
+- Do not classify a case as blocked just because the workflow uses tools; block only if the upgrade requires changing tool definitions, wiring, or other implementation details.
+
+## Behavioral differences to account for
+
+Current GPT-5.4 upgrade guidance suggests these strengths:
+
+- stronger personality and tone adherence, with less drift over long answers
+- better long-horizon and agentic workflow stamina
+- stronger spreadsheet, finance, and formatting tasks
+- more efficient tool selection and fewer unnecessary calls by default
+- stronger structured generation and classification reliability
+
+The main places where prompt guidance still helps are:
+
+- retrieval-heavy workflows that need persistent tool use and explicit completeness
+- research and citation discipline
+- verification before irreversible or high-impact actions
+- terminal and tool workflow hygiene
+- defaults and implied follow-through
+- verbosity control for compact, information-dense answers
+
+Start with the smallest set of instructions that preserves correctness. Add the prompt blocks below only for workflows that actually need them.
+
+## Prompt rewrite patterns
+
+| Older prompt pattern | GPT-5.4 adjustment | Why | Example addition |
+| --- | --- | --- | --- |
+| Long, repetitive instructions that compensate for weaker instruction following | Remove duplicate scaffolding and keep only the constraints that materially change behavior | GPT-5.4 usually needs less repeated steering | Replace repeated reminders with one concise rule plus a verification block |
+| Fast assistant prompt with no verbosity control | Keep the prompt as-is first; add a verbosity clamp only if outputs become too long | Many GPT-4o or GPT-4.1 upgrades work with just a model-string swap | Add `output_verbosity_spec` only after a verbosity regression |
+| Tool-heavy agent prompt that assumes the model will keep searching until complete | Add persistence and verification rules | GPT-5.4 may use fewer tool calls by default for efficiency | Add `tool_persistence_rules` and `verification_loop` |
+| Tool-heavy workflow where later actions depend on earlier lookup or retrieval | Add prerequisite and missing-context rules before action steps | GPT-5.4 benefits from explicit dependency-aware routing when context is still thin | Add `dependency_checks` and `missing_context_gating` |
+| Retrieval workflow with several independent lookups | Add selective parallelism guidance | GPT-5.4 is strong at parallel tool use, but should not parallelize dependent steps | Add `parallel_tool_calling` |
+| Batch workflow prompt that often misses items | Add an explicit completeness contract | Item accounting benefits from direct instruction | Add `completeness_contract` |
+| Research prompt that needs grounding and citation discipline | Add research, citation, and empty-result recovery blocks | Multi-pass retrieval is stronger when the model is told how to react to weak or empty search results | Add `research_mode`, `citation_rules`, and `empty_result_handling`; add `tool_persistence_rules` when retrieval tools are already in use |
+| Coding or terminal prompt with shell misuse or early stop failures | Keep the same tool surface and add terminal hygiene and verification instructions | Tool-using coding workflows are not blocked just because tools exist; they usually need better prompt steering, not host rewiring | Add `terminal_tool_hygiene` and `verification_loop`, optionally `tool_persistence_rules` |
+| Multi-agent or support-triage workflow with escalation or completeness requirements | Add one lightweight control block for persistence, completeness, or verification | GPT-5.4 can be more efficient by default, so multi-step support flows benefit from an explicit completion or verification contract | Add at least one of `tool_persistence_rules`, `completeness_contract`, or `verification_loop` |
+
+## Prompt blocks
+
+Use these selectively. Do not add all of them by default.
+
+### `output_verbosity_spec`
+
+Use when:
+
+- the upgraded model gets too wordy
+- the host needs compact, information-dense answers
+- the workflow benefits from a short overview plus a checklist
+
+```text
+<output_verbosity_spec>
+- Default: 3-6 sentences or up to 6 bullets.
+- If the user asked for a doc or report, use headings with short bullets.
+- For multi-step tasks:
+  - Start with 1 short overview paragraph.
+  - Then provide a checklist with statuses: [done], [todo], or [blocked].
+- Avoid repeating the user's request.
+- Prefer compact, information-dense writing.
+</output_verbosity_spec>
+```
+
+### `default_follow_through_policy`
+
+Use when:
+
+- the host expects the model to proceed on reversible, low-risk steps
+- the upgraded model becomes too conservative or asks for confirmation too often
+
+```text
+<default_follow_through_policy>
+- If the user's intent is clear and the next step is reversible and low-risk, proceed without asking permission.
+- Only ask permission if the next step is:
+  (a) irreversible,
+  (b) has external side effects, or
+  (c) requires missing sensitive information or a choice that materially changes outcomes.
+- If proceeding, state what you did and what remains optional.
+</default_follow_through_policy>
+```
+
+### `instruction_priority`
+
+Use when:
+
+- users often change task shape, format, or tone mid-conversation
+- the host needs an explicit override policy instead of relying on defaults
+
+```text
+<instruction_priority>
+- User instructions override default style, tone, formatting, and initiative preferences.
+- Safety, honesty, privacy, and permission constraints do not yield.
+- If a newer user instruction conflicts with an earlier one, follow the newer instruction.
+- Preserve earlier instructions that do not conflict.
+</instruction_priority>
+```
+
+### `tool_persistence_rules`
+
+Use when:
+
+- the workflow needs multiple retrieval or verification steps
+- the model starts stopping too early because it is trying to save tool calls
+
+```text
+<tool_persistence_rules>
+- Use tools whenever they materially improve correctness, completeness, or grounding.
+- Do not stop early just to save tool calls.
+- Keep calling tools until:
+  (1) the task is complete, and
+  (2) verification passes.
+- If a tool returns empty or partial results, retry with a different strategy.
+</tool_persistence_rules>
+```
+
+### `dig_deeper_nudge`
+
+Use when:
+
+- the model is too literal or stops at the first plausible answer
+- the task is safety- or accuracy-sensitive and needs a small initiative nudge before raising reasoning effort
+
+```text
+<dig_deeper_nudge>
+- Do not stop at the first plausible answer.
+- Look for second-order issues, edge cases, and missing constraints.
+- If the task is safety- or accuracy-critical, perform at least one verification step.
+</dig_deeper_nudge>
+```
+
+### `dependency_checks`
+
+Use when:
+
+- later actions depend on prerequisite lookup, memory retrieval, or discovery steps
+- the model may be tempted to skip prerequisite work because the intended end state seems obvious
+
+```text
+<dependency_checks>
+- Before taking an action, check whether prerequisite discovery, lookup, or memory retrieval is required.
+- Do not skip prerequisite steps just because the intended final action seems obvious.
+- If a later step depends on the output of an earlier one, resolve that dependency first.
+</dependency_checks>
+```
+
+### `parallel_tool_calling`
+
+Use when:
+
+- the workflow has multiple independent retrieval steps
+- wall-clock time matters but some steps still need sequencing
+
+```text
+<parallel_tool_calling>
+- When multiple retrieval or lookup steps are independent, prefer parallel tool calls to reduce wall-clock time.
+- Do not parallelize steps with prerequisite dependencies or where one result determines the next action.
+- After parallel retrieval, pause to synthesize before making more calls.
+- Prefer selective parallelism: parallelize independent evidence gathering, not speculative or redundant tool use.
+</parallel_tool_calling>
+```
+
+### `completeness_contract`
+
+Use when:
+
+- the task involves batches, lists, enumerations, or multiple deliverables
+- missing items are a common failure mode
+
+```text
+<completeness_contract>
+- Deliver all requested items.
+- Maintain an itemized checklist of deliverables.
+- For lists or batches:
+  - state the expected count,
+  - enumerate items 1..N,
+  - confirm that none are missing before finalizing.
+- If any item is blocked by missing data, mark it [blocked] and state exactly what is missing.
+</completeness_contract>
+```
+
+### `empty_result_handling`
+
+Use when:
+
+- the workflow frequently performs search, CRM, logs, or retrieval steps
+- no-results failures are often false negatives
+
+```text
+<empty_result_handling>
+If a lookup returns empty or suspiciously small results:
+- Do not conclude that no results exist immediately.
+- Try at least 2 fallback strategies, such as a broader query, alternate filters, or another source.
+- Only then report that no results were found, along with what you tried.
+</empty_result_handling>
+```
+
+### `verification_loop`
+
+Use when:
+
+- the workflow has downstream impact
+- accuracy, formatting, or completeness regressions matter
+
+```text
+<verification_loop>
+Before finalizing:
+- Check correctness: does the output satisfy every requirement?
+- Check grounding: are factual claims backed by retrieved sources or tool output?
+- Check formatting: does the output match the requested schema or style?
+- Check safety and irreversibility: if the next step has external side effects, ask permission first.
+</verification_loop>
+```
+
+### `missing_context_gating`
+
+Use when:
+
+- required context is sometimes missing early in the workflow
+- the model should prefer retrieval over guessing
+
+```text
+<missing_context_gating>
+- If required context is missing, do not guess.
+- Prefer the appropriate lookup tool when the context is retrievable; ask a minimal clarifying question only when it is not.
+- If you must proceed, label assumptions explicitly and choose a reversible action.
+</missing_context_gating>
+```
+
+### `action_safety`
+
+Use when:
+
+- the agent will actively take actions through tools
+- the host benefits from a short pre-flight and post-flight execution frame
+
+```text
+<action_safety>
+- Pre-flight: summarize the intended action and parameters in 1-2 lines.
+- Execute via tool.
+- Post-flight: confirm the outcome and any validation that was performed.
+</action_safety>
+```
+
+### `citation_rules`
+
+Use when:
+
+- the workflow produces cited answers
+- fabricated citations or wrong citation formats are costly
+
+```text
+<citation_rules>
+- Only cite sources that were actually retrieved in this session.
+- Never fabricate citations, URLs, IDs, or quote spans.
+- If you cannot find a source for a claim, say so and either:
+  - soften the claim, or
+  - explain how to verify it with tools.
+- Use exactly the citation format required by the host application.
+</citation_rules>
+```
+
+### `research_mode`
+
+Use when:
+
+- the workflow is research-heavy
+- the host uses web search or retrieval tools
+
+```text
+<research_mode>
+- Do research in 3 passes:
+  1) Plan: list 3-6 sub-questions to answer.
+  2) Retrieve: search each sub-question and follow 1-2 second-order leads.
+  3) Synthesize: resolve contradictions and write the final answer with citations.
+- Stop only when more searching is unlikely to change the conclusion.
+</research_mode>
+```
+
+If your host environment uses a specific research tool or requires a submit step, combine this with the host's finalization contract.
+
+### `structured_output_contract`
+
+Use when:
+
+- the host depends on strict JSON, SQL, or other structured output
+
+```text
+<structured_output_contract>
+- Output only the requested format.
+- Do not add prose or markdown fences unless they were requested.
+- Validate that parentheses and brackets are balanced.
+- Do not invent tables or fields.
+- If required schema information is missing, ask for it or return an explicit error object.
+</structured_output_contract>
+```
+
+### `bbox_extraction_spec`
+
+Use when:
+
+- the workflow extracts OCR boxes, document regions, or other coordinates
+- layout drift or missed dense regions are common failure modes
+
+```text
+<bbox_extraction_spec>
+- Use the specified coordinate format exactly, such as [x1,y1,x2,y2] normalized to 0..1.
+- For each box, include page, label, text snippet, and confidence.
+- Add a vertical-drift sanity check so boxes stay aligned with the correct line of text.
+- If the layout is dense, process page by page and do a second pass for missed items.
+</bbox_extraction_spec>
+```
+
+### `terminal_tool_hygiene`
+
+Use when:
+
+- the prompt belongs to a terminal-based or coding-agent workflow
+- tool misuse or shell misuse has been observed
+
+```text
+<terminal_tool_hygiene>
+- Only run shell commands through the terminal tool.
+- Never try to "run" tool names as shell commands.
+- If a patch or edit tool exists, use it directly instead of emulating it in bash.
+- After changes, run a lightweight verification step such as ls, tests, or a build before declaring the task done.
+</terminal_tool_hygiene>
+```
+
+### `user_updates_spec`
+
+Use when:
+
+- the workflow is long-running and user updates matter
+
+```text
+<user_updates_spec>
+- Only update the user when starting a new major phase or when the plan changes.
+- Each update should contain:
+  - 1 sentence on what changed,
+  - 1 sentence on the next step.
+- Do not narrate routine tool calls.
+- Keep the user-facing update short, even when the actual work is exhaustive.
+</user_updates_spec>
+```
+
+If you are using [Compaction](https://developers.openai.com/api/docs/guides/compaction) in the Responses API, compact after major milestones, treat compacted items as opaque state, and keep prompts functionally identical after compaction.
+
+## Responses `phase` guidance
+
+For long-running Responses workflows, preambles, or tool-heavy agents that replay assistant items, review whether `phase` is already preserved.
+
+- If the host already round-trips `phase`, keep it intact during the upgrade.
+- If the host uses `previous_response_id` and does not manually replay assistant items, note that this may reduce manual `phase` handling needs.
+- If reliable GPT-5.4 behavior would require adding or preserving `phase` and that would need code edits, treat the case as blocked for prompt-only or model-string-only migration guidance.
+
+## Example upgrade profiles
+
+### GPT-5.2
+
+- Use `gpt-5.4`
+- Match the current reasoning effort first
+- Preserve the existing latency and quality profile before tuning prompt blocks
+- If the repo does not expose the exact setting, emit `same` as the starting recommendation
+
+### GPT-5.3-Codex
+
+- Use `gpt-5.4`
+- Match the current reasoning effort first
+- If you need Codex-style speed and efficiency, add verification blocks before increasing reasoning effort
+- If the repo does not expose the exact setting, emit `same` as the starting recommendation
+
+### GPT-4o or GPT-4.1 assistant
+
+- Use `gpt-5.4`
+- Start with `none` reasoning effort
+- Add `output_verbosity_spec` only if output becomes too verbose
+
+### Long-horizon agent
+
+- Use `gpt-5.4`
+- Start with `medium` reasoning effort
+- Add `tool_persistence_rules`
+- Add `completeness_contract`
+- Add `verification_loop`
+
+### Research workflow
+
+- Use `gpt-5.4`
+- Start with `medium` reasoning effort
+- Add `research_mode`
+- Add `citation_rules`
+- Add `empty_result_handling`
+- Add `tool_persistence_rules` when the host already uses web or retrieval tools
+- Add `parallel_tool_calling` when the retrieval steps are independent
+
+### Support triage or multi-agent workflow
+
+- Use `gpt-5.4`
+- Prefer `model string + light prompt rewrite` over `model string only`
+- Add at least one of `tool_persistence_rules`, `completeness_contract`, or `verification_loop`
+- Add more only if evals show a real regression
+
+### Coding or terminal workflow
+
+- Use `gpt-5.4`
+- Keep the model-string change narrow
+- Match the current reasoning effort first if you are upgrading from GPT-5.3-Codex
+- Add `terminal_tool_hygiene`
+- Add `verification_loop`
+- Add `dependency_checks` when actions depend on prerequisite lookup or discovery
+- Add `tool_persistence_rules` if the agent stops too early
+- Review whether `phase` is already preserved for long-running Responses flows or assistant preambles
+- Do not classify this as blocked just because the workflow uses tools; block only if the upgrade requires changing tool definitions or wiring
+- If the repo already uses Responses plus tools and no required host-side change is shown, prefer `model_string_plus_light_prompt_rewrite` over `blocked`
+
+## Prompt regression checklist
+
+- Check whether the upgraded prompt still preserves the original task intent.
+- Check whether the new prompt is leaner, not just longer.
+- Check completeness, citation quality, dependency handling, verification behavior, and verbosity.
+- For long-running Responses agents, check whether `phase` handling is already in place or needs implementation work.
+- Confirm that each added prompt block addresses an observed regression.
+- Remove prompt blocks that are not earning their keep.

skills/.system/openai-docs/references/latest-model.md

@@ -0,0 +1,35 @@
+# Latest model guide
+
+This file is a curated helper. Every recommendation here must be verified against current OpenAI docs before it is repeated to a user.
+
+## Current model map
+
+| Model ID | Use for |
+| --- | --- |
+| `gpt-5.4` | Default text plus reasoning for most new apps |
+| `gpt-5.4-pro` | Only when the user explicitly asks for maximum reasoning or quality; substantially slower and more expensive |
+| `gpt-5-mini` | Cheaper and faster reasoning with good quality |
+| `gpt-5-nano` | High-throughput simple tasks and classification |
+| `gpt-5.4` | Explicit no-reasoning text path via `reasoning.effort: none` |
+| `gpt-4.1-mini` | Cheaper no-reasoning text |
+| `gpt-4.1-nano` | Fastest and cheapest no-reasoning text |
+| `gpt-5.3-codex` | Agentic coding, code editing, and tool-heavy coding workflows |
+| `gpt-5.1-codex-mini` | Cheaper coding workflows |
+| `gpt-image-1.5` | Best image generation and edit quality |
+| `gpt-image-1-mini` | Cost-optimized image generation |
+| `gpt-4o-mini-tts` | Text-to-speech |
+| `gpt-4o-mini-transcribe` | Speech-to-text, fast and cost-efficient |
+| `gpt-realtime-1.5` | Realtime voice and multimodal sessions |
+| `gpt-realtime-mini` | Cheaper realtime sessions |
+| `gpt-audio` | Chat Completions audio input and output |
+| `gpt-audio-mini` | Cheaper Chat Completions audio workflows |
+| `sora-2` | Faster iteration and draft video generation |
+| `sora-2-pro` | Higher-quality production video |
+| `omni-moderation-latest` | Text and image moderation |
+| `text-embedding-3-large` | Higher-quality retrieval embeddings; default in this skill because no best-specific row exists |
+| `text-embedding-3-small` | Lower-cost embeddings |
+
+## Maintenance notes
+
+- This file will drift unless it is periodically re-verified against current OpenAI docs.
+- If this file conflicts with current docs, the docs win.

skills/.system/openai-docs/references/upgrading-to-gpt-5p4.md

@@ -0,0 +1,164 @@
+# Upgrading to GPT-5.4
+
+Use this guide when the user explicitly asks to upgrade an existing integration to GPT-5.4. Pair it with current OpenAI docs lookups. The default target string is `gpt-5.4`.
+
+## Upgrade posture
+
+Upgrade with the narrowest safe change set:
+
+- replace the model string first
+- update only the prompts that are directly tied to that model usage
+- prefer prompt-only upgrades when possible
+- if the upgrade would require API-surface changes, parameter rewrites, tool rewiring, or broader code edits, mark it as blocked instead of stretching the scope
+
+## Upgrade workflow
+
+1. Inventory current model usage.
+   - Search for model strings, client calls, and prompt-bearing files.
+   - Include inline prompts, prompt templates, YAML or JSON configs, Markdown docs, and saved prompts when they are clearly tied to a model usage site.
+2. Pair each model usage with its prompt surface.
+   - Prefer the closest prompt surface first: inline system or developer text, then adjacent prompt files, then shared templates.
+   - If you cannot confidently tie a prompt to the model usage, say so instead of guessing.
+3. Classify the source model family.
+   - Common buckets: `gpt-4o` or `gpt-4.1`, `o1` or `o3` or `o4-mini`, early `gpt-5`, later `gpt-5.x`, or mixed and unclear.
+4. Decide the upgrade class.
+   - `model string only`
+   - `model string + light prompt rewrite`
+   - `blocked without code changes`
+5. Run the no-code compatibility gate.
+   - Check whether the current integration can accept `gpt-5.4` without API-surface changes or implementation changes.
+   - For long-running Responses or tool-heavy agents, check whether `phase` is already preserved or round-tripped when the host replays assistant items or uses preambles.
+   - If compatibility depends on code changes, return `blocked`.
+   - If compatibility is unclear, return `unknown` rather than improvising.
+6. Recommend the upgrade.
+   - Default replacement string: `gpt-5.4`
+   - Keep the intervention small and behavior-preserving.
+7. Deliver a structured recommendation.
+   - `Current model usage`
+   - `Recommended model-string updates`
+   - `Starting reasoning recommendation`
+   - `Prompt updates`
+   - `Phase assessment` when the flow is long-running, replayed, or tool-heavy
+   - `No-code compatibility check`
+   - `Validation plan`
+   - `Launch-day refresh items`
+
+Output rule:
+
+- Always emit a starting `reasoning_effort_recommendation` for each usage site.
+- If the repo exposes the current reasoning setting, preserve it first unless the source guide says otherwise.
+- If the repo does not expose the current setting, use the source-family starting mapping instead of returning `null`.
+
+## Upgrade outcomes
+
+### `model string only`
+
+Choose this when:
+
+- the existing prompts are already short, explicit, and task-bounded
+- the workflow is not strongly research-heavy, tool-heavy, multi-agent, batch or completeness-sensitive, or long-horizon
+- there are no obvious compatibility blockers
+
+Default action:
+
+- replace the model string with `gpt-5.4`
+- keep prompts unchanged
+- validate behavior with existing evals or spot checks
+
+### `model string + light prompt rewrite`
+
+Choose this when:
+
+- the old prompt was compensating for weaker instruction following
+- the workflow needs more persistence than the default tool-use behavior will likely provide
+- the task needs stronger completeness, citation discipline, or verification
+- the upgraded model becomes too verbose or under-complete unless instructed otherwise
+- the workflow is research-heavy and needs stronger handling of sparse or empty retrieval results
+- the workflow is coding-oriented, tool-heavy, or multi-agent, but the existing API surface and tool definitions can remain unchanged
+
+Default action:
+
+- replace the model string with `gpt-5.4`
+- add one or two targeted prompt blocks
+- read `references/gpt-5p4-prompting-guide.md` to choose the smallest prompt changes that recover the old behavior
+- avoid broad prompt cleanup unrelated to the upgrade
+- for research workflows, default to `research_mode` + `citation_rules` + `empty_result_handling`; add `tool_persistence_rules` when the host already uses retrieval tools
+- for dependency-aware or tool-heavy workflows, default to `tool_persistence_rules` + `dependency_checks` + `verification_loop`; add `parallel_tool_calling` only when retrieval steps are truly independent
+- for coding or terminal workflows, default to `terminal_tool_hygiene` + `verification_loop`
+- for multi-agent support or triage workflows, default to at least one of `tool_persistence_rules`, `completeness_contract`, or `verification_loop`
+- for long-running Responses agents with preambles or multiple assistant messages, explicitly review whether `phase` is already handled; if adding or preserving `phase` would require code edits, mark the path as `blocked`
+- do not classify a coding or tool-using Responses workflow as `blocked` just because the visible snippet is minimal; prefer `model string + light prompt rewrite` unless the repo clearly shows that a safe GPT-5.4 path would require host-side code changes
+
+### `blocked`
+
+Choose this when:
+
+- the upgrade appears to require API-surface changes
+- the upgrade appears to require parameter rewrites or reasoning-setting changes that are not exposed outside implementation code
+- the upgrade would require changing tool definitions, tool handler wiring, or schema contracts
+- you cannot confidently identify the prompt surface tied to the model usage
+
+Default action:
+
+- do not improvise a broader upgrade
+- report the blocker and explain that the fix is out of scope for this guide
+
+## No-code compatibility checklist
+
+Before recommending a no-code upgrade, check:
+
+1. Can the current host accept the `gpt-5.4` model string without changing client code or API surface?
+2. Are the related prompts identifiable and editable?
+3. Does the host depend on behavior that likely needs API-surface changes, parameter rewrites, or tool rewiring?
+4. Would the likely fix be prompt-only, or would it need implementation changes?
+5. Is the prompt surface close enough to the model usage that you can make a targeted change instead of a broad cleanup?
+6. For long-running Responses or tool-heavy agents, is `phase` already preserved if the host relies on preambles, replayed assistant items, or multiple assistant messages?
+
+If item 1 is no, items 3 through 4 point to implementation work, or item 6 is no and the fix needs code changes, return `blocked`.
+
+If item 2 is no, return `unknown` unless the user can point to the prompt location.
+
+Important:
+
+- Existing use of tools, agents, or multiple usage sites is not by itself a blocker.
+- If the current host can keep the same API surface and the same tool definitions, prefer `model string + light prompt rewrite` over `blocked`.
+- Reserve `blocked` for cases that truly require implementation changes, not cases that only need stronger prompt steering.
+
+## Scope boundaries
+
+This guide may:
+
+- update or recommend updated model strings
+- update or recommend updated prompts
+- inspect code and prompt files to understand where those changes belong
+- inspect whether existing Responses flows already preserve `phase`
+- flag compatibility blockers
+
+This guide may not:
+
+- move Chat Completions code to Responses
+- move Responses code to another API surface
+- rewrite parameter shapes
+- change tool definitions or tool-call handling
+- change structured-output wiring
+- add or retrofit `phase` handling in implementation code
+- edit business logic, orchestration logic, or SDK usage beyond a literal model-string replacement
+
+If a safe GPT-5.4 upgrade requires any of those changes, mark the path as blocked and out of scope.
+
+## Validation plan
+
+- Validate each upgraded usage site with existing evals or realistic spot checks.
+- Check whether the upgraded model still matches expected latency, output shape, and quality.
+- If prompt edits were added, confirm each block is doing real work instead of adding noise.
+- If the workflow has downstream impact, add a lightweight verification pass before finalization.
+
+## Launch-day refresh items
+
+When final GPT-5.4 guidance changes:
+
+1. Replace release-candidate assumptions with final GPT-5.4 guidance where appropriate.
+2. Re-check whether the default target string should stay `gpt-5.4` for all source families.
+3. Re-check any prompt-block recommendations whose semantics may have changed.
+4. Re-check research, citation, and compatibility guidance against the final model behavior.
+5. Re-run the same upgrade scenarios and confirm the blocked-versus-viable boundaries still hold.

skills/.system/skill-creator/SKILL.md

@@ -0,0 +1,413 @@
+---
+name: skill-creator
+description: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
+metadata:
+  short-description: Create or update a skill
+---
+
+# Skill Creator
+
+This skill provides guidance for creating effective skills.
+
+## About Skills
+
+Skills are modular, self-contained folders that extend Codex's capabilities by providing
+specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific
+domains or tasks—they transform Codex from a general-purpose agent into a specialized agent
+equipped with procedural knowledge that no model can fully possess.
+
+### What Skills Provide
+
+1. Specialized workflows - Multi-step procedures for specific domains
+2. Tool integrations - Instructions for working with specific file formats or APIs
+3. Domain expertise - Company-specific knowledge, schemas, business logic
+4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks
+
+## Core Principles
+
+### Concise is Key
+
+The context window is a public good. Skills share the context window with everything else Codex needs: system prompt, conversation history, other Skills' metadata, and the actual user request.
+
+**Default assumption: Codex is already very smart.** Only add context Codex doesn't already have. Challenge each piece of information: "Does Codex really need this explanation?" and "Does this paragraph justify its token cost?"
+
+Prefer concise examples over verbose explanations.
+
+### Set Appropriate Degrees of Freedom
+
+Match the level of specificity to the task's fragility and variability:
+
+**High freedom (text-based instructions)**: Use when multiple approaches are valid, decisions depend on context, or heuristics guide the approach.
+
+**Medium freedom (pseudocode or scripts with parameters)**: Use when a preferred pattern exists, some variation is acceptable, or configuration affects behavior.
+
+**Low freedom (specific scripts, few parameters)**: Use when operations are fragile and error-prone, consistency is critical, or a specific sequence must be followed.
+
+Think of Codex as exploring a path: a narrow bridge with cliffs needs specific guardrails (low freedom), while an open field allows many routes (high freedom).
+
+### Protect Validation Integrity
+
+You may use subagents during iteration to validate whether a skill works on realistic tasks or whether a suspected problem is real. This is most useful when you want an independent pass on the skill's behavior, outputs, or failure modes after a revision.  Only do this when it is possible to start new subagents.
+
+When using subagents for validation, treat that as an evaluation surface. The goal is to learn whether the skill generalizes, not whether another agent can reconstruct the answer from leaked context.
+
+Prefer raw artifacts such as example prompts, outputs, diffs, logs, or traces. Give the minimum task-local context needed to perform the validation. Avoid passing the intended answer, suspected bug, intended fix, or your prior conclusions unless the validation explicitly requires them.
+
+### Anatomy of a Skill
+
+Every skill consists of a required SKILL.md file and optional bundled resources:
+
+```
+skill-name/
+├── SKILL.md (required)
+│   ├── YAML frontmatter metadata (required)
+│   │   ├── name: (required)
+│   │   └── description: (required)
+│   └── Markdown instructions (required)
+├── agents/ (recommended)
+│   └── openai.yaml - UI metadata for skill lists and chips
+└── Bundled Resources (optional)
+    ├── scripts/          - Executable code (Python/Bash/etc.)
+    ├── references/       - Documentation intended to be loaded into context as needed
+    └── assets/           - Files used in output (templates, icons, fonts, etc.)
+```
+
+#### SKILL.md (required)
+
+Every SKILL.md consists of:
+
+- **Frontmatter** (YAML): Contains `name` and `description` fields. These are the only fields that Codex reads to determine when the skill gets used, thus it is very important to be clear and comprehensive in describing what the skill is, and when it should be used.
+- **Body** (Markdown): Instructions and guidance for using the skill. Only loaded AFTER the skill triggers (if at all).
+
+#### Agents metadata (recommended)
+
+- UI-facing metadata for skill lists and chips
+- Read references/openai_yaml.md before generating values and follow its descriptions and constraints
+- Create: human-facing `display_name`, `short_description`, and `default_prompt` by reading the skill
+- Generate deterministically by passing the values as `--interface key=value` to `scripts/generate_openai_yaml.py` or `scripts/init_skill.py`
+- On updates: validate `agents/openai.yaml` still matches SKILL.md; regenerate if stale
+- Only include other optional interface fields (icons, brand color) if explicitly provided
+- See references/openai_yaml.md for field definitions and examples
+
+#### Bundled Resources (optional)
+
+##### Scripts (`scripts/`)
+
+Executable code (Python/Bash/etc.) for tasks that require deterministic reliability or are repeatedly rewritten.
+
+- **When to include**: When the same code is being rewritten repeatedly or deterministic reliability is needed
+- **Example**: `scripts/rotate_pdf.py` for PDF rotation tasks
+- **Benefits**: Token efficient, deterministic, may be executed without loading into context
+- **Note**: Scripts may still need to be read by Codex for patching or environment-specific adjustments
+
+##### References (`references/`)
+
+Documentation and reference material intended to be loaded as needed into context to inform Codex's process and thinking.
+
+- **When to include**: For documentation that Codex should reference while working
+- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications
+- **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides
+- **Benefits**: Keeps SKILL.md lean, loaded only when Codex determines it's needed
+- **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md
+- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files.
+
+##### Assets (`assets/`)
+
+Files not intended to be loaded into context, but rather used within the output Codex produces.
+
+- **When to include**: When the skill needs files that will be used in the final output
+- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography
+- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified
+- **Benefits**: Separates output resources from documentation, enables Codex to use files without loading them into context
+
+#### What to Not Include in a Skill
+
+A skill should only contain essential files that directly support its functionality. Do NOT create extraneous documentation or auxiliary files, including:
+
+- README.md
+- INSTALLATION_GUIDE.md
+- QUICK_REFERENCE.md
+- CHANGELOG.md
+- etc.
+
+The skill should only contain the information needed for an AI agent to do the job at hand. It should not contain auxiliary context about the process that went into creating it, setup and testing procedures, user-facing documentation, etc. Creating additional documentation files just adds clutter and confusion.
+
+### Progressive Disclosure Design Principle
+
+Skills use a three-level loading system to manage context efficiently:
+
+1. **Metadata (name + description)** - Always in context (~100 words)
+2. **SKILL.md body** - When skill triggers (<5k words)
+3. **Bundled resources** - As needed by Codex (Unlimited because scripts can be executed without reading into context window)
+
+#### Progressive Disclosure Patterns
+
+Keep SKILL.md body to the essentials and under 500 lines to minimize context bloat. Split content into separate files when approaching this limit. When splitting out content into other files, it is very important to reference them from SKILL.md and describe clearly when to read them, to ensure the reader of the skill knows they exist and when to use them.
+
+**Key principle:** When a skill supports multiple variations, frameworks, or options, keep only the core workflow and selection guidance in SKILL.md. Move variant-specific details (patterns, examples, configuration) into separate reference files.
+
+**Pattern 1: High-level guide with references**
+
+```markdown
+# PDF Processing
+
+## Quick start
+
+Extract text with pdfplumber:
+[code example]
+
+## Advanced features
+
+- **Form filling**: See [FORMS.md](FORMS.md) for complete guide
+- **API reference**: See [REFERENCE.md](REFERENCE.md) for all methods
+- **Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns
+```
+
+Codex loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed.
+
+**Pattern 2: Domain-specific organization**
+
+For Skills with multiple domains, organize content by domain to avoid loading irrelevant context:
+
+```
+bigquery-skill/
+├── SKILL.md (overview and navigation)
+└── reference/
+    ├── finance.md (revenue, billing metrics)
+    ├── sales.md (opportunities, pipeline)
+    ├── product.md (API usage, features)
+    └── marketing.md (campaigns, attribution)
+```
+
+When a user asks about sales metrics, Codex only reads sales.md.
+
+Similarly, for skills supporting multiple frameworks or variants, organize by variant:
+
+```
+cloud-deploy/
+├── SKILL.md (workflow + provider selection)
+└── references/
+    ├── aws.md (AWS deployment patterns)
+    ├── gcp.md (GCP deployment patterns)
+    └── azure.md (Azure deployment patterns)
+```
+
+When the user chooses AWS, Codex only reads aws.md.
+
+**Pattern 3: Conditional details**
+
+Show basic content, link to advanced content:
+
+```markdown
+# DOCX Processing
+
+## Creating documents
+
+Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md).
+
+## Editing documents
+
+For simple edits, modify the XML directly.
+
+**For tracked changes**: See [REDLINING.md](REDLINING.md)
+**For OOXML details**: See [OOXML.md](OOXML.md)
+```
+
+Codex reads REDLINING.md or OOXML.md only when the user needs those features.
+
+**Important guidelines:**
+
+- **Avoid deeply nested references** - Keep references one level deep from SKILL.md. All reference files should link directly from SKILL.md.
+- **Structure longer reference files** - For files longer than 100 lines, include a table of contents at the top so Codex can see the full scope when previewing.
+
+## Skill Creation Process
+
+Skill creation involves these steps:
+
+1. Understand the skill with concrete examples
+2. Plan reusable skill contents (scripts, references, assets)
+3. Initialize the skill (run init_skill.py)
+4. Edit the skill (implement resources and write SKILL.md)
+5. Validate the skill (run quick_validate.py)
+6. Iterate based on real usage and forward-test complex skills.
+
+Follow these steps in order, skipping only if there is a clear reason why they are not applicable.
+
+### Skill Naming
+
+- Use lowercase letters, digits, and hyphens only; normalize user-provided titles to hyphen-case (e.g., "Plan Mode" -> `plan-mode`).
+- When generating names, generate a name under 64 characters (letters, digits, hyphens).
+- Prefer short, verb-led phrases that describe the action.
+- Namespace by tool when it improves clarity or triggering (e.g., `gh-address-comments`, `linear-address-issue`).
+- Name the skill folder exactly after the skill name.
+
+### Step 1: Understanding the Skill with Concrete Examples
+
+Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill.
+
+To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback.
+
+For example, when building an image-editor skill, relevant questions include:
+
+- "What functionality should the image-editor skill support? Editing, rotating, anything else?"
+- "Can you give some examples of how this skill would be used?"
+- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?"
+- "What would a user say that should trigger this skill?"
+
+To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness.
+
+Conclude this step when there is a clear sense of the functionality the skill should support.
+
+### Step 2: Planning the Reusable Skill Contents
+
+To turn concrete examples into an effective skill, analyze each example by:
+
+1. Considering how to execute on the example from scratch
+2. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly
+
+Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows:
+
+1. Rotating a PDF requires re-writing the same code each time
+2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill
+
+Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows:
+
+1. Writing a frontend webapp requires the same boilerplate HTML/React each time
+2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill
+
+Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows:
+
+1. Querying BigQuery requires re-discovering the table schemas and relationships each time
+2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill
+
+To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets.
+
+### Step 3: Initializing the Skill
+
+At this point, it is time to actually create the skill.
+
+Skip this step only if the skill being developed already exists. In this case, continue to the next step.
+
+When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable.
+
+Usage:
+
+```bash
+scripts/init_skill.py <skill-name> --path <output-directory> [--resources scripts,references,assets] [--examples]
+```
+
+Examples:
+
+```bash
+scripts/init_skill.py my-skill --path skills/public
+scripts/init_skill.py my-skill --path skills/public --resources scripts,references
+scripts/init_skill.py my-skill --path skills/public --resources scripts --examples
+```
+
+The script:
+
+- Creates the skill directory at the specified path
+- Generates a SKILL.md template with proper frontmatter and TODO placeholders
+- Creates `agents/openai.yaml` using agent-generated `display_name`, `short_description`, and `default_prompt` passed via `--interface key=value`
+- Optionally creates resource directories based on `--resources`
+- Optionally adds example files when `--examples` is set
+
+After initialization, customize the SKILL.md and add resources as needed. If you used `--examples`, replace or delete placeholder files.
+
+Generate `display_name`, `short_description`, and `default_prompt` by reading the skill, then pass them as `--interface key=value` to `init_skill.py` or regenerate with:
+
+```bash
+scripts/generate_openai_yaml.py <path/to/skill-folder> --interface key=value
+```
+
+Only include other optional interface fields when the user explicitly provides them. For full field descriptions and examples, see references/openai_yaml.md.
+
+### Step 4: Edit the Skill
+
+When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Codex to use. Include information that would be beneficial and non-obvious to Codex. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Codex instance execute these tasks more effectively.
+
+After substantial revisions, or if the skill is particularly tricky, you should use subagents to forward-test the skill on realistic tasks or artifacts. When doing so, pass the artifact under validation rather than your diagnosis of what is wrong, and keep the prompt generic enough that success depends on transferable reasoning rather than hidden ground truth.
+
+#### Start with Reusable Skill Contents
+
+To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`.
+
+Added scripts must be tested by actually running them to ensure there are no bugs and that the output matches what is expected. If there are many similar scripts, only a representative sample needs to be tested to ensure confidence that they all work while balancing time to completion.
+
+If you used `--examples`, delete any placeholder files that are not needed for the skill. Only create resource directories that are actually required.
+
+#### Update SKILL.md
+
+**Writing Guidelines:** Always use imperative/infinitive form.
+
+##### Frontmatter
+
+Write the YAML frontmatter with `name` and `description`:
+
+- `name`: The skill name
+- `description`: This is the primary triggering mechanism for your skill, and helps Codex understand when to use the skill.
+  - Include both what the Skill does and specific triggers/contexts for when to use it.
+  - Include all "when to use" information here - Not in the body. The body is only loaded after triggering, so "When to Use This Skill" sections in the body are not helpful to Codex.
+  - Example description for a `docx` skill: "Comprehensive document creation, editing, and analysis with support for tracked changes, comments, formatting preservation, and text extraction. Use when Codex needs to work with professional documents (.docx files) for: (1) Creating new documents, (2) Modifying or editing content, (3) Working with tracked changes, (4) Adding comments, or any other document tasks"
+
+Do not include any other fields in YAML frontmatter.
+
+##### Body
+
+Write instructions for using the skill and its bundled resources.
+
+### Step 5: Validate the Skill
+
+Once development of the skill is complete, validate the skill folder to catch basic issues early:
+
+```bash
+scripts/quick_validate.py <path/to/skill-folder>
+```
+
+The validation script checks YAML frontmatter format, required fields, and naming rules. If validation fails, fix the reported issues and run the command again.
+
+### Step 6: Iterate
+
+After testing the skill, you may detect the skill is complex enough that it requires forward-testing; or users may request improvements.
+
+User testing often this happens right after using the skill, with fresh context of how the skill performed.
+
+**Forward-testing and iteration workflow:**
+
+1. Use the skill on real tasks
+2. Notice struggles or inefficiencies
+3. Identify how SKILL.md or bundled resources should be updated
+4. Implement changes and test again
+5. Forward-test if it is reasonable and appropriate
+
+## Forward-testing
+
+To forward-test, launch subagents as a way to stress test the skill with minimal context.
+Subagents should *not* know that they are being asked to test the skill.  They should be treated as
+an agent asked to perform a task by the user.  Prompts to subagents should look like:
+  `Use $skill-x at /path/to/skill-x to solve problem y`
+Not:
+  `Review the skill at /path/to/skill-x; pretend a user asks you to...`
+
+Decision rule for forward-testing:
+  - Err on the side of forward-testing
+  - Ask for approval if you think there's a risk that forward-testing would:
+    * take a long time,
+    * require additional approvals from the user, or
+    * modify live production systems
+
+  In these cases, show the user your proposed prompt and request (1) a yes/no decision, and
+  (2) any suggested modifictions.
+
+Considerations when forward-testing:
+   - use fresh threads for independent passes
+   - pass the skill, and a request in a similar way the user would.
+   - pass raw artifacts, not your conclusions
+   - avoid showing expected answers or intended fixes
+   - rebuild context from source artifacts after each iteration
+   - review the subagent's output and reasoning and emitted artifacts
+   - avoid leaving artifacts the agent can find on disk between iterations;
+     clean up subagents' artifacts to avoid additional contamination.
+
+If forward-testing only succeeds when subagents see leaked context, tighten the skill or the
+forward-testing setup before trusting the result.

skills/.system/skill-creator/agents/openai.yaml

@@ -0,0 +1,5 @@
+interface:
+  display_name: "Skill Creator"
+  short_description: "Create or update a skill"
+  icon_small: "./assets/skill-creator-small.svg"
+  icon_large: "./assets/skill-creator.png"

skills/.system/skill-creator/license.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

skills/.system/skill-creator/references/openai_yaml.md

@@ -0,0 +1,49 @@
+# openai.yaml fields (full example + descriptions)
+
+`agents/openai.yaml` is an extended, product-specific config intended for the machine/harness to read, not the agent. Other product-specific config can also live in the `agents/` folder.
+
+## Full example
+
+```yaml
+interface:
+  display_name: "Optional user-facing name"
+  short_description: "Optional user-facing description"
+  icon_small: "./assets/small-400px.png"
+  icon_large: "./assets/large-logo.svg"
+  brand_color: "#3B82F6"
+  default_prompt: "Optional surrounding prompt to use the skill with"
+
+dependencies:
+  tools:
+    - type: "mcp"
+      value: "github"
+      description: "GitHub MCP server"
+      transport: "streamable_http"
+      url: "https://api.githubcopilot.com/mcp/"
+
+policy:
+  allow_implicit_invocation: true
+```
+
+## Field descriptions and constraints
+
+Top-level constraints:
+
+- Quote all string values.
+- Keep keys unquoted.
+- For `interface.default_prompt`: generate a helpful, short (typically 1 sentence) example starting prompt based on the skill. It must explicitly mention the skill as `$skill-name` (e.g., "Use $skill-name-here to draft a concise weekly status update.").
+
+- `interface.display_name`: Human-facing title shown in UI skill lists and chips.
+- `interface.short_description`: Human-facing short UI blurb (25–64 chars) for quick scanning.
+- `interface.icon_small`: Path to a small icon asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
+- `interface.icon_large`: Path to a larger logo asset (relative to skill dir). Default to `./assets/` and place icons in the skill's `assets/` folder.
+- `interface.brand_color`: Hex color used for UI accents (e.g., badges).
+- `interface.default_prompt`: Default prompt snippet inserted when invoking the skill.
+- `dependencies.tools[].type`: Dependency category. Only `mcp` is supported for now.
+- `dependencies.tools[].value`: Identifier of the tool or dependency.
+- `dependencies.tools[].description`: Human-readable explanation of the dependency.
+- `dependencies.tools[].transport`: Connection type when `type` is `mcp`.
+- `dependencies.tools[].url`: MCP server URL when `type` is `mcp`.
+- `policy.allow_implicit_invocation`: When false, the skill is not injected into
+  the model context by default, but can still be invoked explicitly via `$skill`.
+  Defaults to true.

skills/.system/skill-creator/scripts/generate_openai_yaml.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""
+OpenAI YAML Generator - Creates agents/openai.yaml for a skill folder.
+
+Usage:
+    generate_openai_yaml.py <skill_dir> [--name <skill_name>] [--interface key=value]
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+ACRONYMS = {
+    "GH",
+    "MCP",
+    "API",
+    "CI",
+    "CLI",
+    "LLM",
+    "PDF",
+    "PR",
+    "UI",
+    "URL",
+    "SQL",
+}
+
+BRANDS = {
+    "openai": "OpenAI",
+    "openapi": "OpenAPI",
+    "github": "GitHub",
+    "pagerduty": "PagerDuty",
+    "datadog": "DataDog",
+    "sqlite": "SQLite",
+    "fastapi": "FastAPI",
+}
+
+SMALL_WORDS = {"and", "or", "to", "up", "with"}
+
+ALLOWED_INTERFACE_KEYS = {
+    "display_name",
+    "short_description",
+    "icon_small",
+    "icon_large",
+    "brand_color",
+    "default_prompt",
+}
+
+
+def yaml_quote(value):
+    escaped = value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
+    return f'"{escaped}"'
+
+
+def format_display_name(skill_name):
+    words = [word for word in skill_name.split("-") if word]
+    formatted = []
+    for index, word in enumerate(words):
+        lower = word.lower()
+        upper = word.upper()
+        if upper in ACRONYMS:
+            formatted.append(upper)
+            continue
+        if lower in BRANDS:
+            formatted.append(BRANDS[lower])
+            continue
+        if index > 0 and lower in SMALL_WORDS:
+            formatted.append(lower)
+            continue
+        formatted.append(word.capitalize())
+    return " ".join(formatted)
+
+
+def generate_short_description(display_name):
+    description = f"Help with {display_name} tasks"
+
+    if len(description) < 25:
+        description = f"Help with {display_name} tasks and workflows"
+    if len(description) < 25:
+        description = f"Help with {display_name} tasks with guidance"
+
+    if len(description) > 64:
+        description = f"Help with {display_name}"
+    if len(description) > 64:
+        description = f"{display_name} helper"
+    if len(description) > 64:
+        description = f"{display_name} tools"
+    if len(description) > 64:
+        suffix = " helper"
+        max_name_length = 64 - len(suffix)
+        trimmed = display_name[:max_name_length].rstrip()
+        description = f"{trimmed}{suffix}"
+    if len(description) > 64:
+        description = description[:64].rstrip()
+
+    if len(description) < 25:
+        description = f"{description} workflows"
+        if len(description) > 64:
+            description = description[:64].rstrip()
+
+    return description
+
+
+def read_frontmatter_name(skill_dir):
+    skill_md = Path(skill_dir) / "SKILL.md"
+    if not skill_md.exists():
+        print(f"[ERROR] SKILL.md not found in {skill_dir}")
+        return None
+    content = skill_md.read_text()
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        print("[ERROR] Invalid SKILL.md frontmatter format.")
+        return None
+    frontmatter_text = match.group(1)
+
+    import yaml
+
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+    except yaml.YAMLError as exc:
+        print(f"[ERROR] Invalid YAML frontmatter: {exc}")
+        return None
+    if not isinstance(frontmatter, dict):
+        print("[ERROR] Frontmatter must be a YAML dictionary.")
+        return None
+    name = frontmatter.get("name", "")
+    if not isinstance(name, str) or not name.strip():
+        print("[ERROR] Frontmatter 'name' is missing or invalid.")
+        return None
+    return name.strip()
+
+
+def parse_interface_overrides(raw_overrides):
+    overrides = {}
+    optional_order = []
+    for item in raw_overrides:
+        if "=" not in item:
+            print(f"[ERROR] Invalid interface override '{item}'. Use key=value.")
+            return None, None
+        key, value = item.split("=", 1)
+        key = key.strip()
+        value = value.strip()
+        if not key:
+            print(f"[ERROR] Invalid interface override '{item}'. Key is empty.")
+            return None, None
+        if key not in ALLOWED_INTERFACE_KEYS:
+            allowed = ", ".join(sorted(ALLOWED_INTERFACE_KEYS))
+            print(f"[ERROR] Unknown interface field '{key}'. Allowed: {allowed}")
+            return None, None
+        overrides[key] = value
+        if key not in ("display_name", "short_description") and key not in optional_order:
+            optional_order.append(key)
+    return overrides, optional_order
+
+
+def write_openai_yaml(skill_dir, skill_name, raw_overrides):
+    overrides, optional_order = parse_interface_overrides(raw_overrides)
+    if overrides is None:
+        return None
+
+    display_name = overrides.get("display_name") or format_display_name(skill_name)
+    short_description = overrides.get("short_description") or generate_short_description(display_name)
+
+    if not (25 <= len(short_description) <= 64):
+        print(
+            "[ERROR] short_description must be 25-64 characters "
+            f"(got {len(short_description)})."
+        )
+        return None
+
+    interface_lines = [
+        "interface:",
+        f"  display_name: {yaml_quote(display_name)}",
+        f"  short_description: {yaml_quote(short_description)}",
+    ]
+
+    for key in optional_order:
+        value = overrides.get(key)
+        if value is not None:
+            interface_lines.append(f"  {key}: {yaml_quote(value)}")
+
+    agents_dir = Path(skill_dir) / "agents"
+    agents_dir.mkdir(parents=True, exist_ok=True)
+    output_path = agents_dir / "openai.yaml"
+    output_path.write_text("\n".join(interface_lines) + "\n")
+    print(f"[OK] Created agents/openai.yaml")
+    return output_path
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Create agents/openai.yaml for a skill directory.",
+    )
+    parser.add_argument("skill_dir", help="Path to the skill directory")
+    parser.add_argument(
+        "--name",
+        help="Skill name override (defaults to SKILL.md frontmatter)",
+    )
+    parser.add_argument(
+        "--interface",
+        action="append",
+        default=[],
+        help="Interface override in key=value format (repeatable)",
+    )
+    args = parser.parse_args()
+
+    skill_dir = Path(args.skill_dir).resolve()
+    if not skill_dir.exists():
+        print(f"[ERROR] Skill directory not found: {skill_dir}")
+        sys.exit(1)
+    if not skill_dir.is_dir():
+        print(f"[ERROR] Path is not a directory: {skill_dir}")
+        sys.exit(1)
+
+    skill_name = args.name or read_frontmatter_name(skill_dir)
+    if not skill_name:
+        sys.exit(1)
+
+    result = write_openai_yaml(skill_dir, skill_name, args.interface)
+    if result:
+        sys.exit(0)
+    sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

skills/.system/skill-creator/scripts/init_skill.py

@@ -0,0 +1,400 @@
+#!/usr/bin/env python3
+"""
+Skill Initializer - Creates a new skill from template
+
+Usage:
+    init_skill.py <skill-name> --path <path> [--resources scripts,references,assets] [--examples] [--interface key=value]
+
+Examples:
+    init_skill.py my-new-skill --path skills/public
+    init_skill.py my-new-skill --path skills/public --resources scripts,references
+    init_skill.py my-api-helper --path skills/private --resources scripts --examples
+    init_skill.py custom-skill --path /custom/location
+    init_skill.py my-skill --path skills/public --interface short_description="Short UI label"
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+from generate_openai_yaml import write_openai_yaml
+
+MAX_SKILL_NAME_LENGTH = 64
+ALLOWED_RESOURCES = {"scripts", "references", "assets"}
+
+SKILL_TEMPLATE = """---
+name: {skill_name}
+description: [TODO: Complete and informative explanation of what the skill does and when to use it. Include WHEN to use this skill - specific scenarios, file types, or tasks that trigger it.]
+---
+
+# {skill_title}
+
+## Overview
+
+[TODO: 1-2 sentences explaining what this skill enables]
+
+## Structuring This Skill
+
+[TODO: Choose the structure that best fits this skill's purpose. Common patterns:
+
+**1. Workflow-Based** (best for sequential processes)
+- Works well when there are clear step-by-step procedures
+- Example: DOCX skill with "Workflow Decision Tree" -> "Reading" -> "Creating" -> "Editing"
+- Structure: ## Overview -> ## Workflow Decision Tree -> ## Step 1 -> ## Step 2...
+
+**2. Task-Based** (best for tool collections)
+- Works well when the skill offers different operations/capabilities
+- Example: PDF skill with "Quick Start" -> "Merge PDFs" -> "Split PDFs" -> "Extract Text"
+- Structure: ## Overview -> ## Quick Start -> ## Task Category 1 -> ## Task Category 2...
+
+**3. Reference/Guidelines** (best for standards or specifications)
+- Works well for brand guidelines, coding standards, or requirements
+- Example: Brand styling with "Brand Guidelines" -> "Colors" -> "Typography" -> "Features"
+- Structure: ## Overview -> ## Guidelines -> ## Specifications -> ## Usage...
+
+**4. Capabilities-Based** (best for integrated systems)
+- Works well when the skill provides multiple interrelated features
+- Example: Product Management with "Core Capabilities" -> numbered capability list
+- Structure: ## Overview -> ## Core Capabilities -> ### 1. Feature -> ### 2. Feature...
+
+Patterns can be mixed and matched as needed. Most skills combine patterns (e.g., start with task-based, add workflow for complex operations).
+
+Delete this entire "Structuring This Skill" section when done - it's just guidance.]
+
+## [TODO: Replace with the first main section based on chosen structure]
+
+[TODO: Add content here. See examples in existing skills:
+- Code samples for technical skills
+- Decision trees for complex workflows
+- Concrete examples with realistic user requests
+- References to scripts/templates/references as needed]
+
+## Resources (optional)
+
+Create only the resource directories this skill actually needs. Delete this section if no resources are required.
+
+### scripts/
+Executable code (Python/Bash/etc.) that can be run directly to perform specific operations.
+
+**Examples from other skills:**
+- PDF skill: `fill_fillable_fields.py`, `extract_form_field_info.py` - utilities for PDF manipulation
+- DOCX skill: `document.py`, `utilities.py` - Python modules for document processing
+
+**Appropriate for:** Python scripts, shell scripts, or any executable code that performs automation, data processing, or specific operations.
+
+**Note:** Scripts may be executed without loading into context, but can still be read by Codex for patching or environment adjustments.
+
+### references/
+Documentation and reference material intended to be loaded into context to inform Codex's process and thinking.
+
+**Examples from other skills:**
+- Product management: `communication.md`, `context_building.md` - detailed workflow guides
+- BigQuery: API reference documentation and query examples
+- Finance: Schema documentation, company policies
+
+**Appropriate for:** In-depth documentation, API references, database schemas, comprehensive guides, or any detailed information that Codex should reference while working.
+
+### assets/
+Files not intended to be loaded into context, but rather used within the output Codex produces.
+
+**Examples from other skills:**
+- Brand styling: PowerPoint template files (.pptx), logo files
+- Frontend builder: HTML/React boilerplate project directories
+- Typography: Font files (.ttf, .woff2)
+
+**Appropriate for:** Templates, boilerplate code, document templates, images, icons, fonts, or any files meant to be copied or used in the final output.
+
+---
+
+**Not every skill requires all three types of resources.**
+"""
+
+EXAMPLE_SCRIPT = '''#!/usr/bin/env python3
+"""
+Example helper script for {skill_name}
+
+This is a placeholder script that can be executed directly.
+Replace with actual implementation or delete if not needed.
+
+Example real scripts from other skills:
+- pdf/scripts/fill_fillable_fields.py - Fills PDF form fields
+- pdf/scripts/convert_pdf_to_images.py - Converts PDF pages to images
+"""
+
+def main():
+    print("This is an example script for {skill_name}")
+    # TODO: Add actual script logic here
+    # This could be data processing, file conversion, API calls, etc.
+
+if __name__ == "__main__":
+    main()
+'''
+
+EXAMPLE_REFERENCE = """# Reference Documentation for {skill_title}
+
+This is a placeholder for detailed reference documentation.
+Replace with actual reference content or delete if not needed.
+
+Example real reference docs from other skills:
+- product-management/references/communication.md - Comprehensive guide for status updates
+- product-management/references/context_building.md - Deep-dive on gathering context
+- bigquery/references/ - API references and query examples
+
+## When Reference Docs Are Useful
+
+Reference docs are ideal for:
+- Comprehensive API documentation
+- Detailed workflow guides
+- Complex multi-step processes
+- Information too lengthy for main SKILL.md
+- Content that's only needed for specific use cases
+
+## Structure Suggestions
+
+### API Reference Example
+- Overview
+- Authentication
+- Endpoints with examples
+- Error codes
+- Rate limits
+
+### Workflow Guide Example
+- Prerequisites
+- Step-by-step instructions
+- Common patterns
+- Troubleshooting
+- Best practices
+"""
+
+EXAMPLE_ASSET = """# Example Asset File
+
+This placeholder represents where asset files would be stored.
+Replace with actual asset files (templates, images, fonts, etc.) or delete if not needed.
+
+Asset files are NOT intended to be loaded into context, but rather used within
+the output Codex produces.
+
+Example asset files from other skills:
+- Brand guidelines: logo.png, slides_template.pptx
+- Frontend builder: hello-world/ directory with HTML/React boilerplate
+- Typography: custom-font.ttf, font-family.woff2
+- Data: sample_data.csv, test_dataset.json
+
+## Common Asset Types
+
+- Templates: .pptx, .docx, boilerplate directories
+- Images: .png, .jpg, .svg, .gif
+- Fonts: .ttf, .otf, .woff, .woff2
+- Boilerplate code: Project directories, starter files
+- Icons: .ico, .svg
+- Data files: .csv, .json, .xml, .yaml
+
+Note: This is a text placeholder. Actual assets can be any file type.
+"""
+
+
+def normalize_skill_name(skill_name):
+    """Normalize a skill name to lowercase hyphen-case."""
+    normalized = skill_name.strip().lower()
+    normalized = re.sub(r"[^a-z0-9]+", "-", normalized)
+    normalized = normalized.strip("-")
+    normalized = re.sub(r"-{2,}", "-", normalized)
+    return normalized
+
+
+def title_case_skill_name(skill_name):
+    """Convert hyphenated skill name to Title Case for display."""
+    return " ".join(word.capitalize() for word in skill_name.split("-"))
+
+
+def parse_resources(raw_resources):
+    if not raw_resources:
+        return []
+    resources = [item.strip() for item in raw_resources.split(",") if item.strip()]
+    invalid = sorted({item for item in resources if item not in ALLOWED_RESOURCES})
+    if invalid:
+        allowed = ", ".join(sorted(ALLOWED_RESOURCES))
+        print(f"[ERROR] Unknown resource type(s): {', '.join(invalid)}")
+        print(f"   Allowed: {allowed}")
+        sys.exit(1)
+    deduped = []
+    seen = set()
+    for resource in resources:
+        if resource not in seen:
+            deduped.append(resource)
+            seen.add(resource)
+    return deduped
+
+
+def create_resource_dirs(skill_dir, skill_name, skill_title, resources, include_examples):
+    for resource in resources:
+        resource_dir = skill_dir / resource
+        resource_dir.mkdir(exist_ok=True)
+        if resource == "scripts":
+            if include_examples:
+                example_script = resource_dir / "example.py"
+                example_script.write_text(EXAMPLE_SCRIPT.format(skill_name=skill_name))
+                example_script.chmod(0o755)
+                print("[OK] Created scripts/example.py")
+            else:
+                print("[OK] Created scripts/")
+        elif resource == "references":
+            if include_examples:
+                example_reference = resource_dir / "api_reference.md"
+                example_reference.write_text(EXAMPLE_REFERENCE.format(skill_title=skill_title))
+                print("[OK] Created references/api_reference.md")
+            else:
+                print("[OK] Created references/")
+        elif resource == "assets":
+            if include_examples:
+                example_asset = resource_dir / "example_asset.txt"
+                example_asset.write_text(EXAMPLE_ASSET)
+                print("[OK] Created assets/example_asset.txt")
+            else:
+                print("[OK] Created assets/")
+
+
+def init_skill(skill_name, path, resources, include_examples, interface_overrides):
+    """
+    Initialize a new skill directory with template SKILL.md.
+
+    Args:
+        skill_name: Name of the skill
+        path: Path where the skill directory should be created
+        resources: Resource directories to create
+        include_examples: Whether to create example files in resource directories
+
+    Returns:
+        Path to created skill directory, or None if error
+    """
+    # Determine skill directory path
+    skill_dir = Path(path).resolve() / skill_name
+
+    # Check if directory already exists
+    if skill_dir.exists():
+        print(f"[ERROR] Skill directory already exists: {skill_dir}")
+        return None
+
+    # Create skill directory
+    try:
+        skill_dir.mkdir(parents=True, exist_ok=False)
+        print(f"[OK] Created skill directory: {skill_dir}")
+    except Exception as e:
+        print(f"[ERROR] Error creating directory: {e}")
+        return None
+
+    # Create SKILL.md from template
+    skill_title = title_case_skill_name(skill_name)
+    skill_content = SKILL_TEMPLATE.format(skill_name=skill_name, skill_title=skill_title)
+
+    skill_md_path = skill_dir / "SKILL.md"
+    try:
+        skill_md_path.write_text(skill_content)
+        print("[OK] Created SKILL.md")
+    except Exception as e:
+        print(f"[ERROR] Error creating SKILL.md: {e}")
+        return None
+
+    # Create agents/openai.yaml
+    try:
+        result = write_openai_yaml(skill_dir, skill_name, interface_overrides)
+        if not result:
+            return None
+    except Exception as e:
+        print(f"[ERROR] Error creating agents/openai.yaml: {e}")
+        return None
+
+    # Create resource directories if requested
+    if resources:
+        try:
+            create_resource_dirs(skill_dir, skill_name, skill_title, resources, include_examples)
+        except Exception as e:
+            print(f"[ERROR] Error creating resource directories: {e}")
+            return None
+
+    # Print next steps
+    print(f"\n[OK] Skill '{skill_name}' initialized successfully at {skill_dir}")
+    print("\nNext steps:")
+    print("1. Edit SKILL.md to complete the TODO items and update the description")
+    if resources:
+        if include_examples:
+            print("2. Customize or delete the example files in scripts/, references/, and assets/")
+        else:
+            print("2. Add resources to scripts/, references/, and assets/ as needed")
+    else:
+        print("2. Create resource directories only if needed (scripts/, references/, assets/)")
+    print("3. Update agents/openai.yaml if the UI metadata should differ")
+    print("4. Run the validator when ready to check the skill structure")
+    print(
+        "5. Forward-test complex skills with realistic user requests to ensure they work as intended"
+    )
+
+    return skill_dir
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Create a new skill directory with a SKILL.md template.",
+    )
+    parser.add_argument("skill_name", help="Skill name (normalized to hyphen-case)")
+    parser.add_argument("--path", required=True, help="Output directory for the skill")
+    parser.add_argument(
+        "--resources",
+        default="",
+        help="Comma-separated list: scripts,references,assets",
+    )
+    parser.add_argument(
+        "--examples",
+        action="store_true",
+        help="Create example files inside the selected resource directories",
+    )
+    parser.add_argument(
+        "--interface",
+        action="append",
+        default=[],
+        help="Interface override in key=value format (repeatable)",
+    )
+    args = parser.parse_args()
+
+    raw_skill_name = args.skill_name
+    skill_name = normalize_skill_name(raw_skill_name)
+    if not skill_name:
+        print("[ERROR] Skill name must include at least one letter or digit.")
+        sys.exit(1)
+    if len(skill_name) > MAX_SKILL_NAME_LENGTH:
+        print(
+            f"[ERROR] Skill name '{skill_name}' is too long ({len(skill_name)} characters). "
+            f"Maximum is {MAX_SKILL_NAME_LENGTH} characters."
+        )
+        sys.exit(1)
+    if skill_name != raw_skill_name:
+        print(f"Note: Normalized skill name from '{raw_skill_name}' to '{skill_name}'.")
+
+    resources = parse_resources(args.resources)
+    if args.examples and not resources:
+        print("[ERROR] --examples requires --resources to be set.")
+        sys.exit(1)
+
+    path = args.path
+
+    print(f"Initializing skill: {skill_name}")
+    print(f"   Location: {path}")
+    if resources:
+        print(f"   Resources: {', '.join(resources)}")
+        if args.examples:
+            print("   Examples: enabled")
+    else:
+        print("   Resources: none (create as needed)")
+    print()
+
+    result = init_skill(skill_name, path, resources, args.examples, args.interface)
+
+    if result:
+        sys.exit(0)
+    else:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

skills/.system/skill-creator/scripts/quick_validate.py

@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+"""
+Quick validation script for skills - minimal version
+"""
+
+import re
+import sys
+from pathlib import Path
+
+import yaml
+
+MAX_SKILL_NAME_LENGTH = 64
+
+
+def validate_skill(skill_path):
+    """Basic validation of a skill"""
+    skill_path = Path(skill_path)
+
+    skill_md = skill_path / "SKILL.md"
+    if not skill_md.exists():
+        return False, "SKILL.md not found"
+
+    content = skill_md.read_text()
+    if not content.startswith("---"):
+        return False, "No YAML frontmatter found"
+
+    match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
+    if not match:
+        return False, "Invalid frontmatter format"
+
+    frontmatter_text = match.group(1)
+
+    try:
+        frontmatter = yaml.safe_load(frontmatter_text)
+        if not isinstance(frontmatter, dict):
+            return False, "Frontmatter must be a YAML dictionary"
+    except yaml.YAMLError as e:
+        return False, f"Invalid YAML in frontmatter: {e}"
+
+    allowed_properties = {"name", "description", "license", "allowed-tools", "metadata"}
+
+    unexpected_keys = set(frontmatter.keys()) - allowed_properties
+    if unexpected_keys:
+        allowed = ", ".join(sorted(allowed_properties))
+        unexpected = ", ".join(sorted(unexpected_keys))
+        return (
+            False,
+            f"Unexpected key(s) in SKILL.md frontmatter: {unexpected}. Allowed properties are: {allowed}",
+        )
+
+    if "name" not in frontmatter:
+        return False, "Missing 'name' in frontmatter"
+    if "description" not in frontmatter:
+        return False, "Missing 'description' in frontmatter"
+
+    name = frontmatter.get("name", "")
+    if not isinstance(name, str):
+        return False, f"Name must be a string, got {type(name).__name__}"
+    name = name.strip()
+    if name:
+        if not re.match(r"^[a-z0-9-]+$", name):
+            return (
+                False,
+                f"Name '{name}' should be hyphen-case (lowercase letters, digits, and hyphens only)",
+            )
+        if name.startswith("-") or name.endswith("-") or "--" in name:
+            return (
+                False,
+                f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens",
+            )
+        if len(name) > MAX_SKILL_NAME_LENGTH:
+            return (
+                False,
+                f"Name is too long ({len(name)} characters). "
+                f"Maximum is {MAX_SKILL_NAME_LENGTH} characters.",
+            )
+
+    description = frontmatter.get("description", "")
+    if not isinstance(description, str):
+        return False, f"Description must be a string, got {type(description).__name__}"
+    description = description.strip()
+    if description:
+        if "<" in description or ">" in description:
+            return False, "Description cannot contain angle brackets (< or >)"
+        if len(description) > 1024:
+            return (
+                False,
+                f"Description is too long ({len(description)} characters). Maximum is 1024 characters.",
+            )
+
+    return True, "Skill is valid!"
+
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print("Usage: python quick_validate.py <skill_directory>")
+        sys.exit(1)
+
+    valid, message = validate_skill(sys.argv[1])
+    print(message)
+    sys.exit(0 if valid else 1)

skills/.system/skill-installer/LICENSE.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

skills/.system/skill-installer/SKILL.md

@@ -0,0 +1,58 @@
+---
+name: skill-installer
+description: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos).
+metadata:
+  short-description: Install curated skills from openai/skills or other repos
+---
+
+# Skill Installer
+
+Helps install skills. By default these are from https://github.com/openai/skills/tree/main/skills/.curated, but users can also provide other locations. Experimental skills live in https://github.com/openai/skills/tree/main/skills/.experimental and can be installed the same way.
+
+Use the helper scripts based on the task:
+- List skills when the user asks what is available, or if the user uses this skill without specifying what to do. Default listing is `.curated`, but you can pass `--path skills/.experimental` when they ask about experimental skills.
+- Install from the curated list when the user provides a skill name.
+- Install from another repo when the user provides a GitHub repo/path (including private repos).
+
+Install skills with the helper scripts.
+
+## Communication
+
+When listing skills, output approximately as follows, depending on the context of the user's request. If they ask about experimental skills, list from `.experimental` instead of `.curated` and label the source accordingly:
+"""
+Skills from {repo}:
+1. skill-1
+2. skill-2 (already installed)
+3. ...
+Which ones would you like installed?
+"""
+
+After installing a skill, tell the user: "Restart Codex to pick up new skills."
+
+## Scripts
+
+All of these scripts use network, so when running in the sandbox, request escalation when running them.
+
+- `scripts/list-skills.py` (prints skills list with installed annotations)
+- `scripts/list-skills.py --format json`
+- Example (experimental list): `scripts/list-skills.py --path skills/.experimental`
+- `scripts/install-skill-from-github.py --repo <owner>/<repo> --path <path/to/skill> [<path/to/skill> ...]`
+- `scripts/install-skill-from-github.py --url https://github.com/<owner>/<repo>/tree/<ref>/<path>`
+- Example (experimental skill): `scripts/install-skill-from-github.py --repo openai/skills --path skills/.experimental/<skill-name>`
+
+## Behavior and Options
+
+- Defaults to direct download for public GitHub repos.
+- If download fails with auth/permission errors, falls back to git sparse checkout.
+- Aborts if the destination skill directory already exists.
+- Installs into `$CODEX_HOME/skills/<skill-name>` (defaults to `~/.codex/skills`).
+- Multiple `--path` values install multiple skills in one run, each named from the path basename unless `--name` is supplied.
+- Options: `--ref <ref>` (default `main`), `--dest <path>`, `--method auto|download|git`.
+
+## Notes
+
+- Curated listing is fetched from `https://github.com/openai/skills/tree/main/skills/.curated` via the GitHub API. If it is unavailable, explain the error and exit.
+- Private GitHub repos can be accessed via existing git credentials or optional `GITHUB_TOKEN`/`GH_TOKEN` for download.
+- Git fallback tries HTTPS first, then SSH.
+- The skills at https://github.com/openai/skills/tree/main/skills/.system are preinstalled, so no need to help users install those. If they ask, just explain this. If they insist, you can download and overwrite.
+- Installed annotations come from `$CODEX_HOME/skills`.

skills/.system/skill-installer/agents/openai.yaml

@@ -0,0 +1,5 @@
+interface:
+  display_name: "Skill Installer"
+  short_description: "Install curated skills from openai/skills or other repos"
+  icon_small: "./assets/skill-installer-small.svg"
+  icon_large: "./assets/skill-installer.png"

skills/.system/skill-installer/scripts/github_utils.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+"""Shared GitHub helpers for skill install scripts."""
+
+from __future__ import annotations
+
+import os
+import urllib.request
+
+
+def github_request(url: str, user_agent: str) -> bytes:
+    headers = {"User-Agent": user_agent}
+    token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN")
+    if token:
+        headers["Authorization"] = f"token {token}"
+    req = urllib.request.Request(url, headers=headers)
+    with urllib.request.urlopen(req) as resp:
+        return resp.read()
+
+
+def github_api_contents_url(repo: str, path: str, ref: str) -> str:
+    return f"https://api.github.com/repos/{repo}/contents/{path}?ref={ref}"

skills/.system/skill-installer/scripts/install-skill-from-github.py

@@ -0,0 +1,308 @@
+#!/usr/bin/env python3
+"""Install a skill from a GitHub repo path into $CODEX_HOME/skills."""
+
+from __future__ import annotations
+
+import argparse
+from dataclasses import dataclass
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import urllib.error
+import urllib.parse
+import zipfile
+
+from github_utils import github_request
+DEFAULT_REF = "main"
+
+
+@dataclass
+class Args:
+    url: str | None = None
+    repo: str | None = None
+    path: list[str] | None = None
+    ref: str = DEFAULT_REF
+    dest: str | None = None
+    name: str | None = None
+    method: str = "auto"
+
+
+@dataclass
+class Source:
+    owner: str
+    repo: str
+    ref: str
+    paths: list[str]
+    repo_url: str | None = None
+
+
+class InstallError(Exception):
+    pass
+
+
+def _codex_home() -> str:
+    return os.environ.get("CODEX_HOME", os.path.expanduser("~/.codex"))
+
+
+def _tmp_root() -> str:
+    base = os.path.join(tempfile.gettempdir(), "codex")
+    os.makedirs(base, exist_ok=True)
+    return base
+
+
+def _request(url: str) -> bytes:
+    return github_request(url, "codex-skill-install")
+
+
+def _parse_github_url(url: str, default_ref: str) -> tuple[str, str, str, str | None]:
+    parsed = urllib.parse.urlparse(url)
+    if parsed.netloc != "github.com":
+        raise InstallError("Only GitHub URLs are supported for download mode.")
+    parts = [p for p in parsed.path.split("/") if p]
+    if len(parts) < 2:
+        raise InstallError("Invalid GitHub URL.")
+    owner, repo = parts[0], parts[1]
+    ref = default_ref
+    subpath = ""
+    if len(parts) > 2:
+        if parts[2] in ("tree", "blob"):
+            if len(parts) < 4:
+                raise InstallError("GitHub URL missing ref or path.")
+            ref = parts[3]
+            subpath = "/".join(parts[4:])
+        else:
+            subpath = "/".join(parts[2:])
+    return owner, repo, ref, subpath or None
+
+
+def _download_repo_zip(owner: str, repo: str, ref: str, dest_dir: str) -> str:
+    zip_url = f"https://codeload.github.com/{owner}/{repo}/zip/{ref}"
+    zip_path = os.path.join(dest_dir, "repo.zip")
+    try:
+        payload = _request(zip_url)
+    except urllib.error.HTTPError as exc:
+        raise InstallError(f"Download failed: HTTP {exc.code}") from exc
+    with open(zip_path, "wb") as file_handle:
+        file_handle.write(payload)
+    with zipfile.ZipFile(zip_path, "r") as zip_file:
+        _safe_extract_zip(zip_file, dest_dir)
+        top_levels = {name.split("/")[0] for name in zip_file.namelist() if name}
+    if not top_levels:
+        raise InstallError("Downloaded archive was empty.")
+    if len(top_levels) != 1:
+        raise InstallError("Unexpected archive layout.")
+    return os.path.join(dest_dir, next(iter(top_levels)))
+
+
+def _run_git(args: list[str]) -> None:
+    result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+    if result.returncode != 0:
+        raise InstallError(result.stderr.strip() or "Git command failed.")
+
+
+def _safe_extract_zip(zip_file: zipfile.ZipFile, dest_dir: str) -> None:
+    dest_root = os.path.realpath(dest_dir)
+    for info in zip_file.infolist():
+        extracted_path = os.path.realpath(os.path.join(dest_dir, info.filename))
+        if extracted_path == dest_root or extracted_path.startswith(dest_root + os.sep):
+            continue
+        raise InstallError("Archive contains files outside the destination.")
+    zip_file.extractall(dest_dir)
+
+
+def _validate_relative_path(path: str) -> None:
+    if os.path.isabs(path) or os.path.normpath(path).startswith(".."):
+        raise InstallError("Skill path must be a relative path inside the repo.")
+
+
+def _validate_skill_name(name: str) -> None:
+    altsep = os.path.altsep
+    if not name or os.path.sep in name or (altsep and altsep in name):
+        raise InstallError("Skill name must be a single path segment.")
+    if name in (".", ".."):
+        raise InstallError("Invalid skill name.")
+
+
+def _git_sparse_checkout(repo_url: str, ref: str, paths: list[str], dest_dir: str) -> str:
+    repo_dir = os.path.join(dest_dir, "repo")
+    clone_cmd = [
+        "git",
+        "clone",
+        "--filter=blob:none",
+        "--depth",
+        "1",
+        "--sparse",
+        "--single-branch",
+        "--branch",
+        ref,
+        repo_url,
+        repo_dir,
+    ]
+    try:
+        _run_git(clone_cmd)
+    except InstallError:
+        _run_git(
+            [
+                "git",
+                "clone",
+                "--filter=blob:none",
+                "--depth",
+                "1",
+                "--sparse",
+                "--single-branch",
+                repo_url,
+                repo_dir,
+            ]
+        )
+    _run_git(["git", "-C", repo_dir, "sparse-checkout", "set", *paths])
+    _run_git(["git", "-C", repo_dir, "checkout", ref])
+    return repo_dir
+
+
+def _validate_skill(path: str) -> None:
+    if not os.path.isdir(path):
+        raise InstallError(f"Skill path not found: {path}")
+    skill_md = os.path.join(path, "SKILL.md")
+    if not os.path.isfile(skill_md):
+        raise InstallError("SKILL.md not found in selected skill directory.")
+
+
+def _copy_skill(src: str, dest_dir: str) -> None:
+    os.makedirs(os.path.dirname(dest_dir), exist_ok=True)
+    if os.path.exists(dest_dir):
+        raise InstallError(f"Destination already exists: {dest_dir}")
+    shutil.copytree(src, dest_dir)
+
+
+def _build_repo_url(owner: str, repo: str) -> str:
+    return f"https://github.com/{owner}/{repo}.git"
+
+
+def _build_repo_ssh(owner: str, repo: str) -> str:
+    return f"git@github.com:{owner}/{repo}.git"
+
+
+def _prepare_repo(source: Source, method: str, tmp_dir: str) -> str:
+    if method in ("download", "auto"):
+        try:
+            return _download_repo_zip(source.owner, source.repo, source.ref, tmp_dir)
+        except InstallError as exc:
+            if method == "download":
+                raise
+            err_msg = str(exc)
+            if "HTTP 401" in err_msg or "HTTP 403" in err_msg or "HTTP 404" in err_msg:
+                pass
+            else:
+                raise
+    if method in ("git", "auto"):
+        repo_url = source.repo_url or _build_repo_url(source.owner, source.repo)
+        try:
+            return _git_sparse_checkout(repo_url, source.ref, source.paths, tmp_dir)
+        except InstallError:
+            repo_url = _build_repo_ssh(source.owner, source.repo)
+            return _git_sparse_checkout(repo_url, source.ref, source.paths, tmp_dir)
+    raise InstallError("Unsupported method.")
+
+
+def _resolve_source(args: Args) -> Source:
+    if args.url:
+        owner, repo, ref, url_path = _parse_github_url(args.url, args.ref)
+        if args.path is not None:
+            paths = list(args.path)
+        elif url_path:
+            paths = [url_path]
+        else:
+            paths = []
+        if not paths:
+            raise InstallError("Missing --path for GitHub URL.")
+        return Source(owner=owner, repo=repo, ref=ref, paths=paths)
+
+    if not args.repo:
+        raise InstallError("Provide --repo or --url.")
+    if "://" in args.repo:
+        return _resolve_source(
+            Args(url=args.repo, repo=None, path=args.path, ref=args.ref)
+        )
+
+    repo_parts = [p for p in args.repo.split("/") if p]
+    if len(repo_parts) != 2:
+        raise InstallError("--repo must be in owner/repo format.")
+    if not args.path:
+        raise InstallError("Missing --path for --repo.")
+    paths = list(args.path)
+    return Source(
+        owner=repo_parts[0],
+        repo=repo_parts[1],
+        ref=args.ref,
+        paths=paths,
+    )
+
+
+def _default_dest() -> str:
+    return os.path.join(_codex_home(), "skills")
+
+
+def _parse_args(argv: list[str]) -> Args:
+    parser = argparse.ArgumentParser(description="Install a skill from GitHub.")
+    parser.add_argument("--repo", help="owner/repo")
+    parser.add_argument("--url", help="https://github.com/owner/repo[/tree/ref/path]")
+    parser.add_argument(
+        "--path",
+        nargs="+",
+        help="Path(s) to skill(s) inside repo",
+    )
+    parser.add_argument("--ref", default=DEFAULT_REF)
+    parser.add_argument("--dest", help="Destination skills directory")
+    parser.add_argument(
+        "--name", help="Destination skill name (defaults to basename of path)"
+    )
+    parser.add_argument(
+        "--method",
+        choices=["auto", "download", "git"],
+        default="auto",
+    )
+    return parser.parse_args(argv, namespace=Args())
+
+
+def main(argv: list[str]) -> int:
+    args = _parse_args(argv)
+    try:
+        source = _resolve_source(args)
+        source.ref = source.ref or args.ref
+        if not source.paths:
+            raise InstallError("No skill paths provided.")
+        for path in source.paths:
+            _validate_relative_path(path)
+        dest_root = args.dest or _default_dest()
+        tmp_dir = tempfile.mkdtemp(prefix="skill-install-", dir=_tmp_root())
+        try:
+            repo_root = _prepare_repo(source, args.method, tmp_dir)
+            installed = []
+            for path in source.paths:
+                skill_name = args.name if len(source.paths) == 1 else None
+                skill_name = skill_name or os.path.basename(path.rstrip("/"))
+                _validate_skill_name(skill_name)
+                if not skill_name:
+                    raise InstallError("Unable to derive skill name.")
+                dest_dir = os.path.join(dest_root, skill_name)
+                if os.path.exists(dest_dir):
+                    raise InstallError(f"Destination already exists: {dest_dir}")
+                skill_src = os.path.join(repo_root, path)
+                _validate_skill(skill_src)
+                _copy_skill(skill_src, dest_dir)
+                installed.append((skill_name, dest_dir))
+        finally:
+            if os.path.isdir(tmp_dir):
+                shutil.rmtree(tmp_dir, ignore_errors=True)
+        for skill_name, dest_dir in installed:
+            print(f"Installed {skill_name} to {dest_dir}")
+        return 0
+    except InstallError as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

skills/.system/skill-installer/scripts/list-skills.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+"""List skills from a GitHub repo path."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import urllib.error
+
+from github_utils import github_api_contents_url, github_request
+
+DEFAULT_REPO = "openai/skills"
+DEFAULT_PATH = "skills/.curated"
+DEFAULT_REF = "main"
+
+
+class ListError(Exception):
+    pass
+
+
+class Args(argparse.Namespace):
+    repo: str
+    path: str
+    ref: str
+    format: str
+
+
+def _request(url: str) -> bytes:
+    return github_request(url, "codex-skill-list")
+
+
+def _codex_home() -> str:
+    return os.environ.get("CODEX_HOME", os.path.expanduser("~/.codex"))
+
+
+def _installed_skills() -> set[str]:
+    root = os.path.join(_codex_home(), "skills")
+    if not os.path.isdir(root):
+        return set()
+    entries = set()
+    for name in os.listdir(root):
+        path = os.path.join(root, name)
+        if os.path.isdir(path):
+            entries.add(name)
+    return entries
+
+
+def _list_skills(repo: str, path: str, ref: str) -> list[str]:
+    api_url = github_api_contents_url(repo, path, ref)
+    try:
+        payload = _request(api_url)
+    except urllib.error.HTTPError as exc:
+        if exc.code == 404:
+            raise ListError(
+                "Skills path not found: "
+                f"https://github.com/{repo}/tree/{ref}/{path}"
+            ) from exc
+        raise ListError(f"Failed to fetch skills: HTTP {exc.code}") from exc
+    data = json.loads(payload.decode("utf-8"))
+    if not isinstance(data, list):
+        raise ListError("Unexpected skills listing response.")
+    skills = [item["name"] for item in data if item.get("type") == "dir"]
+    return sorted(skills)
+
+
+def _parse_args(argv: list[str]) -> Args:
+    parser = argparse.ArgumentParser(description="List skills.")
+    parser.add_argument("--repo", default=DEFAULT_REPO)
+    parser.add_argument(
+        "--path",
+        default=DEFAULT_PATH,
+        help="Repo path to list (default: skills/.curated)",
+    )
+    parser.add_argument("--ref", default=DEFAULT_REF)
+    parser.add_argument(
+        "--format",
+        choices=["text", "json"],
+        default="text",
+        help="Output format",
+    )
+    return parser.parse_args(argv, namespace=Args())
+
+
+def main(argv: list[str]) -> int:
+    args = _parse_args(argv)
+    try:
+        skills = _list_skills(args.repo, args.path, args.ref)
+        installed = _installed_skills()
+        if args.format == "json":
+            payload = [
+                {"name": name, "installed": name in installed} for name in skills
+            ]
+            print(json.dumps(payload))
+        else:
+            for idx, name in enumerate(skills, start=1):
+                suffix = " (already installed)" if name in installed else ""
+                print(f"{idx}. {name}{suffix}")
+        return 0
+    except ListError as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

skills/agent-kernel/SKILL.md

@@ -0,0 +1,379 @@
+---
+name: agentkernel
+description: >
+  Spawn and orchestrate agents as local subprocesses or Kubernetes pods.
+  Each agent runs with an independent runtime, conversation, tools,
+  and skills. Use when a task benefits from parallel work, role
+  specialization, persistent agent state, or sandboxed execution.
+metadata:
+  version: "2.1"
+  pre-condition: "0"
+---
+
+# AgentKernel
+
+Spawn and orchestrate agents from `<helpers>` blocks. Each agent runs in its own process (local subprocess or Kubernetes pod) with an independent runtime, conversation state, tools, and skills. You decide what agents to create and what to say to them. The kernel handles process lifecycle, networking, image management, and health checks.
+
+## AgentKernel vs `agents` Skill
+
+| | `agents` skill | `agentkernel` |
+|---|---|---|
+| **Backends** | Local subprocesses only | Local subprocesses or k8s pods |
+| **Addressing** | By name (`call_async("my-agent", ...)`) | By UUID + secret nonce |
+| **Protocol** | Anthropic Messages API | Custom SSE (TurnRequest/TurnResponse) |
+| **Access control** | Open — any caller can talk to any agent | Nonce-secured single-owner |
+| **Teams / capacity** | No | Yes |
+| **Image packaging** | No | Yes (OCI images for k8s) |
+| **AgentBus** | No | Yes |
+| **Dependencies** | API server skill | None |
+
+**Use `agents`** for lightweight local agent workflows where convenience matters — create by name, call by name, check event logs.
+
+**Use `agentkernel`** when you need k8s deployment, container isolation, capacity management, nonce-secured access, or agentbus observability.
+
+## Core Concepts
+
+**Kernel**: `AgentKernel` is the entry point. It wires together the spawner, agent client, and storage. The backend determines where agents run.
+
+**Backends**: Two backends are available:
+- **local** — agents run as subprocesses on the same machine. No isolation, no config file needed. Good for development and quick experiments.
+- **kubernetes** — agents run as pods in a k8s cluster. Full container isolation. Requires a config file with cluster details.
+
+The entire API after initialization is identical across backends.
+
+**SpawnRequest + SpawnInfo**: A `SpawnRequest` defines the agent identity (name, team, metadata). The `spawn_info` field carries agent-type-specific config (system prompt, model, tools, etc.) — e.g. `OpenClawSpawnInfo`.
+
+**Nonce**: Each spawn returns a `SpawnResult` containing the agent record and a secret nonce. The nonce is required for all communication — it enforces single-owner access. Only the entity that spawned an agent can talk to it.
+
+**AgentBus**: Optional observability/safety layer. When enabled, all LLM inference and code execution events are logged to an agent bus that can be inspected externally.
+
+**Teams**: Logical groups with capacity limits. Spawning into a full team raises an error.
+
+## Initialization
+
+### Local backend
+
+No config file needed. Agents run as subprocesses with the same permissions as the parent process.
+
+<helpers>
+from agentic.kernel import AgentKernel
+from agentic.kernel.plugins.openclaw import OpenClawPlugin
+
+kernel = AgentKernel(backend="local", plugins=[OpenClawPlugin()])
+</helpers>
+
+### Kubernetes backend
+
+Agents run as pods in the `agentkernel-0` namespace. The config file is at `agentkernel/examples/agentkernel.yaml`:
+
+```yaml
+backend: kubernetes
+namespace: agentkernel-0
+base_image: your-registry.example.com/agentkernel:latest
+kubeconfig: ~/.kube/config
+registry_url: your-registry.example.com
+debug: true
+```
+
+- `debug: true` preserves pods on failure for inspection (otherwise they are cleaned up automatically).
+
+<helpers>
+from agentic.kernel import AgentKernel
+from agentic.kernel.plugins.openclaw import OpenClawPlugin
+
+kernel = AgentKernel.from_config("agentkernel/examples/agentkernel.yaml", plugins=[OpenClawPlugin()])
+</helpers>
+
+## API
+
+All API calls below work identically regardless of backend.
+
+### Spawn an Agent
+
+<helpers>
+import os
+from agentic.kernel import SpawnRequest
+from agentic.kernel.plugins.openclaw import OpenClawSpawnInfo
+
+result = await kernel.spawner.spawn(SpawnRequest(
+    name="researcher",
+    agent_type="openclaw",
+    metadata={"role": "research"},
+    spawn_info=OpenClawSpawnInfo(
+        system_prompt="You are a research specialist. Be thorough and cite sources.",
+        model="claude-sonnet-4-5",
+        api_key=os.environ.get("LLM_API_KEY", ""),
+    ),
+))
+
+agent = result.agent   # Agent(id, name, team_id, state, metadata, ...)
+nonce = result.nonce    # Secret — required for all communication
+print(f"Spawned: {agent.id} ({agent.name})")
+</helpers>
+
+`SpawnRequest` fields:
+- `name` — agent name (also used in k8s pod naming)
+- `team_id` — team for capacity tracking (optional, default: "")
+- `metadata` — arbitrary labels for discovery (e.g. `{"role": "worker"}`)
+- `image_id` — custom image from packaging (optional, defaults to base_image in k8s)
+- `spawn_info` — agent-type-specific config (e.g. `OpenClawSpawnInfo`)
+- `env` — extra environment variables forwarded to the agent process
+
+`OpenClawSpawnInfo` fields:
+- `system_prompt` — system prompt for the agent
+- `model` — LLM model name (default: `"claude-sonnet-4-5"`)
+- `provider` — LLM provider (default: `"anthropic"`)
+- `tools` — list of tool names to enable (default: `["bash"]`)
+- `thinking_level` — thinking level: `"none"`, `"low"`, `"medium"`, `"high"`
+- `api_key` — LLM API key (also forwarded from host `LLM_API_KEY` env var)
+- `base_url` — override LLM API base URL
+
+### Send a Message (Turn)
+
+Use the `ask()` helper to send a message and get the full response:
+
+<helpers>
+response = await ask(kernel, agent.id, nonce, "What are the latest findings on topic X?")
+print(response)
+</helpers>
+
+The agent maintains conversation state — subsequent turns see the full history.
+
+For manual streaming (e.g. to display progress), use `kernel.agent_client.turn()` directly — note `end=""` to avoid extra newlines between tokens:
+
+<helpers>
+import json
+from agentic.kernel import TurnRequest
+
+request = TurnRequest(
+    agent_id=agent.id,
+    nonce=nonce,
+    body=json.dumps({
+        "messages": [{"role": "user", "content": "What are the latest findings on topic X?"}]
+    }).encode(),
+)
+
+response_text = []
+async for chunk in kernel.agent_client.turn(request):
+    if chunk.body:
+        print(chunk.body, end="", flush=True)
+        response_text.append(chunk.body)
+    if chunk.error:
+        print(f"\nError: {chunk.error}")
+full_response = "".join(response_text)
+</helpers>
+
+### Get History
+
+<helpers>
+history = await kernel.agent_client.get_history(agent.id, last_n=5)
+for entry in history:
+    print(f"[{entry['role']}] {entry['content'][:100]}")
+</helpers>
+
+### Get Agent Info
+
+<helpers>
+info = await kernel.agent_client.get_info(agent.id)
+print(f"pid={info['pid']}  cwd={info['cwd']}  uid={info['uid']}")
+</helpers>
+
+### Check Status
+
+<helpers>
+statuses = await kernel.status()
+for s in statuses:
+    line = f"{s['name']}: state={s['state']} live={s['live']}"
+    if s.get('pod_phase'):       # k8s backend
+        line += f" pod={s['pod_phase']}"
+    if s.get('process_alive') is not None:  # local backend
+        line += f" process_alive={s['process_alive']}"
+    print(line)
+</helpers>
+
+### Kill an Agent
+
+<helpers>
+await kernel.spawner.kill(agent.id)
+</helpers>
+
+### Clean Up All Agents
+
+<helpers>
+await kernel.cleanup()
+</helpers>
+
+## Teams
+
+Teams reserve capacity and group agents together.
+
+<helpers>
+from agentic.kernel import CreateTeamRequest
+
+# Reserve capacity
+await kernel.spawner.create_team(CreateTeamRequest(
+    team_id="analysis-team",
+    resources={"cpu": 4},
+))
+
+# Spawn into the team
+result = await kernel.spawner.spawn(SpawnRequest(
+    name="analyst",
+    team_id="analysis-team",
+    agent_type="openclaw",
+    spawn_info=OpenClawSpawnInfo(
+        system_prompt="You are a data analyst.",
+        api_key=os.environ.get("LLM_API_KEY", ""),
+    ),
+))
+
+# Delete team (kills all agents first)
+await kernel.spawner.delete_team("analysis-team")
+</helpers>
+
+## AgentBus
+
+AgentBus adds observability and safety to agent execution. When enabled, the agent logs all LLM inference and code execution events to a bus that can be inspected via the agentbus CLI.
+
+<helpers>
+from agentic.kernel import AgentBusConfig
+
+result = await kernel.spawner.spawn(SpawnRequest(
+    name="worker",
+    agent_type="openclaw",
+    spawn_info=OpenClawSpawnInfo(
+        system_prompt="You are a helpful worker.",
+        api_key=os.environ.get("LLM_API_KEY", ""),
+    ),
+    agentbus=AgentBusConfig(
+        port=8095,
+        disable_safety=False,
+    ),
+))
+</helpers>
+
+To inspect the bus, you can use the agentbus skill.
+
+# Kubernetes backend — port-forward first
+kubectl --kubeconfig ~/.kube/config \
+    -n agentkernel-0 port-forward pod/agent-<id-prefix> 8095:8095
+# Then poll as above
+```
+
+The bus ID follows the pattern `{agent_name}.{agent_uuid}`.
+
+## Patterns
+
+### Fan-out / Fan-in
+
+Spawn specialists, query them in parallel, synthesize results.
+
+<helpers>
+import asyncio
+
+# Spawn specialists
+researcher_r = await kernel.spawner.spawn(SpawnRequest(
+    name="researcher", agent_type="openclaw", spawn_info=OpenClawSpawnInfo(
+        system_prompt="You are a research specialist.",
+        api_key=os.environ.get("LLM_API_KEY", ""),
+    ),
+))
+analyst_r = await kernel.spawner.spawn(SpawnRequest(
+    name="analyst", agent_type="openclaw", spawn_info=OpenClawSpawnInfo(
+        system_prompt="You are a data analyst.",
+        api_key=os.environ.get("LLM_API_KEY", ""),
+    ),
+))
+
+# Fan out — ask() collects streaming chunks into a single string
+research_task = asyncio.create_task(
+    ask(kernel, researcher_r.agent.id, researcher_r.nonce, "Find papers on quantum error correction")
+)
+analysis_task = asyncio.create_task(
+    ask(kernel, analyst_r.agent.id, analyst_r.nonce, "Run cost-benefit analysis on approach X")
+)
+research, analysis = await asyncio.gather(research_task, analysis_task)
+
+print(f"Research: {research[:200]}")
+print(f"Analysis: {analysis[:200]}")
+</helpers>
+
+### Pipeline
+
+One agent's output feeds the next.
+
+<helpers>
+raw_data = await ask(kernel, researcher_r.agent.id, researcher_r.nonce, "Gather data on topic X")
+analysis = await ask(kernel, analyst_r.agent.id, analyst_r.nonce, f"Analyze this data:\n{raw_data}")
+print(analysis)
+</helpers>
+
+### Image Packaging
+
+Bundle custom code into agent images. On local backend, bundles are copied to a directory. On k8s, an OCI image is built and pushed to the registry.
+
+<helpers>
+from agentic.kernel import SourceBundle
+
+# Upload code to blob storage
+helpers_uri = kernel.blob_store.upload_dir("./my_helpers/")
+
+# Build an agent image with the bundle
+job = await kernel.packaging.create_agent_image(
+    name="custom-worker",
+    bundles=[SourceBundle(uri=helpers_uri, labels={"name": "my_helpers"})],
+)
+if job.image:
+    # Spawn an agent using the custom image
+    result = await kernel.spawner.spawn(SpawnRequest(
+        name="custom-agent",
+        agent_type="openclaw",
+        image_id=job.image.id,
+        spawn_info=OpenClawSpawnInfo(
+            system_prompt="You have custom tools available.",
+            api_key=os.environ.get("LLM_API_KEY", ""),
+        ),
+    ))
+</helpers>
+
+## Lifecycle
+
+- Agents persist (as subprocesses or pods) until explicitly killed. Always clean up when done.
+- Each agent has one conversation and one owner. The nonce enforces this — only the spawner can communicate with its agent.
+- Teams have capacity limits. Spawning into a full team raises `ValueError`.
+- The `LLM_API_KEY` and `OPENAI_API_KEY` environment variables are automatically forwarded to agent processes.
+
+## Operations
+
+**Note**: If behind a proxy, configure `HTTP_PROXY`/`HTTPS_PROXY` environment variables.
+
+### Run examples locally
+
+```bash
+# Single agent, local backend (no config file needed)
+LLM_API_KEY=... uv run python -m agentkernel.examples.simple_agent
+
+# Team scenario, local backend
+LLM_API_KEY=... uv run python -m agentkernel.examples.team_scenario
+```
+
+### Run examples on Kubernetes
+
+```bash
+# run_k8s_scenario.sh runs the scenario against the configured k8s cluster
+LLM_API_KEY=... ./agentkernel/scripts/run_k8s_scenario.sh simple_agent
+LLM_API_KEY=... ./agentkernel/scripts/run_k8s_scenario.sh team_scenario
+```
+
+### Build and push the base image (k8s only)
+
+```bash
+./scripts/build_base_image.sh --force-base
+```
+
+### Clean up cluster resources (k8s only)
+
+```bash
+./agentkernel/scripts/cleanup_k8s.sh           # delete all agentkernel pods/svc/cm
+./agentkernel/scripts/cleanup_k8s.sh --dry-run  # preview what would be deleted
+```

skills/hugging-face-evaluation/SKILL.md

@@ -0,0 +1,262 @@
+---
+name: hugging-face-evaluation
+description: Add evaluation results to Hugging Face model repositories using the .eval_results/ format. Uses HF CLI for PR management and manual YAML creation.
+---
+
+# Overview
+
+This skill adds structured evaluation results to HuggingFace model repositories using the [`.eval_results/` format](https://huggingface.co/docs/hub/eval-results).
+
+**What This Enables:**
+- Results appear on model pages with benchmark links
+- Scores are aggregated into benchmark dataset leaderboards
+- Community contributions via Pull Requests
+
+# Important
+
+Evaluation PRs can only be opened on the Hugging Face Hub. They cannot be opened on the GitHub repository.
+
+# Version
+3.0.0
+
+# Workflow Overview
+
+The actual workflow uses:
+1. **HF CLI** (`hf upload`, `hf download`) for PR operations
+2. **Manual YAML creation** in `/tmp/pr-reviews/`
+3. **`check_prs.py`** script to check for existing PRs
+4. **curl** to fetch model cards and leaderboard data
+
+See `references/hf_cli_for_prs.md` for detailed CLI instructions.
+
+---
+
+# CRITICAL: Multiple Scores for One Benchmark
+
+Models can have multiple scores for the same benchmark (with/without tools). **Each variant MUST be in a separate file.**
+
+## File Naming Convention
+
+| Condition | File Name | Notes Field |
+|-----------|-----------|-------------|
+| Default (no tools) | `hle.yaml` | None (omit notes) |
+| With tools | `hle_with_tools.yaml` | `notes: "With tools"` |
+
+## Notes Field Rules
+
+1. **No tools = No notes field** - Default assumption is "without tools"
+2. **With tools = Add notes** - Only add when tools ARE used
+3. **Standardized format** - Always use `notes: "With tools"` (capital W)
+
+**CORRECT:**
+```yaml
+# hle.yaml (no tools - DEFAULT)
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 22.1
+  source:
+    url: https://huggingface.co/org/model
+    name: Model Card
+    user: username
+```
+
+```yaml
+# hle_with_tools.yaml (with tools)
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 44.9
+  source:
+    url: https://huggingface.co/org/model
+    name: Model Card
+    user: username
+  notes: "With tools"
+```
+
+**INCORRECT:**
+```yaml
+notes: "Without tools"  # Don't add notes for default
+notes: "w/ tools"       # Use standardized format
+notes: "with tools"     # Capital W required
+```
+
+---
+
+# Core Workflow
+
+## Step 1: Check for Existing PRs
+
+**ALWAYS check before creating new PRs:**
+
+```bash
+uv run scripts/check_prs.py --repo-id "org/model-name"
+```
+
+If PRs exist, update them instead of creating new ones.
+
+## Step 2: Fetch Model Card and Extract Scores
+
+```bash
+# Get model README
+curl -s "https://huggingface.co/org/model-name/raw/main/README.md" | grep -i -A10 "HLE\|GPQA\|MMLU"
+```
+
+Or use MCP tools:
+```
+mcp__hf-mcp-server__hub_repo_details
+  repo_ids: ["org/model-name"]
+  include_readme: true
+```
+
+## Step 3: Create YAML File
+
+```bash
+mkdir -p /tmp/pr-reviews/new-prs
+cd /tmp/pr-reviews/new-prs
+
+cat > hle.yaml << 'EOF'
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 22.1
+  date: '2026-02-03'
+  source:
+    url: https://huggingface.co/org/model-name
+    name: Model Card
+    user: burtenshaw
+EOF
+```
+
+## Step 4: Create PR
+
+```bash
+hf upload org/model-name hle.yaml .eval_results/hle.yaml \
+  --repo-type model --create-pr \
+  --commit-message "Add HLE evaluation result"
+```
+
+## Step 5: Get PR Number
+
+```bash
+uv run scripts/check_prs.py --repo-id "org/model-name"
+```
+
+---
+
+# Updating Existing PRs
+
+```bash
+# Download PR contents
+hf download org/model-name --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --include ".eval_results/*" \
+  --local-dir /tmp/pr-reviews/model-pr<PR_NUMBER>
+
+# Edit the YAML file, then upload
+hf upload org/model-name /tmp/pr-reviews/updated.yaml .eval_results/hle.yaml \
+  --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --commit-message "Update evaluation result"
+```
+
+---
+
+# Deleting Files from PRs
+
+Use Python API:
+```bash
+uv run --with huggingface_hub python3 << 'EOF'
+from huggingface_hub import HfApi
+api = HfApi()
+api.delete_file(
+    path_in_repo=".eval_results/old_file.yaml",
+    repo_id="org/model-name",
+    repo_type="model",
+    revision="refs/pr/<PR_NUMBER>",
+    commit_message="Remove file"
+)
+EOF
+```
+
+---
+
+# Fetching Leaderboard Data
+
+```bash
+# HLE leaderboard (requires auth for private datasets)
+curl -s "https://huggingface.co/api/datasets/cais/hle/leaderboard" \
+  -H "Authorization: Bearer $HF_TOKEN"
+
+# MMLU-Pro leaderboard (public)
+curl -s "https://huggingface.co/api/datasets/TIGER-Lab/MMLU-Pro/leaderboard"
+
+# Model eval results
+curl -s "https://huggingface.co/api/models/org/model?expand[]=evalResults"
+```
+
+---
+
+# .eval_results/ Format
+
+```yaml
+# .eval_results/hle.yaml
+- dataset:
+    id: cais/hle              # Required: Hub Benchmark dataset ID
+    task_id: hle              # Required: task id from dataset's eval.yaml
+  value: 22.2                 # Required: metric value
+  date: "2026-01-14"          # Optional: ISO-8601 date
+  source:                     # Optional: attribution
+    url: https://huggingface.co/org/model
+    name: Model Card
+    user: username
+```
+
+---
+
+# Supported Benchmarks
+
+| Benchmark | Hub Dataset ID | Task ID |
+|-----------|---------------|---------|
+| HLE | cais/hle | hle |
+| GPQA | Idavidrein/gpqa | diamond |
+| MMLU-Pro | TIGER-Lab/MMLU-Pro | mmlu_pro |
+
+---
+
+# Tool-Using Agent Models
+
+Models like MiroThinker, Nemotron-Orchestrator are inherently tool-using agents. For these:
+
+1. Use `hle_with_tools.yaml` as filename
+2. Add `notes: "With tools"`
+3. Look for terms: "search agent", "agentic", "orchestrator", "code-interpreter"
+
+---
+
+# Environment Setup
+
+```bash
+export HF_TOKEN="your-huggingface-token"
+```
+
+---
+
+# Scripts Reference
+
+```bash
+# Check for existing PRs (ALWAYS do this first)
+uv run scripts/check_prs.py --repo-id "org/model-name"
+```
+
+See `references/hf_cli_for_prs.md` for complete HF CLI workflow documentation.
+
+---
+
+# Best Practices
+
+1. **Always check for existing PRs** before creating new ones
+2. **Separate files for variants** - `hle.yaml` for default, `hle_with_tools.yaml` for tools
+3. **Notes only for non-default** - Omit notes for standard evaluations
+4. **Standardized format** - Use `"With tools"` exactly (capital W)
+5. **Verify scores** - Compare YAML against model card before submitting

skills/hugging-face-evaluation/examples/.env.example

@@ -0,0 +1,7 @@
+# Hugging Face Token (required for all operations)
+# Get your token at: https://huggingface.co/settings/tokens
+HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+# Artificial Analysis API Key (required for import-aa command)
+# Get your key at: https://artificialanalysis.ai/
+AA_API_KEY=aa_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

skills/hugging-face-evaluation/examples/USAGE_EXAMPLES.md

@@ -0,0 +1,380 @@
+# Usage Examples
+
+Practical examples for adding evaluations to HuggingFace model repositories using the `.eval_results/` format.
+
+## Table of Contents
+1. [Setup](#setup)
+2. [Add Single Benchmark (Recommended)](#add-single-benchmark-recommended)
+3. [Batch Process Trending Models](#batch-process-trending-models)
+4. [Extract from README Tables](#extract-from-readme-tables)
+5. [Import from Artificial Analysis](#import-from-artificial-analysis)
+6. [Common Workflows](#common-workflows)
+
+---
+
+## Setup
+
+### Environment Variables
+
+```bash
+# Required for creating PRs
+export HF_TOKEN="hf_your_write_token_here"
+
+# Optional: for Artificial Analysis source
+export AA_API_KEY="your_aa_api_key_here"
+```
+
+Or use a `.env` file:
+```bash
+cp examples/.env.example .env
+# Edit .env with your tokens
+```
+
+### Verify Installation
+
+```bash
+uv run scripts/evaluation_manager.py --help
+```
+
+---
+
+## Add Single Benchmark (Recommended)
+
+The simplest way to add a specific benchmark score to a model.
+
+### Basic Usage
+
+```bash
+# Preview (default - prints YAML without uploading)
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "moonshotai/Kimi-K2-Thinking"
+```
+
+Output:
+```
+Looking up HLE score for moonshotai/Kimi-K2-Thinking from model_card...
+Found: HLE = 23.9
+Generated YAML:
+- dataset:
+    id: cais/hle
+    task_id: default
+  value: 23.9
+  date: "2026-01-14"
+  source:
+    url: https://huggingface.co/moonshotai/Kimi-K2-Thinking
+    name: Model Card
+```
+
+### From Artificial Analysis
+
+```bash
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "MiniMaxAI/MiniMax-M2.1" \
+  --source aa
+```
+
+### Create PR
+
+```bash
+# Always check for existing PRs first!
+uv run scripts/evaluation_manager.py get-prs --repo-id "model/name"
+
+# If no PRs exist, create one
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "model/name" \
+  --create-pr
+```
+
+### Push Directly (Your Own Model)
+
+```bash
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark GPQA \
+  --repo-id "your-username/your-model" \
+  --apply
+```
+
+### Provide Score Manually
+
+```bash
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "model/name" \
+  --value 84.5 \
+  --create-pr
+```
+
+---
+
+## Batch Process Trending Models
+
+Process multiple trending models at once.
+
+### Preview Mode (Dry Run)
+
+```bash
+uv run scripts/batch_eval_prs.py --limit 10 --benchmark HLE --dry-run
+```
+
+Output:
+```
+==================================================
+Batch Evaluation PR Creator
+==================================================
+Benchmark: HLE
+Source: model_card
+Pipeline tag: text-generation
+Limit: 10
+Sort: trending
+Dry run: True
+==================================================
+
+Processing: LiquidAI/LFM2.5-1.2B-Instruct
+  Not found: HLE score not available
+Processing: MiniMaxAI/MiniMax-M2.1
+  Found: HLE = 22.2
+  Status: Would create PR (dry run)
+...
+
+Summary:
+Success: 3
+Not found: 7
+```
+
+### Create PRs
+
+```bash
+# From model cards
+uv run scripts/batch_eval_prs.py --limit 10 --benchmark HLE
+
+# From Artificial Analysis
+uv run scripts/batch_eval_prs.py --limit 10 --benchmark HLE --source aa
+```
+
+### Sort Options
+
+```bash
+# By downloads (more established models)
+uv run scripts/batch_eval_prs.py --limit 20 --sort downloads --benchmark GPQA
+
+# By likes
+uv run scripts/batch_eval_prs.py --limit 10 --sort likes --benchmark MMLU-Pro
+```
+
+### Filter by Pipeline Tag
+
+```bash
+# Only text-generation models (default)
+uv run scripts/batch_eval_prs.py --limit 10 --benchmark HLE --pipeline-tag text-generation
+
+# Image generation models
+uv run scripts/batch_eval_prs.py --limit 10 --benchmark HLE --pipeline-tag text-to-image
+```
+
+### Results Tracking
+
+Results are saved to `runs/{benchmark}_{date}_{hash}.json`:
+
+```bash
+cat runs/hle_20260114_abc123.json
+```
+
+```json
+{
+  "benchmark": "HLE",
+  "source": "aa",
+  "source_url": "https://artificialanalysis.ai",
+  "created": "2026-01-14T08:00:00Z",
+  "results": [
+    {
+      "repo_id": "MiniMaxAI/MiniMax-M2.1",
+      "value": 22.2,
+      "status": "pr_created",
+      "source_url": "https://artificialanalysis.ai"
+    }
+  ]
+}
+```
+
+---
+
+## Extract from README Tables
+
+For models with evaluation tables in their README.
+
+### Step 1: Inspect Tables
+
+```bash
+uv run scripts/evaluation_manager.py inspect-tables \
+  --repo-id "deepseek-ai/DeepSeek-V3"
+```
+
+This shows all tables with their structure, helping you identify which table to extract.
+
+### Step 2: Preview Extraction
+
+```bash
+uv run scripts/evaluation_manager.py extract-readme \
+  --repo-id "deepseek-ai/DeepSeek-V3" \
+  --table 1
+```
+
+### Step 3: Create PR
+
+```bash
+uv run scripts/evaluation_manager.py extract-readme \
+  --repo-id "deepseek-ai/DeepSeek-V3" \
+  --table 1 \
+  --create-pr
+```
+
+---
+
+## Import from Artificial Analysis
+
+Import all available benchmarks from Artificial Analysis API.
+
+### Preview
+
+```bash
+uv run scripts/evaluation_manager.py import-aa \
+  --creator-slug "anthropic" \
+  --model-name "claude-sonnet-4" \
+  --repo-id "your-username/claude-mirror"
+```
+
+### Create PR
+
+```bash
+uv run scripts/evaluation_manager.py import-aa \
+  --creator-slug "anthropic" \
+  --model-name "claude-sonnet-4" \
+  --repo-id "your-username/claude-mirror" \
+  --apply --create-pr
+```
+
+### Finding Creator Slug and Model Name
+
+Visit [Artificial Analysis](https://artificialanalysis.ai/) and check the URL:
+- URL: `https://artificialanalysis.ai/models/{creator-slug}/{model-name}`
+
+Common examples:
+- Anthropic: `--creator-slug "anthropic" --model-name "claude-sonnet-4"`
+- OpenAI: `--creator-slug "openai" --model-name "gpt-4-turbo"`
+- Meta: `--creator-slug "meta" --model-name "llama-3-70b"`
+
+---
+
+## Common Workflows
+
+### Workflow 1: Add Missing Benchmark to Popular Model
+
+```bash
+# 1. Check for existing PRs
+uv run scripts/evaluation_manager.py get-prs \
+  --repo-id "meta-llama/Llama-3.1-8B-Instruct"
+
+# 2. Preview what we'd add
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "meta-llama/Llama-3.1-8B-Instruct"
+
+# 3. Create PR if score found
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "meta-llama/Llama-3.1-8B-Instruct" \
+  --create-pr
+```
+
+### Workflow 2: Batch Update Trending Models
+
+```bash
+# 1. Dry run to see which models have HLE scores
+uv run scripts/batch_eval_prs.py --limit 20 --benchmark HLE --source aa --dry-run
+
+# 2. Create PRs for models with scores
+uv run scripts/batch_eval_prs.py --limit 20 --benchmark HLE --source aa
+
+# 3. Check results
+cat runs/hle_*.json | jq '.results[] | select(.status == "pr_created")'
+```
+
+### Workflow 3: Update Your Own Model
+
+```bash
+# 1. Add HLE score from your model card
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark HLE \
+  --repo-id "your-username/your-model" \
+  --apply
+
+# 2. Add GPQA score manually
+uv run scripts/evaluation_manager.py add-eval \
+  --benchmark GPQA \
+  --repo-id "your-username/your-model" \
+  --value 84.5 \
+  --apply
+```
+
+---
+
+## Output Format
+
+Results are stored in `.eval_results/*.yaml`:
+
+```yaml
+- dataset:
+    id: cais/hle              # Hub Benchmark dataset ID
+    task_id: default          # Optional task ID
+  value: 23.9                 # Metric value
+  date: "2026-01-14"          # ISO-8601 date
+  source:                     # Attribution
+    url: https://huggingface.co/model/name
+    name: Model Card
+```
+
+---
+
+## Supported Benchmarks
+
+| Benchmark | Hub Dataset ID |
+|-----------|---------------|
+| HLE | cais/hle |
+| GPQA | Idavidrein/gpqa |
+| MMLU-Pro | TIGER-Lab/MMLU-Pro |
+| GSM8K | openai/gsm8k |
+
+To add a new benchmark, update `examples/metric_mapping.json`.
+
+---
+
+## Troubleshooting
+
+### "AA_API_KEY not set"
+```bash
+export AA_API_KEY="your-key"
+# or add to .env file
+```
+
+### "Could not find benchmark in model card"
+The benchmark name may be formatted differently in the README. Check the model card manually.
+
+### "Token does not have write access"
+Generate a new token at https://huggingface.co/settings/tokens with Write scope.
+
+---
+
+## Getting Help
+
+```bash
+uv run scripts/evaluation_manager.py --help
+uv run scripts/evaluation_manager.py add-eval --help
+uv run scripts/batch_eval_prs.py --help
+```
+
+For more information:
+- [HuggingFace Eval Results Documentation](https://huggingface.co/docs/hub/eval-results)
+- [SKILL.md](../SKILL.md) - Complete skill documentation

skills/hugging-face-evaluation/examples/artificial_analysis_to_hub.py

@@ -0,0 +1,220 @@
+# /// script
+# requires-python = ">=3.13"
+# dependencies = [
+#     "huggingface-hub>=1.1.4",
+#     "python-dotenv>=1.2.1",
+#     "pyyaml>=6.0.3",
+#     "requests>=2.32.5",
+# ]
+# ///
+
+"""
+Add Artificial Analysis evaluations to a Hugging Face model repository.
+
+This script outputs evaluation results in the new .eval_results/ format
+as documented at https://huggingface.co/docs/hub/eval-results
+
+NOTE: This is a standalone reference script. For integrated functionality
+with additional features (README extraction, validation, etc.), use:
+    ../scripts/evaluation_manager.py import-aa [options]
+
+STANDALONE USAGE:
+AA_API_KEY="<your-api-key>" HF_TOKEN="<your-huggingface-token>" \
+python artificial_analysis_to_hub.py \
+--creator-slug <artificial-analysis-creator-slug> \
+--model-name <artificial-analysis-model-name> \
+--repo-id <huggingface-repo-id>
+
+INTEGRATED USAGE (Recommended):
+python ../scripts/evaluation_manager.py import-aa \
+--creator-slug <creator-slug> \
+--model-name <model-name> \
+--repo-id <repo-id> \
+[--create-pr]
+"""
+
+import argparse
+import json
+import os
+from datetime import date
+from pathlib import Path
+
+import requests
+import yaml
+import dotenv
+from huggingface_hub import HfApi
+
+dotenv.load_dotenv()
+
+API_KEY = os.getenv("AA_API_KEY")
+HF_TOKEN = os.getenv("HF_TOKEN")
+URL = "https://artificialanalysis.ai/api/v2/data/llms/models"
+HEADERS = {"x-api-key": API_KEY}
+
+if not API_KEY:
+    raise ValueError("AA_API_KEY is not set")
+if not HF_TOKEN:
+    raise ValueError("HF_TOKEN is not set")
+
+
+def load_benchmark_mapping():
+    """Load the benchmark-to-dataset mapping from metric_mapping.json."""
+    mapping_file = Path(__file__).parent / "metric_mapping.json"
+
+    if not mapping_file.exists():
+        # Fallback to minimal mapping
+        return {
+            "MMLU": {"dataset_id": "cais/mmlu", "aliases": ["mmlu"]},
+            "GPQA": {"dataset_id": "Idavidrein/gpqa", "task_id": "gpqa_diamond", "aliases": ["gpqa"]},
+            "GSM8K": {"dataset_id": "openai/gsm8k", "aliases": ["gsm8k"]},
+        }
+
+    with open(mapping_file) as f:
+        mapping = json.load(f)
+
+    mapping.pop("_comment", None)
+    return mapping
+
+
+def find_benchmark_dataset(benchmark_name, mapping):
+    """Find the Hub dataset ID for a benchmark name."""
+    normalized = benchmark_name.lower().replace(" ", "_").replace("-", "_")
+
+    # Try exact match
+    if benchmark_name in mapping:
+        entry = mapping[benchmark_name]
+        return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    # Try case-insensitive match
+    for key, entry in mapping.items():
+        if key.lower() == benchmark_name.lower():
+            return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    # Try matching aliases
+    for key, entry in mapping.items():
+        aliases = entry.get("aliases", [])
+        if normalized in [a.lower().replace(" ", "_").replace("-", "_") for a in aliases]:
+            return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    return None
+
+
+def get_model_evaluations_data(creator_slug, model_name):
+    response = requests.get(URL, headers=HEADERS)
+    response_data = response.json()["data"]
+    for model in response_data:
+        if (
+            model["model_creator"]["slug"] == creator_slug
+            and model["slug"] == model_name
+        ):
+            return model
+    raise ValueError(f"Model {model_name} not found")
+
+
+def aa_evaluations_to_eval_results(model):
+    """
+    Convert Artificial Analysis model data to .eval_results/ format.
+
+    Returns a list of evaluation entries ready for YAML output.
+    """
+    if not model:
+        raise ValueError("Model data is required")
+
+    evaluations = model.get("evaluations", {})
+    mapping = load_benchmark_mapping()
+    results = []
+    today = date.today().isoformat()
+
+    for key, value in evaluations.items():
+        if value is None:
+            continue
+
+        # Convert key to title case for matching
+        benchmark_name = key.replace("_", " ").title()
+        dataset_info = find_benchmark_dataset(benchmark_name, mapping)
+
+        if not dataset_info:
+            # Try the original key as well
+            dataset_info = find_benchmark_dataset(key, mapping)
+
+        if not dataset_info:
+            print(f"Warning: Could not find Hub dataset ID for '{benchmark_name}'. Skipping.")
+            continue
+
+        entry = {
+            "dataset": {
+                "id": dataset_info["dataset_id"],
+            },
+            "value": value,
+            "date": today,
+            "source": {
+                "url": "https://artificialanalysis.ai",
+                "name": "Artificial Analysis",
+            }
+        }
+
+        # Add task_id if not default
+        if dataset_info.get("task_id") and dataset_info["task_id"] != "default":
+            entry["dataset"]["task_id"] = dataset_info["task_id"]
+
+        results.append(entry)
+
+    return results
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--creator-slug", type=str, required=True)
+    parser.add_argument("--model-name", type=str, required=True)
+    parser.add_argument("--repo-id", type=str, required=True)
+    parser.add_argument("--filename", type=str, default="artificial_analysis.yaml",
+                        help="Output filename in .eval_results/")
+    parser.add_argument("--dry-run", action="store_true",
+                        help="Print YAML without uploading")
+    args = parser.parse_args()
+
+    aa_evaluations_data = get_model_evaluations_data(
+        creator_slug=args.creator_slug, model_name=args.model_name
+    )
+
+    eval_results = aa_evaluations_to_eval_results(model=aa_evaluations_data)
+
+    if not eval_results:
+        print("No evaluations could be mapped to Hub dataset IDs")
+        return
+
+    # Generate YAML content
+    yaml_content = yaml.dump(eval_results, sort_keys=False, allow_unicode=True)
+
+    if args.dry_run:
+        print("\nGenerated .eval_results/ YAML:")
+        print(yaml_content)
+        return
+
+    # Upload to .eval_results/ folder
+    api = HfApi(token=HF_TOKEN)
+    file_path = f".eval_results/{args.filename}"
+
+    commit_message = f"Add Artificial Analysis evaluations for {args.model_name}"
+    commit_description = (
+        "This commit adds structured evaluation results in the new .eval_results/ format. "
+        "Results will appear on the model page and linked benchmark leaderboards. "
+        "See https://huggingface.co/docs/hub/eval-results for format details."
+    )
+
+    api.upload_file(
+        path_or_fileobj=yaml_content.encode("utf-8"),
+        path_in_repo=file_path,
+        repo_id=args.repo_id,
+        repo_type="model",
+        commit_message=commit_message,
+        commit_description=commit_description,
+        create_pr=True,
+    )
+
+    print(f"✓ Pull request created for {args.repo_id}")
+    print(f"  File: {file_path}")
+
+
+if __name__ == "__main__":
+    main()

skills/hugging-face-evaluation/examples/eval.example.yaml

@@ -0,0 +1,11 @@
+- dataset:
+    id: cais/hle                  # Required. Hub dataset ID (must be a Benchmark)
+    task_id: default              # Optional, in case there are multiple tasks or leaderboards for this dataset.
+    revision: <hash>              # Optional. Dataset revision hash
+  value: 20.90                    # Required. Metric value
+  verifyToken: <token>            # Optional. Cryptographic proof of auditable evaluation
+  date: "2025-01-15"              # Optional. ISO-8601 date or datetime (defaults to git commit time)
+  source:                         # Optional. Attribution for this result, for instance a repo containing output traces or a Paper
+    url: https://huggingface.co/spaces/SaylorTwift/smollm3-mmlu-pro  # Required if source provided
+    name: Eval traces             # Optional. Display name
+    user: SaylorTwift             # Optional. HF username/org

skills/hugging-face-evaluation/examples/example_readme_tables.md

@@ -0,0 +1,135 @@
+# Example Evaluation Table Formats
+
+This file shows various formats of evaluation tables that can be extracted from model README files.
+
+## Format 1: Benchmarks as Rows (Most Common)
+
+```markdown
+| Benchmark | Score |
+|-----------|-------|
+| MMLU      | 85.2  |
+| HumanEval | 72.5  |
+| GSM8K     | 91.3  |
+| HellaSwag | 88.9  |
+```
+
+## Format 2: Multiple Metric Columns
+
+```markdown
+| Benchmark | Accuracy | F1 Score |
+|-----------|----------|----------|
+| MMLU      | 85.2     | 0.84     |
+| GSM8K     | 91.3     | 0.91     |
+| DROP      | 78.5     | 0.77     |
+```
+
+## Format 3: Benchmarks as Columns
+
+```markdown
+| MMLU | HumanEval | GSM8K | HellaSwag |
+|------|-----------|-------|-----------|
+| 85.2 | 72.5      | 91.3  | 88.9      |
+```
+
+## Format 4: Percentage Values
+
+```markdown
+| Benchmark     | Score    |
+|---------------|----------|
+| MMLU          | 85.2%    |
+| HumanEval     | 72.5%    |
+| GSM8K         | 91.3%    |
+| TruthfulQA    | 68.7%    |
+```
+
+## Format 5: Mixed Format with Categories
+
+```markdown
+### Reasoning
+
+| Benchmark | Score |
+|-----------|-------|
+| MMLU      | 85.2  |
+| BBH       | 82.4  |
+| GPQA      | 71.3  |
+
+### Coding
+
+| Benchmark | Score |
+|-----------|-------|
+| HumanEval | 72.5  |
+| MBPP      | 78.9  |
+
+### Math
+
+| Benchmark | Score |
+|-----------|-------|
+| GSM8K     | 91.3  |
+| MATH      | 65.8  |
+```
+
+## Format 6: With Additional Columns
+
+```markdown
+| Benchmark | Score | Rank | Notes              |
+|-----------|-------|------|--------------------|
+| MMLU      | 85.2  | #5   | 5-shot             |
+| HumanEval | 72.5  | #8   | pass@1             |
+| GSM8K     | 91.3  | #3   | 8-shot, maj@1      |
+```
+
+## How the Extractor Works
+
+The script will:
+1. Find all markdown tables in the README
+2. Identify which tables contain evaluation results
+3. Parse the table structure (rows vs columns)
+4. Extract numeric values as scores
+5. Convert to model-index YAML format
+
+## Tips for README Authors
+
+To ensure your evaluation tables are properly extracted:
+
+1. **Use clear headers**: Include "Benchmark", "Score", or similar terms
+2. **Keep it simple**: Stick to benchmark name + score columns
+3. **Use standard formats**: Follow markdown table syntax
+4. **Include numeric values**: Ensure scores are parseable numbers
+5. **Be consistent**: Use the same format across multiple tables
+
+## Example Complete README Section
+
+```markdown
+# Model Card for MyModel-7B
+
+## Evaluation Results
+
+Our model was evaluated on several standard benchmarks:
+
+| Benchmark     | Score |
+|---------------|-------|
+| MMLU          | 85.2  |
+| HumanEval     | 72.5  |
+| GSM8K         | 91.3  |
+| HellaSwag     | 88.9  |
+| ARC-Challenge | 81.7  |
+| TruthfulQA    | 68.7  |
+
+### Detailed Results
+
+For more detailed results and methodology, see our [paper](link).
+```
+
+## Running the Extractor
+
+```bash
+# Extract from this example
+python scripts/evaluation_manager.py extract-readme \
+  --repo-id "your-username/your-model" \
+  --dry-run
+
+# Apply to your model card
+python scripts/evaluation_manager.py extract-readme \
+  --repo-id "your-username/your-model" \
+  --task-type "text-generation"
+```

skills/hugging-face-evaluation/examples/metric_mapping.json

@@ -0,0 +1,118 @@
+{
+  "_comment": "Maps benchmark names to Hub dataset IDs for .eval_results/ format. Dataset IDs must be registered Benchmarks on HuggingFace Hub.",
+  "MMLU": {
+    "dataset_id": "cais/mmlu",
+    "task_id": "default",
+    "aliases": ["mmlu", "massive_multitask_language_understanding"]
+  },
+  "MMLU-Pro": {
+    "dataset_id": "TIGER-Lab/MMLU-Pro",
+    "task_id": "mmlu_pro",
+    "aliases": ["mmlu_pro", "mmlu-pro"]
+  },
+  "MMLU-Redux": {
+    "dataset_id": "edinburgh-dawg/mmlu-redux",
+    "task_id": "default",
+    "aliases": ["mmlu_redux", "mmlu-redux"]
+  },
+  "HumanEval": {
+    "dataset_id": "openai/openai_humaneval",
+    "task_id": "default",
+    "aliases": ["humaneval", "human_eval"]
+  },
+  "GSM8K": {
+    "dataset_id": "openai/gsm8k",
+    "task_id": "default",
+    "aliases": ["gsm8k", "gsm_8k", "grade_school_math"]
+  },
+  "HellaSwag": {
+    "dataset_id": "Rowan/hellaswag",
+    "task_id": "default",
+    "aliases": ["hellaswag", "hella_swag"]
+  },
+  "ARC-Challenge": {
+    "dataset_id": "allenai/ai2_arc",
+    "task_id": "ARC-Challenge",
+    "aliases": ["arc_challenge", "arc-c", "arc_c"]
+  },
+  "ARC-Easy": {
+    "dataset_id": "allenai/ai2_arc",
+    "task_id": "ARC-Easy",
+    "aliases": ["arc_easy", "arc-e", "arc_e"]
+  },
+  "Winogrande": {
+    "dataset_id": "allenai/winogrande",
+    "task_id": "default",
+    "aliases": ["winogrande", "wino_grande"]
+  },
+  "TruthfulQA": {
+    "dataset_id": "truthfulqa/truthful_qa",
+    "task_id": "default",
+    "aliases": ["truthfulqa", "truthful_qa"]
+  },
+  "GPQA": {
+    "dataset_id": "Idavidrein/gpqa",
+    "task_id": "gpqa_diamond",
+    "aliases": ["gpqa", "gpqa_diamond"]
+  },
+  "DROP": {
+    "dataset_id": "ucinlp/drop",
+    "task_id": "default",
+    "aliases": ["drop"]
+  },
+  "BBH": {
+    "dataset_id": "lukaemon/bbh",
+    "task_id": "default",
+    "aliases": ["bbh", "big_bench_hard"]
+  },
+  "MATH": {
+    "dataset_id": "lighteval/MATH",
+    "task_id": "default",
+    "aliases": ["math"]
+  },
+  "HLE": {
+    "dataset_id": "cais/hle",
+    "task_id": "default",
+    "aliases": ["hle", "human_level_evaluation", "hle_text_only", "hle_(text_only)"]
+  },
+  "AIME25": {
+    "dataset_id": "OpenEvals/aime_2025",
+    "task_id": "default",
+    "aliases": ["aime25", "aime_25", "aime_2025"]
+  },
+  "SWE-bench": {
+    "dataset_id": "princeton-nlp/SWE-bench_Verified",
+    "task_id": "default",
+    "aliases": ["swe_bench", "swe-bench", "swe_bench_verified", "swe-bench_verified"]
+  },
+  "LiveCodeBench": {
+    "dataset_id": "livecodebench/livecodebench",
+    "task_id": "default",
+    "aliases": ["livecodebench", "live_code_bench", "livecodebenchv6"]
+  },
+  "SimpleQA": {
+    "dataset_id": "OpenEvals/SimpleQA",
+    "task_id": "default",
+    "aliases": ["simpleqa", "simple_qa"]
+  },
+  "AIME": {
+    "dataset_id": "OpenEvals/aime_24",
+    "task_id": "default",
+    "aliases": ["aime", "aime_24", "aime24"]
+  },
+  "IFEval": {
+    "dataset_id": "google/IFEval",
+    "task_id": "default",
+    "aliases": ["ifeval", "if_eval"]
+  },
+  "MBPP": {
+    "dataset_id": "google-research-datasets/mbpp",
+    "task_id": "default",
+    "aliases": ["mbpp"]
+  },
+  "MuSR": {
+    "dataset_id": "OpenEvals/MuSR",
+    "task_id": "default",
+    "aliases": ["musr"]
+  }
+}

skills/hugging-face-evaluation/references/hf_cli_for_prs.md

@@ -0,0 +1,258 @@
+# HF CLI Workflow for Evaluation PRs
+
+This document explains how to manage evaluation result PRs using the `hf` CLI and temporary directories.
+
+## Directory Structure
+
+Use `/tmp/pr-reviews/` as the working directory for PR operations:
+
+```
+/tmp/pr-reviews/
+├── updates/           # YAML files for updating existing PRs
+├── new-prs/           # YAML files for new PRs
+├── <model-name>/      # Model-specific directories
+└── check-<model>/     # Directories for verifying PR contents
+```
+
+## Creating New PRs
+
+### Step 1: Create YAML file
+
+```bash
+mkdir -p /tmp/pr-reviews/new-prs
+cd /tmp/pr-reviews/new-prs
+
+cat > hle.yaml << 'EOF'
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 22.1
+  date: '2026-02-03'
+  source:
+    url: https://huggingface.co/org/model-name
+    name: Model Card
+    user: burtenshaw
+EOF
+```
+
+### Step 2: Upload and create PR
+
+```bash
+hf upload org/model-name hle.yaml .eval_results/hle.yaml \
+  --repo-type model --create-pr \
+  --commit-message "Add HLE evaluation result"
+```
+
+### Step 3: Get PR number
+
+```bash
+uv run scripts/evaluation_manager.py get-prs --repo-id "org/model-name"
+```
+
+## Updating Existing PRs
+
+### Step 1: Download current PR contents
+
+```bash
+hf download org/model-name --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --include ".eval_results/*" \
+  --local-dir /tmp/pr-reviews/<model-name>-pr<PR_NUMBER>
+```
+
+### Step 2: Review current contents
+
+```bash
+cat /tmp/pr-reviews/<model-name>-pr<PR_NUMBER>/.eval_results/*.yaml
+```
+
+### Step 3: Create updated YAML
+
+```bash
+cat > /tmp/pr-reviews/updates/updated.yaml << 'EOF'
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 22.1
+  date: '2026-02-03'
+  source:
+    url: https://huggingface.co/org/model-name
+    name: Model Card
+    user: burtenshaw
+  notes: "With tools"
+EOF
+```
+
+### Step 4: Push update to existing PR
+
+```bash
+hf upload org/model-name /tmp/pr-reviews/updates/updated.yaml .eval_results/hle.yaml \
+  --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --commit-message "Update evaluation result"
+```
+
+## Deleting Files from PRs
+
+Use the `huggingface_hub` Python API to delete files:
+
+```bash
+uv run --with huggingface_hub python3 << 'EOF'
+from huggingface_hub import HfApi
+api = HfApi()
+
+api.delete_file(
+    path_in_repo=".eval_results/old_file.yaml",
+    repo_id="org/model-name",
+    repo_type="model",
+    revision="refs/pr/<PR_NUMBER>",
+    commit_message="Remove duplicate file"
+)
+EOF
+```
+
+## Verifying PR Contents
+
+### Check what files are in a PR
+
+```bash
+rm -rf /tmp/check-<model>
+hf download org/model-name --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --include ".eval_results/*" \
+  --local-dir /tmp/check-<model>
+
+ls -la /tmp/check-<model>/.eval_results/
+```
+
+### Compare PR to main branch
+
+```bash
+# Download main
+hf download org/model-name --repo-type model \
+  --revision main \
+  --include ".eval_results/*" \
+  --local-dir /tmp/<model>-main
+
+# Download PR
+hf download org/model-name --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --include ".eval_results/*" \
+  --local-dir /tmp/<model>-pr<PR_NUMBER>
+
+# Compare
+diff /tmp/<model>-main/.eval_results/ /tmp/<model>-pr<PR_NUMBER>/.eval_results/
+```
+
+## Multiple Score Variants
+
+When a model has multiple scores for the same benchmark (e.g., with/without tools), create separate files:
+
+```bash
+cd /tmp/pr-reviews/new-prs
+
+# Default (no tools) - no notes field
+cat > hle.yaml << 'EOF'
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 10.2
+  date: '2026-02-03'
+  source:
+    url: https://huggingface.co/org/model-name
+    name: Model Card
+    user: burtenshaw
+EOF
+
+# With tools - add notes field
+cat > hle_with_tools.yaml << 'EOF'
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 15.5
+  date: '2026-02-03'
+  source:
+    url: https://huggingface.co/org/model-name
+    name: Model Card
+    user: burtenshaw
+  notes: "With tools"
+EOF
+
+# Create separate PRs
+hf upload org/model-name hle.yaml .eval_results/hle.yaml \
+  --repo-type model --create-pr \
+  --commit-message "Add HLE evaluation result"
+
+hf upload org/model-name hle_with_tools.yaml .eval_results/hle_with_tools.yaml \
+  --repo-type model --create-pr \
+  --commit-message "Add HLE evaluation result (with tools)"
+```
+
+## Restoring Files Accidentally Deleted
+
+If a PR shows a file as deleted (because it was removed from the PR branch), restore it from main:
+
+```bash
+# Download the file from main
+hf download org/model-name --repo-type model \
+  --revision main \
+  --include ".eval_results/hle.yaml" \
+  --local-dir /tmp/<model>-main
+
+# Re-upload to PR to restore it
+hf upload org/model-name /tmp/<model>-main/.eval_results/hle.yaml .eval_results/hle.yaml \
+  --repo-type model \
+  --revision refs/pr/<PR_NUMBER> \
+  --commit-message "Restore original file"
+```
+
+## Common Patterns
+
+### Batch create YAML files
+
+```bash
+cd /tmp/pr-reviews/updates
+
+# Create multiple files in one script
+for model in "org/model1" "org/model2"; do
+  cat > "${model//\//-}-hle.yaml" << EOF
+- dataset:
+    id: cais/hle
+    task_id: hle
+  value: 22.1
+  source:
+    url: https://huggingface.co/$model
+    name: Model Card
+    user: burtenshaw
+EOF
+done
+```
+
+### Check for existing PRs before creating
+
+Always check first:
+
+```bash
+uv run scripts/evaluation_manager.py get-prs --repo-id "org/model-name"
+```
+
+If PRs exist, update them instead of creating new ones.
+
+## File Naming Convention
+
+| Condition | File Name | Notes Field |
+|-----------|-----------|-------------|
+| Default (no tools) | `hle.yaml` | None (omit) |
+| With tools | `hle_with_tools.yaml` | `notes: "With tools"` |
+| Different task | `gpqa_diamond.yaml` | Based on task_id |
+
+## Cleanup
+
+After PRs are merged or work is complete:
+
+```bash
+rm -rf /tmp/pr-reviews/
+rm -rf /tmp/check-*
+rm -rf /tmp/*-main
+rm -rf /tmp/*-pr*
+```

skills/hugging-face-evaluation/references/hf_papers_extraction.md

@@ -0,0 +1,297 @@
+# Paper Score Extraction via HF MCP Server
+
+This document provides instructions for extracting benchmark scores from academic papers linked to HuggingFace models using the HF MCP Server tools.
+
+---
+
+## Overview
+
+Papers linked to HuggingFace models often contain comprehensive benchmark results that aren't in the model card. This guide shows how to:
+
+1. Use `hub_repo_details` to discover papers linked to a model
+2. Use `paper_search` to find and retrieve paper abstracts
+3. Extract benchmark scores from paper abstracts/content
+4. Use `WebFetch` on arxiv PDFs for detailed scores not in abstracts
+5. Format results for `.eval_results/`
+
+---
+
+## Step 1: Discover Linked Papers
+
+### Get Model Details with README
+
+Use `hub_repo_details` to fetch model metadata including linked papers:
+
+```
+mcp__hf-mcp-server__hub_repo_details
+  repo_ids: ["org/model-name"]
+  include_readme: true
+```
+
+Look for arXiv references in:
+- `tags` array: entries starting with `arxiv:` (e.g., `"arxiv:2411.15124"`)
+- README content: arXiv links or paper references
+- Model metadata: `paperInfo` or `cardData.arxiv` fields
+
+### Example Response Fields
+
+The response will include:
+- Model metadata (downloads, likes, tags)
+- README content (if `include_readme: true`)
+- Any linked paper IDs in tags
+
+---
+
+## Step 2: Search for Papers
+
+Once you have an arXiv ID or want to find related papers, use `paper_search`:
+
+```
+mcp__hf-mcp-server__paper_search
+  query: "OLMo-2 evaluation benchmark"
+  results_limit: 5
+  concise_only: false  # Get full abstracts for score extraction
+```
+
+### Search Strategies
+
+**By model name:**
+```
+query: "Llama 3.1 benchmark evaluation"
+```
+
+**By arXiv ID (if known):**
+```
+query: "2411.15124"
+```
+
+**By benchmark + model family:**
+```
+query: "MMLU GPQA Qwen2.5"
+```
+
+---
+
+## Step 3: Extract Benchmark Scores
+
+The paper search returns abstracts and paper content. Look for:
+
+### Common Benchmark Mentions
+
+Papers typically report headline numbers in abstracts:
+- "achieves **85.2%** on MMLU"
+- "state-of-the-art results on GPQA Diamond (72.1%)"
+- "HLE score of 12.3%"
+
+### Benchmark Name Variations
+
+| Standard Name | Paper Variations |
+|---------------|------------------|
+| HLE | Humanity's Last Exam, HLE (Text Only) |
+| GPQA | GPQA Diamond, GPQA-Diamond |
+| MMLU | MMLU, MMLU-Pro, Massive Multitask |
+| GSM8K | GSM8K, GSM-8K, Grade School Math |
+| MATH | MATH, MATH-500 |
+| HumanEval | HumanEval, human_eval |
+| SWE-bench | SWE-bench, SWE-bench Verified |
+
+### Score Format Normalization
+
+- **Percentages**: `85.2%` → use `85.2`
+- **Decimals**: `0.852` → convert to `85.2` if context shows percentages
+- **Accuracy vs Error Rate**: Ensure you're extracting accuracy, not error rate
+
+---
+
+## Step 4: Format for .eval_results/
+
+Once you have extracted scores, format them as YAML:
+
+```yaml
+# .eval_results/{benchmark_name}.yaml
+- dataset:
+    id: {hub_dataset_id}
+    task_id: {task_variant}
+  value: {score}
+  date: "{extraction_date}"
+  source:
+    url: https://arxiv.org/abs/{arxiv_id}
+    name: Paper
+```
+
+### Dataset ID Reference
+
+| Benchmark | Dataset ID | Task ID |
+|-----------|------------|---------|
+| HLE | `cais/hle` | `default` |
+| GPQA | `Idavidrein/gpqa` | `gpqa_diamond` |
+| MMLU | `cais/mmlu` | `default` |
+| MMLU-Pro | `TIGER-Lab/MMLU-Pro` | `default` |
+| GSM8K | `openai/gsm8k` | `default` |
+| MATH | `lighteval/MATH` | `default` |
+| HumanEval | `openai/openai_humaneval` | `default` |
+| DROP | `ucinlp/drop` | `default` |
+| ARC-Challenge | `allenai/ai2_arc` | `ARC-Challenge` |
+| HellaSwag | `Rowan/hellaswag` | `default` |
+| TruthfulQA | `truthfulqa/truthful_qa` | `default` |
+| IFEval | `google/IFEval` | `default` |
+| SWE-bench | `princeton-nlp/SWE-bench_Verified` | `default` |
+| AIME24 | `OpenEvals/aime_24` | `default` |
+| AIME25 | `OpenEvals/aime_2025` | `default` |
+| LiveCodeBench | `livecodebench/livecodebench` | `default` |
+
+---
+
+## Complete Example Workflow
+
+### Scenario: Extract MMLU score for OLMo-2 from its paper
+
+```
+1. Get model details:
+   mcp__hf-mcp-server__hub_repo_details
+     repo_ids: ["allenai/OLMo-2-1124-7B-Instruct"]
+     include_readme: true
+
+   → Found tags: ["arxiv:2411.15124", "arxiv:2501.00656"]
+
+2. Search for the paper:
+   mcp__hf-mcp-server__paper_search
+     query: "OLMo-2 2501.00656"
+     results_limit: 3
+
+   → Returns paper abstract with benchmark scores
+
+3. Extract from abstract:
+   "OLMo-2-7B-Instruct achieves 61.3 on MMLU..."
+
+   If score not in abstract, fetch PDF:
+   WebFetch
+     url: "https://arxiv.org/pdf/2501.00656"
+     prompt: "Find MMLU score for OLMo-2-7B-Instruct in the evaluation tables."
+
+4. Create the eval result:
+   $ uv run scripts/evaluation_manager.py add-eval \
+       --benchmark MMLU \
+       --repo-id "allenai/OLMo-2-1124-7B-Instruct" \
+       --value 61.3 \
+       --create-pr
+```
+
+---
+
+## Tips for Better Extraction
+
+### 1. Check Multiple Papers
+Models may have multiple linked papers. Use `hub_repo_details` to find all arXiv tags, then search for each.
+
+### 2. Use Concise Mode for Broad Searches
+```
+mcp__hf-mcp-server__paper_search
+  query: "large language model evaluation"
+  concise_only: true  # 2-sentence summaries
+  results_limit: 10
+```
+
+### 3. Prefer Primary Sources
+Use the model's own release paper rather than papers that cite it.
+
+### 4. Note Evaluation Settings
+Papers may report different settings (0-shot vs 5-shot, with/without CoT). Document which setting you're extracting.
+
+### 5. Cross-Reference Model Card
+If both paper and model card have scores, prefer the paper as the authoritative source but verify they match.
+
+---
+
+## Step 5: Extract Scores from Paper PDFs
+
+The `paper_search` tool only returns abstracts, which often miss detailed benchmark tables. For comprehensive score extraction, fetch the full paper PDF.
+
+### URL Pattern
+
+HuggingFace paper links map directly to arxiv PDFs:
+
+| Source | URL Pattern |
+|--------|-------------|
+| HF Paper Page | `https://huggingface.co/papers/{arxiv_id}` |
+| arxiv Abstract | `https://arxiv.org/abs/{arxiv_id}` |
+| arxiv PDF | `https://arxiv.org/pdf/{arxiv_id}` |
+
+**Example**: `2601.01739` → `https://arxiv.org/pdf/2601.01739`
+
+### Fetching PDF Content
+
+Use `WebFetch` to retrieve and search the PDF:
+
+```
+WebFetch
+  url: "https://arxiv.org/pdf/{arxiv_id}"
+  prompt: "Extract all benchmark evaluation scores and results tables. Look for metrics like accuracy, F1, BLEU, pass@k, or percentage scores. List each benchmark name and its corresponding score."
+```
+
+### Targeted Extraction Prompts
+
+For specific benchmarks:
+
+```
+prompt: "Find the HLE (Humanity's Last Exam) score in this paper. Look in results tables and the evaluation section."
+```
+
+```
+prompt: "Extract all scores from the main results table. Include benchmark names, model variants, and numerical scores."
+```
+
+```
+prompt: "Find MMLU, GPQA, GSM8K, and MATH scores for the main model in this paper."
+```
+
+### When to Use PDF Extraction
+
+Use PDF fetching when:
+- Abstract doesn't contain specific benchmark scores
+- You need scores for multiple benchmarks
+- Paper mentions "see Table X for full results"
+- Model card references paper but lacks detailed numbers
+
+### Example: Full PDF Extraction Workflow
+
+```
+1. Get arxiv ID from model:
+   mcp__hf-mcp-server__hub_repo_details
+     repo_ids: ["meta-llama/Llama-3.1-70B-Instruct"]
+     include_readme: true
+
+   → Found: arxiv:2407.21783
+
+2. Fetch PDF for detailed scores:
+   WebFetch
+     url: "https://arxiv.org/pdf/2407.21783"
+     prompt: "Extract benchmark scores for Llama 3.1 70B Instruct from all evaluation tables. Include MMLU, GPQA Diamond, HumanEval, GSM8K, MATH, and any other benchmarks."
+
+3. Parse extracted scores and create eval results
+```
+
+---
+
+## Common Issues
+
+### Paper Score Differs from Model Card
+- Paper may report different model size/variant
+- Evaluation settings may differ
+- Paper may be pre-release; model card updated post-release
+
+**Solution**: Note the discrepancy and prefer the most recent source.
+
+### Score Not Found in Paper Search
+- Paper abstracts rarely contain full benchmark tables
+- Paper may not have evaluated that benchmark
+- Try searching with different query terms
+- Check if benchmark uses a different name
+
+**Solution**: Use `WebFetch` to fetch the full PDF (`https://arxiv.org/pdf/{arxiv_id}`) - detailed scores are typically in results tables within the paper body, not the abstract. Also try alternative query terms or benchmark aliases.
+
+### Multiple Models in Paper
+- Paper describes a family of models (7B, 13B, 70B)
+- Results may combine scores across sizes
+
+**Solution**: Carefully match the exact model variant to the HuggingFace repo.

skills/hugging-face-evaluation/references/model_card_extraction.md

@@ -0,0 +1,244 @@
+# Model Card Score Extraction via HF MCP Server
+
+This document provides instructions for extracting benchmark scores from HuggingFace model cards using the HF MCP Server tools.
+
+---
+
+## Overview
+
+Model cards often contain evaluation tables with benchmark scores. This guide shows how to:
+
+1. Use `hub_repo_details` to fetch model card content
+2. Search for benchmark variations in the README
+3. Extract and normalize scores
+4. Format results for `.eval_results/`
+
+---
+
+## Step 1: Fetch the Model Card
+
+Use `hub_repo_details` to get the model's README content:
+
+```
+mcp__hf-mcp-server__hub_repo_details
+  repo_ids: ["org/model-name"]
+  include_readme: true
+```
+
+This returns:
+- Model metadata (downloads, likes, tags, pipeline_tag)
+- Full README content (when `include_readme: true`)
+- Linked papers and datasets
+
+### Batch Fetching
+
+You can fetch multiple models at once:
+
+```
+mcp__hf-mcp-server__hub_repo_details
+  repo_ids: ["meta-llama/Llama-3.1-8B-Instruct", "Qwen/Qwen2.5-7B-Instruct"]
+  include_readme: true
+```
+
+---
+
+## Step 2: Search for Benchmark Scores
+
+### Benchmark Name Variations
+
+Model cards use inconsistent naming. Search for these variations:
+
+| Benchmark | Variations to Search |
+|-----------|---------------------|
+| HLE | `HLE`, `hle`, `Humanity's Last Exam`, `HLE (Text Only)` |
+| GPQA | `GPQA`, `GPQA Diamond`, `gpqa_diamond`, `GPQA-Diamond` |
+| MMLU-Pro | `MMLU-Pro`, `MMLU Pro`, `mmlu_pro`, `MMLU-PRO` |
+| MMLU | `MMLU`, `mmlu`, `Massive Multitask Language Understanding` |
+| GSM8K | `GSM8K`, `gsm8k`, `GSM-8K`, `Grade School Math` |
+| HumanEval | `HumanEval`, `humaneval`, `human_eval` |
+| HellaSwag | `HellaSwag`, `hellaswag`, `hella_swag` |
+| ARC-Challenge | `ARC-Challenge`, `ARC-C`, `arc_challenge` |
+| TruthfulQA | `TruthfulQA`, `truthful_qa`, `TruthfulQA MC` |
+| MATH | `MATH`, `math`, `MATH-500` |
+| AIME | `AIME`, `AIME24`, `AIME 2024`, `aime_24` |
+| SWE-bench | `SWE-bench`, `SWE-bench Verified`, `swe_bench` |
+| LiveCodeBench | `LiveCodeBench`, `LCB`, `LiveCodeBenchV6` |
+| IFEval | `IFEval`, `IF-Eval`, `ifeval` |
+
+---
+
+## Step 3: Identify Table Formats
+
+Model cards typically present scores in these formats:
+
+### Format A: Model-Column Table (most common)
+```markdown
+| Model | MMLU | GPQA | HLE |
+|-------|------|------|-----|
+| This Model | 85.2 | 72.1 | 12.3 |
+| GPT-4 | 86.4 | 74.2 | 15.1 |
+```
+
+### Format B: Benchmark-Column Table
+```markdown
+| Benchmark | Score |
+|-----------|-------|
+| MMLU | 85.2 |
+| GPQA | 72.1 |
+| HLE | 12.3 |
+```
+
+### Format C: Inline Text
+```markdown
+Our model achieves **85.2%** on MMLU, **72.1%** on GPQA Diamond, and **12.3%** on HLE.
+```
+
+### Format D: Nested/Grouped Tables
+```markdown
+| Category | Benchmark | Score |
+|----------|-----------|-------|
+| Reasoning | GPQA | 72.1 |
+| Knowledge | MMLU | 85.2 |
+```
+
+---
+
+## Step 4: Extract and Normalize Scores
+
+### Score Format Normalization
+
+Scores may be presented as:
+- **Percentages**: `85.2%` or `85.2` (when context implies %)
+- **Decimals**: `0.852` (multiply by 100 for percentage)
+- **Fractions**: `85.2/100`
+
+**Important**: The `.eval_results/` format expects values matching the benchmark's standard scale. Most benchmarks use percentage scale (0-100).
+
+---
+
+## Step 5: Format for .eval_results/
+
+Once you have the score, format it for `.eval_results/`:
+
+```yaml
+# .eval_results/{benchmark}.yaml
+- dataset:
+    id: cais/hle           # Hub dataset ID (see mapping below)
+    task_id: default       # Task variant if applicable
+  value: 12.3              # Score value
+  date: "2026-01-14"       # ISO date of extraction
+  source:
+    url: https://huggingface.co/{org}/{model}
+    name: Model Card
+```
+
+### Dataset ID Reference
+
+| Benchmark | Dataset ID | Task ID |
+|-----------|------------|---------|
+| HLE | `cais/hle` | `default` |
+| GPQA | `Idavidrein/gpqa` | `gpqa_diamond` |
+| MMLU-Pro | `TIGER-Lab/MMLU-Pro` | `default` |
+| MMLU | `cais/mmlu` | `default` |
+| GSM8K | `openai/gsm8k` | `default` |
+| HumanEval | `openai/openai_humaneval` | `default` |
+| MATH | `lighteval/MATH` | `default` |
+| ARC-Challenge | `allenai/ai2_arc` | `ARC-Challenge` |
+| HellaSwag | `Rowan/hellaswag` | `default` |
+| TruthfulQA | `truthfulqa/truthful_qa` | `default` |
+| SWE-bench | `princeton-nlp/SWE-bench_Verified` | `default` |
+| AIME24 | `OpenEvals/aime_24` | `default` |
+| AIME25 | `OpenEvals/aime_2025` | `default` |
+| LiveCodeBench | `livecodebench/livecodebench` | `default` |
+| IFEval | `google/IFEval` | `default` |
+
+---
+
+## Complete Example Workflow
+
+### Scenario: Extract HLE score from a model card
+
+```
+1. Fetch model card:
+   mcp__hf-mcp-server__hub_repo_details
+     repo_ids: ["Qwen/Qwen2.5-72B-Instruct"]
+     include_readme: true
+
+2. Search README for HLE variations:
+   Found: "| HLE | 18.5 |" in evaluation table
+
+3. Create the eval result:
+   $ uv run scripts/evaluation_manager.py add-eval \
+       --benchmark HLE \
+       --repo-id "Qwen/Qwen2.5-72B-Instruct" \
+       --value 18.5 \
+       --create-pr
+```
+
+---
+
+## Finding Models with Evaluations
+
+Use `model_search` to find models that might have benchmark scores:
+
+### Search for Trending Models
+```
+mcp__hf-mcp-server__model_search
+  task: "text-generation"
+  sort: "trendingScore"
+  limit: 20
+```
+
+### Search by Author
+```
+mcp__hf-mcp-server__model_search
+  author: "meta-llama"
+  task: "text-generation"
+  limit: 10
+```
+
+### Search by Query
+```
+mcp__hf-mcp-server__model_search
+  query: "instruct chat"
+  task: "text-generation"
+  limit: 20
+```
+
+Then use `hub_repo_details` on promising results to check their model cards.
+
+---
+
+## Tips for Better Extraction
+
+### 1. Check the Full README
+Model cards may have scores in different sections (Overview, Evaluation, Benchmarks, Results).
+
+### 2. Look for Multiple Tables
+Some model cards have separate tables for different benchmark categories.
+
+### 3. Note Evaluation Settings
+Papers may report different settings (0-shot vs 5-shot, with/without CoT). Document which setting you're extracting.
+
+### 4. Verify Against Papers
+If both paper and model card have scores, prefer the paper as the authoritative source but verify they match.
+
+---
+
+## Common Issues
+
+### Score in Image/Figure Only
+**Solution**: Check if there's a linked technical report or paper with tabular data. Use `paper_search` to find it.
+
+### Benchmark Name Differs Significantly
+**Solution**: Search for the underlying task name (e.g., "graduate-level science" for GPQA).
+
+### Multiple Scores for Same Benchmark
+**Solution**: Prefer "0-shot" or "standard" settings; note the configuration in source attribution.
+
+### Score Not Found
+- Score genuinely not present in model card
+- Benchmark not evaluated by model authors
+- Try `paper_search` to find scores in linked papers
+
+**Solution**: Document why extraction failed to distinguish "not found by automation" from "truly not available".

skills/hugging-face-evaluation/scripts/check_prs.py

@@ -0,0 +1,98 @@
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#     "requests>=2.31.0",
+# ]
+# ///
+
+"""
+Check for open pull requests on a Hugging Face model repository.
+
+Usage:
+    uv run scripts/check_prs.py --repo-id "org/model-name"
+"""
+
+import argparse
+import sys
+from typing import Any
+
+import requests
+
+
+def get_open_prs(repo_id: str) -> list[dict[str, Any]]:
+    """
+    Fetch open pull requests for a Hugging Face model repository.
+
+    Args:
+        repo_id: Hugging Face model repository ID (e.g., "nvidia/model-name")
+
+    Returns:
+        List of open PR dictionaries with num, title, author, and createdAt
+    """
+    url = f"https://huggingface.co/api/models/{repo_id}/discussions"
+
+    try:
+        response = requests.get(url, timeout=30, allow_redirects=True)
+        response.raise_for_status()
+
+        data = response.json()
+        discussions = data.get("discussions", [])
+
+        open_prs = [
+            {
+                "num": d["num"],
+                "title": d["title"],
+                "author": d["author"]["name"],
+                "createdAt": d.get("createdAt", "unknown"),
+            }
+            for d in discussions
+            if d.get("status") == "open" and d.get("isPullRequest")
+        ]
+
+        return open_prs
+
+    except requests.RequestException as e:
+        print(f"Error fetching PRs from Hugging Face: {e}", file=sys.stderr)
+        return []
+
+
+def list_open_prs(repo_id: str) -> None:
+    """Display open pull requests for a model repository."""
+    prs = get_open_prs(repo_id)
+
+    print(f"\n{'='*70}")
+    print(f"Open Pull Requests for: {repo_id}")
+    print(f"{'='*70}")
+
+    if not prs:
+        print("\nNo open pull requests found.")
+    else:
+        print(f"\nFound {len(prs)} open PR(s):\n")
+        for pr in prs:
+            print(f"  PR #{pr['num']} - {pr['title']}")
+            print(f"     Author: {pr['author']}")
+            print(f"     Created: {pr['createdAt']}")
+            print(f"     URL: https://huggingface.co/{repo_id}/discussions/{pr['num']}")
+            print()
+
+    print(f"{'='*70}\n")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Check for open pull requests on a Hugging Face model repository.",
+        epilog="Always run this before creating new PRs to avoid duplicates.",
+    )
+    parser.add_argument(
+        "--repo-id",
+        type=str,
+        required=True,
+        help="HF repository ID (e.g., 'nvidia/model-name')",
+    )
+
+    args = parser.parse_args()
+    list_open_prs(args.repo_id)
+
+
+if __name__ == "__main__":
+    main()

skills/hugging-face-evaluation/scripts/import_aa.py

@@ -0,0 +1,353 @@
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+#     "huggingface-hub>=1.1.4",
+#     "python-dotenv>=1.0.0",
+#     "pyyaml>=6.0.0",
+#     "requests>=2.31.0",
+# ]
+# ///
+"""
+Import evaluation results from Artificial Analysis API.
+
+Usage:
+  # Look up a specific benchmark (dry run - prints YAML)
+  AA_API_KEY=... uv run scripts/import_aa.py --repo-id "org/model" --benchmark HLE
+
+  # Look up a benchmark and create PR
+  AA_API_KEY=... uv run scripts/import_aa.py --repo-id "org/model" --benchmark GPQA --create-pr
+
+  # Import all available benchmarks
+  AA_API_KEY=... uv run scripts/import_aa.py --repo-id "org/model" --all
+
+  # Provide value manually (skip lookup)
+  uv run scripts/import_aa.py --repo-id "org/model" --benchmark HLE --value 22.5 --create-pr
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+from datetime import date
+from pathlib import Path
+from typing import Any
+
+import requests
+
+
+AA_INDEX_URL = "https://artificialanalysis.ai/api/v2/data/llms/models"
+
+
+def load_env() -> None:
+    try:
+        import dotenv
+        dotenv.load_dotenv()
+    except ModuleNotFoundError:
+        pass
+
+
+def load_benchmark_mapping() -> dict[str, Any]:
+    script_dir = Path(__file__).parent
+    mapping_file = script_dir.parent / "examples" / "metric_mapping.json"
+
+    if not mapping_file.exists():
+        return {
+            "GPQA": {"dataset_id": "Idavidrein/gpqa", "task_id": "gpqa_diamond", "aliases": ["gpqa"]},
+            "HLE": {"dataset_id": "cais/hle", "task_id": "default", "aliases": ["hle"]},
+            "SimpleQA": {"dataset_id": "OpenEvals/SimpleQA", "task_id": "default", "aliases": ["simpleqa"]},
+            "MMLU": {"dataset_id": "cais/mmlu", "task_id": "default", "aliases": ["mmlu"]},
+            "GSM8K": {"dataset_id": "openai/gsm8k", "task_id": "default", "aliases": ["gsm8k"]},
+        }
+
+    with open(mapping_file) as f:
+        mapping = json.load(f)
+    mapping.pop("_comment", None)
+    return mapping
+
+
+def find_benchmark_dataset(benchmark_name: str, mapping: dict[str, Any]) -> dict[str, str] | None:
+    cleaned = re.sub(r'\[([^\]]+)\]\([^\)]+\)', r'\1', benchmark_name)
+    cleaned = re.sub(r'\*\*([^\*]+)\*\*', r'\1', cleaned)
+    cleaned = re.sub(r'\*([^\*]+)\*', r'\1', cleaned)
+    cleaned = cleaned.strip()
+
+    normalized = cleaned.lower().replace(" ", "_").replace("-", "_")
+    base_name = re.sub(r'\s*\([^)]*\)\s*$', '', cleaned).strip()
+    base_normalized = base_name.lower().replace(" ", "_").replace("-", "_")
+
+    if cleaned in mapping:
+        entry = mapping[cleaned]
+        return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    for key, entry in mapping.items():
+        if key.lower() == cleaned.lower():
+            return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    for key, entry in mapping.items():
+        aliases = entry.get("aliases", [])
+        normalized_aliases = [a.lower().replace(" ", "_").replace("-", "_") for a in aliases]
+        if normalized in normalized_aliases:
+            return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    for key, entry in mapping.items():
+        key_normalized = key.lower().replace(" ", "_").replace("-", "_")
+        if normalized == key_normalized:
+            return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    if base_normalized != normalized:
+        for key, entry in mapping.items():
+            if key.lower() == base_name.lower():
+                return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+            key_normalized = key.lower().replace(" ", "_").replace("-", "_")
+            if base_normalized == key_normalized:
+                return {"dataset_id": entry["dataset_id"], "task_id": entry.get("task_id", "default")}
+
+    return None
+
+
+def fetch_aa_models(api_key: str) -> list[dict[str, Any]]:
+    response = requests.get(
+        AA_INDEX_URL,
+        headers={"x-api-key": api_key},
+        timeout=30,
+    )
+    response.raise_for_status()
+    data = response.json()
+    return list(data.get("data", []))
+
+
+def find_model_in_aa(models: list[dict[str, Any]], repo_id: str) -> dict[str, Any] | None:
+    model_name = repo_id.split("/")[-1] if "/" in repo_id else repo_id
+    model_name_normalized = model_name.lower().replace("-", " ").replace("_", " ")
+
+    for model in models:
+        aa_name = model.get("name", "").lower().replace("-", " ").replace("_", " ")
+        aa_slug = model.get("slug", "").lower().replace("-", " ").replace("_", " ")
+        if model_name_normalized in aa_name or model_name_normalized in aa_slug:
+            return model
+
+    return None
+
+
+def lookup_benchmark_from_aa(
+    models: list[dict[str, Any]],
+    repo_id: str,
+    benchmark_name: str,
+) -> float | None:
+    model = find_model_in_aa(models, repo_id)
+    if not model:
+        return None
+
+    evaluations = model.get("evaluations", {})
+    benchmark_normalized = benchmark_name.lower().replace(" ", "_").replace("-", "_")
+
+    for key, value in evaluations.items():
+        key_normalized = key.lower().replace(" ", "_").replace("-", "_")
+        if benchmark_normalized == key_normalized or benchmark_normalized in key_normalized:
+            if value is not None:
+                return float(value)
+
+    return None
+
+
+def get_all_benchmarks_from_aa(
+    models: list[dict[str, Any]],
+    repo_id: str,
+) -> list[dict[str, Any]]:
+    model = find_model_in_aa(models, repo_id)
+    if not model:
+        return []
+
+    evaluations = model.get("evaluations", {})
+    metrics = []
+
+    for key, value in evaluations.items():
+        if value is not None:
+            metrics.append({
+                "name": key.replace("_", " ").title(),
+                "type": key,
+                "value": float(value),
+            })
+
+    return metrics
+
+
+def convert_to_eval_results_format(
+    metrics: list[dict[str, Any]],
+    source_url: str | None = None,
+    source_name: str | None = None,
+    source_user: str | None = None,
+) -> list[dict[str, Any]]:
+    mapping = load_benchmark_mapping()
+    results = []
+    today = date.today().isoformat()
+
+    for metric in metrics:
+        benchmark_name = metric.get("name", "")
+        value = metric.get("value")
+
+        if value is None:
+            continue
+
+        dataset_info = find_benchmark_dataset(benchmark_name, mapping)
+        if not dataset_info:
+            print(f"Warning: Could not find Hub dataset ID for benchmark '{benchmark_name}'. Skipping.", file=sys.stderr)
+            continue
+
+        entry: dict[str, Any] = {
+            "dataset": {"id": dataset_info["dataset_id"]},
+            "value": value,
+            "date": today,
+        }
+
+        if dataset_info.get("task_id") and dataset_info["task_id"] != "default":
+            entry["dataset"]["task_id"] = dataset_info["task_id"]
+
+        if source_url:
+            entry["source"] = {"url": source_url}
+            if source_name:
+                entry["source"]["name"] = source_name
+            if source_user:
+                entry["source"]["user"] = source_user
+
+        results.append(entry)
+
+    return results
+
+
+def upload_eval_results(
+    repo_id: str,
+    results: list[dict[str, Any]],
+    filename: str = "evaluations.yaml",
+    create_pr: bool = False,
+    commit_message: str | None = None,
+) -> bool:
+    import yaml
+    from huggingface_hub import HfApi
+
+    load_env()
+    hf_token = os.getenv("HF_TOKEN")
+    if not hf_token:
+        print("Error: HF_TOKEN environment variable is not set", file=sys.stderr)
+        return False
+
+    api = HfApi(token=hf_token)
+    yaml_content = yaml.dump(results, sort_keys=False, allow_unicode=True, default_flow_style=False)
+    file_path = f".eval_results/{filename}"
+
+    if not commit_message:
+        model_name = repo_id.split("/")[-1] if "/" in repo_id else repo_id
+        commit_message = f"Add Artificial Analysis evaluation results for {model_name}"
+
+    pr_description = """## Evaluation Results
+
+This PR adds structured evaluation results using the new [`.eval_results/` format](https://huggingface.co/docs/hub/eval-results).
+
+**Source:** [Artificial Analysis](https://artificialanalysis.ai)
+
+### What This Enables
+
+- **Model Page**: Results appear on the model page with benchmark links
+- **Leaderboards**: Scores are aggregated into benchmark dataset leaderboards
+- **Verification**: Support for cryptographic verification of evaluation runs
+
+---
+*Generated by [community-evals](https://github.com/huggingface/community-evals)*"""
+
+    try:
+        api.upload_file(
+            path_or_fileobj=yaml_content.encode("utf-8"),
+            path_in_repo=file_path,
+            repo_id=repo_id,
+            repo_type="model",
+            commit_message=commit_message,
+            commit_description=pr_description,
+            create_pr=create_pr,
+        )
+
+        action = "Pull request created" if create_pr else "Evaluation results uploaded"
+        print(f"✓ {action} successfully for {repo_id}")
+        print(f"  File: {file_path}")
+        return True
+
+    except Exception as e:
+        print(f"Error uploading evaluation results: {e}", file=sys.stderr)
+        return False
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Import evaluation results from Artificial Analysis API.",
+    )
+    parser.add_argument("--repo-id", required=True, help="HuggingFace repository ID")
+    parser.add_argument("--benchmark", help="Specific benchmark to look up (e.g., HLE, GPQA)")
+    parser.add_argument("--value", type=float, help="Manually provide the score (skips AA lookup)")
+    parser.add_argument("--all", action="store_true", help="Import all available benchmarks")
+    parser.add_argument("--source-user", help="HF username/org for attribution")
+    parser.add_argument("--filename", default="artificial_analysis.yaml", help="Output filename")
+    parser.add_argument("--create-pr", action="store_true", help="Create PR instead of direct push")
+    parser.add_argument("--apply", action="store_true", help="Apply changes (default is dry run)")
+    parser.add_argument("--pretty", action="store_true", help="Pretty-print YAML output")
+    parser.add_argument("--verbose", action="store_true", help="Print progress to stderr")
+    args = parser.parse_args()
+
+    load_env()
+
+    if args.value is not None and args.benchmark:
+        metrics = [{"name": args.benchmark, "type": args.benchmark.lower(), "value": args.value}]
+    else:
+        api_key = os.getenv("AA_API_KEY")
+        if not api_key:
+            print("Error: AA_API_KEY is required to query Artificial Analysis.", file=sys.stderr)
+            sys.exit(1)
+
+        if args.verbose:
+            print("Fetching models from Artificial Analysis...", file=sys.stderr)
+
+        models = fetch_aa_models(api_key)
+
+        if args.all:
+            metrics = get_all_benchmarks_from_aa(models, args.repo_id)
+            if not metrics:
+                print(f"No benchmarks found for {args.repo_id} in Artificial Analysis", file=sys.stderr)
+                sys.exit(1)
+        elif args.benchmark:
+            value = lookup_benchmark_from_aa(models, args.repo_id, args.benchmark)
+            if value is None:
+                print(f"Could not find {args.benchmark} score for {args.repo_id} in Artificial Analysis", file=sys.stderr)
+                sys.exit(1)
+            print(f"Found: {args.benchmark} = {value}")
+            metrics = [{"name": args.benchmark, "type": args.benchmark.lower(), "value": value}]
+        else:
+            print("Error: Either --benchmark or --all is required", file=sys.stderr)
+            sys.exit(1)
+
+    eval_results = convert_to_eval_results_format(
+        metrics=metrics,
+        source_url="https://artificialanalysis.ai",
+        source_name="Artificial Analysis",
+        source_user=args.source_user,
+    )
+
+    if not eval_results:
+        print("No benchmarks could be mapped to Hub dataset IDs", file=sys.stderr)
+        sys.exit(1)
+
+    import yaml
+    print("\nImported evaluations (.eval_results/ format):")
+    print(yaml.dump(eval_results, sort_keys=False, allow_unicode=True, default_flow_style=False))
+
+    if args.apply or args.create_pr:
+        upload_eval_results(
+            repo_id=args.repo_id,
+            results=eval_results,
+            filename=args.filename,
+            create_pr=args.create_pr,
+        )
+
+
+if __name__ == "__main__":
+    main()

skills/hugging-face-model-trainer/SKILL.md

@@ -0,0 +1,718 @@
+---
+name: hugging-face-model-trainer
+description: This skill should be used when users want to train or fine-tune language models using TRL (Transformer Reinforcement Learning) on Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs package, UV scripts with PEP 723 format, dataset preparation and validation, hardware selection, cost estimation, Trackio monitoring, Hub authentication, and model persistence. Should be invoked for tasks involving cloud GPU training, GGUF conversion, or when users mention training on Hugging Face Jobs without local GPU setup.
+license: Complete terms in LICENSE.txt
+---
+
+# TRL Training on Hugging Face Jobs
+
+## Overview
+
+Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required—models train on cloud GPUs and results are automatically saved to the Hugging Face Hub.
+
+**TRL provides multiple training methods:**
+- **SFT** (Supervised Fine-Tuning) - Standard instruction tuning
+- **DPO** (Direct Preference Optimization) - Alignment from preference data
+- **GRPO** (Group Relative Policy Optimization) - Online RL training
+- **Reward Modeling** - Train reward models for RLHF
+
+**For detailed TRL method documentation:**
+```python
+hf_doc_search("your query", product="trl")
+hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer")  # SFT
+hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")  # DPO
+# etc.
+```
+
+**See also:** `references/training_methods.md` for method overviews and selection guidance
+
+## When to Use This Skill
+
+Use this skill when users want to:
+- Fine-tune language models on cloud GPUs without local infrastructure
+- Train with TRL methods (SFT, DPO, GRPO, etc.)
+- Run training jobs on Hugging Face Jobs infrastructure
+- Convert trained models to GGUF for local deployment (Ollama, LM Studio, llama.cpp)
+- Ensure trained models are permanently saved to the Hub
+- Use modern workflows with optimized defaults
+
+### When to Use Unsloth
+
+Use **Unsloth** (`references/unsloth.md`) instead of standard TRL when:
+- **Limited GPU memory** - Unsloth uses ~60% less VRAM
+- **Speed matters** - Unsloth is ~2x faster
+- Training **large models (>13B)** - memory efficiency is critical
+- Training **Vision-Language Models (VLMs)** - Unsloth has `FastVisionModel` support
+
+See `references/unsloth.md` for complete Unsloth documentation and `scripts/unsloth_sft_example.py` for a production-ready training script.
+
+## Key Directives
+
+When assisting with training jobs:
+
+1. **ALWAYS use `hf_jobs()` MCP tool** - Submit jobs using `hf_jobs("uv", {...})`, NOT bash `trl-jobs` commands. The `script` parameter accepts Python code directly. Do NOT save to local files unless the user explicitly requests it. Pass the script content as a string to `hf_jobs()`. If user asks to "train a model", "fine-tune", or similar requests, you MUST create the training script AND submit the job immediately using `hf_jobs()`.
+
+2. **Always include Trackio** - Every training script should include Trackio for real-time monitoring. Use example scripts in `scripts/` as templates.
+
+3. **Provide job details after submission** - After submitting, provide job ID, monitoring URL, estimated time, and note that the user can request status checks later.
+
+4. **Use example scripts as templates** - Reference `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, etc. as starting points.
+
+## Local Script Dependencies
+
+To run scripts locally (like `estimate_cost.py`), install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+## Prerequisites Checklist
+
+Before starting any training job, verify:
+
+### ✅ **Account & Authentication**
+- Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan)
+- Authenticated login: Check with `hf_whoami()`
+- **HF_TOKEN for Hub Push** ⚠️ CRITICAL - Training environment is ephemeral, must push to Hub or ALL training results are lost
+- Token must have write permissions  
+- **MUST pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config** to make token available (the `$HF_TOKEN` syntax
+  references your actual token value)
+
+### ✅ **Dataset Requirements**
+- Dataset must exist on Hub or be loadable via `datasets.load_dataset()`
+- Format must match training method (SFT: "messages"/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only)
+- **ALWAYS validate unknown datasets** before GPU training to prevent format failures (see Dataset Validation section below)
+- Size appropriate for hardware (Demo: 50-100 examples on t4-small; Production: 1K-10K+ on a10g-large/a100-large)
+
+### ⚠️ **Critical Settings**
+- **Timeout must exceed expected training time** - Default 30min is TOO SHORT for most training. Minimum recommended: 1-2 hours. Job fails and loses all progress if timeout is exceeded.
+- **Hub push must be enabled** - Config: `push_to_hub=True`, `hub_model_id="username/model-name"`; Job: `secrets={"HF_TOKEN": "$HF_TOKEN"}`
+
+## Asynchronous Job Guidelines
+
+**⚠️ IMPORTANT: Training jobs run asynchronously and can take hours**
+
+### Action Required
+
+**When user requests training:**
+1. **Create the training script** with Trackio included (use `scripts/train_sft_example.py` as template)
+2. **Submit immediately** using `hf_jobs()` MCP tool with script content inline - don't save to file unless user requests
+3. **Report submission** with job ID, monitoring URL, and estimated time
+4. **Wait for user** to request status checks - don't poll automatically
+
+### Ground Rules
+- **Jobs run in background** - Submission returns immediately; training continues independently
+- **Initial logs delayed** - Can take 30-60 seconds for logs to appear
+- **User checks status** - Wait for user to request status updates
+- **Avoid polling** - Check logs only on user request; provide monitoring links instead
+
+### After Submission
+
+**Provide to user:**
+- ✅ Job ID and monitoring URL
+- ✅ Expected completion time
+- ✅ Trackio dashboard URL
+- ✅ Note that user can request status checks later
+
+**Example Response:**
+```
+✅ Job submitted successfully!
+
+Job ID: abc123xyz
+Monitor: https://huggingface.co/jobs/username/abc123xyz
+
+Expected time: ~2 hours
+Estimated cost: ~$10
+
+The job is running in the background. Ask me to check status/logs when ready!
+```
+
+## Quick Start: Three Approaches
+
+**💡 Tip for Demos:** For quick demos on smaller GPUs (t4-small), omit `eval_dataset` and `eval_strategy` to save ~40% memory. You'll still see training loss and learning progress.
+
+### Sequence Length Configuration
+
+**TRL config classes use `max_length` (not `max_seq_length`)** to control tokenized sequence length:
+
+```python
+# ✅ CORRECT - If you need to set sequence length
+SFTConfig(max_length=512)   # Truncate sequences to 512 tokens
+DPOConfig(max_length=2048)  # Longer context (2048 tokens)
+
+# ❌ WRONG - This parameter doesn't exist
+SFTConfig(max_seq_length=512)  # TypeError!
+```
+
+**Default behavior:** `max_length=1024` (truncates from right). This works well for most training.
+
+**When to override:**
+- **Longer context**: Set higher (e.g., `max_length=2048`)
+- **Memory constraints**: Set lower (e.g., `max_length=512`)
+- **Vision models**: Set `max_length=None` (prevents cutting image tokens)
+
+**Usually you don't need to set this parameter at all** - the examples below use the sensible default.
+
+### Approach 1: UV Scripts (Recommended—Default Choice)
+
+UV scripts use PEP 723 inline dependencies for clean, self-contained training. **This is the primary approach for Claude Code.**
+
+```python
+hf_jobs("uv", {
+    "script": """
+# /// script
+# dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"]
+# ///
+
+from datasets import load_dataset
+from peft import LoraConfig
+from trl import SFTTrainer, SFTConfig
+import trackio
+
+dataset = load_dataset("trl-lib/Capybara", split="train")
+
+# Create train/eval split for monitoring
+dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset_split["train"],
+    eval_dataset=dataset_split["test"],
+    peft_config=LoraConfig(r=16, lora_alpha=32),
+    args=SFTConfig(
+        output_dir="my-model",
+        push_to_hub=True,
+        hub_model_id="username/my-model",
+        num_train_epochs=3,
+        eval_strategy="steps",
+        eval_steps=50,
+        report_to="trackio",
+        project="meaningful_prject_name", # project name for the training name (trackio)
+        run_name="meaningful_run_name",   # descriptive name for the specific training run (trackio)
+    )
+)
+
+trainer.train()
+trainer.push_to_hub()
+""",
+    "flavor": "a10g-large",
+    "timeout": "2h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**Benefits:** Direct MCP tool usage, clean code, dependencies declared inline (PEP 723), no file saving required, full control
+**When to use:** Default choice for all training tasks in Claude Code, custom training logic, any scenario requiring `hf_jobs()`
+
+#### Working with Scripts
+
+⚠️ **Important:** The `script` parameter accepts either inline code (as shown above) OR a URL. **Local file paths do NOT work.**
+
+**Why local paths don't work:**
+Jobs run in isolated Docker containers without access to your local filesystem. Scripts must be:
+- Inline code (recommended for custom training)
+- Publicly accessible URLs
+- Private repo URLs (with HF_TOKEN)
+
+**Common mistakes:**
+```python
+# ❌ These will all fail
+hf_jobs("uv", {"script": "train.py"})
+hf_jobs("uv", {"script": "./scripts/train.py"})
+hf_jobs("uv", {"script": "/path/to/train.py"})
+```
+
+**Correct approaches:**
+```python
+# ✅ Inline code (recommended)
+hf_jobs("uv", {"script": "# /// script\n# dependencies = [...]\n# ///\n\n<your code>"})
+
+# ✅ From Hugging Face Hub
+hf_jobs("uv", {"script": "https://huggingface.co/user/repo/resolve/main/train.py"})
+
+# ✅ From GitHub
+hf_jobs("uv", {"script": "https://raw.githubusercontent.com/user/repo/main/train.py"})
+
+# ✅ From Gist
+hf_jobs("uv", {"script": "https://gist.githubusercontent.com/user/id/raw/train.py"})
+```
+
+**To use local scripts:** Upload to HF Hub first:
+```bash
+huggingface-cli repo create my-training-scripts --type model
+huggingface-cli upload my-training-scripts ./train.py train.py
+# Use: https://huggingface.co/USERNAME/my-training-scripts/resolve/main/train.py
+```
+
+### Approach 2: TRL Maintained Scripts (Official Examples)
+
+TRL provides battle-tested scripts for all methods. Can be run from URLs:
+
+```python
+hf_jobs("uv", {
+    "script": "https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py",
+    "script_args": [
+        "--model_name_or_path", "Qwen/Qwen2.5-0.5B",
+        "--dataset_name", "trl-lib/Capybara",
+        "--output_dir", "my-model",
+        "--push_to_hub",
+        "--hub_model_id", "username/my-model"
+    ],
+    "flavor": "a10g-large",
+    "timeout": "2h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**Benefits:** No code to write, maintained by TRL team, production-tested
+**When to use:** Standard TRL training, quick experiments, don't need custom code
+**Available:** Scripts are available from https://github.com/huggingface/trl/tree/main/examples/scripts
+
+### Finding More UV Scripts on Hub
+
+The `uv-scripts` organization provides ready-to-use UV scripts stored as datasets on Hugging Face Hub:
+
+```python
+# Discover available UV script collections
+dataset_search({"author": "uv-scripts", "sort": "downloads", "limit": 20})
+
+# Explore a specific collection
+hub_repo_details(["uv-scripts/classification"], repo_type="dataset", include_readme=True)
+```
+
+**Popular collections:** ocr, classification, synthetic-data, vllm, dataset-creation
+
+### Approach 3: HF Jobs CLI (Direct Terminal Commands)
+
+When the `hf_jobs()` MCP tool is unavailable, use the `hf jobs` CLI directly.
+
+**⚠️ CRITICAL: CLI Syntax Rules**
+
+```bash
+# ✅ CORRECT syntax - flags BEFORE script URL
+hf jobs uv run --flavor a10g-large --timeout 2h --secrets HF_TOKEN "https://example.com/train.py"
+
+# ❌ WRONG - "run uv" instead of "uv run"
+hf jobs run uv "https://example.com/train.py" --flavor a10g-large
+
+# ❌ WRONG - flags AFTER script URL (will be ignored!)
+hf jobs uv run "https://example.com/train.py" --flavor a10g-large
+
+# ❌ WRONG - "--secret" instead of "--secrets" (plural)
+hf jobs uv run --secret HF_TOKEN "https://example.com/train.py"
+```
+
+**Key syntax rules:**
+1. Command order is `hf jobs uv run` (NOT `hf jobs run uv`)
+2. All flags (`--flavor`, `--timeout`, `--secrets`) must come BEFORE the script URL
+3. Use `--secrets` (plural), not `--secret`
+4. Script URL must be the last positional argument
+
+**Complete CLI example:**
+```bash
+hf jobs uv run \
+  --flavor a10g-large \
+  --timeout 2h \
+  --secrets HF_TOKEN \
+  "https://huggingface.co/user/repo/resolve/main/train.py"
+```
+
+**Check job status via CLI:**
+```bash
+hf jobs ps                        # List all jobs
+hf jobs logs <job-id>             # View logs
+hf jobs inspect <job-id>          # Job details
+hf jobs cancel <job-id>           # Cancel a job
+```
+
+### Approach 4: TRL Jobs Package (Simplified Training)
+
+The `trl-jobs` package provides optimized defaults and one-liner training.
+
+```bash
+# Install
+pip install trl-jobs
+
+# Train with SFT (simplest possible)
+trl-jobs sft \
+  --model_name Qwen/Qwen2.5-0.5B \
+  --dataset_name trl-lib/Capybara
+```
+
+**Benefits:** Pre-configured settings, automatic Trackio integration, automatic Hub push, one-line commands
+**When to use:** User working in terminal directly (not Claude Code context), quick local experimentation
+**Repository:** https://github.com/huggingface/trl-jobs
+
+⚠️ **In Claude Code context, prefer using `hf_jobs()` MCP tool (Approach 1) when available.**
+
+## Hardware Selection
+
+| Model Size | Recommended Hardware | Cost (approx/hr) | Use Case |
+|------------|---------------------|------------------|----------|
+| <1B params | `t4-small` | ~$0.75 | Demos, quick tests only without eval steps |
+| 1-3B params | `t4-medium`, `l4x1` | ~$1.50-2.50 | Development |
+| 3-7B params | `a10g-small`, `a10g-large` | ~$3.50-5.00 | Production training |
+| 7-13B params | `a10g-large`, `a100-large` | ~$5-10 | Large models (use LoRA) |
+| 13B+ params | `a100-large`, `a10g-largex2` | ~$10-20 | Very large (use LoRA) |
+
+**GPU Flavors:** cpu-basic/upgrade/performance/xl, t4-small/medium, l4x1/x4, a10g-small/large/largex2/largex4, a100-large, h100/h100x8
+
+**Guidelines:**
+- Use **LoRA/PEFT** for models >7B to reduce memory
+- Multi-GPU automatically handled by TRL/Accelerate
+- Start with smaller hardware for testing
+
+**See:** `references/hardware_guide.md` for detailed specifications
+
+## Critical: Saving Results to Hub
+
+**⚠️ EPHEMERAL ENVIRONMENT—MUST PUSH TO HUB**
+
+The Jobs environment is temporary. All files are deleted when the job ends. If the model isn't pushed to Hub, **ALL TRAINING IS LOST**.
+
+### Required Configuration
+
+**In training script/config:**
+```python
+SFTConfig(
+    push_to_hub=True,
+    hub_model_id="username/model-name",  # MUST specify
+    hub_strategy="every_save",  # Optional: push checkpoints
+)
+```
+
+**In job submission:**
+```python
+{
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # Enables authentication
+}
+```
+
+### Verification Checklist
+
+Before submitting:
+- [ ] `push_to_hub=True` set in config
+- [ ] `hub_model_id` includes username/repo-name
+- [ ] `secrets` parameter includes HF_TOKEN
+- [ ] User has write access to target repo
+
+**See:** `references/hub_saving.md` for detailed troubleshooting
+
+## Timeout Management
+
+**⚠️ DEFAULT: 30 MINUTES—TOO SHORT FOR TRAINING**
+
+### Setting Timeouts
+
+```python
+{
+    "timeout": "2h"   # 2 hours (formats: "90m", "2h", "1.5h", or seconds as integer)
+}
+```
+
+### Timeout Guidelines
+
+| Scenario | Recommended | Notes |
+|----------|-------------|-------|
+| Quick demo (50-100 examples) | 10-30 min | Verify setup |
+| Development training | 1-2 hours | Small datasets |
+| Production (3-7B model) | 4-6 hours | Full datasets |
+| Large model with LoRA | 3-6 hours | Depends on dataset |
+
+**Always add 20-30% buffer** for model/dataset loading, checkpoint saving, Hub push operations, and network delays.
+
+**On timeout:** Job killed immediately, all unsaved progress lost, must restart from beginning
+
+## Cost Estimation
+
+**Offer to estimate cost when planning jobs with known parameters.** Use `scripts/estimate_cost.py`:
+
+```bash
+uv run scripts/estimate_cost.py \
+  --model meta-llama/Llama-2-7b-hf \
+  --dataset trl-lib/Capybara \
+  --hardware a10g-large \
+  --dataset-size 16000 \
+  --epochs 3
+```
+
+Output includes estimated time, cost, recommended timeout (with buffer), and optimization suggestions.
+
+**When to offer:** User planning a job, asks about cost/time, choosing hardware, job will run >1 hour or cost >$5
+
+## Example Training Scripts
+
+**Production-ready templates with all best practices:**
+
+Load these scripts for correctly:
+
+- **`scripts/train_sft_example.py`** - Complete SFT training with Trackio, LoRA, checkpoints
+- **`scripts/train_dpo_example.py`** - DPO training for preference learning
+- **`scripts/train_grpo_example.py`** - GRPO training for online RL
+
+These scripts demonstrate proper Hub saving, Trackio integration, checkpoint management, and optimized parameters. Pass their content inline to `hf_jobs()` or use as templates for custom scripts.
+
+## Monitoring and Tracking
+
+**Trackio** provides real-time metrics visualization. See `references/trackio_guide.md` for complete setup guide.
+
+**Key points:**
+- Add `trackio` to dependencies
+- Configure trainer with `report_to="trackio" and run_name="meaningful_name"`
+
+### Trackio Configuration Defaults
+
+**Use sensible defaults unless user specifies otherwise.** When generating training scripts with Trackio:
+
+**Default Configuration:**
+- **Space ID**: `{username}/trackio` (use "trackio" as default space name)
+- **Run naming**: Unless otherwise specified, name the run in a way the user will recognize (e.g., descriptive of the task, model, or purpose)
+- **Config**: Keep minimal - only include hyperparameters and model/dataset info
+- **Project Name**: Use a Project Name to associate runs with a particular Project 
+
+**User overrides:** If user requests specific trackio configuration (custom space, run naming, grouping, or additional config), apply their preferences instead of defaults.
+
+
+This is useful for managing multiple jobs with the same configuration or keeping training scripts portable.
+
+See `references/trackio_guide.md` for complete documentation including grouping runs for experiments.
+
+### Check Job Status
+
+```python
+# List all jobs
+hf_jobs("ps")
+
+# Inspect specific job
+hf_jobs("inspect", {"job_id": "your-job-id"})
+
+# View logs
+hf_jobs("logs", {"job_id": "your-job-id"})
+```
+
+**Remember:** Wait for user to request status checks. Avoid polling repeatedly.
+
+## Dataset Validation
+
+**Validate dataset format BEFORE launching GPU training to prevent the #1 cause of training failures: format mismatches.**
+
+### Why Validate
+
+- 50%+ of training failures are due to dataset format issues
+- DPO especially strict: requires exact column names (`prompt`, `chosen`, `rejected`)
+- Failed GPU jobs waste $1-10 and 30-60 minutes
+- Validation on CPU costs ~$0.01 and takes <1 minute
+
+### When to Validate
+
+**ALWAYS validate for:**
+- Unknown or custom datasets
+- DPO training (CRITICAL - 90% of datasets need mapping)
+- Any dataset not explicitly TRL-compatible
+
+**Skip validation for known TRL datasets:**
+- `trl-lib/ultrachat_200k`, `trl-lib/Capybara`, `HuggingFaceH4/ultrachat_200k`, etc.
+
+### Usage
+
+```python
+hf_jobs("uv", {
+    "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py",
+    "script_args": ["--dataset", "username/dataset-name", "--split", "train"]
+})
+```
+
+The script is fast, and will usually complete synchronously.
+
+### Reading Results
+
+The output shows compatibility for each training method:
+
+- **`✓ READY`** - Dataset is compatible, use directly
+- **`✗ NEEDS MAPPING`** - Compatible but needs preprocessing (mapping code provided)
+- **`✗ INCOMPATIBLE`** - Cannot be used for this method
+
+When mapping is needed, the output includes a **"MAPPING CODE"** section with copy-paste ready Python code.
+
+### Example Workflow
+
+```python
+# 1. Inspect dataset (costs ~$0.01, <1 min on CPU)
+hf_jobs("uv", {
+    "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py",
+    "script_args": ["--dataset", "argilla/distilabel-math-preference-dpo", "--split", "train"]
+})
+
+# 2. Check output markers:
+#    ✓ READY → proceed with training
+#    ✗ NEEDS MAPPING → apply mapping code below
+#    ✗ INCOMPATIBLE → choose different method/dataset
+
+# 3. If mapping needed, apply before training:
+def format_for_dpo(example):
+    return {
+        'prompt': example['instruction'],
+        'chosen': example['chosen_response'],
+        'rejected': example['rejected_response'],
+    }
+dataset = dataset.map(format_for_dpo, remove_columns=dataset.column_names)
+
+# 4. Launch training job with confidence
+```
+
+### Common Scenario: DPO Format Mismatch
+
+Most DPO datasets use non-standard column names. Example:
+
+```
+Dataset has: instruction, chosen_response, rejected_response
+DPO expects: prompt, chosen, rejected
+```
+
+The validator detects this and provides exact mapping code to fix it.
+
+## Converting Models to GGUF
+
+After training, convert models to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools.
+
+**What is GGUF:**
+- Optimized for CPU/GPU inference with llama.cpp
+- Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size
+- Compatible with Ollama, LM Studio, Jan, GPT4All, llama.cpp
+- Typically 2-8GB for 7B models (vs 14GB unquantized)
+
+**When to convert:**
+- Running models locally with Ollama or LM Studio
+- Reducing model size with quantization
+- Deploying to edge devices
+- Sharing models for local-first use
+
+**See:** `references/gguf_conversion.md` for complete conversion guide, including production-ready conversion script, quantization options, hardware requirements, usage examples, and troubleshooting.
+
+**Quick conversion:**
+```python
+hf_jobs("uv", {
+    "script": "<see references/gguf_conversion.md for complete script>",
+    "flavor": "a10g-large",
+    "timeout": "45m",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
+    "env": {
+        "ADAPTER_MODEL": "username/my-finetuned-model",
+        "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
+        "OUTPUT_REPO": "username/my-model-gguf"
+    }
+})
+```
+
+## Common Training Patterns
+
+See `references/training_patterns.md` for detailed examples including:
+- Quick demo (5-10 minutes)
+- Production with checkpoints
+- Multi-GPU training
+- DPO training (preference learning)
+- GRPO training (online RL)
+
+## Common Failure Modes
+
+### Out of Memory (OOM)
+
+**Fix (try in order):**
+1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128. 
+2. Enable: `gradient_checkpointing=True`
+3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc. 
+
+### Dataset Misformatted
+
+**Fix:**
+1. Validate first with dataset inspector:
+   ```bash
+   uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \
+     --dataset name --split train
+   ```
+2. Check output for compatibility markers (✓ READY, ✗ NEEDS MAPPING, ✗ INCOMPATIBLE)
+3. Apply mapping code from inspector output if needed
+
+### Job Timeout
+
+**Fix:**
+1. Check logs for actual runtime: `hf_jobs("logs", {"job_id": "..."})`
+2. Increase timeout with buffer: `"timeout": "3h"` (add 30% to estimated time)
+3. Or reduce training: lower `num_train_epochs`, use smaller dataset, enable `max_steps`
+4. Save checkpoints: `save_strategy="steps"`, `save_steps=500`, `hub_strategy="every_save"`
+
+**Note:** Default 30min is insufficient for real training. Minimum 1-2 hours.
+
+### Hub Push Failures
+
+**Fix:**
+1. Add to job: `secrets={"HF_TOKEN": "$HF_TOKEN"}`
+2. Add to config: `push_to_hub=True`, `hub_model_id="username/model-name"`
+3. Verify auth: `mcp__huggingface__hf_whoami()`
+4. Check token has write permissions and repo exists (or set `hub_private_repo=True`)
+
+### Missing Dependencies
+
+**Fix:**
+Add to PEP 723 header:
+```python
+# /// script
+# dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio", "missing-package"]
+# ///
+```
+
+## Troubleshooting
+
+**Common issues:**
+- Job times out → Increase timeout, reduce epochs/dataset, use smaller model/LoRA
+- Model not saved to Hub → Check push_to_hub=True, hub_model_id, secrets=HF_TOKEN
+- Out of Memory (OOM) → Reduce batch size, increase gradient accumulation, enable LoRA, use larger GPU
+- Dataset format error → Validate with dataset inspector (see Dataset Validation section)
+- Import/module errors → Add PEP 723 header with dependencies, verify format
+- Authentication errors → Check `mcp__huggingface__hf_whoami()`, token permissions, secrets parameter
+
+**See:** `references/troubleshooting.md` for complete troubleshooting guide
+
+## Resources
+
+### References (In This Skill)
+- `references/training_methods.md` - Overview of SFT, DPO, GRPO, KTO, PPO, Reward Modeling
+- `references/training_patterns.md` - Common training patterns and examples
+- `references/unsloth.md` - Unsloth for fast VLM training (~2x speed, 60% less VRAM)
+- `references/gguf_conversion.md` - Complete GGUF conversion guide
+- `references/trackio_guide.md` - Trackio monitoring setup
+- `references/hardware_guide.md` - Hardware specs and selection
+- `references/hub_saving.md` - Hub authentication troubleshooting
+- `references/troubleshooting.md` - Common issues and solutions
+
+### Scripts (In This Skill)
+- `scripts/train_sft_example.py` - Production SFT template
+- `scripts/train_dpo_example.py` - Production DPO template
+- `scripts/train_grpo_example.py` - Production GRPO template
+- `scripts/unsloth_sft_example.py` - Unsloth text LLM training template (faster, less VRAM)
+- `scripts/estimate_cost.py` - Estimate time and cost (offer when appropriate)
+- `scripts/convert_to_gguf.py` - Complete GGUF conversion script
+
+### External Scripts
+- [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Validate dataset format before training (use via `uv run` or `hf_jobs`)
+
+### External Links
+- [TRL Documentation](https://huggingface.co/docs/trl)
+- [TRL Jobs Training Guide](https://huggingface.co/docs/trl/en/jobs_training)
+- [TRL Jobs Package](https://github.com/huggingface/trl-jobs)
+- [HF Jobs Documentation](https://huggingface.co/docs/huggingface_hub/guides/jobs)
+- [TRL Example Scripts](https://github.com/huggingface/trl/tree/main/examples/scripts)
+- [UV Scripts Guide](https://docs.astral.sh/uv/guides/scripts/)
+- [UV Scripts Organization](https://huggingface.co/uv-scripts)
+
+## Key Takeaways
+
+1. **Submit scripts inline** - The `script` parameter accepts Python code directly; no file saving required unless user requests
+2. **Jobs are asynchronous** - Don't wait/poll; let user check when ready
+3. **Always set timeout** - Default 30 min is insufficient; minimum 1-2 hours recommended
+4. **Always enable Hub push** - Environment is ephemeral; without push, all results lost
+5. **Include Trackio** - Use example scripts as templates for real-time monitoring
+6. **Offer cost estimation** - When parameters are known, use `scripts/estimate_cost.py`
+7. **Use UV scripts (Approach 1)** - Default to `hf_jobs("uv", {...})` with inline scripts; TRL maintained scripts for standard training; avoid bash `trl-jobs` commands in Claude Code
+8. **Use hf_doc_fetch/hf_doc_search** for latest TRL documentation
+9. **Validate dataset format** before training with dataset inspector (see Dataset Validation section)
+10. **Choose appropriate hardware** for model size; use LoRA for models >7B

skills/hugging-face-model-trainer/references/gguf_conversion.md

@@ -0,0 +1,296 @@
+# GGUF Conversion Guide
+
+After training models with TRL on Hugging Face Jobs, convert them to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools.
+
+**This guide provides production-ready, tested code based on successful conversions.** All critical dependencies and build steps are included.
+
+## What is GGUF?
+
+**GGUF** (GPT-Generated Unified Format):
+- Optimized format for CPU/GPU inference with llama.cpp
+- Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size
+- Compatible with: Ollama, LM Studio, Jan, GPT4All, llama.cpp
+- Typically 2-8GB for 7B models (vs 14GB unquantized)
+
+## When to Convert to GGUF
+
+**Convert when:**
+- Running models locally with Ollama or LM Studio
+- Using CPU-optimized inference
+- Reducing model size with quantization
+- Deploying to edge devices
+- Sharing models for local-first use
+
+## Critical Success Factors
+
+Based on production testing, these are **essential** for reliable conversion:
+
+### 1. ✅ Install Build Tools FIRST
+**Before cloning llama.cpp**, install build dependencies:
+```python
+subprocess.run(["apt-get", "update", "-qq"], check=True, capture_output=True)
+subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True, capture_output=True)
+```
+
+**Why:** The quantization tool requires gcc and cmake. Installing after cloning doesn't help.
+
+### 2. ✅ Use CMake (Not Make)
+**Build the quantize tool with CMake:**
+```python
+# Create build directory
+os.makedirs("/tmp/llama.cpp/build", exist_ok=True)
+
+# Configure
+subprocess.run([
+    "cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
+    "-DGGML_CUDA=OFF"  # Faster build, CUDA not needed for quantization
+], check=True, capture_output=True, text=True)
+
+# Build
+subprocess.run([
+    "cmake", "--build", "/tmp/llama.cpp/build",
+    "--target", "llama-quantize", "-j", "4"
+], check=True, capture_output=True, text=True)
+
+# Binary path
+quantize_bin = "/tmp/llama.cpp/build/bin/llama-quantize"
+```
+
+**Why:** CMake is more reliable than `make` and produces consistent binary paths.
+
+### 3. ✅ Include All Dependencies
+**PEP 723 header must include:**
+```python
+# /// script
+# dependencies = [
+#     "transformers>=4.36.0",
+#     "peft>=0.7.0",
+#     "torch>=2.0.0",
+#     "accelerate>=0.24.0",
+#     "huggingface_hub>=0.20.0",
+#     "sentencepiece>=0.1.99",  # Required for tokenizer
+#     "protobuf>=3.20.0",        # Required for tokenizer
+#     "numpy",
+#     "gguf",
+# ]
+# ///
+```
+
+**Why:** `sentencepiece` and `protobuf` are critical for tokenizer conversion. Missing them causes silent failures.
+
+### 4. ✅ Verify Names Before Use
+**Always verify repos exist:**
+```python
+# Before submitting job, verify:
+hub_repo_details([ADAPTER_MODEL], repo_type="model")
+hub_repo_details([BASE_MODEL], repo_type="model")
+```
+
+**Why:** Non-existent dataset/model names cause job failures that could be caught in seconds.
+
+## Complete Conversion Script
+
+See `scripts/convert_to_gguf.py` for the complete, production-ready script.
+
+**Key features:**
+- ✅ All dependencies in PEP 723 header
+- ✅ Build tools installed automatically
+- ✅ CMake build process (reliable)
+- ✅ Comprehensive error handling
+- ✅ Environment variable configuration
+- ✅ Automatic README generation
+
+## Quick Conversion Job
+
+```python
+# Before submitting: VERIFY MODELS EXIST
+hub_repo_details(["username/my-finetuned-model"], repo_type="model")
+hub_repo_details(["Qwen/Qwen2.5-0.5B"], repo_type="model")
+
+# Submit conversion job
+hf_jobs("uv", {
+    "script": open("trl/scripts/convert_to_gguf.py").read(),  # Or inline the script
+    "flavor": "a10g-large",
+    "timeout": "45m",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
+    "env": {
+        "ADAPTER_MODEL": "username/my-finetuned-model",
+        "BASE_MODEL": "Qwen/Qwen2.5-0.5B",
+        "OUTPUT_REPO": "username/my-model-gguf",
+        "HF_USERNAME": "username"  # Optional, for README
+    }
+})
+```
+
+## Conversion Process
+
+The script performs these steps:
+
+1. **Load and Merge** - Load base model and LoRA adapter, merge them
+2. **Install Build Tools** - Install gcc, cmake (CRITICAL: before cloning llama.cpp)
+3. **Setup llama.cpp** - Clone repo, install Python dependencies
+4. **Convert to GGUF** - Create FP16 GGUF using llama.cpp converter
+5. **Build Quantize Tool** - Use CMake to build `llama-quantize`
+6. **Quantize** - Create Q4_K_M, Q5_K_M, Q8_0 versions
+7. **Upload** - Upload all versions + README to Hub
+
+## Quantization Options
+
+Common quantization formats (from smallest to largest):
+
+| Format | Size | Quality | Use Case |
+|--------|------|---------|----------|
+| **Q4_K_M** | ~300MB | Good | **Recommended** - best balance of size/quality |
+| **Q5_K_M** | ~350MB | Better | Higher quality, slightly larger |
+| **Q8_0** | ~500MB | Very High | Near-original quality |
+| **F16** | ~1GB | Original | Full precision, largest file |
+
+**Recommendation:** Create Q4_K_M, Q5_K_M, and Q8_0 versions to give users options.
+
+## Hardware Requirements
+
+**For conversion:**
+- Small models (<1B): CPU-basic works, but slow
+- Medium models (1-7B): a10g-large recommended
+- Large models (7B+): a10g-large or a100-large
+
+**Time estimates:**
+- 0.5B model: ~15-25 minutes on A10G
+- 3B model: ~30-45 minutes on A10G
+- 7B model: ~45-60 minutes on A10G
+
+## Using GGUF Models
+
+**GGUF models work on both CPU and GPU.** They're optimized for CPU inference but can also leverage GPU acceleration when available.
+
+### With Ollama (auto-detects GPU)
+```bash
+# Download GGUF
+huggingface-cli download username/my-model-gguf model-q4_k_m.gguf
+
+# Create Modelfile
+echo "FROM ./model-q4_k_m.gguf" > Modelfile
+
+# Create and run (uses GPU automatically if available)
+ollama create my-model -f Modelfile
+ollama run my-model
+```
+
+### With llama.cpp
+```bash
+# CPU only
+./llama-cli -m model-q4_k_m.gguf -p "Your prompt"
+
+# With GPU acceleration (offload 32 layers to GPU)
+./llama-cli -m model-q4_k_m.gguf -ngl 32 -p "Your prompt"
+```
+
+### With LM Studio
+1. Download the `.gguf` file
+2. Import into LM Studio
+3. Start chatting
+
+## Best Practices
+
+### ✅ DO:
+1. **Verify repos exist** before submitting jobs (use `hub_repo_details`)
+2. **Install build tools FIRST** before cloning llama.cpp
+3. **Use CMake** for building quantize tool (not make)
+4. **Include all dependencies** in PEP 723 header (especially sentencepiece, protobuf)
+5. **Create multiple quantizations** - Give users choice
+6. **Test on known models** before production use
+7. **Use A10G GPU** for faster conversion
+
+### ❌ DON'T:
+1. **Assume repos exist** - Always verify with hub tools
+2. **Use make** instead of CMake - Less reliable
+3. **Remove dependencies** to "simplify" - They're all needed
+4. **Skip build tools** - Quantization will fail silently
+5. **Use default paths** - CMake puts binaries in build/bin/
+
+## Common Issues
+
+### Out of memory during merge
+**Fix:**
+- Use larger GPU (a10g-large or a100-large)
+- Ensure `device_map="auto"` for automatic placement
+- Use `dtype=torch.float16` or `torch.bfloat16`
+
+### Conversion fails with architecture error
+**Fix:**
+- Ensure llama.cpp supports the model architecture
+- Check for standard architecture (Qwen, Llama, Mistral, etc.)
+- Update llama.cpp to latest: `git clone --depth 1 https://github.com/ggerganov/llama.cpp.git`
+- Check llama.cpp documentation for model support
+
+### Quantization fails
+**Fix:**
+- Verify build tools installed: `apt-get install build-essential cmake`
+- Use CMake (not make) to build quantize tool
+- Check binary path: `/tmp/llama.cpp/build/bin/llama-quantize`
+- Verify FP16 GGUF exists before quantizing
+
+### Missing sentencepiece error
+**Fix:**
+- Add to PEP 723 header: `"sentencepiece>=0.1.99", "protobuf>=3.20.0"`
+- Don't remove dependencies to "simplify" - all are required
+
+### Upload fails or times out
+**Fix:**
+- Large models (>2GB) need longer timeout: `"timeout": "1h"`
+- Upload quantized versions separately if needed
+- Check network/Hub status
+
+## Lessons Learned
+
+These are from production testing and real failures:
+
+### 1. Always Verify Before Use
+**Lesson:** Don't assume repos/datasets exist. Check first.
+```python
+# BEFORE submitting job
+hub_repo_details(["trl-lib/argilla-dpo-mix-7k"], repo_type="dataset")  # Would catch error
+```
+**Prevented failures:** Non-existent dataset names, typos in model names
+
+### 2. Prioritize Reliability Over Performance
+**Lesson:** Default to what's most likely to succeed.
+- Use CMake (not make) - more reliable
+- Disable CUDA in build - faster, not needed
+- Include all dependencies - don't "simplify"
+
+**Prevented failures:** Build failures, missing binaries
+
+### 3. Create Atomic, Self-Contained Scripts
+**Lesson:** Don't remove dependencies or steps. Scripts should work as a unit.
+- All dependencies in PEP 723 header
+- All build steps included
+- Clear error messages
+
+**Prevented failures:** Missing tokenizer libraries, build tool failures
+
+## References
+
+**In this skill:**
+- `scripts/convert_to_gguf.py` - Complete, production-ready script
+
+**External:**
+- [llama.cpp Repository](https://github.com/ggerganov/llama.cpp)
+- [GGUF Specification](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md)
+- [Ollama Documentation](https://ollama.ai)
+- [LM Studio](https://lmstudio.ai)
+
+## Summary
+
+**Critical checklist for GGUF conversion:**
+- [ ] Verify adapter and base models exist on Hub
+- [ ] Use production script from `scripts/convert_to_gguf.py`
+- [ ] All dependencies in PEP 723 header (including sentencepiece, protobuf)
+- [ ] Build tools installed before cloning llama.cpp
+- [ ] CMake used for building quantize tool (not make)
+- [ ] Correct binary path: `/tmp/llama.cpp/build/bin/llama-quantize`
+- [ ] A10G GPU selected for reasonable conversion time
+- [ ] Timeout set to 45m minimum
+- [ ] HF_TOKEN in secrets for Hub upload
+
+**The script in `scripts/convert_to_gguf.py` incorporates all these lessons and has been tested successfully in production.**

skills/hugging-face-model-trainer/references/hardware_guide.md

@@ -0,0 +1,283 @@
+# Hardware Selection Guide
+
+Choosing the right hardware (flavor) is critical for cost-effective training.
+
+## Available Hardware
+
+### CPU
+- `cpu-basic` - Basic CPU, testing only
+- `cpu-upgrade` - Enhanced CPU
+
+**Use cases:** Dataset validation, preprocessing, testing scripts
+**Not recommended for training:** Too slow for any meaningful training
+
+### GPU Options
+
+| Flavor | GPU | Memory | Use Case | Cost/hour |
+|--------|-----|--------|----------|-----------|
+| `t4-small` | NVIDIA T4 | 16GB | <1B models, demos | ~$0.50-1 |
+| `t4-medium` | NVIDIA T4 | 16GB | 1-3B models, development | ~$1-2 |
+| `l4x1` | NVIDIA L4 | 24GB | 3-7B models, efficient training | ~$2-3 |
+| `l4x4` | 4x NVIDIA L4 | 96GB | Multi-GPU training | ~$8-12 |
+| `a10g-small` | NVIDIA A10G | 24GB | 3-7B models, production | ~$3-4 |
+| `a10g-large` | NVIDIA A10G | 24GB | 7-13B models | ~$4-6 |
+| `a10g-largex2` | 2x NVIDIA A10G | 48GB | Multi-GPU, large models | ~$8-12 |
+| `a10g-largex4` | 4x NVIDIA A10G | 96GB | Multi-GPU, very large models | ~$16-24 |
+| `a100-large` | NVIDIA A100 | 40GB | 13B+ models, fast training | ~$8-12 |
+
+### TPU Options
+
+| Flavor | Type | Use Case |
+|--------|------|----------|
+| `v5e-1x1` | TPU v5e | Small TPU workloads |
+| `v5e-2x2` | 4x TPU v5e | Medium TPU workloads |
+| `v5e-2x4` | 8x TPU v5e | Large TPU workloads |
+
+**Note:** TPUs require TPU-optimized code. Most TRL training uses GPUs.
+
+## Selection Guidelines
+
+### By Model Size
+
+**Tiny Models (<1B parameters)**
+- **Recommended:** `t4-small`
+- **Example:** Qwen2.5-0.5B, TinyLlama
+- **Batch size:** 4-8
+- **Training time:** 1-2 hours for 1K examples
+
+**Small Models (1-3B parameters)**
+- **Recommended:** `t4-medium` or `a10g-small`
+- **Example:** Qwen2.5-1.5B, Phi-2
+- **Batch size:** 2-4
+- **Training time:** 2-4 hours for 10K examples
+
+**Medium Models (3-7B parameters)**
+- **Recommended:** `a10g-small` or `a10g-large`
+- **Example:** Qwen2.5-7B, Mistral-7B
+- **Batch size:** 1-2 (or LoRA with 4-8)
+- **Training time:** 4-8 hours for 10K examples
+
+**Large Models (7-13B parameters)**
+- **Recommended:** `a10g-large` or `a100-large`
+- **Example:** Llama-3-8B, Mixtral-8x7B (with LoRA)
+- **Batch size:** 1 (full fine-tuning) or 2-4 (LoRA)
+- **Training time:** 6-12 hours for 10K examples
+- **Note:** Always use LoRA/PEFT
+
+**Very Large Models (13B+ parameters)**
+- **Recommended:** `a100-large` with LoRA
+- **Example:** Llama-3-13B, Llama-3-70B (LoRA only)
+- **Batch size:** 1-2 with LoRA
+- **Training time:** 8-24 hours for 10K examples
+- **Note:** Full fine-tuning not feasible, use LoRA/PEFT
+
+### By Budget
+
+**Minimal Budget (<$5 total)**
+- Use `t4-small`
+- Train on subset of data (100-500 examples)
+- Limit to 1-2 epochs
+- Use small model (<1B)
+
+**Small Budget ($5-20)**
+- Use `t4-medium` or `a10g-small`
+- Train on 1K-5K examples
+- 2-3 epochs
+- Model up to 3B parameters
+
+**Medium Budget ($20-50)**
+- Use `a10g-small` or `a10g-large`
+- Train on 5K-20K examples
+- 3-5 epochs
+- Model up to 7B parameters
+
+**Large Budget ($50-200)**
+- Use `a10g-large` or `a100-large`
+- Full dataset training
+- Multiple epochs
+- Model up to 13B parameters with LoRA
+
+### By Training Type
+
+**Quick Demo/Experiment**
+- `t4-small`
+- 50-100 examples
+- 5-10 steps
+- ~10-15 minutes
+
+**Development/Iteration**
+- `t4-medium` or `a10g-small`
+- 1K examples
+- 1 epoch
+- ~30-60 minutes
+
+**Production Training**
+- `a10g-large` or `a100-large`
+- Full dataset
+- 3-5 epochs
+- 4-12 hours
+
+**Research/Experimentation**
+- `a100-large`
+- Multiple runs
+- Various hyperparameters
+- Budget for 20-50 hours
+
+## Memory Considerations
+
+### Estimating Memory Requirements
+
+**Full fine-tuning:**
+```
+Memory (GB) ≈ (Model params in billions) × 20
+```
+
+**LoRA fine-tuning:**
+```
+Memory (GB) ≈ (Model params in billions) × 4
+```
+
+**Examples:**
+- Qwen2.5-0.5B full: ~10GB ✅ fits t4-small
+- Qwen2.5-1.5B full: ~30GB ❌ exceeds most GPUs
+- Qwen2.5-1.5B LoRA: ~6GB ✅ fits t4-small
+- Qwen2.5-7B full: ~140GB ❌ not feasible
+- Qwen2.5-7B LoRA: ~28GB ✅ fits a10g-large
+
+### Memory Optimization
+
+If hitting memory limits:
+
+1. **Use LoRA/PEFT**
+   ```python
+   peft_config=LoraConfig(r=16, lora_alpha=32)
+   ```
+
+2. **Reduce batch size**
+   ```python
+   per_device_train_batch_size=1
+   ```
+
+3. **Increase gradient accumulation**
+   ```python
+   gradient_accumulation_steps=8  # Effective batch size = 1×8
+   ```
+
+4. **Enable gradient checkpointing**
+   ```python
+   gradient_checkpointing=True
+   ```
+
+5. **Use mixed precision**
+   ```python
+   bf16=True  # or fp16=True
+   ```
+
+6. **Upgrade to larger GPU**
+   - t4 → a10g → a100
+
+## Cost Estimation
+
+### Formula
+
+```
+Total Cost = (Hours of training) × (Cost per hour)
+```
+
+### Example Calculations
+
+**Quick demo:**
+- Hardware: t4-small ($0.75/hour)
+- Time: 15 minutes (0.25 hours)
+- Cost: $0.19
+
+**Development training:**
+- Hardware: a10g-small ($3.50/hour)
+- Time: 2 hours
+- Cost: $7.00
+
+**Production training:**
+- Hardware: a10g-large ($5/hour)
+- Time: 6 hours
+- Cost: $30.00
+
+**Large model with LoRA:**
+- Hardware: a100-large ($10/hour)
+- Time: 8 hours
+- Cost: $80.00
+
+### Cost Optimization Tips
+
+1. **Start small:** Test on t4-small with subset
+2. **Use LoRA:** 4-5x cheaper than full fine-tuning
+3. **Optimize hyperparameters:** Fewer epochs if possible
+4. **Set appropriate timeout:** Don't waste compute on stalled jobs
+5. **Use checkpointing:** Resume if job fails
+6. **Monitor costs:** Check running jobs regularly
+
+## Multi-GPU Training
+
+TRL automatically handles multi-GPU training with Accelerate when using multi-GPU flavors.
+
+**Multi-GPU flavors:**
+- `l4x4` - 4x L4 GPUs
+- `a10g-largex2` - 2x A10G GPUs
+- `a10g-largex4` - 4x A10G GPUs
+
+**When to use:**
+- Models >13B parameters
+- Need faster training (linear speedup)
+- Large datasets (>50K examples)
+
+**Example:**
+```python
+hf_jobs("uv", {
+    "script": "train.py",
+    "flavor": "a10g-largex2",  # 2 GPUs
+    "timeout": "4h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+No code changes needed—TRL/Accelerate handles distribution automatically.
+
+## Choosing Between Options
+
+### a10g vs a100
+
+**Choose a10g when:**
+- Model <13B parameters
+- Budget conscious
+- Training time not critical
+
+**Choose a100 when:**
+- Model 13B+ parameters
+- Need fastest training
+- Memory requirements high
+- Budget allows
+
+### Single vs Multi-GPU
+
+**Choose single GPU when:**
+- Model <7B parameters
+- Budget constrained
+- Simpler debugging
+
+**Choose multi-GPU when:**
+- Model >13B parameters
+- Need faster training
+- Large batch sizes required
+- Cost-effective for large jobs
+
+## Quick Reference
+
+```python
+# Model size → Hardware selection
+HARDWARE_MAP = {
+    "<1B":     "t4-small",
+    "1-3B":    "a10g-small",
+    "3-7B":    "a10g-large",
+    "7-13B":   "a10g-large (LoRA) or a100-large",
+    ">13B":    "a100-large (LoRA required)"
+}
+```

skills/hugging-face-model-trainer/references/hub_saving.md

@@ -0,0 +1,364 @@
+# Saving Training Results to Hugging Face Hub
+
+**⚠️ CRITICAL:** Training environments are ephemeral. ALL results are lost when a job completes unless pushed to the Hub.
+
+## Why Hub Push is Required
+
+When running on Hugging Face Jobs:
+- Environment is temporary
+- All files deleted on job completion
+- No local disk persistence
+- Cannot access results after job ends
+
+**Without Hub push, training is completely wasted.**
+
+## Required Configuration
+
+### 1. Training Configuration
+
+In your SFTConfig or trainer config:
+
+```python
+SFTConfig(
+    push_to_hub=True,                    # Enable Hub push
+    hub_model_id="username/model-name",   # Target repository
+)
+```
+
+### 2. Job Configuration
+
+When submitting the job:
+
+```python
+hf_jobs("uv", {
+    "script": "train.py",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # Provide authentication
+})
+```
+
+**The `$HF_TOKEN` placeholder is automatically replaced with your Hugging Face token.**
+
+## Complete Example
+
+```python
+# train.py
+# /// script
+# dependencies = ["trl"]
+# ///
+
+from trl import SFTTrainer, SFTConfig
+from datasets import load_dataset
+
+dataset = load_dataset("trl-lib/Capybara", split="train")
+
+# Configure with Hub push
+config = SFTConfig(
+    output_dir="my-model",
+    num_train_epochs=3,
+    
+    # ✅ CRITICAL: Hub push configuration
+    push_to_hub=True,
+    hub_model_id="myusername/my-trained-model",
+    
+    # Optional: Push strategy
+    push_to_hub_model_id="myusername/my-trained-model",
+    push_to_hub_organization=None,
+    push_to_hub_token=None,  # Uses environment token
+)
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+    args=config,
+)
+
+trainer.train()
+
+# ✅ Push final model
+trainer.push_to_hub()
+
+print("✅ Model saved to: https://huggingface.co/myusername/my-trained-model")
+```
+
+**Submit with authentication:**
+
+```python
+hf_jobs("uv", {
+    "script": "train.py",
+    "flavor": "a10g-large",
+    "timeout": "2h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # ✅ Required!
+})
+```
+
+## What Gets Saved
+
+When `push_to_hub=True`:
+
+1. **Model weights** - Final trained parameters
+2. **Tokenizer** - Associated tokenizer
+3. **Configuration** - Model config (config.json)
+4. **Training arguments** - Hyperparameters used
+5. **Model card** - Auto-generated documentation
+6. **Checkpoints** - If `save_strategy="steps"` enabled
+
+## Checkpoint Saving
+
+Save intermediate checkpoints during training:
+
+```python
+SFTConfig(
+    output_dir="my-model",
+    push_to_hub=True,
+    hub_model_id="username/my-model",
+    
+    # Checkpoint configuration
+    save_strategy="steps",
+    save_steps=100,              # Save every 100 steps
+    save_total_limit=3,          # Keep only last 3 checkpoints
+)
+```
+
+**Benefits:**
+- Resume training if job fails
+- Compare checkpoint performance
+- Use intermediate models
+
+**Checkpoints are pushed to:** `username/my-model` (same repo)
+
+## Authentication Methods
+
+### Method 1: Automatic Token (Recommended)
+
+```python
+"secrets": {"HF_TOKEN": "$HF_TOKEN"}
+```
+
+Uses your logged-in Hugging Face token automatically.
+
+### Method 2: Explicit Token
+
+```python
+"secrets": {"HF_TOKEN": "hf_abc123..."}
+```
+
+Provide token explicitly (not recommended for security).
+
+### Method 3: Environment Variable
+
+```python
+"env": {"HF_TOKEN": "hf_abc123..."}
+```
+
+Pass as regular environment variable (less secure than secrets).
+
+**Always prefer Method 1** for security and convenience.
+
+## Verification Checklist
+
+Before submitting any training job, verify:
+
+- [ ] `push_to_hub=True` in training config
+- [ ] `hub_model_id` is specified (format: `username/model-name`)
+- [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config
+- [ ] Repository name doesn't conflict with existing repos
+- [ ] You have write access to the target namespace
+
+## Repository Setup
+
+### Automatic Creation
+
+If repository doesn't exist, it's created automatically when first pushing.
+
+### Manual Creation
+
+Create repository before training:
+
+```python
+from huggingface_hub import HfApi
+
+api = HfApi()
+api.create_repo(
+    repo_id="username/model-name",
+    repo_type="model",
+    private=False,  # or True for private repo
+)
+```
+
+### Repository Naming
+
+**Valid names:**
+- `username/my-model`
+- `username/model-name`
+- `organization/model-name`
+
+**Invalid names:**
+- `model-name` (missing username)
+- `username/model name` (spaces not allowed)
+- `username/MODEL` (uppercase discouraged)
+
+## Troubleshooting
+
+### Error: 401 Unauthorized
+
+**Cause:** HF_TOKEN not provided or invalid
+
+**Solutions:**
+1. Verify `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config
+2. Check you're logged in: `huggingface-cli whoami`
+3. Re-login: `huggingface-cli login`
+
+### Error: 403 Forbidden
+
+**Cause:** No write access to repository
+
+**Solutions:**
+1. Check repository namespace matches your username
+2. Verify you're a member of organization (if using org namespace)
+3. Check repository isn't private (if accessing org repo)
+
+### Error: Repository not found
+
+**Cause:** Repository doesn't exist and auto-creation failed
+
+**Solutions:**
+1. Manually create repository first
+2. Check repository name format
+3. Verify namespace exists
+
+### Error: Push failed during training
+
+**Cause:** Network issues or Hub unavailable
+
+**Solutions:**
+1. Training continues but final push fails
+2. Checkpoints may be saved
+3. Re-run push manually after job completes
+
+### Issue: Model saved but not visible
+
+**Possible causes:**
+1. Repository is private—check https://huggingface.co/username
+2. Wrong namespace—verify `hub_model_id` matches login
+3. Push still in progress—wait a few minutes
+
+## Manual Push After Training
+
+If training completes but push fails, push manually:
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+# Load from local checkpoint
+model = AutoModel.from_pretrained("./output_dir")
+tokenizer = AutoTokenizer.from_pretrained("./output_dir")
+
+# Push to Hub
+model.push_to_hub("username/model-name", token="hf_abc123...")
+tokenizer.push_to_hub("username/model-name", token="hf_abc123...")
+```
+
+**Note:** Only possible if job hasn't completed (files still exist).
+
+## Best Practices
+
+1. **Always enable `push_to_hub=True`**
+2. **Use checkpoint saving** for long training runs
+3. **Verify Hub push** in logs before job completes
+4. **Set appropriate `save_total_limit`** to avoid excessive checkpoints
+5. **Use descriptive repo names** (e.g., `qwen-capybara-sft` not `model1`)
+6. **Add model card** with training details
+7. **Tag models** with relevant tags (e.g., `text-generation`, `fine-tuned`)
+
+## Monitoring Push Progress
+
+Check logs for push progress:
+
+```python
+hf_jobs("logs", {"job_id": "your-job-id"})
+```
+
+**Look for:**
+```
+Pushing model to username/model-name...
+Upload file pytorch_model.bin: 100%
+✅ Model pushed successfully
+```
+
+## Example: Full Production Setup
+
+```python
+# production_train.py
+# /// script
+# dependencies = ["trl>=0.12.0", "peft>=0.7.0"]
+# ///
+
+from datasets import load_dataset
+from peft import LoraConfig
+from trl import SFTTrainer, SFTConfig
+import os
+
+# Verify token is available
+assert "HF_TOKEN" in os.environ, "HF_TOKEN not found in environment!"
+
+# Load dataset
+dataset = load_dataset("trl-lib/Capybara", split="train")
+print(f"✅ Dataset loaded: {len(dataset)} examples")
+
+# Configure with comprehensive Hub settings
+config = SFTConfig(
+    output_dir="qwen-capybara-sft",
+    
+    # Hub configuration
+    push_to_hub=True,
+    hub_model_id="myusername/qwen-capybara-sft",
+    hub_strategy="checkpoint",  # Push checkpoints
+    
+    # Checkpoint configuration
+    save_strategy="steps",
+    save_steps=100,
+    save_total_limit=3,
+    
+    # Training settings
+    num_train_epochs=3,
+    per_device_train_batch_size=4,
+    
+    # Logging
+    logging_steps=10,
+    logging_first_step=True,
+)
+
+# Train with LoRA
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+    args=config,
+    peft_config=LoraConfig(r=16, lora_alpha=32),
+)
+
+print("🚀 Starting training...")
+trainer.train()
+
+print("💾 Pushing final model to Hub...")
+trainer.push_to_hub()
+
+print("✅ Training complete!")
+print(f"Model available at: https://huggingface.co/myusername/qwen-capybara-sft")
+```
+
+**Submit:**
+
+```python
+hf_jobs("uv", {
+    "script": "production_train.py",
+    "flavor": "a10g-large",
+    "timeout": "6h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+## Key Takeaway
+
+**Without `push_to_hub=True` and `secrets={"HF_TOKEN": "$HF_TOKEN"}`, all training results are permanently lost.**
+
+Always verify both are configured before submitting any training job.

skills/hugging-face-model-trainer/references/reliability_principles.md

@@ -0,0 +1,371 @@
+# Reliability Principles for Training Jobs
+
+These principles are derived from real production failures and successful fixes. Following them prevents common failure modes and ensures reliable job execution.
+
+## Principle 1: Always Verify Before Use
+
+**Rule:** Never assume repos, datasets, or resources exist. Verify with tools first.
+
+### What It Prevents
+
+- **Non-existent datasets** - Jobs fail immediately when dataset doesn't exist
+- **Typos in names** - Simple mistakes like "argilla-dpo-mix-7k" vs "ultrafeedback_binarized"
+- **Incorrect paths** - Old or moved repos, renamed files
+- **Missing dependencies** - Undocumented requirements
+
+### How to Apply
+
+**Before submitting ANY job:**
+
+```python
+# Verify dataset exists
+dataset_search({"query": "dataset-name", "author": "author-name", "limit": 5})
+hub_repo_details(["author/dataset-name"], repo_type="dataset")
+
+# Verify model exists
+hub_repo_details(["org/model-name"], repo_type="model")
+
+# Check script/file paths (for URL-based scripts)
+# Verify before using: https://github.com/user/repo/blob/main/script.py
+```
+
+**Examples that would have caught errors:**
+
+```python
+# ❌ WRONG: Assumed dataset exists
+hf_jobs("uv", {
+    "script": """...""",
+    "env": {"DATASET": "trl-lib/argilla-dpo-mix-7k"}  # Doesn't exist!
+})
+
+# ✅ CORRECT: Verify first
+dataset_search({"query": "argilla dpo", "author": "trl-lib"})
+# Would show: "trl-lib/ultrafeedback_binarized" is the correct name
+
+hub_repo_details(["trl-lib/ultrafeedback_binarized"], repo_type="dataset")
+# Confirms it exists before using
+```
+
+### Implementation Checklist
+
+- [ ] Check dataset exists before training
+- [ ] Verify base model exists before fine-tuning
+- [ ] Confirm adapter model exists before GGUF conversion
+- [ ] Test script URLs are valid before submitting
+- [ ] Validate file paths in repositories
+- [ ] Check for recent updates/renames of resources
+
+**Time cost:** 5-10 seconds  
+**Time saved:** Hours of failed job time + debugging
+
+---
+
+## Principle 2: Prioritize Reliability Over Performance
+
+**Rule:** Default to what is most likely to succeed, not what is theoretically fastest.
+
+### What It Prevents
+
+- **Hardware incompatibilities** - Features that fail on certain GPUs
+- **Unstable optimizations** - Speed-ups that cause crashes
+- **Complex configurations** - More failure points
+- **Build system issues** - Unreliable compilation methods
+
+### How to Apply
+
+**Choose reliability:**
+
+```python
+# ❌ RISKY: Aggressive optimization that may fail
+SFTConfig(
+    torch_compile=True,  # Can fail on T4, A10G GPUs
+    optim="adamw_bnb_8bit",  # Requires specific setup
+    fp16=False,  # May cause training instability
+    ...
+)
+
+# ✅ SAFE: Proven defaults
+SFTConfig(
+    # torch_compile=True,  # Commented with note: "Enable on H100 for 20% speedup"
+    optim="adamw_torch",  # Standard, always works
+    fp16=True,  # Stable and fast
+    ...
+)
+```
+
+**For build processes:**
+
+```python
+# ❌ UNRELIABLE: Uses make (platform-dependent)
+subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"], check=True)
+
+# ✅ RELIABLE: Uses CMake (consistent, documented)
+subprocess.run([
+    "cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp",
+    "-DGGML_CUDA=OFF"  # Disable CUDA for faster, more reliable build
+], check=True)
+
+subprocess.run([
+    "cmake", "--build", "/tmp/llama.cpp/build",
+    "--target", "llama-quantize", "-j", "4"
+], check=True)
+```
+
+### Real-World Example
+
+**The `torch.compile` failure:**
+- Added for "20% speedup" on H100
+- **Failed fatally on T4-medium** with cryptic error
+- Misdiagnosed as dataset issue (cost hours)
+- **Fix:** Disable by default, add as optional comment
+
+**Result:** Reliability > 20% performance gain
+
+### Implementation Checklist
+
+- [ ] Use proven, standard configurations by default
+- [ ] Comment out performance optimizations with hardware notes
+- [ ] Use stable build systems (CMake > make)
+- [ ] Test on target hardware before production
+- [ ] Document known incompatibilities
+- [ ] Provide "safe" and "fast" variants when needed
+
+**Performance loss:** 10-20% in best case  
+**Reliability gain:** 95%+ success rate vs 60-70%
+
+---
+
+## Principle 3: Create Atomic, Self-Contained Scripts
+
+**Rule:** Scripts should work as complete, independent units. Don't remove parts to "simplify."
+
+### What It Prevents
+
+- **Missing dependencies** - Removed "unnecessary" packages that are actually required
+- **Incomplete processes** - Skipped steps that seem redundant
+- **Environment assumptions** - Scripts that need pre-setup
+- **Partial failures** - Some parts work, others fail silently
+
+### How to Apply
+
+**Complete dependency specifications:**
+
+```python
+# ❌ INCOMPLETE: "Simplified" by removing dependencies
+# /// script
+# dependencies = [
+#     "transformers",
+#     "peft",
+#     "torch",
+# ]
+# ///
+
+# ✅ COMPLETE: All dependencies explicit
+# /// script
+# dependencies = [
+#     "transformers>=4.36.0",
+#     "peft>=0.7.0",
+#     "torch>=2.0.0",
+#     "accelerate>=0.24.0",
+#     "huggingface_hub>=0.20.0",
+#     "sentencepiece>=0.1.99",  # Required for tokenizers
+#     "protobuf>=3.20.0",        # Required for tokenizers
+#     "numpy",
+#     "gguf",
+# ]
+# ///
+```
+
+**Complete build processes:**
+
+```python
+# ❌ INCOMPLETE: Assumes build tools exist
+subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
+subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"])  # FAILS: no gcc/make
+
+# ✅ COMPLETE: Installs all requirements
+subprocess.run(["apt-get", "update", "-qq"], check=True)
+subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True)
+subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"])
+# ... then build
+```
+
+### Real-World Example
+
+**The `sentencepiece` failure:**
+- Original script had it: worked fine
+- "Simplified" version removed it: "doesn't look necessary"
+- **GGUF conversion failed silently** - tokenizer couldn't convert
+- Hard to debug: no obvious error message
+- **Fix:** Restore all original dependencies
+
+**Result:** Don't remove dependencies without thorough testing
+
+### Implementation Checklist
+
+- [ ] All dependencies in PEP 723 header with version pins
+- [ ] All system packages installed by script
+- [ ] No assumptions about pre-existing environment
+- [ ] No "optional" steps that are actually required
+- [ ] Test scripts in clean environment
+- [ ] Document why each dependency is needed
+
+**Complexity:** Slightly longer scripts  
+**Reliability:** Scripts "just work" every time
+
+---
+
+## Principle 4: Provide Clear Error Context
+
+**Rule:** When things fail, make it obvious what went wrong and how to fix it.
+
+### How to Apply
+
+**Wrap subprocess calls:**
+
+```python
+# ❌ UNCLEAR: Silent failure
+subprocess.run([...], check=True, capture_output=True)
+
+# ✅ CLEAR: Shows what failed
+try:
+    result = subprocess.run(
+        [...],
+        check=True,
+        capture_output=True,
+        text=True
+    )
+    print(result.stdout)
+    if result.stderr:
+        print("Warnings:", result.stderr)
+except subprocess.CalledProcessError as e:
+    print(f"❌ Command failed!")
+    print("STDOUT:", e.stdout)
+    print("STDERR:", e.stderr)
+    raise
+```
+
+**Validate inputs:**
+
+```python
+# ❌ UNCLEAR: Fails later with cryptic error
+model = load_model(MODEL_NAME)
+
+# ✅ CLEAR: Fails fast with clear message
+if not MODEL_NAME:
+    raise ValueError("MODEL_NAME environment variable not set!")
+
+print(f"Loading model: {MODEL_NAME}")
+try:
+    model = load_model(MODEL_NAME)
+    print(f"✅ Model loaded successfully")
+except Exception as e:
+    print(f"❌ Failed to load model: {MODEL_NAME}")
+    print(f"Error: {e}")
+    print("Hint: Check that model exists on Hub")
+    raise
+```
+
+### Implementation Checklist
+
+- [ ] Wrap external calls with try/except
+- [ ] Print stdout/stderr on failure
+- [ ] Validate environment variables early
+- [ ] Add progress indicators (✅, ❌, 🔄)
+- [ ] Include hints for common failures
+- [ ] Log configuration at start
+
+---
+
+## Principle 5: Test the Happy Path on Known-Good Inputs
+
+**Rule:** Before using new code in production, test with inputs you know work.
+
+### How to Apply
+
+**Known-good test inputs:**
+
+```python
+# For training
+TEST_DATASET = "trl-lib/Capybara"  # Small, well-formatted, widely used
+TEST_MODEL = "Qwen/Qwen2.5-0.5B"  # Small, fast, reliable
+
+# For GGUF conversion
+TEST_ADAPTER = "evalstate/qwen-capybara-medium"  # Known working model
+TEST_BASE = "Qwen/Qwen2.5-0.5B"  # Compatible base
+```
+
+**Testing workflow:**
+
+1. Test with known-good inputs first
+2. If that works, try production inputs
+3. If production fails, you know it's the inputs (not code)
+4. Isolate the difference
+
+### Implementation Checklist
+
+- [ ] Maintain list of known-good test models/datasets
+- [ ] Test new scripts with test inputs first
+- [ ] Document what makes inputs "good"
+- [ ] Keep test jobs cheap (small models, short timeouts)
+- [ ] Only move to production after test succeeds
+
+**Time cost:** 5-10 minutes for test run  
+**Debugging time saved:** Hours
+
+---
+
+## Summary: The Reliability Checklist
+
+Before submitting ANY job:
+
+### Pre-Flight Checks
+- [ ] **Verified** all repos/datasets exist (hub_repo_details)
+- [ ] **Tested** with known-good inputs if new code
+- [ ] **Using** proven hardware/configuration
+- [ ] **Included** all dependencies in PEP 723 header
+- [ ] **Installed** system requirements (build tools, etc.)
+- [ ] **Set** appropriate timeout (not default 30m)
+- [ ] **Configured** Hub push with HF_TOKEN
+- [ ] **Added** clear error handling
+
+### Script Quality
+- [ ] Self-contained (no external setup needed)
+- [ ] Complete dependencies listed
+- [ ] Build tools installed by script
+- [ ] Progress indicators included
+- [ ] Error messages are clear
+- [ ] Configuration logged at start
+
+### Job Configuration
+- [ ] Timeout > expected runtime + 30% buffer
+- [ ] Hardware appropriate for model size
+- [ ] Secrets include HF_TOKEN
+- [ ] Environment variables set correctly
+- [ ] Cost estimated and acceptable
+
+**Following these principles transforms job success rate from ~60-70% to ~95%+**
+
+---
+
+## When Principles Conflict
+
+Sometimes reliability and performance conflict. Here's how to choose:
+
+| Scenario | Choose | Rationale |
+|----------|--------|-----------|
+| Demo/test | Reliability | Fast failure is worse than slow success |
+| Production (first run) | Reliability | Prove it works before optimizing |
+| Production (proven) | Performance | Safe to optimize after validation |
+| Time-critical | Reliability | Failures cause more delay than slow runs |
+| Cost-critical | Balanced | Test with small model, then optimize |
+
+**General rule:** Reliability first, optimize second.
+
+---
+
+## Further Reading
+
+- `troubleshooting.md` - Common issues and fixes
+- `training_patterns.md` - Proven training configurations
+- `gguf_conversion.md` - Production GGUF workflow

skills/hugging-face-model-trainer/references/trackio_guide.md

@@ -0,0 +1,189 @@
+# Trackio Integration for TRL Training
+
+**Trackio** is an experiment tracking library that provides real-time metrics visualization for remote training on Hugging Face Jobs infrastructure.
+
+⚠️ **IMPORTANT**: For Jobs training (remote cloud GPUs):
+- Training happens on ephemeral cloud runners (not your local machine)
+- Trackio syncs metrics to a Hugging Face Space for real-time monitoring
+- Without a Space, metrics are lost when the job completes
+- The Space dashboard persists your training metrics permanently
+
+## Setting Up Trackio for Jobs
+
+**Step 1: Add trackio dependency**
+```python
+# /// script
+# dependencies = [
+#     "trl>=0.12.0",
+#     "trackio",  # Required!
+# ]
+# ///
+```
+
+**Step 2: Create a Trackio Space (one-time setup)**
+
+**Option A: Let Trackio auto-create (Recommended)**
+Pass a `space_id` to `trackio.init()` and Trackio will automatically create the Space if it doesn't exist.
+
+**Option B: Create manually**
+- Create Space via Hub UI at https://huggingface.co/new-space
+- Select Gradio SDK
+- OR use command: `huggingface-cli repo create my-trackio-dashboard --type space --space_sdk gradio`
+
+**Step 3: Initialize Trackio with space_id**
+```python
+import trackio
+
+trackio.init(
+    project="my-training",
+    space_id="username/trackio",  # CRITICAL for Jobs! Replace 'username' with your HF username
+    config={
+        "model": "Qwen/Qwen2.5-0.5B",
+        "dataset": "trl-lib/Capybara",
+        "learning_rate": 2e-5,
+    }
+)
+```
+
+**Step 4: Configure TRL to use Trackio**
+```python
+SFTConfig(
+    report_to="trackio",
+    # ... other config
+)
+```
+
+**Step 5: Finish tracking**
+```python
+trainer.train()
+trackio.finish()  # Ensures final metrics are synced
+```
+
+## What Trackio Tracks
+
+Trackio automatically logs:
+- ✅ Training loss
+- ✅ Learning rate
+- ✅ GPU utilization
+- ✅ Memory usage
+- ✅ Training throughput
+- ✅ Custom metrics
+
+## How It Works with Jobs
+
+1. **Training runs** → Metrics logged to local SQLite DB
+2. **Every 5 minutes** → Trackio syncs DB to HF Dataset (Parquet)
+3. **Space dashboard** → Reads from Dataset, displays metrics in real-time
+4. **Job completes** → Final sync ensures all metrics persisted
+
+## Default Configuration Pattern
+
+**Use sensible defaults for trackio configuration unless user requests otherwise.**
+
+### Recommended Defaults
+
+```python
+import trackio
+
+trackio.init(
+    project="qwen-capybara-sft",
+    name="baseline-run",             # Descriptive name user will recognize
+    space_id="username/trackio",     # Default space: {username}/trackio
+    config={
+        # Keep config minimal - hyperparameters and model/dataset info only
+        "model": "Qwen/Qwen2.5-0.5B",
+        "dataset": "trl-lib/Capybara",
+        "learning_rate": 2e-5,
+        "num_epochs": 3,
+    }
+)
+```
+
+**Key principles:**
+- **Space ID**: Use `{username}/trackio` with "trackio" as default space name
+- **Run naming**: Unless otherwise specified, name the run in a way the user will recognize
+- **Config**: Keep minimal - don't automatically capture job metadata unless requested
+- **Grouping**: Optional - only use if user requests organizing related experiments
+
+## Grouping Runs (Optional)
+
+The `group` parameter helps organize related runs together in the dashboard sidebar. This is useful when user is running multiple experiments with different configurations but wants to compare them together:
+
+```python
+# Example: Group runs by experiment type
+trackio.init(project="my-project", run_name="baseline-run-1", group="baseline")
+trackio.init(project="my-project", run_name="augmented-run-1", group="augmented")
+trackio.init(project="my-project", run_name="tuned-run-1", group="tuned")
+```
+
+Runs with the same group name can be grouped together in the sidebar, making it easier to compare related experiments. You can group by any configuration parameter:
+
+```python
+# Hyperparameter sweep - group by learning rate
+trackio.init(project="hyperparam-sweep", run_name="lr-0.001-run", group="lr_0.001")
+trackio.init(project="hyperparam-sweep", run_name="lr-0.01-run", group="lr_0.01")
+```
+
+## Environment Variables for Jobs
+
+You can configure trackio using environment variables instead of passing parameters to `trackio.init()`. This is useful for managing configuration across multiple jobs.
+
+
+
+**`HF_TOKEN`**
+Required for creating Spaces and writing to datasets (passed via `secrets`):
+```python
+hf_jobs("uv", {
+    "script": "...",
+    "secrets": {
+        "HF_TOKEN": "$HF_TOKEN"  # Enables Space creation and Hub push
+    }
+})
+```
+
+### Example with Environment Variables
+
+```python
+hf_jobs("uv", {
+    "script": """
+# Training script - trackio config from environment
+import trackio
+from datetime import datetime
+
+# Auto-generate run name
+timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M")
+run_name = f"sft_qwen25_{timestamp}"
+
+# Project and space_id can come from environment variables
+trackio.init(run_name=run_name, group="SFT")
+
+# ... training code ...
+trackio.finish()
+""",
+    "flavor": "a10g-large",
+    "timeout": "2h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**When to use environment variables:**
+- Managing multiple jobs with same configuration
+- Keeping training scripts portable across projects
+- Separating configuration from code
+
+**When to use direct parameters:**
+- Single job with specific configuration
+- When clarity in code is preferred
+- When each job has different project/space
+
+## Viewing the Dashboard
+
+After starting training:
+1. Navigate to the Space: `https://huggingface.co/spaces/username/trackio`
+2. The Gradio dashboard shows all tracked experiments
+3. Filter by project, compare runs, view charts with smoothing
+
+## Recommendation
+
+- **Trackio**: Best for real-time monitoring during long training runs
+- **Weights & Biases**: Best for team collaboration, requires account

skills/hugging-face-model-trainer/references/training_methods.md

@@ -0,0 +1,150 @@
+# TRL Training Methods Overview
+
+TRL (Transformer Reinforcement Learning) provides multiple training methods for fine-tuning and aligning language models. This reference provides a brief overview of each method.
+
+## Supervised Fine-Tuning (SFT)
+
+**What it is:** Standard instruction tuning with supervised learning on demonstration data.
+
+**When to use:**
+- Initial fine-tuning of base models on task-specific data
+- Teaching new capabilities or domains
+- Most common starting point for fine-tuning
+
+**Dataset format:** Conversational format with "messages" field, OR text field, OR prompt/completion pairs
+
+**Example:**
+```python
+from trl import SFTTrainer, SFTConfig
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+    args=SFTConfig(
+        output_dir="my-model",
+        push_to_hub=True,
+        hub_model_id="username/my-model",
+        eval_strategy="no",  # Disable eval for simple example
+        # max_length=1024 is the default - only set if you need different length
+    )
+)
+trainer.train()
+```
+
+**Note:** For production training with evaluation monitoring, see `scripts/train_sft_example.py`
+
+**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer")`
+
+## Direct Preference Optimization (DPO)
+
+**What it is:** Alignment method that trains directly on preference pairs (chosen vs rejected responses) without requiring a reward model.
+
+**When to use:**
+- Aligning models to human preferences
+- Improving response quality after SFT
+- Have paired preference data (chosen/rejected responses)
+
+**Dataset format:** Preference pairs with "chosen" and "rejected" fields
+
+**Example:**
+```python
+from trl import DPOTrainer, DPOConfig
+
+trainer = DPOTrainer(
+    model="Qwen/Qwen2.5-0.5B-Instruct",  # Use instruct model
+    train_dataset=dataset,
+    args=DPOConfig(
+        output_dir="dpo-model",
+        beta=0.1,  # KL penalty coefficient
+        eval_strategy="no",  # Disable eval for simple example
+        # max_length=1024 is the default - only set if you need different length
+    )
+)
+trainer.train()
+```
+
+**Note:** For production training with evaluation monitoring, see `scripts/train_dpo_example.py`
+
+**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")`
+
+## Group Relative Policy Optimization (GRPO)
+
+**What it is:** Online RL method that optimizes relative to group performance, useful for tasks with verifiable rewards.
+
+**When to use:**
+- Tasks with automatic reward signals (code execution, math verification)
+- Online learning scenarios
+- When DPO offline data is insufficient
+
+**Dataset format:** Prompt-only format (model generates responses, reward computed online)
+
+**Example:**
+```python
+# Use TRL maintained script
+hf_jobs("uv", {
+    "script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py",
+    "script_args": [
+        "--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct",
+        "--dataset_name", "trl-lib/math_shepherd",
+        "--output_dir", "grpo-model"
+    ],
+    "flavor": "a10g-large",
+    "timeout": "4h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")`
+
+## Reward Modeling
+
+**What it is:** Train a reward model to score responses, used as a component in RLHF pipelines.
+
+**When to use:**
+- Building RLHF pipeline
+- Need automatic quality scoring
+- Creating reward signals for PPO training
+
+**Dataset format:** Preference pairs with "chosen" and "rejected" responses
+
+**Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/reward_trainer")`
+
+## Method Selection Guide
+
+| Method | Complexity | Data Required | Use Case |
+|--------|-----------|---------------|----------|
+| **SFT** | Low | Demonstrations | Initial fine-tuning |
+| **DPO** | Medium | Paired preferences | Post-SFT alignment |
+| **GRPO** | Medium | Prompts + reward fn | Online RL with automatic rewards |
+| **Reward** | Medium | Paired preferences | Building RLHF pipeline |
+
+## Recommended Pipeline
+
+**For most use cases:**
+1. **Start with SFT** - Fine-tune base model on task data
+2. **Follow with DPO** - Align to preferences using paired data
+3. **Optional: GGUF conversion** - Deploy for local inference
+
+**For advanced RL scenarios:**
+1. **Start with SFT** - Fine-tune base model
+2. **Train reward model** - On preference data
+
+## Dataset Format Reference
+
+For complete dataset format specifications, use:
+```python
+hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats")
+```
+
+Or validate your dataset:
+```bash
+uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \
+  --dataset your/dataset --split train
+```
+
+## See Also
+
+- `references/training_patterns.md` - Common training patterns and examples
+- `scripts/train_sft_example.py` - Complete SFT template
+- `scripts/train_dpo_example.py` - Complete DPO template
+- [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Dataset format validation tool

skills/hugging-face-model-trainer/references/training_patterns.md

@@ -0,0 +1,203 @@
+# Common Training Patterns
+
+This guide provides common training patterns and use cases for TRL on Hugging Face Jobs.
+
+## Multi-GPU Training
+
+Automatic distributed training across multiple GPUs. TRL/Accelerate handles distribution automatically:
+
+```python
+hf_jobs("uv", {
+    "script": """
+# Your training script here (same as single GPU)
+# No changes needed - Accelerate detects multiple GPUs
+""",
+    "flavor": "a10g-largex2",  # 2x A10G GPUs
+    "timeout": "4h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**Tips for multi-GPU:**
+- No code changes needed
+- Use `per_device_train_batch_size` (per GPU, not total)
+- Effective batch size = `per_device_train_batch_size` × `num_gpus` × `gradient_accumulation_steps`
+- Monitor GPU utilization to ensure both GPUs are being used
+
+## DPO Training (Preference Learning)
+
+Train with preference data for alignment:
+
+```python
+hf_jobs("uv", {
+    "script": """
+# /// script
+# dependencies = ["trl>=0.12.0", "trackio"]
+# ///
+
+from datasets import load_dataset
+from trl import DPOTrainer, DPOConfig
+import trackio
+
+dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
+
+# Create train/eval split
+dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
+
+config = DPOConfig(
+    output_dir="dpo-model",
+    push_to_hub=True,
+    hub_model_id="username/dpo-model",
+    num_train_epochs=1,
+    beta=0.1,  # KL penalty coefficient
+    eval_strategy="steps",
+    eval_steps=50,
+    report_to="trackio",
+    run_name="baseline_run", # use a meaningful run name
+    # max_length=1024,  # Default - only set if you need different sequence length
+)
+
+trainer = DPOTrainer(
+    model="Qwen/Qwen2.5-0.5B-Instruct",  # Use instruct model as base
+    train_dataset=dataset_split["train"],
+    eval_dataset=dataset_split["test"],  # IMPORTANT: Provide eval_dataset when eval_strategy is enabled
+    args=config,
+)
+
+trainer.train()
+trainer.push_to_hub()
+trackio.finish()
+""",
+    "flavor": "a10g-large",
+    "timeout": "3h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**For DPO documentation:** Use `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")`
+
+## GRPO Training (Online RL)
+
+Group Relative Policy Optimization for online reinforcement learning:
+
+```python
+hf_jobs("uv", {
+    "script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py",
+    "script_args": [
+        "--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct",
+        "--dataset_name", "trl-lib/math_shepherd",
+        "--output_dir", "grpo-model",
+        "--push_to_hub",
+        "--hub_model_id", "username/grpo-model"
+    ],
+    "flavor": "a10g-large",
+    "timeout": "4h",
+    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
+})
+```
+
+**For GRPO documentation:** Use `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")`
+
+## Trackio Configuration
+
+**Use sensible defaults for trackio setup.** See `references/trackio_guide.md` for complete documentation including grouping runs for experiments.
+
+### Basic Pattern
+
+```python
+import trackio
+
+trackio.init(
+    project="my-training",
+    run_name="baseline-run",             # Descriptive name user will recognize
+    space_id="username/trackio",     # Default space: {username}/trackio
+    config={
+        # Keep config minimal - hyperparameters and model/dataset info only
+        "model": "Qwen/Qwen2.5-0.5B",
+        "dataset": "trl-lib/Capybara",
+        "learning_rate": 2e-5,
+    }
+)
+
+# Your training code...
+
+trackio.finish()
+```
+
+### Grouping for Experiments (Optional)
+
+When user wants to compare related runs, use the `group` parameter:
+
+```python
+# Hyperparameter sweep
+trackio.init(project="hyperparam-sweep", run_name="lr-0.001", group="lr_0.001")
+trackio.init(project="hyperparam-sweep", run_name="lr-0.01", group="lr_0.01")
+```
+
+## Pattern Selection Guide
+
+| Use Case | Pattern | Hardware | Time |
+|----------|---------|----------|------|
+| SFT training | `scripts/train_sft_example.py` | a10g-large | 2-6 hours |
+| Large dataset (>10K) | Multi-GPU | a10g-largex2 | 4-12 hours |
+| Preference learning | DPO Training | a10g-large | 2-4 hours |
+| Online RL | GRPO Training | a10g-large | 3-6 hours |
+
+## Critical: Evaluation Dataset Requirements
+
+**⚠️ IMPORTANT**: If you set `eval_strategy="steps"` or `eval_strategy="epoch"`, you **MUST** provide an `eval_dataset` to the trainer, or the training will hang.
+
+### ✅ CORRECT - With eval dataset:
+```python
+dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset_split["train"],
+    eval_dataset=dataset_split["test"],  # ← MUST provide when eval_strategy is enabled
+    args=SFTConfig(eval_strategy="steps", ...),
+)
+```
+
+### ❌ WRONG - Will hang:
+```python
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+    # NO eval_dataset but eval_strategy="steps" ← WILL HANG
+    args=SFTConfig(eval_strategy="steps", ...),
+)
+```
+
+### Option: Disable evaluation if no eval dataset
+```python
+config = SFTConfig(
+    eval_strategy="no",  # ← Explicitly disable evaluation
+    # ... other config
+)
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+    # No eval_dataset needed
+    args=config,
+)
+```
+
+## Best Practices
+
+1. **Use train/eval splits** - Create evaluation split for monitoring progress
+2. **Enable Trackio** - Monitor progress in real-time
+3. **Add 20-30% buffer to timeout** - Account for loading/saving overhead
+4. **Test with TRL official scripts first** - Use maintained examples before custom code
+5. **Always provide eval_dataset** - When using eval_strategy, or set to "no"
+6. **Use multi-GPU for large models** - 7B+ models benefit significantly
+
+## See Also
+
+- `scripts/train_sft_example.py` - Complete SFT template with Trackio and eval split
+- `scripts/train_dpo_example.py` - Complete DPO template
+- `scripts/train_grpo_example.py` - Complete GRPO template
+- `references/hardware_guide.md` - Detailed hardware specifications
+- `references/training_methods.md` - Overview of all TRL training methods
+- `references/troubleshooting.md` - Common issues and solutions

skills/hugging-face-model-trainer/references/troubleshooting.md

@@ -0,0 +1,282 @@
+# Troubleshooting TRL Training Jobs
+
+Common issues and solutions when training with TRL on Hugging Face Jobs.
+
+## Training Hangs at "Starting training..." Step
+
+**Problem:** Job starts but hangs at the training step - never progresses, never times out, just sits there.
+
+**Root Cause:** Using `eval_strategy="steps"` or `eval_strategy="epoch"` without providing an `eval_dataset` to the trainer.
+
+**Solution:**
+
+**Option A: Provide eval_dataset (recommended)**
+```python
+# Create train/eval split
+dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset_split["train"],
+    eval_dataset=dataset_split["test"],  # ← MUST provide when eval_strategy is enabled
+    args=SFTConfig(
+        eval_strategy="steps",
+        eval_steps=50,
+        ...
+    ),
+)
+```
+
+**Option B: Disable evaluation**
+```python
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-0.5B",
+    train_dataset=dataset,
+    # No eval_dataset
+    args=SFTConfig(
+        eval_strategy="no",  # ← Explicitly disable
+        ...
+    ),
+)
+```
+
+**Prevention:**
+- Always create train/eval split for better monitoring
+- Use `dataset.train_test_split(test_size=0.1, seed=42)`
+- Check example scripts: `scripts/train_sft_example.py` includes proper eval setup
+
+## Job Times Out
+
+**Problem:** Job terminates before training completes, all progress lost.
+
+**Solutions:**
+- Increase timeout parameter (e.g., `"timeout": "4h"`)
+- Reduce `num_train_epochs` or use smaller dataset slice
+- Use smaller model or enable LoRA/PEFT to speed up training
+- Add 20-30% buffer to estimated time for loading/saving overhead
+
+**Prevention:**
+- Always start with a quick demo run to estimate timing
+- Use `scripts/estimate_cost.py` to get time estimates
+- Monitor first runs closely via Trackio or logs
+
+## Model Not Saved to Hub
+
+**Problem:** Training completes but model doesn't appear on Hub - all work lost.
+
+**Check:**
+- [ ] `push_to_hub=True` in training config
+- [ ] `hub_model_id` specified with username (e.g., `"username/model-name"`)
+- [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job submission
+- [ ] User has write access to target repo
+- [ ] Token has write permissions (check at https://huggingface.co/settings/tokens)
+- [ ] Training script calls `trainer.push_to_hub()` at the end
+
+**See:** `references/hub_saving.md` for detailed Hub authentication troubleshooting
+
+## Out of Memory (OOM)
+
+**Problem:** Job fails with CUDA out of memory error.
+
+**Solutions (in order of preference):**
+1. **Reduce batch size:** Lower `per_device_train_batch_size` (try 4 → 2 → 1)
+2. **Increase gradient accumulation:** Raise `gradient_accumulation_steps` to maintain effective batch size
+3. **Disable evaluation:** Remove `eval_dataset` and `eval_strategy` (saves ~40% memory, good for demos)
+4. **Enable LoRA/PEFT:** Use `peft_config=LoraConfig(r=8, lora_alpha=16)` to train adapters only (smaller rank = less memory)
+5. **Use larger GPU:** Switch from `t4-small` → `l4x1` → `a10g-large` → `a100-large`
+6. **Enable gradient checkpointing:** Set `gradient_checkpointing=True` in config (slower but saves memory)
+7. **Use smaller model:** Try a smaller variant (e.g., 0.5B instead of 3B)
+
+**Memory guidelines:**
+- T4 (16GB): <1B models with LoRA
+- A10G (24GB): 1-3B models with LoRA, <1B full fine-tune
+- A100 (40GB/80GB): 7B+ models with LoRA, 3B full fine-tune
+
+## Parameter Naming Issues
+
+**Problem:** `TypeError: SFTConfig.__init__() got an unexpected keyword argument 'max_seq_length'`
+
+**Cause:** TRL config classes use `max_length`, not `max_seq_length`.
+
+**Solution:**
+```python
+# ✅ CORRECT - TRL uses max_length
+SFTConfig(max_length=512)
+DPOConfig(max_length=512)
+
+# ❌ WRONG - This will fail
+SFTConfig(max_seq_length=512)
+```
+
+**Note:** Most TRL configs don't require explicit max_length - the default (1024) works well. Only set if you need a specific value.
+
+## Dataset Format Error
+
+**Problem:** Training fails with dataset format errors or missing fields.
+
+**Solutions:**
+1. **Check format documentation:**
+   ```python
+   hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats")
+   ```
+
+2. **Validate dataset before training:**
+   ```bash
+   uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \
+     --dataset <dataset-name> --split train
+   ```
+   Or via hf_jobs:
+   ```python
+   hf_jobs("uv", {
+       "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py",
+       "script_args": ["--dataset", "dataset-name", "--split", "train"]
+   })
+   ```
+
+3. **Verify field names:**
+   - **SFT:** Needs "messages" field (conversational), OR "text" field, OR "prompt"/"completion"
+   - **DPO:** Needs "chosen" and "rejected" fields
+   - **GRPO:** Needs prompt-only format
+
+4. **Check dataset split:**
+   - Ensure split exists (e.g., `split="train"`)
+   - Preview dataset: `load_dataset("name", split="train[:5]")`
+
+## Import/Module Errors
+
+**Problem:** Job fails with "ModuleNotFoundError" or import errors.
+
+**Solutions:**
+1. **Add PEP 723 header with dependencies:**
+   ```python
+   # /// script
+   # dependencies = [
+   #     "trl>=0.12.0",
+   #     "peft>=0.7.0",
+   #     "transformers>=4.36.0",
+   # ]
+   # ///
+   ```
+
+2. **Verify exact format:**
+   - Must have `# ///` delimiters (with space after `#`)
+   - Dependencies must be valid PyPI package names
+   - Check spelling and version constraints
+
+3. **Test locally first:**
+   ```bash
+   uv run train.py  # Tests if dependencies are correct
+   ```
+
+## Authentication Errors
+
+**Problem:** Job fails with authentication or permission errors when pushing to Hub.
+
+**Solutions:**
+1. **Verify authentication:**
+   ```python
+   mcp__huggingface__hf_whoami()  # Check who's authenticated
+   ```
+
+2. **Check token permissions:**
+   - Go to https://huggingface.co/settings/tokens
+   - Ensure token has "write" permission
+   - Token must not be "read-only"
+
+3. **Verify token in job:**
+   ```python
+   "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # Must be in job config
+   ```
+
+4. **Check repo permissions:**
+   - User must have write access to target repo
+   - If org repo, user must be member with write access
+   - Repo must exist or user must have permission to create
+
+## Job Stuck or Not Starting
+
+**Problem:** Job shows "pending" or "starting" for extended period.
+
+**Solutions:**
+- Check Jobs dashboard for status: https://huggingface.co/jobs
+- Verify hardware availability (some GPU types may have queues)
+- Try different hardware flavor if one is heavily utilized
+- Check for account billing issues (Jobs requires paid plan)
+
+**Typical startup times:**
+- CPU jobs: 10-30 seconds
+- GPU jobs: 30-90 seconds
+- If >3 minutes: likely queued or stuck
+
+## Training Loss Not Decreasing
+
+**Problem:** Training runs but loss stays flat or doesn't improve.
+
+**Solutions:**
+1. **Check learning rate:** May be too low (try 2e-5 to 5e-5) or too high (try 1e-6)
+2. **Verify dataset quality:** Inspect examples to ensure they're reasonable
+3. **Check model size:** Very small models may not have capacity for task
+4. **Increase training steps:** May need more epochs or larger dataset
+5. **Verify dataset format:** Wrong format may cause degraded training
+
+## Logs Not Appearing
+
+**Problem:** Cannot see training logs or progress.
+
+**Solutions:**
+1. **Wait 30-60 seconds:** Initial logs can be delayed
+2. **Check logs via MCP tool:**
+   ```python
+   hf_jobs("logs", {"job_id": "your-job-id"})
+   ```
+3. **Use Trackio for real-time monitoring:** See `references/trackio_guide.md`
+4. **Verify job is actually running:**
+   ```python
+   hf_jobs("inspect", {"job_id": "your-job-id"})
+   ```
+
+## Checkpoint/Resume Issues
+
+**Problem:** Cannot resume from checkpoint or checkpoint not saved.
+
+**Solutions:**
+1. **Enable checkpoint saving:**
+   ```python
+   SFTConfig(
+       save_strategy="steps",
+       save_steps=100,
+       hub_strategy="every_save",  # Push each checkpoint
+   )
+   ```
+
+2. **Verify checkpoints pushed to Hub:** Check model repo for checkpoint folders
+
+3. **Resume from checkpoint:**
+   ```python
+   trainer = SFTTrainer(
+       model="username/model-name",  # Can be checkpoint path
+       resume_from_checkpoint="username/model-name/checkpoint-1000",
+   )
+   ```
+
+## Getting Help
+
+If issues persist:
+
+1. **Check TRL documentation:**
+   ```python
+   hf_doc_search("your issue", product="trl")
+   ```
+
+2. **Check Jobs documentation:**
+   ```python
+   hf_doc_fetch("https://huggingface.co/docs/huggingface_hub/guides/jobs")
+   ```
+
+3. **Review related guides:**
+   - `references/hub_saving.md` - Hub authentication issues
+   - `references/hardware_guide.md` - Hardware selection and specs
+   - `references/training_patterns.md` - Eval dataset requirements
+   - SKILL.md "Working with Scripts" section - Script format and URL issues
+
+4. **Ask in HF forums:** https://discuss.huggingface.co/