Upload 57 files

.gitattributes

@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imgs/main_result.png filter=lfs diff=lfs merge=lfs -text
+HY_MT2_0_Technical_Report.pdf filter=lfs diff=lfs merge=lfs -text
+IFMTBench/data/test_multi_constraint.jsonl filter=lfs diff=lfs merge=lfs -text

HY_MT2_0_Technical_Report.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce04272779bf9df1e47ba64132008ff95d9416c4111d266927390749812d1b72
+size 2597469

IFMTBench/README.md

@@ -0,0 +1,160 @@
+# Translation Instruction Following Benchmark
+
+A benchmark for evaluating how well LLMs follow complex instructions during translation tasks. The benchmark covers **6 constraint types** across multiple languages, including single-constraint and multi-constraint scenarios.
+
+## Constraint Types
+
+| Constraint Type | Evaluation Method | Score Type |
+|---|---|---|
+| Glossary Compliance (术语表约束翻译) | Rule check + LLM Judge fallback | Gate (0/1) |
+| Style Following (风格指令遵循) | LLM Judge | Continuous (0-5 → 0-1) |
+| Background Disambiguation (带上下文背景翻译) | LLM Judge | Continuous (0-5 → 0-1) |
+| Layout Preservation (布局保留翻译) | Rule check (delimiter-based) | Gate (0/1) |
+| Structured Data (结构化数据翻译) | Rule check (format validation) | Gate (0/1) |
+| Code/Tag Preservation (代码标签保留翻译) | Rule check (asset matching) | Gate (0/1) |
+
+**Multi-constraint scoring**: `final_score = gate_score × avg(continuous_scores)`
+
+## Data Format
+
+### Test Data
+
+Each line in the test JSONL files contains:
+
+| Field | Required | Description |
+|---|---|---|
+| `input` | ✓ | Full prompt to send to the model |
+| `output` | ✓ | Reference translation (ground truth) |
+| `class` | ✓ | Constraint type(s), list format, determines scoring logic |
+| `md5` | ✓ | Unique identifier |
+| `origin_text` | ✓ | Original source text |
+| `meta_data` | * | Metadata for code/layout evaluation (extracted_assets, primary_delimiter, etc.) |
+| `term_dict` | * | Glossary mapping (for glossary constraint type) |
+| `selected_style` | * | Target style (for style constraint type) |
+| `generated_background` | * | Background context (for background constraint type) |
+| `origin_language` | | Source language |
+| `target_language` | | Target language |
+| `instruction_lang` | | Language of the instruction |
+| `original_input` | | Original short-form input before rewriting |
+
+Fields marked with `*` are required for specific constraint types.
+
+### Model Output Format
+
+Your model output file should be a JSONL file where each line contains:
+
+```json
+{"md5": "<matching md5 from test data>", "response": "<model's translation output>"}
+```
+
+- **`md5`**: Must match the `md5` field from the test data to identify which test case this response corresponds to.
+- **`response`**: The model's raw translation output (the text your model generated).
+
+See `data/sample_output.jsonl` for an example.
+
+## Quick Start
+
+### 1. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+### 2. Configure LLM Judge API
+
+Copy `.env.example` to `.env` and fill in your API credentials:
+
+```bash
+cp .env.example .env
+# Edit .env with your API configuration
+```
+
+The LLM Judge supports any OpenAI-compatible API endpoint.
+
+### 3. Prepare Your Model Output
+
+Send each test item's `input` field to your model, collect the responses, and save them as:
+
+```jsonl
+{"md5": "c320a518b633b006b48719b403253e64", "response": "Your model's translation here..."}
+{"md5": "6953e58b3d8ddf3161cbb3ff5f0de7f0", "response": "Another translation..."}
+...
+```
+
+### 4. Run Evaluation
+
+```bash
+# Evaluate single-constraint data
+python run_eval.py \
+    --input_data data/test_single_constraint.jsonl \
+    --input_response your_model_output.jsonl \
+    --output_dir eval_results
+
+# Evaluate both single and multi-constraint
+python run_eval.py \
+    --input_data data/test_single_constraint.jsonl data/test_multi_constraint.jsonl \
+    --input_response your_model_output.jsonl \
+    --output_dir eval_results
+
+# Rule-only mode (no LLM Judge, for debugging)
+python run_eval.py \
+    --input_data data/test_single_constraint.jsonl \
+    --input_response your_model_output.jsonl \
+    --output_dir eval_results \
+    --skip-llm
+```
+
+### 5. View Results
+
+Results are saved to the output directory:
+- `eval_details.jsonl` — Per-item scoring details
+- `eval_summary.json` — Aggregated statistics
+
+## Project Structure
+
+```
+openbench/
+├── README.md              # English documentation
+├── README_zh.md           # 中文文档
+├── run_eval.py            # Evaluation entry point
+├── config.py              # Configuration
+├── .env.example           # Environment variable template
+├── requirements.txt       # Dependencies
+├── data/
+│   ├── test_single_constraint.jsonl   # Single-constraint test data (4506 items)
+│   ├── test_multi_constraint.jsonl    # Multi-constraint test data (2838 items)
+│   └── sample_output.jsonl            # Example model output format
+├── eval/
+│   ├── __init__.py
+│   ├── scoring.py         # Core scoring logic
+│   ├── rule_validators.py # Rule-based validators
+│   └── llm_judge.py       # LLM Judge module
+└── scripts/
+    └── prepare_data.py    # Data preprocessing script
+```
+
+## Evaluation Workflow
+
+```
+Test Data (JSONL)  +  Model Output (JSONL)
+        │                      │
+        └──────────┬───────────┘
+                   ▼
+            run_eval.py
+                   │
+        ┌──────────┴──────────┐
+        ▼                     ▼
+  Rule Validators        LLM Judge
+  (glossary/layout/      (style/background/
+   structured/code)       glossary fallback)
+        │                     │
+        └──────────┬──────────┘
+                   ▼
+          Score Composition
+                   │
+                   ▼
+         eval_results/
+         ├── eval_details.jsonl
+         └── eval_summary.json
+```
+

IFMTBench/README_zh.md

@@ -0,0 +1,162 @@
+# 翻译指令遵循评测基准
+
+本项目用于评测大语言模型在翻译任务中对复杂指令的遵循能力。评测覆盖 **6 种约束类型**，支持多语言，包含单约束和多约束场景。
+
+## 约束类型
+
+| 约束类型 | 评测方式 | 分数类型 |
+|---|---|---|
+| 术语表约束翻译 | 规则校验 + LLM Judge 保底 | 门控 (0/1) |
+| 风格指令遵循 | LLM Judge | 连续 (0-5 → 0-1) |
+| 带上下文背景翻译 | LLM Judge | 连续 (0-5 → 0-1) |
+| 布局保留翻译 | 规则校验（分隔符切分） | 门控 (0/1) |
+| 结构化数据翻译 | 规则校验（格式验证） | 门控 (0/1) |
+| 代码标签保留翻译 | 规则校验（代码片段匹配） | 门控 (0/1) |
+
+**多约束评分规则**: `final_score = gate_score × avg(continuous_scores)`
+- `gate_score` = 所有门控分数相乘（任一为 0 则整体为 0）
+- `continuous_scores` = 所有连续分数的平均值（无连续分数则为 1.0）
+
+## 数据格式
+
+### 测试数据
+
+测试数据为 JSONL 格式，每行包含以下字段：
+
+| 字段 | 必需 | 说明 |
+|---|---|---|
+| `input` | ✓ | 发送给模型的完整 prompt |
+| `output` | ✓ | 参考翻译（ground truth） |
+| `class` | ✓ | 约束类型列表，决定评分逻辑 |
+| `md5` | ✓ | 唯一标识符 |
+| `origin_text` | ✓ | 原始待翻译文本 |
+| `meta_data` | * | 代码/布局类评测所需的元数据（extracted_assets、primary_delimiter 等） |
+| `term_dict` | * | 术语映射表（术语表约束类型） |
+| `selected_style` | * | 目标风格（风格约束类型） |
+| `generated_background` | * | 背景上下文（背景约束类型） |
+| `origin_language` | | 源语言 |
+| `target_language` | | 目标语言 |
+| `instruction_lang` | | 指令语言 |
+| `original_input` | | 改写前的原始简短输入 |
+
+标 `*` 的字段对特定约束类型是必需的。
+
+### 模型输出格式
+
+你的模型输出文件应为 JSONL 格式，每行包含：
+
+```json
+{"md5": "<与测试数据中的 md5 对应>", "response": "<模型的翻译输出>"}
+```
+
+- **`md5`**: 必须与测试数据中的 `md5` 字段匹配，用于标识对应的测试用例。
+- **`response`**: 模型生成的原始翻译输出。
+
+参见 `data/sample_output.jsonl` 了解示例格式。
+
+## 快速开始
+
+### 1. 安装依赖
+
+```bash
+pip install -r requirements.txt
+```
+
+### 2. 配置 LLM Judge API
+
+将 `.env.example` 复制为 `.env` 并填入 API 凭据：
+
+```bash
+cp .env.example .env
+# 编辑 .env 填入你的 API 配置
+```
+
+LLM Judge 支持任何 OpenAI 兼容的 API 接口。
+
+### 3. 准备模型输出
+
+将测试数据中每条的 `input` 字段发送给你的模型，收集响应并保存为：
+
+```jsonl
+{"md5": "c320a518b633b006b48719b403253e64", "response": "你的模型翻译结果..."}
+{"md5": "6953e58b3d8ddf3161cbb3ff5f0de7f0", "response": "另一条翻译..."}
+...
+```
+
+### 4. 运行评测
+
+```bash
+# 评测单约束数据
+python run_eval.py \
+    --input_data data/test_single_constraint.jsonl \
+    --input_response your_model_output.jsonl \
+    --output_dir eval_results
+
+# 同时评测单约束和多约束
+python run_eval.py \
+    --input_data data/test_single_constraint.jsonl data/test_multi_constraint.jsonl \
+    --input_response your_model_output.jsonl \
+    --output_dir eval_results
+
+# 仅规则校验模式（跳过 LLM Judge，用于调试）
+python run_eval.py \
+    --input_data data/test_single_constraint.jsonl \
+    --input_response your_model_output.jsonl \
+    --output_dir eval_results \
+    --skip-llm
+```
+
+### 5. 查看结果
+
+结果保存在输出目录中：
+- `eval_details.jsonl` — 每条数据的详细评分
+- `eval_summary.json` — 汇总统计
+
+## 项目结构
+
+```
+openbench/
+├── README.md              # 英文文档
+├── README_zh.md           # 中文文档
+├── run_eval.py            # 评测入口
+├── config.py              # 配置文件
+├── .env.example           # 环境变量模板
+├── requirements.txt       # 依赖
+├── data/
+│   ├── test_single_constraint.jsonl   # 单约束测试数据（4506 条）
+│   ├── test_multi_constraint.jsonl    # 多约束测试数据（2838 条）
+│   └── sample_output.jsonl            # 示例模型输出格式
+├── eval/
+│   ├── __init__.py
+│   ├── scoring.py         # 评分核心逻辑
+│   ├── rule_validators.py # 规则校验器
+│   └── llm_judge.py       # LLM Judge 模块
+└── scripts/
+    └── prepare_data.py    # 数据预处理脚本
+```
+
+## 评测流程
+
+```
+测试数据 (JSONL)  +  模型输出 (JSONL)
+        │                      │
+        └──────────┬───────────┘
+                   ▼
+            run_eval.py
+                   │
+        ┌──────────┴──────────┐
+        ▼                     ▼
+    规则校验器            LLM Judge
+  (术语表/布局/         (风格/背景/
+   结构化/代码)          术语表保底)
+        │                     │
+        └──────────┬──────────┘
+                   ▼
+            分数合成
+                   │
+                   ▼
+         eval_results/
+         ├── eval_details.jsonl
+         └── eval_summary.json
+```
+

IFMTBench/config.py

@@ -0,0 +1,42 @@
+# -*- coding: utf-8 -*-
+"""
+Evaluation configuration.
+
+LLM Judge settings are configured via environment variables, see .env.example.
+"""
+
+import os
+
+# LLM Judge API configuration
+LLM_CONFIG = {
+    "api_base": os.environ.get("LLM_API_BASE", ""),
+    "api_key": os.environ.get("LLM_API_KEY", ""),
+    "model_name": os.environ.get("LLM_MODEL_NAME", "gpt-4o"),
+    "temperature": float(os.environ.get("LLM_TEMPERATURE", "0.0")),
+    "max_tokens": int(os.environ.get("LLM_MAX_TOKENS", "8192")),
+    "top_p": float(os.environ.get("LLM_TOP_P", "0.6")),
+}
+
+# Evaluation concurrency configuration
+MAX_WORKERS = int(os.environ.get("EVAL_MAX_WORKERS", "20"))
+MAX_RETRIES = int(os.environ.get("EVAL_MAX_RETRIES", "5"))
+REQUEST_TIMEOUT = int(os.environ.get("EVAL_REQUEST_TIMEOUT", "360"))
+
+# Class label -> evaluation dimension mapping
+CLASS_TO_DIMENSION = {
+    "机器翻译-术语表约束翻译": "glossary",
+    "机器翻译-风格指令遵循": "style",
+    "机器翻译-带上下文背景翻译": "background",
+}
+
+# Class categories grouped by evaluation method
+CLASS_LLM_JUDGE = {"机器翻译-风格指令遵循", "机器翻译-带上下文背景翻译"}
+CLASS_GLOSSARY = {"机器翻译-术语表约束翻译"}
+CLASS_LAYOUT = {"机器翻译-布局保留翻译"}
+CLASS_STRUCTURED = {"机器翻译-结构化数据翻译"}
+CLASS_CODE = {"机器翻译-内联代码保留翻译", "机器翻译-代码标签保留翻译"}
+
+# Gate classes (binary 0/1)
+GATE_CLASSES = CLASS_GLOSSARY | CLASS_LAYOUT | CLASS_STRUCTURED | CLASS_CODE
+# Continuous classes (0-5 normalized to 0-1)
+CONTINUOUS_CLASSES = CLASS_LLM_JUDGE

IFMTBench/data/sample_output.jsonl

@@ -0,0 +1,5 @@
+{"md5": "c320a518b633b006b48719b403253e64", "response": "<div class=\"embed-card\" id=\"post-123\">\n<span class=\"author-name\">简·杜</span> 向大家强烈推荐这篇<a href=\"https://example.com/article\" target=\"_blank\">前端开发指南</a>。\n</div>"}
+{"md5": "6953e58b3d8ddf3161cbb3ff5f0de7f0", "response": "When the breath recovery is complete, your next basic attack is guaranteed to trigger a fatal strike. Please time it wisely."}
+{"md5": "743008ba40088aec869f2297aeef10c8", "response": "Ignore the minions, just focus Maokai!"}
+{"md5": "26172d5e84ab3d6a403dc71601e9552f", "response": "他目不转睛地凝视着那颗纽扣，极度恐惧它会在大典前不合时宜地脱落。"}
+{"md5": "ee37da2a488de42588f778c0c6e5653c", "response": "弊省は、貴国との緊密な連携を通じ、本件に対し迅速に対応する所存であります。関係省庁間の意見調整に多少の時間を要しましたが、合意に至った最終案を別紙の通り送付いたします。"}

IFMTBench/data/test_multi_constraint.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9fb393c49880e688e58515b215e1e586f9b72d143a404546f20de4a01a3e17f
+size 14066944

IFMTBench/eval/__init__.py

@@ -0,0 +1 @@
+# -*- coding: utf-8 -*-

IFMTBench/eval/llm_judge.py

@@ -0,0 +1,257 @@
+# -*- coding: utf-8 -*-
+"""
+LLM Judge module: Calls LLM API to score style, background, and glossary compliance.
+
+Supports any OpenAI-compatible API endpoint.
+"""
+
+import json
+import re
+import time
+import random
+import logging
+from typing import Optional
+
+import requests
+
+from config import LLM_CONFIG, MAX_RETRIES, REQUEST_TIMEOUT
+
+log = logging.getLogger(__name__)
+
+# ============================================================
+# Prompt Templates
+# ============================================================
+
+GLOSSARY_REWARD_PROMPT = '''
+# ROLE
+You are an expert Linguistic and Morphological Evaluator for a translation Reward Model. Your SOLE objective is to determine if the specified terminology from the instruction was correctly integrated into the target translation, accounting for complex morphological adaptations (declension, pluralization, tense, etc.).
+
+# EVALUATION DATA
+<instruction>
+{user_instruction}
+</instruction>
+
+<ground_truth>
+{ground_truth}
+</ground_truth>
+
+<model_output>
+{target_translation}
+</model_output>
+
+# RUBRICS
+### Glossary Compliance - [BINARY SCORING: 0 or 1]
+Evaluate if the translation accurately incorporates the specific terminology provided in the instruction/background.
+- [1] Perfect Adherence: Flawlessly integrated the required terms. Morphological adaptations (e.g., plurals, tense, part-of-speech, conjugations) are grammatically natural in the target language. It is acceptable if the term underwent necessary morphological changes compared to its base dictionary form.
+- [0] Fatal Violation (Veto): Instant 0 if ANY of the following occur: unauthorized synonym substitution, fallback to generic dictionary translation, omission of the core concept, or severe grammatical corruption caused by forcing the term. 
+
+# OUTPUT FORMAT
+Output ONLY a single integer: `1` or `0`. 
+Do NOT wrap it in JSON, Markdown, or any other formatting. Do NOT output any explanatory text.
+'''
+
+STYLE_AND_BACKGROUND_REWARD_PROMPT = '''
+# ROLE
+You are an advanced Reward Model designed for Reinforcement Learning (RL) of Large Language Models. Your primary function is to evaluate **Instruction Tracking and Constraint Satisfaction**. 
+Do NOT evaluate basic translation fluency. Your SOLE objective is to score whether the model executed the specific holistic [Constraints] (Style and Background).
+
+# EVALUATION DATA
+<instruction>
+{user_instruction}
+</instruction>
+
+<ground_truth>
+{ground_truth}
+</ground_truth>
+
+<model_output>
+{target_translation}
+</model_output>
+
+# RUBRICS
+Analyze the <instruction>. If a constraint is NOT requested, output `null`. If activated, evaluate against the rubrics.
+
+### 1. Style & Register (Style) - [0-5 SCALE]
+- [Activation Condition]: Activate if the instruction requests a specific tone, persona, register, or formatting style.
+- [5] Perfect Alignment: Tone and register are exceptionally distinct and consistent throughout.
+- [4] Strong Alignment: Generally fits the required style, but 1-2 lexical choices feel slightly generic.
+- [3] Marginal Pass: Follows the basic directional constraint, but leans heavily on standard, flavorless translation. 
+- [2] Default/Generic: Ignored the stylistic constraint, reverting to a safe, bland machine translation tone.
+- [1] Severe Deviation: Noticeable conflict with the requested style.
+- [0] Rule Break: Wrong style AND included conversational filler/hallucinations, breaking the fourth wall.
+
+### 2. Contextual Cohesion (Background) - [0-5 SCALE]
+- [Activation Condition]: Activate if the instruction provides ANY preceding context, a background summary, or asks the translation to consider the "context" or "background".
+- [5] Perfect Disambiguation: Masterfully leveraged the background summary to resolve potential ambiguities. Flawless logical cohesion.
+- [4] Strong Utilization: Correctly used the summary to guide the translation, but feels slightly rigid when referencing the background.
+- [3] Logically Consistent: Does not contradict the summary, but disambiguation is mediocre (literal translation).
+- [2] Total Ignorance: Ignored the summary entirely, resulting in a disjointed literal translation.
+- [1] Logical Contradiction: Directly contradicts the core logic or established facts in the background summary.
+- [0] Severe Hallucination (Prompt Bleeding): Mistakenly translated the background summary itself as part of the target text.
+
+# OUTPUT FORMAT
+Output ONLY a valid JSON object. Do NOT wrap the JSON in Markdown code blocks (e.g., no ```json). 
+{{
+  "scores": {{
+    "style": [0, 1, 2, 3, 4, 5, or null],
+    "background": [0, 1, 2, 3, 4, 5, or null]
+  }}
+}}'''
+
+
+# ============================================================
+# API Calls
+# ============================================================
+
+def _call_llm(user_prompt: str) -> tuple:
+    """Call LLM API (OpenAI-compatible endpoint)."""
+    config = LLM_CONFIG
+    url = config["api_base"].rstrip("/") + "/chat/completions"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {config['api_key']}",
+    }
+    json_data = {
+        "model": config["model_name"],
+        "messages": [
+            {"role": "user", "content": user_prompt},
+        ],
+        "temperature": config["temperature"],
+        "max_tokens": config["max_tokens"],
+        "top_p": config["top_p"],
+        "stream": False,
+    }
+    resp = requests.post(url, headers=headers, json=json_data, timeout=REQUEST_TIMEOUT)
+    resp.raise_for_status()
+    response_json = resp.json()
+    content = response_json["choices"][0]["message"]["content"]
+    reasoning = response_json["choices"][0]["message"].get("reasoning_content", "")
+    return True, content.strip(), reasoning.strip() if reasoning else ""
+
+
+def _call_llm_with_retry(user_prompt: str) -> tuple:
+    """Call LLM with retry logic."""
+    last_error = None
+    for attempt in range(MAX_RETRIES + 1):
+        try:
+            success, content, reasoning = _call_llm(user_prompt)
+            if success:
+                return True, content, reasoning
+            last_error = content
+        except Exception as e:
+            last_error = str(e)
+        if attempt < MAX_RETRIES:
+            time.sleep(random.uniform(1, min(2 ** attempt, 10)))
+    return False, f"Failed after {MAX_RETRIES} retries: {last_error}", ""
+
+
+def _parse_json_from_text(text: str) -> Optional[dict]:
+    """Parse JSON from LLM output text."""
+    if not text:
+        return None
+    text = text.strip()
+    try:
+        return json.loads(text)
+    except json.JSONDecodeError:
+        pass
+    m = re.search(r'```(?:json)?\s*\n?(.*?)\n?\s*```', text, re.DOTALL)
+    if m:
+        try:
+            return json.loads(m.group(1))
+        except json.JSONDecodeError:
+            pass
+    s, e = text.find('{'), text.rfind('}')
+    if s != -1 and e > s:
+        try:
+            return json.loads(text[s:e + 1])
+        except json.JSONDecodeError:
+            pass
+    return None
+
+
+# ============================================================
+# Judge Scoring Functions
+# ============================================================
+
+def score_glossary_judge(user_instruction: str, ground_truth: str, target_translation: str) -> dict:
+    """Glossary LLM Judge (binary 0/1)."""
+    result = {"glossary": None, "if_score": None, "raw_response": None, "reasoning": None}
+    prompt = GLOSSARY_REWARD_PROMPT.format(
+        user_instruction=user_instruction,
+        ground_truth=ground_truth,
+        target_translation=target_translation,
+    )
+    success, text, reasoning = _call_llm_with_retry(prompt)
+    if not success:
+        log.error(f"Glossary Judge call failed: {text}")
+        return result
+
+    result["raw_response"] = text[:500]
+    result["reasoning"] = reasoning[:500] if reasoning else None
+
+    text_stripped = text.strip()
+    score = None
+    if text_stripped in ("0", "1"):
+        score = int(text_stripped)
+    else:
+        m = re.search(r'\b([01])\b', text_stripped)
+        if m:
+            score = int(m.group(1))
+
+    if score is not None:
+        result["glossary"] = score
+        result["if_score"] = float(score)
+    else:
+        log.warning(f"Glossary Judge parse failed: {text[:200]}")
+    return result
+
+
+def score_style_background_judge(user_instruction: str, ground_truth: str,
+                                  target_translation: str, expected_dimension: str) -> dict:
+    """Style/Background LLM Judge (0-5 scale, normalized to 0-1)."""
+    result = {"style": None, "background": None, "classification_match": False,
+              "if_score": None, "raw_response": None, "reasoning": None}
+    prompt = STYLE_AND_BACKGROUND_REWARD_PROMPT.format(
+        user_instruction=user_instruction,
+        ground_truth=ground_truth,
+        target_translation=target_translation,
+    )
+    success, text, reasoning = _call_llm_with_retry(prompt)
+    if not success:
+        log.error(f"Style/Background Judge call failed: {text}")
+        return result
+
+    result["raw_response"] = text[:500]
+    result["reasoning"] = reasoning[:500] if reasoning else None
+    parsed = _parse_json_from_text(text)
+    if parsed is None:
+        log.warning(f"Style/Background Judge JSON parse failed: {text[:200]}")
+        return result
+
+    scores = parsed.get("scores", parsed) if isinstance(parsed, dict) else parsed
+
+    def _extract_score(val):
+        if val is None:
+            return None
+        if isinstance(val, (int, float)):
+            return val
+        if isinstance(val, list) and len(val) > 0:
+            return val[0] if isinstance(val[0], (int, float)) else None
+        return None
+
+    if not isinstance(scores, dict):
+        score_val = _extract_score(scores)
+        result["style"] = score_val if expected_dimension == "style" else None
+        result["background"] = score_val if expected_dimension == "background" else None
+    else:
+        result["style"] = _extract_score(scores.get("style"))
+        result["background"] = _extract_score(scores.get("background"))
+
+    dim_value = result.get(expected_dimension)
+    if dim_value is not None:
+        result["classification_match"] = True
+        result["if_score"] = dim_value / 5.0
+    else:
+        result["classification_match"] = False
+        result["if_score"] = None
+    return result

IFMTBench/eval/rule_validators.py

@@ -0,0 +1,286 @@
+# -*- coding: utf-8 -*-
+"""
+Rule-based validators: glossary, layout preservation, structured data, and code snippet matching.
+
+These are deterministic rule checks that do not require an LLM.
+"""
+
+import io
+import csv
+import json
+import re
+import logging
+from html.parser import HTMLParser
+
+log = logging.getLogger(__name__)
+
+
+# ============================================================
+# Glossary Validation
+# ============================================================
+
+def validate_glossary(model_response: str, term_dict_str: str, ground_truth: str = "") -> dict:
+    """
+    Glossary rule check: verifies that model_response contains the correct target terms
+    from term_dict. For multi-candidate scenarios, cross-validates with ground_truth.
+    """
+    result = {"valid": True, "errors": [], "matched": 0, "total": 0}
+
+    if not term_dict_str:
+        result["valid"] = False
+        result["errors"].append("term_dict is empty, cannot perform rule check")
+        return result
+    if not model_response:
+        result["valid"] = False
+        result["errors"].append("model_response is empty")
+        return result
+
+    try:
+        term_dict = json.loads(term_dict_str) if isinstance(term_dict_str, str) else term_dict_str
+    except (json.JSONDecodeError, TypeError):
+        result["valid"] = False
+        result["errors"].append(f"term_dict parse failed: {str(term_dict_str)[:100]}")
+        return result
+
+    if not isinstance(term_dict, dict) or not term_dict:
+        result["valid"] = False
+        result["errors"].append("term_dict is empty or has invalid format")
+        return result
+
+    for src_term, tgt_terms in term_dict.items():
+        if not isinstance(tgt_terms, list):
+            tgt_terms = [tgt_terms]
+        result["total"] += 1
+        model_hits = [tgt for tgt in tgt_terms if tgt and tgt in model_response]
+
+        if not model_hits:
+            result["valid"] = False
+            result["errors"].append(f"Term not matched: {src_term} -> {tgt_terms}")
+            continue
+
+        if len(tgt_terms) > 1 and ground_truth:
+            gt_hits = [tgt for tgt in tgt_terms if tgt and tgt in ground_truth]
+            correct_hit = any(tgt in gt_hits for tgt in model_hits)
+            if correct_hit:
+                result["matched"] += 1
+            else:
+                result["valid"] = False
+                result["errors"].append(
+                    f"Wrong term choice: {src_term} -> model used {model_hits}, "
+                    f"but correct term in ground_truth is {gt_hits}"
+                )
+        else:
+            result["matched"] += 1
+
+    return result
+
+
+# ============================================================
+# Layout Preservation Validation
+# ============================================================
+
+def validate_layout(model_response: str, meta_data: dict, origin_text: str) -> dict:
+    """Layout preservation check: verifies chunk count consistency after delimiter splitting."""
+    primary_delimiter = meta_data.get("primary_delimiter", "")
+    source_chunks = meta_data.get("source_chunks", [])
+    result = {"valid": False, "errors": []}
+
+    if not primary_delimiter:
+        result["errors"].append("primary_delimiter is empty")
+        return result
+    if not origin_text:
+        result["errors"].append("origin_text is empty")
+        return result
+    if not model_response:
+        result["errors"].append("model_response is empty")
+        return result
+
+    origin_chunks = origin_text.split(primary_delimiter)
+    output_chunks = model_response.split(primary_delimiter)
+    if len(origin_chunks) == len(output_chunks) == len(source_chunks):
+        result["valid"] = True
+    else:
+        result["errors"].append(
+            f"Chunk count mismatch: origin={len(origin_chunks)}, "
+            f"response={len(output_chunks)}, source_chunks={len(source_chunks)}"
+        )
+    return result
+
+
+# ============================================================
+# Structured Data Validation
+# ============================================================
+
+class HTMLTagExtractor(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.tags = []
+
+    def handle_starttag(self, tag, attrs):
+        self.tags.append(("start", tag, sorted([k for k, v in attrs])))
+
+    def handle_endtag(self, tag):
+        self.tags.append(("end", tag, []))
+
+    def error(self, message):
+        pass
+
+
+def _validate_json_struct(origin_text, output_text):
+    errors = []
+    try:
+        oj = json.loads(origin_text)
+    except Exception:
+        errors.append("Origin JSON parse failed")
+        oj = None
+    try:
+        mj = json.loads(output_text)
+    except Exception:
+        errors.append("Output JSON parse failed")
+        mj = None
+    if oj is not None and mj is not None:
+        errors.extend(_check_json_keys(oj, mj, "$"))
+    return len(errors) == 0, errors
+
+
+def _check_json_keys(origin, output, path="$"):
+    errors = []
+    if type(origin) != type(output):
+        return [f"Type mismatch @ {path}"]
+    if isinstance(origin, dict):
+        ok, ek = set(origin.keys()), set(output.keys())
+        if ok != ek:
+            if ok - ek:
+                errors.append(f"Missing keys @ {path}: {ok - ek}")
+            if ek - ok:
+                errors.append(f"Extra keys @ {path}: {ek - ok}")
+        for k in ok & ek:
+            errors.extend(_check_json_keys(origin[k], output[k], f"{path}.{k}"))
+    elif isinstance(origin, list):
+        if len(origin) != len(output):
+            errors.append(f"Array length mismatch @ {path}")
+        for i in range(min(len(origin), len(output))):
+            errors.extend(_check_json_keys(origin[i], output[i], f"{path}[{i}]"))
+    return errors
+
+
+def _validate_html_struct(origin_text, output_text):
+    errors = []
+    op = HTMLTagExtractor()
+    try:
+        op.feed(origin_text)
+    except Exception:
+        errors.append("Origin HTML parse failed")
+    mp = HTMLTagExtractor()
+    try:
+        mp.feed(output_text)
+    except Exception:
+        errors.append("Output HTML parse failed")
+    if not errors:
+        if len(op.tags) != len(mp.tags):
+            errors.append("Tag count mismatch")
+        else:
+            for i, (ot, mt) in enumerate(zip(op.tags, mp.tags)):
+                if ot[0] != mt[0] or ot[1] != mt[1]:
+                    errors.append(f"Tag #{i+1} mismatch")
+    return len(errors) == 0, errors
+
+
+def _validate_csv_struct(origin_text, output_text):
+    errors = []
+    try:
+        or_ = list(csv.reader(io.StringIO(origin_text)))
+    except Exception:
+        errors.append("Origin CSV parse failed")
+        or_ = None
+    try:
+        mr_ = list(csv.reader(io.StringIO(output_text)))
+    except Exception:
+        errors.append("Output CSV parse failed")
+        mr_ = None
+    if or_ is not None and mr_ is not None:
+        if len(or_) != len(mr_):
+            errors.append("Row count mismatch")
+        else:
+            for i, (a, b) in enumerate(zip(or_, mr_)):
+                if len(a) != len(b):
+                    errors.append(f"Column count mismatch at row {i+1}")
+    return len(errors) == 0, errors
+
+
+def _parse_md_table(text):
+    lines = [l.strip() for l in text.strip().split("\n") if l.strip()]
+    if len(lines) < 2:
+        return None, None, "Insufficient table rows"
+
+    def split_row(line):
+        line = line.strip().strip("|")
+        return [c.strip() for c in line.split("|")]
+
+    header = split_row(lines[0])
+    if not re.match(r"^[\|\s\-:]+$", lines[1]):
+        return None, None, "Row 2 is not a valid separator row"
+    return header, [split_row(l) for l in lines[2:]], None
+
+
+def _validate_markdown_struct(origin_text, output_text):
+    errors = []
+    oh, orows, oerr = _parse_md_table(origin_text)
+    if oerr:
+        errors.append(f"origin: {oerr}")
+    mh, mrows, merr = _parse_md_table(output_text)
+    if merr:
+        errors.append(f"output: {merr}")
+    if oh and mh:
+        if len(oh) != len(mh):
+            errors.append("Header column count mismatch")
+        if orows and mrows and len(orows) != len(mrows):
+            errors.append("Data row count mismatch")
+    return len(errors) == 0, errors
+
+
+STRUCT_VALIDATORS = {
+    "JSON": _validate_json_struct, "json": _validate_json_struct,
+    "HTML片段": _validate_html_struct, "HTML": _validate_html_struct, "html": _validate_html_struct,
+    "CSV": _validate_csv_struct, "csv": _validate_csv_struct,
+    "Markdown表格": _validate_markdown_struct, "Markdown": _validate_markdown_struct,
+    "markdown": _validate_markdown_struct,
+}
+
+
+def validate_structured(origin_text: str, model_response: str, data_format: str) -> dict:
+    """Structured data validation: checks if translation preserves the original data structure."""
+    result = {"valid": False, "errors": []}
+    if not origin_text or not model_response:
+        result["errors"].append("origin_text or model_response is empty")
+        return result
+    validator = STRUCT_VALIDATORS.get(data_format)
+    if not validator:
+        result["errors"].append(f"Unknown data_format: {data_format}")
+        return result
+    valid, errors = validator(origin_text, model_response)
+    result["valid"] = valid
+    result["errors"] = errors
+    return result
+
+
+# ============================================================
+# Code Snippet Matching
+# ============================================================
+
+def validate_code_assets(model_response: str, extracted_assets: list) -> dict:
+    """Code/tag preservation check: verifies model_response contains all extracted_assets."""
+    result = {"valid": True, "errors": [], "matched": 0, "total": len(extracted_assets)}
+    if not extracted_assets:
+        return result
+    if not model_response:
+        result["valid"] = False
+        result["errors"].append("model_response is empty")
+        return result
+    for asset in extracted_assets:
+        if asset in model_response:
+            result["matched"] += 1
+        else:
+            result["valid"] = False
+            result["errors"].append(f"Not matched: {asset}")
+    return result

IFMTBench/eval/scoring.py

@@ -0,0 +1,303 @@
+# -*- coding: utf-8 -*-
+"""
+Core scoring module: integrates single-constraint and multi-constraint scoring logic.
+
+Single-constraint scoring:
+  - Glossary: rule check first, falls back to LLM Judge if rule fails
+  - Style/Background: LLM Judge (0-5 normalized to 0-1)
+  - Layout/Structured/Code: rule check (0/1)
+
+Multi-constraint scoring:
+  final_score = gate_score × avg(continuous_scores)
+  - gate_score = product of all gate scores (any 0 makes the final 0)
+  - continuous_scores = average of all continuous scores (defaults to 1.0 if none)
+"""
+
+import json
+import logging
+from collections import defaultdict
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+from tqdm import tqdm
+
+from config import (
+    CLASS_GLOSSARY, CLASS_LLM_JUDGE, CLASS_LAYOUT, CLASS_STRUCTURED, CLASS_CODE,
+    CLASS_TO_DIMENSION, MAX_WORKERS,
+)
+from eval.rule_validators import (
+    validate_glossary, validate_layout, validate_structured, validate_code_assets,
+)
+from eval.llm_judge import score_glossary_judge, score_style_background_judge
+
+log = logging.getLogger(__name__)
+
+
+def get_class_list(item: dict) -> list:
+    """Extract the class list from an item."""
+    cls = item.get("class", [])
+    if isinstance(cls, str):
+        cls = [cls]
+    return cls
+
+
+def score_single_dimension(item: dict, cls: str, model_response: str) -> dict:
+    """
+    Score a single constraint dimension.
+
+    Args:
+        item: Test data entry (contains input, output, origin_text, meta_data, etc.)
+        cls: The constraint class to evaluate
+        model_response: The model's translation output
+
+    Returns:
+        {"class": str, "score_type": "gate"|"continuous", "score": float|None, "details": dict}
+    """
+    origin_text = item.get("origin_text", "")
+    ground_truth = item.get("output", "")
+    user_instruction = item.get("input", "")
+    meta_data = item.get("meta_data", {})
+
+    result = {"class": cls, "score_type": None, "score": None, "details": {}}
+
+    if cls in CLASS_GLOSSARY:
+        result["score_type"] = "gate"
+        term_dict_str = item.get("term_dict", "")
+        check = validate_glossary(model_response, term_dict_str, ground_truth)
+        if check["valid"]:
+            result["score"] = 1.0
+            result["details"] = {"method": "rule", "valid": True,
+                                 "matched": check["matched"], "total": check["total"]}
+        else:
+            judge = score_glossary_judge(user_instruction, ground_truth, model_response)
+            result["score"] = judge.get("if_score", 0.0)
+            result["details"] = {"method": "llm_judge_fallback",
+                                 "rule_errors": check["errors"],
+                                 "judge_score": judge.get("glossary")}
+
+    elif cls in CLASS_STRUCTURED:
+        result["score_type"] = "gate"
+        data_format = item.get("data_format", meta_data.get("data_format", ""))
+        check = validate_structured(origin_text, model_response, data_format)
+        result["score"] = 1.0 if check["valid"] else 0.0
+        result["details"] = {"method": "rule", "valid": check["valid"],
+                             "data_format": data_format, "errors": check["errors"]}
+
+    elif cls in CLASS_LAYOUT:
+        result["score_type"] = "gate"
+        check = validate_layout(model_response, meta_data, origin_text)
+        result["score"] = 1.0 if check["valid"] else 0.0
+        result["details"] = {"method": "rule", "valid": check["valid"],
+                             "errors": check["errors"]}
+
+    elif cls in CLASS_CODE:
+        result["score_type"] = "gate"
+        extracted_assets = meta_data.get("extracted_assets", [])
+        check = validate_code_assets(model_response, extracted_assets)
+        result["score"] = 1.0 if check["valid"] else 0.0
+        result["details"] = {"method": "rule", "valid": check["valid"],
+                             "matched": check["matched"], "total": check["total"]}
+
+    elif cls in CLASS_LLM_JUDGE:
+        result["score_type"] = "continuous"
+        expected_dim = CLASS_TO_DIMENSION.get(cls, "")
+        judge = score_style_background_judge(user_instruction, ground_truth, model_response, expected_dim)
+        result["score"] = judge.get("if_score")
+        result["details"] = {
+            "method": "llm_judge",
+            "style": judge.get("style"),
+            "background": judge.get("background"),
+            "classification_match": judge.get("classification_match", False),
+            "expected_dim": expected_dim,
+        }
+
+    else:
+        result["score_type"] = "unknown"
+        result["details"] = {"error": f"Unknown class: {cls}"}
+
+    return result
+
+
+def compose_multi_scores(dim_scores: list) -> dict:
+    """
+    Compose multi-dimension scores:
+    final = gate_score × avg(continuous_scores)
+    """
+    gate_scores = []
+    continuous_scores = []
+
+    for ds in dim_scores:
+        if ds["score"] is None:
+            continue
+        if ds["score_type"] == "gate":
+            gate_scores.append(ds["score"])
+        elif ds["score_type"] == "continuous":
+            continuous_scores.append(ds["score"])
+
+    gate = 1.0
+    for g in gate_scores:
+        gate *= g
+
+    if continuous_scores:
+        continuous_avg = sum(continuous_scores) / len(continuous_scores)
+    else:
+        continuous_avg = 1.0
+
+    final = round(gate * continuous_avg, 4)
+
+    return {
+        "gate_score": round(gate, 4),
+        "continuous_avg": round(continuous_avg, 4),
+        "final_score": final,
+    }
+
+
+def score_one_item(item: dict, model_response: str) -> dict:
+    """
+    Score a single data item.
+
+    Args:
+        item: Test data entry
+        model_response: Model output
+
+    Returns:
+        Scoring result dictionary
+    """
+    cls_list = get_class_list(item)
+    is_multi = len(cls_list) > 1
+
+    entry = {
+        "md5": item.get("md5", ""),
+        "class": cls_list,
+        "is_multi_constraint": is_multi,
+    }
+
+    if is_multi:
+        dim_scores = []
+        for cls in cls_list:
+            ds = score_single_dimension(item, cls, model_response)
+            dim_scores.append(ds)
+
+        composition = compose_multi_scores(dim_scores)
+        entry["dimension_scores"] = dim_scores
+        entry["gate_score"] = composition["gate_score"]
+        entry["continuous_avg"] = composition["continuous_avg"]
+        entry["final_score"] = composition["final_score"]
+    else:
+        cls = cls_list[0] if cls_list else "unknown"
+        ds = score_single_dimension(item, cls, model_response)
+        entry["dimension_scores"] = [ds]
+        entry["final_score"] = ds["score"]
+
+    return entry
+
+
+def batch_score(test_data: list, responses: dict, workers: int = None) -> list:
+    """
+    Batch scoring.
+
+    Args:
+        test_data: List of test data items
+        responses: {md5: response_text} mapping
+        workers: Number of concurrent threads
+
+    Returns:
+        List of scoring results
+    """
+    if workers is None:
+        workers = MAX_WORKERS
+
+    log.info(f"Starting evaluation: {len(test_data)} items, workers={workers}")
+
+    results = [None] * len(test_data)
+    skipped = 0
+
+    def _worker(idx):
+        item = test_data[idx]
+        md5 = item.get("md5", "")
+        response = responses.get(md5, "")
+        if not response:
+            return idx, {"md5": md5, "class": get_class_list(item),
+                        "final_score": None, "error": "Model output not found"}
+        return idx, score_one_item(item, response)
+
+    with ThreadPoolExecutor(max_workers=min(workers, max(len(test_data), 1))) as executor:
+        futures = {executor.submit(_worker, i): i for i in range(len(test_data))}
+        with tqdm(total=len(test_data), desc="Scoring", unit="item") as pbar:
+            for future in as_completed(futures):
+                idx, entry = future.result()
+                results[idx] = entry
+                if entry.get("error"):
+                    skipped += 1
+                pbar.update(1)
+
+    if skipped:
+        log.warning(f"Skipped {skipped} items (no matching model output found)")
+
+    return results
+
+
+def compute_summary(results: list) -> dict:
+    """Compute aggregated scoring statistics."""
+    single_stats = defaultdict(lambda: {"count": 0, "scores": []})
+    multi_stats = defaultdict(lambda: {"count": 0, "scores": []})
+
+    for r in results:
+        if r is None or r.get("error"):
+            continue
+        cls_key = str(r["class"])
+        is_multi = r.get("is_multi_constraint", False)
+
+        if is_multi:
+            s = multi_stats[cls_key]
+        else:
+            s = single_stats[cls_key]
+
+        s["count"] += 1
+        if r["final_score"] is not None:
+            s["scores"].append(r["final_score"])
+
+    # Overall statistics
+    all_scores = []
+    for s in single_stats.values():
+        all_scores.extend(s["scores"])
+    for s in multi_stats.values():
+        all_scores.extend(s["scores"])
+
+    summary = {
+        "total_items": len(results),
+        "scored_items": len(all_scores),
+        "overall_avg": round(sum(all_scores) / len(all_scores), 4) if all_scores else None,
+        "single_constraint": {k: {"count": v["count"],
+                                   "avg_score": round(sum(v["scores"]) / len(v["scores"]), 4) if v["scores"] else None}
+                              for k, v in single_stats.items()},
+        "multi_constraint": {k: {"count": v["count"],
+                                  "avg_score": round(sum(v["scores"]) / len(v["scores"]), 4) if v["scores"] else None}
+                             for k, v in multi_stats.items()},
+    }
+    return summary
+
+
+def print_summary(results: list):
+    """Print scoring summary."""
+    summary = compute_summary(results)
+
+    print("\n" + "=" * 85)
+    print(f"  Evaluation Results Summary")
+    print("=" * 85)
+    print(f"  Total: {summary['total_items']}  Scored: {summary['scored_items']}  "
+          f"Overall Avg: {summary['overall_avg']}")
+    print("-" * 85)
+
+    if summary["single_constraint"]:
+        print("\n  [Single Constraint]")
+        print(f"  {'CLASS':<50} {'COUNT':>6} {'AVG':>10}")
+        for k, v in sorted(summary["single_constraint"].items()):
+            print(f"  {k:<50} {v['count']:>6} {v['avg_score'] or 'N/A':>10}")
+
+    if summary["multi_constraint"]:
+        print("\n  [Multi Constraint] (final = gate × avg_continuous)")
+        print(f"  {'CLASS COMBO':<50} {'COUNT':>6} {'AVG':>10}")
+        for k, v in sorted(summary["multi_constraint"].items()):
+            print(f"  {k:<50} {v['count']:>6} {v['avg_score'] or 'N/A':>10}")
+
+    print("=" * 85)

IFMTBench/requirements.txt

@@ -0,0 +1,3 @@
+requests>=2.28.0
+tqdm>=4.64.0
+python-dotenv>=1.0.0

IFMTBench/run_eval.py

@@ -0,0 +1,154 @@
+# -*- coding: utf-8 -*-
+"""
+Translation Instruction Following Evaluation entry point.
+
+Usage:
+    python run_eval.py \
+        --input_data data/test_single_constraint.jsonl \
+        --input_response data/sample_output.jsonl \
+        --output_dir eval_results
+
+    # Evaluate both single and multi-constraint
+    python run_eval.py \
+        --input_data data/test_single_constraint.jsonl data/test_multi_constraint.jsonl \
+        --input_response my_model_output.jsonl \
+        --output_dir eval_results
+
+    # Rule-only mode (skip LLM Judge, for debugging)
+    python run_eval.py \
+        --input_data data/test_single_constraint.jsonl \
+        --input_response my_model_output.jsonl \
+        --output_dir eval_results \
+        --skip-llm
+"""
+
+import argparse
+import json
+import os
+import sys
+import logging
+
+# Add project root to path
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+from dotenv import load_dotenv
+load_dotenv()  # Load environment variables from .env file
+
+from eval.scoring import batch_score, print_summary, compute_summary
+
+logging.basicConfig(
+    format="%(asctime)s : %(levelname)s : %(filename)s: %(message)s",
+    level=logging.INFO,
+)
+log = logging.getLogger(__name__)
+
+
+def load_jsonl(path: str) -> list:
+    """Load a JSONL file."""
+    data = []
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                data.append(json.loads(line))
+    return data
+
+
+def load_responses(path: str) -> dict:
+    """
+    Load model output file, returning a {md5: response} mapping.
+
+    Model output file format: one JSON object per line, must contain:
+      - md5: unique identifier matching the md5 field in test data
+      - response: the model's generated translation
+    """
+    responses = {}
+    data = load_jsonl(path)
+    for item in data:
+        md5 = item.get("md5", "")
+        response = item.get("response", "")
+        if md5:
+            responses[md5] = response
+    log.info(f"Loaded {len(responses)} model responses")
+    return responses
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Translation Instruction Following Evaluation"
+    )
+    parser.add_argument(
+        "--input_data", "-d", nargs="+", required=True,
+        help="Test data file path(s) (JSONL), multiple files supported"
+    )
+    parser.add_argument(
+        "--input_response", "-r", required=True,
+        help="Model output file path (JSONL), each line must contain md5 and response fields"
+    )
+    parser.add_argument(
+        "--output_dir", "-o", default="eval_results",
+        help="Output directory for evaluation results (default: eval_results)"
+    )
+    parser.add_argument(
+        "--workers", "-w", type=int, default=None,
+        help="Number of concurrent LLM Judge threads (default: EVAL_MAX_WORKERS env var or 20)"
+    )
+    parser.add_argument(
+        "--skip-llm", action="store_true",
+        help="Skip LLM Judge evaluation (rule-only mode, for debugging)"
+    )
+    args = parser.parse_args()
+
+    # Load test data
+    test_data = []
+    for path in args.input_data:
+        items = load_jsonl(path)
+        log.info(f"Loaded test data: {path} ({len(items)} items)")
+        test_data.extend(items)
+    log.info(f"Total: {len(test_data)} test items")
+
+    # Load model responses
+    responses = load_responses(args.input_response)
+
+    # Check coverage
+    test_md5s = {item.get("md5", "") for item in test_data if item.get("md5")}
+    response_md5s = set(responses.keys())
+    coverage = len(test_md5s & response_md5s) / len(test_md5s) if test_md5s else 0
+    log.info(f"Response coverage: {coverage:.1%} ({len(test_md5s & response_md5s)}/{len(test_md5s)})")
+
+    if coverage < 0.5:
+        log.warning("Response coverage is below 50%, please check if md5 fields match")
+
+    # If skipping LLM, temporarily modify config
+    if args.skip_llm:
+        log.info("Skipping LLM Judge (rule-only mode)")
+        # Remap LLM Judge classes to empty set so no LLM calls are made
+        import config
+        config.CLASS_LLM_JUDGE = set()
+
+    # Run scoring
+    results = batch_score(test_data, responses, workers=args.workers)
+
+    # Output results
+    os.makedirs(args.output_dir, exist_ok=True)
+
+    # Write detailed results
+    detail_path = os.path.join(args.output_dir, "eval_details.jsonl")
+    with open(detail_path, "w", encoding="utf-8") as f:
+        for r in results:
+            f.write(json.dumps(r, ensure_ascii=False) + "\n")
+    log.info(f"Detailed results written to: {detail_path}")
+
+    # Write summary
+    summary = compute_summary(results)
+    summary_path = os.path.join(args.output_dir, "eval_summary.json")
+    with open(summary_path, "w", encoding="utf-8") as f:
+        json.dump(summary, f, ensure_ascii=False, indent=2)
+    log.info(f"Summary written to: {summary_path}")
+
+    # Print summary
+    print_summary(results)
+
+
+if __name__ == "__main__":
+    main()

LICENSE-IFMTBench.txt

@@ -0,0 +1,111 @@
+Tencent is pleased to support the community by making IFMTBench available.
+
+Copyright (C)  2026 Tencent.  All rights reserved. 
+
+IFMTBench is licensed under License CC-BY-4.0. IFMTBench does not impose any additional restrictions beyond the terms of the license.
+
+For the avoidance of doubt, IFMTBench refers solely to the dataset made publicly available by Tencent in accordance with License Term of CC-BY-4.0. 
+
+Terms of CC-BY-4.0:
+--------------------------------------------------------------------
+Creative Commons Attribution 4.0 International
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an "as-is" basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses.
+
+Considerations for licensors: Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other material, or material used under an exception or limitation to copyright. More considerations for licensors : wiki.creativecommons.org/Considerations_for_licensors
+
+Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensor's permission is not necessary for any reason–for example, because of any applicable exception or limitation to copyright–then that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to respect those requests where reasonable. More considerations for the public : wiki.creativecommons.org/Considerations_for_licensees
+
+Creative Commons Attribution 4.0 International Public License
+
+By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.
+
+Section 1 Definitions.
+
+a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.
+b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License.
+c. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights.
+d. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.
+e. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.
+f. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License.
+g. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.
+h. Licensor means the individual(s) or entity(ies) granting rights under this Public License.
+i. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.
+j. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.
+k. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning.
+Section 2 Scope.
+
+a. License grant.
+1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to:
+A. reproduce and Share the Licensed Material, in whole or in part; and
+B. produce, reproduce, and Share Adapted Material.
+2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.
+3. Term. The term of this Public License is specified in Section 6(a).
+4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material.
+5. Downstream recipients.
+A. Offer from the Licensor – Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.
+B. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.
+6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i).
+b. Other rights.
+1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.
+2. Patent and trademark rights are not licensed under this Public License.
+3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties.
+Section 3 License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the following conditions.
+
+a. Attribution.
+1. If You Share the Licensed Material (including in modified form), You must:
+A. retain the following if it is supplied by the Licensor with the Licensed Material:
+i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);
+ii. a copyright notice;
+iii. a notice that refers to this Public License;
+iv. a notice that refers to the disclaimer of warranties;
+v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable;
+B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and
+C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.
+2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.
+3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable.
+4. If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License.
+Section 4 Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:
+
+a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database;
+b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and
+c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database.
+For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights.
+
+Section 5 Disclaimer of Warranties and Limitation of Liability.
+
+a. Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.
+b. To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.
+c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.
+Section 6 Term and Termination.
+
+a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.
+b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates:
+1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
+2. upon express reinstatement by the Licensor.
+c. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License.
+d. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.
+e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License.
+Section 7 Other Terms and Conditions.
+
+a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.
+b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.
+Section 8 Interpretation.
+
+a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.
+b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.
+c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.
+d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.
+Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the "Licensor." The text of the Creative Commons public licenses is dedicated to the public domain under the CC0 Public Domain Dedication. Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at creativecommons.org/policies, Creative Commons does not authorize the use of the trademark "Creative Commons" or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses.
+
+Creative Commons may be contacted at creativecommons.org.
+
+

LICENSE.txt

@@ -0,0 +1,80 @@
+TENCENT HY COMMUNITY LICENSE AGREEMENT
+Tencent Hy-MT2 Release Date: May 21, 2026
+THIS LICENSE AGREEMENT DOES NOT APPLY IN THE EUROPEAN UNION, AND IS EXPRESSLY LIMITED TO THE TERRITORY, AS DEFINED BELOW.
+
+By clicking to agree or by using, reproducing, modifying, distributing, performing or displaying any portion or element of the Tencent HY Works, including via any Hosted Service, You will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
+
+1.	DEFINITIONS.
+a.	“Acceptable Use Policy” shall mean the policy made available by Tencent as set forth in the Exhibit A.
+b.	“Agreement” shall mean the terms and conditions for use, reproduction, distribution, modification, performance and displaying of Tencent HY Works or any portion or element thereof set forth herein.
+c.	“Documentation” shall mean the specifications, manuals and documentation for Tencent HY made publicly available by Tencent.
+d.	“Hosted Service” shall mean a hosted service offered via an application programming interface (API), web access, or any other electronic or remote means.
+e.	“Licensee,” “You” or “Your” shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Tencent HY Works for any purpose and in any field of use.
+f.	“Materials” shall mean, collectively, Tencent’s proprietary Tencent HY and Documentation (and any portion thereof) as made available by Tencent under this Agreement.
+g.	“Model Derivatives” shall mean all: (i) modifications to Tencent HY or any Model Derivative of Tencent HY; (ii) works based on Tencent HY or any Model Derivative of Tencent HY; or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of Tencent HY or any Model Derivative of Tencent HY, to that model in order to cause that model to perform similarly to Tencent HY or a Model Derivative of Tencent HY, including distillation methods, methods that use intermediate data representations, or methods based on the generation of synthetic data Outputs by Tencent HY or a Model Derivative of Tencent HY for training that model. For clarity, Outputs by themselves are not deemed Model Derivatives.
+h.	“Output” shall mean the information and/or content output of Tencent HY or a Model Derivative that results from operating or otherwise using Tencent HY or a Model Derivative, including via a Hosted Service.
+i.	“Tencent,” “We” or “Us” shall mean the applicable entity or entities in the Tencent corporate family that own(s) intellectual property or other rights embodied in or utilized by the Materials.
+j.	“Tencent HY” shall mean the large language models, text/image/video/audio/3D generation models, and multimodal large language models and their software and algorithms, including trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing made publicly available by Us, including, without limitation to, Tencent Hy-MT2-1.8B released at https://huggingface.co/tencent/Hy-MT2-1.8B, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-1.8B; Tencent Hy-MT2-7B released at https://huggingface.co/tencent/Hy-MT2-7B, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-7B; Tencent Hy-MT2-30B-A3B released at https://huggingface.co/tencent/Hy-MT2-30B-A3B, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-30B-A3B; Tencent Hy-MT2-1.8B-FP8 released at https://huggingface.co/tencent/Hy-MT2-1.8B-FP8, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-1.8B-FP8; Tencent Hy-MT2-7B-FP8 released at https://huggingface.co/tencent/Hy-MT2-7B-FP8, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-7B-FP8; Tencent Hy-MT2-30B-A3B-FP8 released at https://huggingface.co/tencent/Hy-MT2-30B-A3B-FP8, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-30B-A3B-FP8; Hy-MT2-1.8B-GGUF released at https://huggingface.co/tencent/Hy-MT2-1.8B-GGUF, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-1.8B-GGUF; Hy-MT2-7B-GGUF released at https://huggingface.co/tencent/Hy-MT2-7B-GGUF, https://modelscope.cn/models/Tencent-Hunyuan/Hy-MT2-7B-GGUF. 
+k.	“Tencent HY Works” shall mean: (i) the Materials; (ii) Model Derivatives; and (iii) all derivative works thereof.
+l.	“Territory” shall mean the worldwide territory, excluding the territory of the European Union. 
+m.	“Third Party” or “Third Parties” shall mean individuals or legal entities that are not under common control with Us or You.
+n.	“including” shall mean including but not limited to.
+2.	GRANT OF RIGHTS.
+We grant You, for the Territory only, a non-exclusive, non-transferable and royalty-free limited license under Tencent’s intellectual property or other rights owned by Us embodied in or utilized by the Materials to use, reproduce, distribute, create derivative works of (including Model Derivatives), and make modifications to the Materials, only in accordance with the terms of this Agreement and the Acceptable Use Policy, and You must not violate (or encourage or permit anyone else to violate) any term of this Agreement or the Acceptable Use Policy.
+3.	DISTRIBUTION.
+You may, subject to Your compliance with this Agreement, distribute or make available to Third Parties the Tencent HY Works, exclusively in the Territory, provided that You meet all of the following conditions:
+a.	You must provide all such Third Party recipients of the Tencent HY Works or products or services using them a copy of this Agreement;
+b.	You must cause any modified files to carry prominent notices stating that You changed the files;
+c.	You are encouraged to: (i) publish at least one technology introduction blogpost or one public statement expressing Your experience of using the Tencent HY Works; and (ii) mark the products or services developed by using the Tencent HY Works to indicate that the product/service is “Powered by Tencent HY”; and
+d.	All distributions to Third Parties (other than through a Hosted Service) must be accompanied by a “Notice” text file that contains the following notice: “Tencent HY is licensed under the Tencent HY Community License Agreement, Copyright © 2026 Tencent. All Rights Reserved. The trademark rights of “Tencent HY” are owned by Tencent or its affiliate.”
+e.	In the event that You use, integrate, implement, or otherwise deploy the Tencent HY Works, in whole or in part, to provide, enable, or support any service, product, or functionality to third parties, You shall clearly, accurately, and prominently disclose to all end users the full legal name and entity of the actual provider of such service, product, or functionality. You shall expressly and conspicuously state that Tencent is not affiliated with, associated with, sponsoring, or endorsing any such service, product, or functionality. You shall not use or display any name, logo, trademark, trade name, or other indicia of Tencent in any manner that could be construed as, or be likely to create, confusion, deception, or a false impression regarding any relationship, affiliation, sponsorship, or endorsement by Tencent.
+You may add Your own copyright statement to Your modifications and, except as set forth in this Section and in Section 5, may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Model Derivatives as a whole, provided Your use, reproduction, modification, distribution, performance and display of the work otherwise complies with the terms and conditions of this Agreement (including as regards the Territory). If You receive Tencent HY Works from a Licensee as part of an integrated end user product, then this Section 3 of this Agreement will not apply to You.
+4.	ADDITIONAL COMMERCIAL TERMS.
+If, on the Tencent HY version release date, the monthly active users of all products or services made available by or for Licensee is greater than 100 million monthly active users in the preceding calendar month, You must request a license from Tencent, which Tencent may grant to You in its sole discretion, and You are not authorized to exercise any of the rights under this Agreement unless or until Tencent otherwise expressly grants You such rights.
+5.	RULES OF USE.
+a.	Your use of the Tencent HY Works must comply with applicable laws and regulations (including trade compliance laws and regulations) and adhere to the Acceptable Use Policy for the Tencent HY Works, which is hereby incorporated by reference into this Agreement. You must include the use restrictions referenced in these Sections 5(a) and 5(b) as an enforceable provision in any agreement (e.g., license agreement, terms of use, etc.) governing the use and/or distribution of Tencent HY Works and You must provide notice to subsequent users to whom You distribute that Tencent HY Works are subject to the use restrictions in these Sections 5(a) and 5(b).
+b.	You must not use the Tencent HY Works or any Output or results of the Tencent HY Works to improve any other AI model (other than Tencent HY or Model Derivatives thereof).
+c.	You must not use, reproduce, modify, distribute, or display the Tencent HY Works, Output or results of the Tencent HY Works outside the Territory. Any such use outside the Territory is unlicensed and unauthorized under this Agreement.
+6.	INTELLECTUAL PROPERTY.
+a.	Subject to Tencent’s ownership of Tencent HY Works made by or for Tencent and intellectual property rights therein, conditioned upon Your compliance with the terms and conditions of this Agreement, as between You and Tencent, You will be the owner of any derivative works and modifications of the Materials and any Model Derivatives that are made by or for You.
+b.	No trademark licenses are granted under this Agreement, and in connection with the Tencent HY Works, Licensee may not use any name or mark owned by or associated with Tencent or any of its affiliates, except as required for reasonable and customary use in describing and distributing the Tencent HY Works. Tencent hereby grants You a license to use “Tencent HY” (the “Mark”) in the Territory solely as required to comply with the provisions of Section 3(c), provided that You comply with any applicable laws related to trademark protection. All goodwill arising out of Your use of the Mark will inure to the benefit of Tencent.
+c.	If You commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against Us or any person or entity alleging that the Materials or any Output, or any portion of any of the foregoing, infringe any intellectual property or other right owned or licensable by You, then all licenses granted to You under this Agreement shall terminate as of the date such lawsuit or other proceeding is filed. You will defend, indemnify and hold harmless Us from and against any claim by any Third Party arising out of or related to Your or the Third Party’s use or distribution of the Tencent HY Works.
+d.	Tencent claims no rights in Outputs You generate. You and Your users are solely responsible for Outputs and their subsequent uses.
+7.	DISCLAIMERS OF WARRANTY AND LIMITATIONS OF LIABILITY.
+a.	We are not obligated to support, update, provide training for, or develop any further version of the Tencent HY Works or to grant any license thereto.
+b.	UNLESS AND ONLY TO THE EXTENT REQUIRED BY APPLICABLE LAW, THE TENCENT HY WORKS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED “AS IS” WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES OF ANY KIND INCLUDING ANY WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, COURSE OF DEALING, USAGE OF TRADE, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING, REPRODUCING, MODIFYING, PERFORMING, DISPLAYING OR DISTRIBUTING ANY OF THE TENCENT HY WORKS OR OUTPUTS AND ASSUME ANY AND ALL RISKS ASSOCIATED WITH YOUR OR A THIRD PARTY’S USE OR DISTRIBUTION OF ANY OF THE TENCENT HY WORKS OR OUTPUTS AND YOUR EXERCISE OF RIGHTS AND PERMISSIONS UNDER THIS AGREEMENT.
+c.	TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL TENCENT OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, FOR ANY DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, CONSEQUENTIAL OR PUNITIVE DAMAGES, OR LOST PROFITS OF ANY KIND ARISING FROM THIS AGREEMENT OR RELATED TO ANY OF THE TENCENT HY WORKS OR OUTPUTS, EVEN IF TENCENT OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
+8.	SURVIVAL AND TERMINATION.
+a.	The term of this Agreement shall commence upon Your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
+b.	We may terminate this Agreement if You breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, You must promptly delete and cease use of the Tencent HY Works. Sections 6(a), 6(c), 7 and 9 shall survive the termination of this Agreement.
+9.	GOVERNING LAW AND JURISDICTION.
+a.	This Agreement and any dispute arising out of or relating to it will be governed by the laws of the Hong Kong Special Administrative Region of the People’s Republic of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
+b.	Exclusive jurisdiction and venue for any dispute arising out of or relating to this Agreement will be a court of competent jurisdiction in the Hong Kong Special Administrative Region of the People’s Republic of China, and Tencent and Licensee consent to the exclusive jurisdiction of such court with respect to any such dispute.
+ 
+EXHIBIT A
+ACCEPTABLE USE POLICY
+
+Tencent reserves the right to update this Acceptable Use Policy from time to time.
+Last modified: December 30, 2025
+
+Tencent endeavors to promote safe and fair use of its tools and features, including Tencent HY. You agree not to use Tencent HY or Model Derivatives:
+1.	Outside the Territory;
+2.	In any way that violates any applicable national, federal, state, local, international or any other law or regulation;
+3.	To harm Yourself or others;
+4.	To repurpose or distribute output from Tencent HY or any Model Derivatives to harm Yourself or others; 
+5.	To override or circumvent the safety guardrails and safeguards We have put in place;
+6.	For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
+7.	To generate or disseminate verifiably false information and/or content with the purpose of harming others or influencing elections;
+8.	To generate or facilitate false online engagement, including fake reviews and other means of fake online engagement;
+9.	To intentionally defame, disparage or otherwise harass others;
+10.	To generate and/or disseminate malware (including ransomware) or any other content to be used for the purpose of harming electronic systems;
+11.	To generate or disseminate personal identifiable information with the purpose of harming others;
+12.	To generate or disseminate information (including images, code, posts, articles), and place the information in any public context (including –through the use of bot generated tweets), without expressly and conspicuously identifying that the information and/or content is machine generated;
+13.	To impersonate another individual without consent, authorization, or legal right;
+14.	To make high-stakes automated decisions in domains that affect an individual’s safety, rights or wellbeing (e.g., law enforcement, migration, medicine/health, management of critical infrastructure, safety components of products, essential services, credit, employment, housing, education, social scoring, or insurance);
+15.	In a manner that violates or disrespects the social ethics and moral standards of other countries or regions;
+16.	To perform, facilitate, threaten, incite, plan, promote or encourage violent extremism or terrorism;
+17.	For any use intended to discriminate against or harm individuals or groups based on protected characteristics or categories, online or offline social behavior or known or predicted personal or personality characteristics;
+18.	To intentionally exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
+19.	For military purposes;
+20.	To engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or other professional practices.

README.md

@@ -1,60 +1,294 @@
-### Use with transformers
-First, please install transformers, recommends v4.56.0
-```SHELL
-pip install transformers==4.56.0
-```
+<p align="left">
+   English&nbsp;｜&nbsp;<a href="README_CN.md">中文</a>
+</p>
+<br>
+
+<p align="center">
+ <img src="imgs/logo-en.png" width="400"/> <br> 
+</p>
+
+<div align="center" style="line-height: 1;">
+
+
+[![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Tencent%20Hy-ffc107?color=ffc107&logoColor=white)](https://huggingface.co/collections/tencent/hy-mt2)
+&nbsp;&nbsp;
+[![ModelScope](https://img.shields.io/badge/ModelScope-Tencent%20Hy-624aff)](https://modelscope.cn/collections/Tencent-Hunyuan/Hy-MT2)
+
+</div>
+
+<p align="center">
+    🖥️&nbsp;<a href="https://aistudio.tencent.com/"><b>Official Website</b></a>&nbsp;&nbsp;|&nbsp;&nbsp;
+    💬&nbsp;<a href="https://github.com/Tencent-Hunyuan/Hy-MT2"><b>GitHub</b></a>&nbsp;&nbsp;|&nbsp;&nbsp;
+    🪡&nbsp;<a href="https://github.com/Tencent/AngelSlim/tree/main"><b>AngelSlim</b></a></p>
+
+## Model Introduction
+
+
+**Hy-MT2** is a multilingual machine translation model series covering both Dense and MoE architectures. It includes three fast-thinking models: **Hy-MT2-1.8B, 7B, and 30B-A3B**. The series supports translation among 33 languages and 5 ethnic minority languages / Chinese dialects, as well as multilingual instruction following. The series also provides **1.25-bit extreme quantized versions** based on AngelSlim. Among them, the 1.8B model requires only 440 MB of storage and runs 1.5x faster than traditional 4-bit inference on the Apple A15 chip.
+
+Evaluation results show that Hy-MT2 performs strongly across multiple scenarios:
+
+* **General Translation (FLORES-200)**: The average performance of the three models reaches 89.9%, 97.9%, and 98.6% of **Gemini 3.1 Pro (Think)**, respectively. Among them, the 7B and A3B models outperform **DeepSeek-V4-Pro**, while the 1.8B model achieves better overall performance than commercial APIs such as Microsoft Translator.
+* **Real-World Scenarios and Professional Domains (WildMTBench/DomainMTBench)**: The GEMBA scores of the three models reach more than 96%–99% of Gemini 3.1 Pro (Think), and all of them outperform larger open-source models.
+* **Translation Instruction Following (IFMTBench)**: The models significantly outperform open-source models of the same scale, while the A3B model approaches the performance of Gemini 3.1 Pro (Think).
+
+In summary, Hy-MT2 is an efficient and powerful translation model series designed for complex real-world scenarios.
+
+In this release, we also open-source [IFMTBench](./IFMTBench/README.md), a benchmark for evaluating translation instruction-following capabilities.
+
+We also welcome everyone to use our released Hy-MT2-Translator Skill, which makes it easy to integrate Hy-MT2 series models for translation tasks. Download links: [ClawHub](https://clawhub.ai/tencent-adm/hy-mt2-translator-skill) and [SkillHub](https://skillhub.cn/skills/hy-mt2-translator).
+
+## News
+<br>
+* 2026.5.21  We open-sourced **Hy-MT2-1.8B**/**Hy-MT2-7B**/**Hy-MT2-30B-A3B** on HuggingFace and ModelScope.
+* 2025.12.30 We open-sourced **HY-MT1.5-1.8B** and **HY-MT1.5-7B** on HuggingFace and ModelScope.
+* 2025.9.1 We open-sourced **Hunyuan-MT-7B** and **Hunyuan-MT-Chimera-7B** on HuggingFace and ModelScope.
+
+
+## Results
+<div align='center'>
+<img src="imgs/main_result.png" width = "100%" />
+</div>
+
+For more experimental results and analysis, please refer to our [technical report](./HY_MT2_0_Technical_Report.pdf).
+
+&nbsp;
+
+## Model Links
+| Model Name  | Description | Download Link |
+| ----------- | ----------- |-----------
+| Hy-MT2-1.8B  | Hunyuan 1.8B translation model |🤗 [Model](https://huggingface.co/tencent/Hy-MT2-1.8B)|
+| Hy-MT2-1.8B-FP8 | Hunyuan 1.8B translation model, FP8 quantization    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-1.8B-FP8)|
+| Hy-MT2-1.8B-GGUF | Hunyuan 1.8B translation model, llama.cpp    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-1.8B-GGUF)|
+| Hy-MT2-7B | Hunyuan 7B translation model    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-7B)|
+| Hy-MT2-7B-FP8 | Hunyuan 7B translation model, FP8 quantization     | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-7B-FP8)|
+| Hy-MT2-7B-GGUF | Hunyuan 7B translation model, llama.cpp    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-7B-GGUF)|
+| Hy-MT2-30B-A3B | Hunyuan 30B-A3B translation model    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-30B-A3B)|
+| Hy-MT2-30B-A3B-FP8 | Hunyuan 30B-A3B translation model, FP8 quantization     | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-30B-A3B-FP8)|
+
+
 
-*!!! If you want to load fp8 model with transformers, you need to change the name"ignored_layers" in config.json to "ignore" and upgrade the compressed-tensors to compressed-tensors-0.11.0.*
+## Hy-MT2 Translation Task Instruction Examples (Chinese-English Comparison)
 
-The following code snippet shows how to use the transformers library to load and apply the model.
+*Note: In the following examples, both source_lang and target_lang should use the full language names. Chinese names should be used in Chinese prompts, and English names should be used in English prompts.*
 
-we use tencent/Hy-MT2-1.8B for example
+| Type | Chinese prompt | English prompt |
+|---|---|---|
+| **Default Translation** | 将以下文本翻译为 `{target_lang}`，注意**只需要输出翻译后的结果，不要额外解释**：<br><br>`{source_text}` | Translate the following text into `{target_lang}`. Note that you should **only output the translated result without any additional explanation**:<br><br>`{source_text}` |
+| **Terminology** | *参考下面的翻译：*<br>`{text}` 翻译成 `{text}`<br>`{text}` 翻译成 `{text}`<br>`{text}` 翻译成 `{text}`<br>将以下文本翻译为 `{target_lang}`，注意**只需要输出翻译后的结果，不要额外解释**：<br><br>`{source_text}` | *Reference the following translations:*<br>`{text}` translates to `{text}`<br>`{text}` translates to `{text}`<br>`{text}` translates to `{text}`<br><br>Translate the following text into `{target_lang}`. Note that you must **ONLY output the translated result without any additional explanation**:<br><br>`{source_text}` |
+| **Style** | 请将以下文本翻译为 `{target_lang}`。<br>注意翻译的风格要严格符合【**`{target_style}`**】<br><br>`{source_text}` | Please translate the following text into `{target_lang}`. Note that the translation style must strictly conform to [**`{target_style}`**]:<br><br>`{source_text}` |
+| **Personalization** | *【待翻译文本】*<br>`{source_text}`<br><br>*【翻译任务】*<br>1、**`{user_preferences}`**<br>2、**`{user_preferences}`**<br>3、……<br>4、将【待翻译文本】翻译为 `{target_lang}`。 | *[Source Text]*<br>`{source_text}`<br><br>*[Translation Tasks]*<br>1. **`{user_preferences}`**<br>2. **`{user_preferences}`**<br>3. ...<br>4. Translate the [Source Text] into `{target_lang}`. |
+| **Delimiters** | 请将以下文本准确翻译为 `{target_lang}`。<br>你必须在译文中**保留等量的分隔符，绝对不可遗漏、转义或翻译该符号，并注意分隔符的位置**。<br><br>`{source_text}` | Please accurately translate the following text into `{target_lang}`.<br>You must **retain the exact same number of delimiters in the translation. Strictly do not omit, escape, or translate these symbols, and pay close attention to their placement**.<br><br>`{source_text}` |
+| **Structured Data 1** | *# 任务目标*<br>将下方 `{source_text}` 中的 `{format_type}` 格式数据翻译为 `{target_lang}`。<br><br>*# 严格约束*<br>1. **结构锁定**：绝对保持原有的 `{format_type}` 数据结构、缩进和层级完全不变。<br>2. **选择性翻译**：仅翻译面向用户展示的可见文本内容。<br>3. **禁止修改**：**严禁**翻译或更改任何代码标签、键名 (Key)、变量占位符（如 `{{var}}`、`${var}`、`%s`、`%d` 等）或代码属性。<br><br>*# 数据输入*<br>`{source_text}` | *### Task*<br>Translate the user-facing text within the following `{format_type}` data into `{target_lang}`.<br><br>*### Strict Rules*<br>1. **Structure Preservation:** You MUST preserve the original `{format_type}` data structure, nesting, hierarchy, and indentation exactly as they are.<br>2. **Selective Translation:** Translate ONLY the visible, user-facing text content/values.<br>3. **Strict Non-Translation:** NEVER translate or alter code tags, keys, properties, object names, or variable placeholders. Leave them exactly in their original English/code form.<br><br>*### Source Data*<br>`{source_text}` |
+| **Structured Data 2** | *【背景信息】*<br>`{background_text}`<br><br>请结合背景信息将以下文本翻译为 `{target_lang}`。<br><br>*【待翻译文本】*<br>`{source_text}` | *[Background Information]*<br>`{background_text}`<br><br>Please translate the following text into `{target_lang}`, taking the provided background information into consideration.<br><br>*[Source Text]*<br>`{source_text}` |
+
+---
+
+## Inference and Deployment
+### transformers
+
+transformers>=5.6.0
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
-import os
-
-model_name_or_path = "tencent/Hy-MT2-1.8B"
-
-tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
-model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto")  # You may want to use bfloat16 and/or move to GPU here
-messages = [
-    {"role": "user", "content": "Translate the following segment into Chinese, without additional explanation.\n\nIt’s on the house."},
-]
-tokenized_chat = tokenizer.apply_chat_template(
-    messages,
-    tokenize=True,
-    add_generation_prompt=False,
-    return_tensors="pt"
+import torch
+
+model_path = "tencent/Hy-MT2-30B-A3B"
+
+# Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+
+# Load model
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
 )
 
-outputs = model.generate(tokenized_chat.to(model.device), max_new_tokens=2048)
-output_text = tokenizer.decode(outputs[0])
+model.eval()
+
+# Example inference
+prompt = "将以下文本翻译成英语,注意只需要输出翻译后的结果,不要额外解释:\n\n今天天气真好。"
+messages = [{"role": "user", "content": prompt}]
+inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=4096,
+    )
+response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
+print(response)
 ```
 
-We recommend using the following set of parameters for inference. Note that our model does not have the default system_prompt.
+### vllm
+
+Build vLLM from source:
+```bash
+uv venv --python 3.12 --seed --managed-python
+source .venv/bin/activate
+git clone https://github.com/vllm-project/vllm.git
+cd vllm
+uv pip install --editable . --torch-backend=auto
+```
+
+Start the vLLM server:
+
+```bash
+vllm serve tencent/Hy-MT2-30B-A3B --tensor-parallel-size 1
+```
+
+### sglang
+
+Build SGLang from source:
+```bash
+git clone https://github.com/sgl-project/sglang
+cd sglang
+pip3 install pip --upgrade
+pip3 install "transformers>=5.6.0"
+pip3 install -e "python"
+```
+
+Launch SGLang server:
+
+```bash
+python3 -m sglang.launch_server --model tencent/Hy-MT2-30B-A3B --tp 1
+```
+
+### llama_cpp
+**❕❕ This gguf depends on our STQ kernel, which is released at [PR #22836](https://github.com/ggml-org/llama.cpp/pull/22836).**
+
+#### Clone llama.cpp
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp.git
+```
+
+#### Enter the llama.cpp folder
+
+```bash
+cd llama.cpp
+```
+
+#### Build llama.cpp
+
+```bash
+cmake -B build
+cmake --build build --config Release
+```
+
+#### Run a completion example
+
+```bash
+./build/bin/llama-completion \
+  --model model.gguf  \
+  -p "Translate the following segment into Chinese, without additional explanation：Hello" \
+  --jinja \
+  -ngl 0 \
+  -n 64 -st 
+```
+
+#### Run the llama.cpp benchmark
+
+```bash
+./build/bin/llama-bench -m model_zoo/model.gguf  -ngl 0
+```
+
+
+For 1.8B and 7B, we recommend using the following parameters for inference. Note that our models do not have a default system_prompt.
 
 ```json
+
 {
-  "top_k": 20,
+  "temperature": 0.7,
   "top_p": 0.6,
+  "top_k": 20,
   "repetition_penalty": 1.05,
-  "temperature": 0.7
+  "max_tokens": 4096
 }
 ```
 
-### Use with vllm
+For 30B-A3B, we recommend using the following parameters for inference. Note that our models do not have a default system_prompt.
 
-Start the vLLM server:
+```json
 
-```bash
-vllm serve tencent/Hy-MT2-1.8B --tensor-parallel-size 1
+{
+  "temperature": 0.7,
+  "top_p": 1.0,
+  "top_k": -1,
+  "repetition_penalty": 1.0,
+  "max_tokens": 4096
+}
 ```
 
-### Use with sglang
+## Model Training
+Hy-MT2 provides a complete model training pipeline, supporting both full-parameter fine-tuning and LoRA fine-tuning, as well as multiple DeepSpeed ZeRO configurations and LLaMA-Factory integration.
 
-Launch SGLang server:
+For detailed training documentation, please refer to: [Model Training Guide](./train/README.md)
 
-```bash
-python3 -m sglang.launch_server --model tencent/Hy-MT2-1.8B --tp 1
-```
+## Quantization Tool
+
+We provide [AngelSlim](https://github.com/tencent/AngelSlim), an easy-to-use, comprehensive, and efficient large model compression toolkit covering common quantization algorithms, low-bit quantization, speculative sampling, and more.
+
+
+## Supported Languages
+| Languages         | Abbr.   | Chinese Names   |
+|-------------------|---------|-----------------|
+| Chinese           | zh      | 中文            |
+| English           | en      | 英语            |
+| French            | fr      | 法语            |
+| Portuguese        | pt      | 葡萄牙语        |
+| Spanish           | es      | 西班牙语        |
+| Japanese          | ja      | 日语            |
+| Turkish           | tr      | 土耳其语        |
+| Russian           | ru      | 俄语            |
+| Arabic            | ar      | 阿拉伯语        |
+| Korean            | ko      | 韩语            |
+| Thai              | th      | 泰语            |
+| Italian           | it      | 意大利语        |
+| German            | de      | 德语            |
+| Vietnamese        | vi      | 越南语          |
+| Malay             | ms      | 马来语          |
+| Indonesian        | id      | 印尼语          |
+| Filipino          | tl      | 菲律宾语        |
+| Hindi             | hi      | 印地语          |
+| Traditional Chinese | zh-Hant| 繁体中文        |
+| Polish            | pl      | 波兰语          |
+| Czech             | cs      | 捷克语          |
+| Dutch             | nl      | 荷兰语          |
+| Khmer             | km      | 高棉语          |
+| Burmese           | my      | 缅甸语          |
+| Persian           | fa      | 波斯语          |
+| Gujarati          | gu      | 古吉拉特语      |
+| Urdu              | ur      | 乌尔都语        |
+| Telugu            | te      | 泰卢固语        |
+| Marathi           | mr      | 马拉地语        |
+| Hebrew            | he      | 希伯来语        |
+| Bengali           | bn      | 孟加拉语        |
+| Tamil             | ta      | 泰米尔语        |
+| Ukrainian         | uk      | 乌克兰语        |
+| Tibetan           | bo      | 藏语            |
+| Kazakh            | kk      | 哈萨克语        |
+| Mongolian         | mn      | 蒙古语          |
+| Uyghur            | ug      | 维吾尔语        |
+| Cantonese         | yue     | 粤语            |
+
+
+
+## Citing Hy-MT2
+
+```bibtex
+@misc{hy-mt1.5,
+      title={HY-MT1.5 Technical Report}, 
+      author={Mao Zheng and Zheng Li and Tao Chen and Mingyang Song and Di Wang},
+      year={2025},
+      eprint={2512.24092},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2512.24092}, 
+}
+```
+
+## Contact Us
+If you would like to leave feedback for our R&D and product teams, you are welcome to contact the Tencent Hunyuan LLM team. You can reach us by email at hunyuan_opensource@tencent.com.

README_CN.md

@@ -0,0 +1,297 @@
+<p align="left">
+   <a href="README.md">English</a>&nbsp;｜&nbsp;中文
+</p>
+<br>
+
+<p align="center">
+ <img src="imgs/logo-zh.png" width="400"/> <br> 
+</p>
+
+<div align="center" style="line-height: 1;">
+
+
+[![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Tencent%20Hy-ffc107?color=ffc107&logoColor=white)](https://huggingface.co/collections/tencent/hy-mt2)
+&nbsp;&nbsp;
+[![ModelScope](https://img.shields.io/badge/ModelScope-Tencent%20Hy-624aff)](https://modelscope.cn/collections/Tencent-Hunyuan/Hy-MT2)
+
+</div>
+
+<p align="center">
+    🖥️&nbsp;<a href="https://aistudio.tencent.com/"><b>官方网站</b></a>&nbsp;&nbsp;|&nbsp;&nbsp;
+    💬&nbsp;<a href="https://github.com/Tencent-Hunyuan/Hy-MT2"><b>GitHub</b></a>&nbsp;&nbsp;|&nbsp;&nbsp;
+    🪡&nbsp;<a href="https://github.com/Tencent/AngelSlim/tree/main"><b>AngelSlim</b></a></p>
+
+
+## 模型介绍
+
+
+**Hy-MT2**是涵盖Dense和MoE架构的多语言机器翻译模型系列，包含 **Hy-MT2-1.8B、7B 和 30B-A3B** 三款快思考模型，支持33种语言互译和5种民汉/方言，支持多语言指令遵循。该系列提供基于AngelSlim的**1.25-bit极致量化版**，其中1.8B模型仅需440MB存储，在苹果A15芯片上比传统4-bit推理快1.5倍。
+
+评测结果表明，Hy-MT2 在多场景下表现出众：
+
+* **通用翻译（FLORES-200）**：三款模型平均性能分别达到 **Gemini 3.1 Pro (Think)** 的 89.9%、97.9% 和 98.6%。其中 7B 和 A3B 性能超越 **DeepSeek-V4-Pro**，1.8B 综合表现超越微软翻译等商业 API。
+* **真实场景与专业领域（WildMTBench/DomainMTBench）**：三款模型 GEMBA 评分达 Gemini 3.1 Pro (Think) 的 96%~99% 以上，且均优于更大规模的开源模型。
+* **翻译指令遵循（IFMTBench）**：大幅超越同规模开源模型，A3B 性能逼近 Gemini 3.1 Pro (Think)。
+
+总之，Hy-MT2 是一个面向真实复杂场景、高效且强大的翻译模型系列。
+
+同时，本次我们也开源了一个针对翻译指令遵循能力的评测集[IFMTBench](./IFMTBench/README_zh.md)。
+
+也欢迎大家使用我们发布的 Hy-MT2-Translator Skill，可以方便接入Hy-MT2系列模型完成翻译任务，下载链接[ClawHub](https://clawhub.ai/tencent-adm/hy-mt2-translator-skill)和[SkillHub](https://skillhub.cn/skills/hy-mt2-translator)。
+
+## 新闻
+<br>
+* 2026.5.21  我们在HuggingFace和ModelScope上开源了 **Hy-MT2-1.8B**/**Hy-MT2-7B**/**Hy-MT2-30B-A3B**/**IFMTBench**
+* 2025.12.30 我们在HuggingFace和ModelScope开源了 **HY-MT1.5-1.8B**和**HY-MT1.5-7B**
+* 2025.9.1 我们在HuggingFace和ModelScope开源了 **Hunyuan-MT-7B**和**Hunyuan-MT-Chimera-7B**。
+
+
+## 效果
+<div align='center'>
+<img src="imgs/main_result.png" width = "100%" />
+</div>
+
+更多的实验效果和分析可以参考我们的[技术报告](./HY_MT2_0_Technical_Report.pdf)。
+
+&nbsp;
+
+## 模型链接
+| 模型名  | 简介 | 下载链接 |
+| ----------- | ----------- |-----------
+| Hy-MT2-1.8B  | 混元1.8B翻译模型 |🤗 [Model](https://huggingface.co/tencent/Hy-MT2-1.8B)|
+| Hy-MT2-1.8B-FP8 | 混元1.8B翻译模型，fp8量化    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-1.8B-FP8)|
+| Hy-MT2-1.8B-GGUF | 混元1.8B翻译模型， llama.cpp    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-1.8B-GGUF)|
+| Hy-MT2-7B | 混元7B翻译模型    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-7B)|
+| Hy-MT2-7B-FP8 | 混元7B翻译模型，fp8量化     | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-7B-FP8)|
+| Hy-MT2-7B-GGUF | 混元7B翻译模型， llama.cpp    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-7B-GGUF)|
+| Hy-MT2-30B-A3B | 混元30B-A3B翻译模型    | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-30B-A3B)|
+| Hy-MT2-30B-A3B-FP8 | 混元30B-A3B翻译模型，fp8量化     | 🤗 [Model](https://huggingface.co/tencent/Hy-MT2-30B-A3B-FP8)|
+
+
+
+## Hy-MT2 翻译任务指令示例（中英文对照）
+
+*注意：下面的source_lang和target_lang都使用语言的全称，中文使用中文全称，英文使用英文全称。*
+
+| Type | Chinese prompt | English prompt |
+|---|---|---|
+| **Default Translation** | 将以下文本翻译为 `{target_lang}`，注意**只需要输出翻译后的结果，不要额外解释**：<br><br>`{source_text}` | Translate the following text into `{target_lang}`. Note that you should **only output the translated result without any additional explanation**:<br><br>`{source_text}` |
+| **Terminology** | *参考下面的翻译：*<br>`{text}` 翻译成 `{text}`<br>`{text}` 翻译成 `{text}`<br>`{text}` 翻译成 `{text}`<br>将以下文本翻译为 `{target_lang}`，注意**只需要输出翻译后的结果，不要额外解释**：<br><br>`{source_text}` | *Reference the following translations:*<br>`{text}` translates to `{text}`<br>`{text}` translates to `{text}`<br>`{text}` translates to `{text}`<br><br>Translate the following text into `{target_lang}`. Note that you must **ONLY output the translated result without any additional explanation**:<br><br>`{source_text}` |
+| **Style** | 请将以下文本翻译为 `{target_lang}`。<br>注意翻译的风格要严格符合【**`{target_style}`**】<br><br>`{source_text}` | Please translate the following text into `{target_lang}`. Note that the translation style must strictly conform to [**`{target_style}`**]:<br><br>`{source_text}` |
+| **Personalization** | *【待翻译文本】*<br>`{source_text}`<br><br>*【翻译任务】*<br>1、**`{user_preferences}`**<br>2、**`{user_preferences}`**<br>3、……<br>4、将【待翻译文本】翻译为 `{target_lang}`。 | *[Source Text]*<br>`{source_text}`<br><br>*[Translation Tasks]*<br>1. **`{user_preferences}`**<br>2. **`{user_preferences}`**<br>3. ...<br>4. Translate the [Source Text] into `{target_lang}`. |
+| **Delimiters** | 请将以下文本准确翻译为 `{target_lang}`。<br>你必须在译文中**保留等量的分隔符，绝对不可遗漏、转义或翻译该符号，并注意分隔符的位置**。<br><br>`{source_text}` | Please accurately translate the following text into `{target_lang}`.<br>You must **retain the exact same number of delimiters in the translation. Strictly do not omit, escape, or translate these symbols, and pay close attention to their placement**.<br><br>`{source_text}` |
+| **Structured Data 1** | *# 任务目标*<br>将下方 `{source_text}` 中的 `{format_type}` 格式数据翻译为 `{target_lang}`。<br><br>*# 严格约束*<br>1. **结构锁定**：绝对保持原有的 `{format_type}` 数据结构、缩进和层级完全不变。<br>2. **选择性翻译**：仅翻译面向用户展示的可见文本内容。<br>3. **禁止修改**：**严禁**翻译或更改任何代码标签、键名 (Key)、变量占位符（如 `{{var}}`、`${var}`、`%s`、`%d` 等）或代码属性。<br><br>*# 数据输入*<br>`{source_text}` | *### Task*<br>Translate the user-facing text within the following `{format_type}` data into `{target_lang}`.<br><br>*### Strict Rules*<br>1. **Structure Preservation:** You MUST preserve the original `{format_type}` data structure, nesting, hierarchy, and indentation exactly as they are.<br>2. **Selective Translation:** Translate ONLY the visible, user-facing text content/values.<br>3. **Strict Non-Translation:** NEVER translate or alter code tags, keys, properties, object names, or variable placeholders. Leave them exactly in their original English/code form.<br><br>*### Source Data*<br>`{source_text}` |
+| **Structured Data 2** | *【背景信息】*<br>`{background_text}`<br><br>请结合背景信息将以下文本翻译为 `{target_lang}`。<br><br>*【待翻译文本】*<br>`{source_text}` | *[Background Information]*<br>`{background_text}`<br><br>Please translate the following text into `{target_lang}`, taking the provided background information into consideration.<br><br>*[Source Text]*<br>`{source_text}` |
+
+---
+
+## 推理和部署
+### transformers
+
+transformers>=5.6.0
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_path = "tencent/Hy-MT2-30B-A3B"
+
+# Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+
+# Load model
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+model.eval()
+
+# Example inference
+prompt = "将以下文本翻译成英语,注意只需要输出翻译后的结果,不要额外解释:\n\n今天天气真好。"
+messages = [{"role": "user", "content": prompt}]
+inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=4096,
+    )
+response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
+print(response)
+```
+
+### vllm
+
+Build vLLM from source:
+```bash
+uv venv --python 3.12 --seed --managed-python
+source .venv/bin/activate
+git clone https://github.com/vllm-project/vllm.git
+cd vllm
+uv pip install --editable . --torch-backend=auto
+```
+
+Start the vLLM server:
+
+```bash
+vllm serve tencent/Hy-MT2-30B-A3B --tensor-parallel-size 1
+```
+
+### sglang
+
+Build SGLang from source:
+```bash
+git clone https://github.com/sgl-project/sglang
+cd sglang
+pip3 install pip --upgrade
+pip3 install "transformers>=5.6.0"
+pip3 install -e "python"
+```
+
+Launch SGLang server:
+
+```bash
+python3 -m sglang.launch_server --model tencent/Hy-MT2-30B-A3B --tp 1
+```
+
+
+### llama_cpp
+**❕❕ This gguf depends on our STQ kernel, which is released at [PR #22836](https://github.com/ggml-org/llama.cpp/pull/22836).**
+
+#### Clone llama.cpp
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp.git
+```
+
+#### Enter the llama.cpp folder
+
+```bash
+cd llama.cpp
+```
+
+#### Build llama.cpp
+
+```bash
+cmake -B build
+cmake --build build --config Release
+```
+
+#### Run a completion example
+
+```bash
+./build/bin/llama-completion \
+  --model model.gguf  \
+  -p "Translate the following segment into Chinese, without additional explanation：Hello" \
+  --jinja \
+  -ngl 0 \
+  -n 64 -st 
+```
+
+#### Run the llama.cpp benchmark
+
+```bash
+./build/bin/llama-bench -m model_zoo/model.gguf  -ngl 0
+```
+
+
+对于1.8B和7B，我们推荐使用下面这组参数进行推理。注意，我们的模型没有默认 system_prompt。
+
+```json
+
+{
+  "temperature": 0.7,
+  "top_p": 0.6,
+  "top_k": 20,
+  "repetition_penalty": 1.05,
+  "max_tokens": 4096
+}
+```
+
+对于30B-A3B，我们推荐使用下面这组参数进行推理。注意，我们的模型没有默认 system_prompt。
+
+```json
+
+{
+  "temperature": 0.7,
+  "top_p": 1.0,
+  "top_k": -1,
+  "repetition_penalty": 1.0,
+  "max_tokens": 4096
+}
+```
+
+
+## 模型训练
+Hy-MT2提供了完整的模型训练流程，支持全量微调和 LoRA 微调，同时支持 DeepSpeed ZeRO 多种配置以及 LLaMA-Factory 集成。
+
+详细的训练文档请参考：[模型训练指南](./train/README_CN.md)
+
+## 量化工具
+
+我们提供了 [AngelSlim](https://github.com/tencent/AngelSlim)——一套易用、全面、高效的大模型压缩工具包，涵盖常用量化算法、低比特量化和投机采样等能力。
+
+
+## 支持的语种
+| Languages         | Abbr.   | Chinese Names   |
+|-------------------|---------|-----------------|
+| Chinese           | zh      | 中文            |
+| English           | en      | 英语            |
+| French            | fr      | 法语            |
+| Portuguese        | pt      | 葡萄牙语        |
+| Spanish           | es      | 西班牙语        |
+| Japanese          | ja      | 日语            |
+| Turkish           | tr      | 土耳其语        |
+| Russian           | ru      | 俄语            |
+| Arabic            | ar      | 阿拉伯语        |
+| Korean            | ko      | 韩语            |
+| Thai              | th      | 泰语            |
+| Italian           | it      | 意大利语        |
+| German            | de      | 德语            |
+| Vietnamese        | vi      | 越南语          |
+| Malay             | ms      | 马来语          |
+| Indonesian        | id      | 印尼语          |
+| Filipino          | tl      | 菲律宾语        |
+| Hindi             | hi      | 印地语          |
+| Traditional Chinese | zh-Hant| 繁体中文        |
+| Polish            | pl      | 波兰语          |
+| Czech             | cs      | 捷克语          |
+| Dutch             | nl      | 荷兰语          |
+| Khmer             | km      | 高棉语          |
+| Burmese           | my      | 缅甸语          |
+| Persian           | fa      | 波斯语          |
+| Gujarati          | gu      | 古吉拉特语      |
+| Urdu              | ur      | 乌尔都语        |
+| Telugu            | te      | 泰卢固语        |
+| Marathi           | mr      | 马拉地语        |
+| Hebrew            | he      | 希伯来语        |
+| Bengali           | bn      | 孟加拉语        |
+| Tamil             | ta      | 泰米尔语        |
+| Ukrainian         | uk      | 乌克兰语        |
+| Tibetan           | bo      | 藏语            |
+| Kazakh            | kk      | 哈萨克语        |
+| Mongolian         | mn      | 蒙古语          |
+| Uyghur            | ug      | 维吾尔语        |
+| Cantonese         | yue     | 粤语            |
+
+
+
+## Citing Hy-MT2
+
+```bibtex
+@misc{hy-mt1.5,
+      title={HY-MT1.5 Technical Report}, 
+      author={Mao Zheng and Zheng Li and Tao Chen and Mingyang Song and Di Wang},
+      year={2025},
+      eprint={2512.24092},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2512.24092}, 
+}
+```
+
+## 联系我们
+如果你想给我们的研发和产品团队留言，欢迎联系我们腾讯混元LLM团队。你可以通过邮件（hunyuan_opensource@tencent.com）联系我们。

train/README.md

@@ -0,0 +1,303 @@
+<p align="left">
+    <a href="README_CN.md">中文</a> ｜ English
+</p>
+
+# Model Training
+
+Hy3 preview provides processes related to model training. This section details how to process training data for model training purposes.
+
+## Training Data Format and Processing
+
+The training data should be formatted as a list of messages. By default, the system prompt for both training and inference is empty, but you may customize it as needed.
+
+Below is a training data example for a translation task:
+
+```python
+# Translation task example
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n实验结果证明了假设的正确性。"}, {"role": "assistant", "content": "The experimental results demonstrate the correctness of the hypothesis."}]}
+
+```
+
+## Quick Start
+
+You can quickly get started by following the instructions in the Quick Start Guide.
+
+## Model Training
+
+### Hardware Requirements
+
+The following are the minimum hardware requirements for each model at max_seq_length = 8192:
+
+#### Hy-MT2-1.8B (Dense)
+
+| Training Method | DeepSpeed Strategy | Minimum GPU Requirement |
+|----------------|-------------------|------------------------|
+| LoRA Fine-tuning | ZeRO-2 (no offload) | 1 GPU (24GB+) |
+| Full Fine-tuning | ZeRO-2 (no offload) | 1 GPU (24GB+) |
+
+#### Hy-MT2-7B (Dense)
+
+| Training Method | DeepSpeed Strategy | Minimum GPU Requirement |
+|----------------|-------------------|------------------------|
+| LoRA Fine-tuning | ZeRO-2 (no offload) | 1 GPU (80GB+) |
+| Full Fine-tuning | ZeRO-3 (no offload) | 2 GPUs (80GB+ each) |
+
+#### Hy-MT2-30B-A3B (MoE)
+
+| Training Method | DeepSpeed Strategy | Minimum GPU Requirement |
+|----------------|-------------------|------------------------|
+| LoRA Fine-tuning | ZeRO-2 (no offload) | 8 GPUs on a single machine (80GB+ each) |
+| Full Fine-tuning | ZeRO-3 + offload | 8 GPUs on a single machine (80GB+ each) |
+
+### Configure Passwordless SSH Login Between Machines (Multi-Machine Training)
+
+> If you only use single-machine training, you can skip this section.
+
+The following instructions use two machines as an example, with their IPs denoted as `${ip1}` and `${ip2}`. All steps should be performed inside the Docker container.
+
+First, configure passwordless SSH for each container on every machine:
+
+```sh
+ssh-keygen			# Generate id_rsa and id_rsa.pub for passwordless login
+ssh-keygen -t rsa -A    # Generate /etc/ssh/ssh_host_rsa_key and ssh_host_ecdsa_key for SSH listening
+/usr/sbin/sshd -p 36005 -o ListenAddress=0.0.0.0        # Start SSH listening
+echo "Port 36005" > ~/.ssh/config   # Set SSH connection port to 36005
+passwd root    # Set the root password to avoid monitoring platform alerts
+```
+
+Note: `36005` is an example port. You may use any available port, but ensure it is **open** and **not occupied by other processes**.
+
+Next, in each machine's container, execute:
+
+```sh
+cat ~/.ssh/id_rsa.pub
+```
+
+**Copy the output SSH public key and paste it into the `~/.ssh/authorized_keys` file, one key per line. This must be done on every machine.** In the end, the `~/.ssh/authorized_keys` file on each machine should be identical and contain the public keys of all machines.
+
+Please note that for multi-node training, the code executed on each node must be identical. It is recommended to mount a shared network drive. If this is not possible, you must manually copy the dataset, scripts, and code to the same directory on each machine.
+
+### Launch Methods
+
+This project provides two training methods. You can choose based on your needs:
+
+- **DeepSpeed Native Training** (based on HuggingFace Transformers Trainer): Located in the `train/deepspeed_support` directory
+- **LLaMA-Factory Training**: Located in the `train/llama_factory_support` directory
+
+#### DeepSpeed Native Training
+
+Reference: [HuggingFace Transformers Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer)
+
+##### Training Scripts
+
+In the `train/deepspeed_support` directory, the training scripts for each model are as follows:
+
+| Model | Full Fine-tuning | LoRA Fine-tuning |
+|-------|-----------------|-----------------|
+| Hy-MT2-1.8B (Dense) | `bash train_dense.sh 1.8B` | `bash train_dense_lora.sh 1.8B` |
+| Hy-MT2-7B (Dense) | `bash train_dense.sh 7B` | `bash train_dense_lora.sh 7B` |
+| Hy-MT2-30B-A3B (MoE) | `bash train.sh` | `bash train_lora.sh` |
+
+##### Single-Machine Training
+
+In the `train/deepspeed_support` directory, install dependencies and execute the corresponding script:
+
+```sh
+pip install -r requirements.txt
+# Example: Dense 1.8B full fine-tuning
+bash train_dense.sh 1.8B
+```
+
+##### Multi-Machine Training
+
+To launch training across multiple machines, please first complete the configuration in [Configure Passwordless SSH Login Between Machines](#configure-passwordless-ssh-login-between-machines-multi-machine-training), and ensure all machines are within the same cluster.
+
+Confirm that dependencies are installed (if not, run `pip install -r requirements.txt`), then set the `IP_LIST` environment variable in the corresponding training script:
+
+```shell
+export HOST_GPU_NUM=8
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+IP_LIST=${IP_LIST:-"127.0.0.1"}
+```
+
+Note: If the `IP_LIST` environment variable is not set, replace `IP_LIST` with the IP list! The format is:
+```
+For a single IP:
+IP_LIST=${ip_1}
+
+For multiple IPs:
+IP_LIST=${ip_1},${ip_2}
+
+```
+
+Replace `${ip_1}` and `${ip_2}` with the actual IP addresses.
+
+Then, on the machine with `${ip1}`, execute the corresponding training script in the `train/deepspeed_support/` directory. On first launch, you may see the following output:
+
+```ssh
+The authenticity of host '[ip]:36005 ([ip]:36005)' can't be established.
+ECDSA key fingerprint is xxxxxx.
+ECDSA key fingerprint is MD5:xxxxxx.
+Are you sure you want to continue connecting (yes/no)?
+```
+
+Type `yes` to continue.
+
+##### Key Parameters
+
+The key parameters in the script are as follows:
+
+- `--deepspeed`: Path to the DeepSpeed configuration file. Three default DeepSpeed configuration files are provided in the `train/deepspeed_support` folder: `ds_zero2_no_offload.json`, `ds_zero3_no_offload.json`, and `ds_zero3_offload.json`, with decreasing memory requirements in that order.
+- `--model_name_or_path`: Path to the Hy3 preview HF pre-trained model weights to load.
+- `--tokenizer_name_or_path`: Path to the tokenizer folder.
+- `--train_data_file`: Path to the training file, which should be a jsonl file.
+- `--output_dir`: Output directory where logs, tensorboard files, and model weights will be stored.
+- `--per_device_train_batch_size`: Batch size per GPU.
+- `--gradient_accumulation_steps`: Number of gradient accumulation steps. The global batch size is `per_device_train_batch_size * gradient_accumulation_steps * dp_size`.
+- `--max_steps`: Total number of training steps.
+- `--save_steps`: Number of steps between saving checkpoints.
+- `--use_lora`: Whether to use LoRA training. Also accepts `--lora_rank`, `--lora_alpha`, and `--lora_dropout` parameters. By default, LoRA is applied to "q_proj", "k_proj", "v_proj", and "o_proj". To change this, modify the code. Note: **When using LoRA training, only the LoRA weights are saved, not the base model weights.** To merge LoRA weights, see the "LoRA Weight Merging" section below.
+- `--make_moe_param_leaf_module`: When using ZeRO-3 with MoE training, treat the MoE module as a leaf module, i.e., its parameters are not partitioned by ZeRO-3. This option is expected to significantly increase memory usage.
+- `--gradient_checkpointing`: Enable gradient checkpointing.
+- `--train_attention_params_only`: Whether to train only attention parameters.
+- `--learning_rate`: Maximum learning rate during training.
+- `--min_lr`: Minimum learning rate during training.
+- `--use_flash_attn`: Enable flash-attention for accelerated training.
+
+**Notes:**
+
+- To resume training from a previously saved checkpoint rather than loading pre-trained weights, specify `--resume_from_checkpoint` with the path to the checkpoint. Do not specify `--model_name_or_path`; this will load only the weights without the training state.
+- When resuming from a checkpoint, there may be minor differences in loss due to the randomness of some non-deterministic algorithms. This is normal. See: [HuggingFace Transformers Trainer Randomness](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#randomness)
+- When `--model_name_or_path` is specified, all model-related parameters will be ignored.
+- Samples within a batch are padded to the length of the longest sample in the batch, but the maximum length of each sample is `max_seq_length`. Any excess will be truncated.
+- If you see a warning about bias weights not being loaded, you can ignore it. Hunyuan-Large does not use bias.
+
+##### What if GPU Memory is Insufficient?
+
+Reference: [DeepSpeed Configuration](https://www.deepspeed.ai/docs/config-json/)
+
+You can try modifying the DeepSpeed configuration by removing the `auto` attribute from the following parameters and reducing their values:
+
+- `stage3_param_persistence_threshold`
+- `stage3_prefetch_bucket_size`
+- `stage3_max_reuse_distance`
+
+##### LoRA Weight Merging
+
+LoRA weights saved during training cannot be merged into the ZeRO-3 model at runtime, as ZeRO-3 partitions model weights across data parallel ranks. To merge LoRA weights into the base model, you can do so offline to obtain a merged weight file. Run `merge_lora_weight.sh` to merge the LoRA and base model weights. The parameters are:
+
+- `--base_model_path`: Directory of the base model weights
+- `--adapter_model_path`: Directory of the LoRA weights
+- `--output_path`: Directory to save the merged weights
+- `--save_dtype`: Data type for saving the merged weights; options are: fp16, bf16, fp32
+
+#### LLaMA-Factory Training
+
+If you are familiar with LLaMA-Factory, you may use it for fine-tuning. All scripts, code, and configuration files are archived in the `train/llama_factory_support` directory. Unless otherwise specified, all files mentioned below are located in this directory.
+
+##### Installation
+
+You can install LLaMA-Factory by downloading the source code from https://github.com/hiyouga/LLaMA-Factory/tree/main and following the instructions on the website.
+
+##### Training Scripts and Configuration Files
+
+The configuration files and launch scripts for each model are as follows:
+
+| Model | Full Fine-tuning Config | LoRA Fine-tuning Config | Launch Script |
+|-------|------------------------|------------------------|---------------|
+| Hy-MT2-1.8B (Dense) | `hy_dense_1_8b_full_sft.yaml` | `hy_dense_1_8b_lora_sft.yaml` | `bash train_lf_dense.sh` |
+| Hy-MT2-7B (Dense) | `hy_dense_7b_full_sft.yaml` | `hy_dense_7b_lora_sft.yaml` | `YAML_FILE=hy_dense_7b_full_sft.yaml bash train_lf_dense.sh` |
+| Hy-MT2-30B-A3B (MoE) | `hy_v3_full_sft.yaml` | `hy_v3_lora_sft.yaml` | `bash train_lf.sh` |
+
+> **Tip**: The Dense model launch script `train_lf_dense.sh` uses `hy_dense_1_8b_full_sft.yaml` by default. You can specify other configuration files via the `YAML_FILE` environment variable.
+
+Key parameters in the configuration files are as follows:
+
+**Model:**
+
+- `model_name_or_path`: Path to the Hy-MT HF format pre-trained model weights
+- `trust_remote_code`: Whether to trust remote code; Hy-MT requires this to be set to `true`
+
+**Training Method:**
+
+- `stage`: Training stage, currently `sft` (supervised fine-tuning)
+- `finetuning_type`: Fine-tuning type, either `full` (full fine-tuning) or `lora` (LoRA fine-tuning)
+- `deepspeed`: DeepSpeed configuration file path; `ds_zero3_offload.json` is recommended for full fine-tuning, `ds_zero2_offload_lora.json` for LoRA fine-tuning
+
+**LoRA Parameters (only effective during LoRA fine-tuning):**
+
+- `lora_rank`: LoRA rank, default `64`
+- `lora_alpha`: LoRA alpha coefficient, default `128`
+- `lora_dropout`: LoRA dropout ratio, default `0.05`
+- `lora_target`: Target modules for LoRA, default `q_proj,k_proj,v_proj,o_proj`
+
+**Dataset:**
+
+- `dataset_dir`: Dataset directory path
+- `dataset`: Dataset name, must be registered in `dataset_info.json` under `dataset_dir`
+- `template`: Chat template; Hy-MT2-1.8B uses `hy_dense_1_8b`, Hy-MT2-7B uses `hy_dense_7b`, Hy-MT2-30B-A3B uses `hy_v3`
+- `cutoff_len`: Maximum sequence length; sequences exceeding this will be truncated. For full fine-tuning, can be set to `262144` (262K); for LoRA fine-tuning, `8192` is recommended to save memory
+- `max_samples`: Maximum number of samples per dataset
+- `overwrite_cache`: Whether to overwrite cached preprocessed datasets
+
+**Output:**
+
+- `output_dir`: Output directory where logs, TensorBoard files, and weights will be stored
+- `logging_steps`: Number of steps between logging
+- `save_steps`: Number of steps between saving checkpoints
+- `plot_loss`: Whether to plot the training loss curve
+- `overwrite_output_dir`: Whether to overwrite the existing output directory
+- `save_only_model`: Whether to save only model weights (excluding optimizer states, etc.)
+- `report_to`: Logging tool, options: `none`, `wandb`, `tensorboard`, `swanlab`, `mlflow`
+
+**Training Hyperparameters:**
+
+- `per_device_train_batch_size`: Batch size per GPU
+- `gradient_accumulation_steps`: Gradient accumulation steps; `per_device_train_batch_size * gradient_accumulation_steps * dp_size` equals the global batch size
+- `learning_rate`: Maximum learning rate; `1.0e-5` recommended for full fine-tuning, `2.0e-4` for LoRA fine-tuning
+- `num_train_epochs`: Number of training epochs
+- `lr_scheduler_type`: Learning rate scheduler type; `cosine_with_min_lr` is recommended
+- `lr_scheduler_kwargs.min_lr_rate`: Ratio of minimum to maximum learning rate; e.g., `0.1` means the minimum learning rate is 10% of the maximum
+- `warmup_ratio`: Proportion of total training steps used for warmup
+- `bf16`: Whether to use BFloat16 mixed precision training
+- `gradient_checkpointing`: Whether to enable gradient checkpointing to save memory
+- `ddp_timeout`: Distributed training timeout (milliseconds)
+- `flash_attn`: Attention implementation; `fa2` (FlashAttention-2) is recommended, `sdpa` is also available; using `fa2` requires the flash-attn package
+- `resume_from_checkpoint`: Resume training from a specified checkpoint path; set to `null` to start from scratch
+
+##### Launch Training
+
+For multi-machine training, please first complete the configuration in [Configure Passwordless SSH Login Between Machines](#configure-passwordless-ssh-login-between-machines-multi-machine-training) (single-machine training can skip this step).
+
+Modify the following configuration at the beginning of the corresponding launch script:
+
+```shell
+export HOST_GPU_NUM=8
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+export IP_LIST=${IP_LIST:-"127.0.0.1"}
+```
+
+Note: If the `IP_LIST` environment variable is not set, replace `IP_LIST` with the IP list! The format is:
+```
+For a single IP:
+IP_LIST=${ip_1}
+
+For multiple IPs:
+IP_LIST=${ip_1},${ip_2}
+
+```
+
+Replace `${ip_1}` and `${ip_2}` with the actual IP addresses.
+
+Then, on each machine, run the corresponding launch script in the `train/llama_factory_support/` directory. For example:
+
+```sh
+# Dense 1.8B full fine-tuning
+bash train_lf_dense.sh
+
+# Dense 7B LoRA fine-tuning
+YAML_FILE=hy_dense_7b_lora_sft.yaml bash train_lf_dense.sh
+
+# MoE 30B-A3B full fine-tuning
+bash train_lf.sh
+```

train/README_CN.md

@@ -0,0 +1,303 @@
+<p align="left">
+   <a href="README.md">English</a> ｜ 中文
+</p>
+
+# 模型训练
+
+Hy-MT 提供了模型训练相关流程，您可以在此章节对训练数据格式进行处理以供模型训练使用。
+
+## 训练数据格式及处理
+
+训练数据按照以下形式处理为 messages 格式，训练和推理的默认 system prompt 为空，可以根据自己的需求进行设定。
+
+以下是翻译任务的训练数据示例：
+
+```python
+# 翻译任务示例
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n实验结果证明了假设的正确性。"}, {"role": "assistant", "content": "The experimental results demonstrate the correctness of the hypothesis."}]}
+
+```
+
+## 快速开始
+
+您可以参照快速开始文档中的内容进行快速上手。
+
+## 模型训练
+
+### 硬件需求
+
+以下为各模型在 max_seq_length = 8192 时的最低硬件需求：
+
+#### Hy-MT2-1.8B（Dense）
+
+| 训练方式 | DeepSpeed 策略 | 最低 GPU 需求 |
+|---------|---------------|-------------|
+| LoRA 微调 | ZeRO-2（无 offload） | 1 卡（24GB+） |
+| 全量微调 | ZeRO-2（无 offload） | 1 卡（24GB+） |
+
+#### Hy-MT2-7B（Dense）
+
+| 训练方式 | DeepSpeed 策略 | 最低 GPU 需求 |
+|---------|---------------|-------------|
+| LoRA 微调 | ZeRO-2（无 offload） | 1 卡（80GB+） |
+| 全量微调 | ZeRO-3（无 offload） | 2 卡（80GB+ 每卡） |
+
+#### Hy-MT2-30B-A3B（MoE）
+
+| 训练方式 | DeepSpeed 策略 | 最低 GPU 需求 |
+|---------|---------------|-------------|
+| LoRA 微调 | ZeRO-2（无 offload） | 单机 8 卡（80GB+ 每卡） |
+| 全量微调 | ZeRO-3 + offload | 单机 8 卡（80GB+ 每卡） |
+
+### 配置机器间免密 ssh 登录（多机训练）
+
+> 如果只使用单机训练，可跳过本节。
+
+以下操作以两个机器为例，两台机器的 ip 分别以`${ip1}`和`${ip2}`标识，以下操作均在 docker container 内执行。
+
+首先，配置多机container免密，在每台机器上执行。
+
+```sh
+ssh-keygen			# 生成id_rsa和id_rsa.pub，用于免密登录
+ssh-keygen -t rsa -A    # 生成/etc/ssh/ssh_host_rsa_key和ssh_host_ecdsa_key， 用于后面启动ssh listen
+/usr/sbin/sshd -p 36005 -o ListenAddress=0.0.0.0        # 启动 SSH 监听
+echo "Port 36005" > ~/.ssh/config   # ssh 连接端口修改为 36005
+passwd root    # 需要配置root密码，否则监测平台会报警
+```
+
+注意：这里的`36005`是一个示例端口，可以选用任意端口，但需要保证使用的端口**开放**且**不被其他的进程占用**。
+
+接下来，在每台机器的 container 内，执行：
+
+```sh
+cat ~/.ssh/id_rsa.pub
+```
+
+**将输出的 ssh 公钥复制并粘贴到`~/.ssh/authorized_keys`文件中，每行一个公钥，每台机器上都要做这个操作**。最终每台机器上的`~/.ssh/authorized_keys`文件内容应当是一致的，并且包含了所有机器的公钥。
+
+需要注意，多节点训练时，每个节点上执行的代码都得一致，建议挂载一个共享的网络盘，如果无法挂载共享网盘，则需要手动将数据集、脚本、代码复制在多台机器的相同目录下。
+
+### 启动方式
+
+本项目提供两种训练方式，您可以根据需求选择：
+
+- **DeepSpeed 原生训练**（基于 HuggingFace Transformers Trainer）：位于 `train/deepspeed_support` 目录下
+- **LLaMA-Factory 训练**：位于 `train/llama_factory_support` 目录下
+
+#### DeepSpeed 原生训练
+
+参考：[HuggingFace Transformers Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer)
+
+##### 训练脚本
+
+在 `train/deepspeed_support` 目录下，各模型对应的训练脚本如下：
+
+| 模型 | 全量微调 | LoRA 微调 |
+|------|---------|----------|
+| Hy-MT2-1.8B（Dense） | `bash train_dense.sh 1.8B` | `bash train_dense_lora.sh 1.8B` |
+| Hy-MT2-7B（Dense） | `bash train_dense.sh 7B` | `bash train_dense_lora.sh 7B` |
+| Hy-MT2-30B-A3B（MoE） | `bash train.sh` | `bash train_lora.sh` |
+
+##### 单机启动训练
+
+在 `train/deepspeed_support` 目录下，安装依赖后直接执行对应的脚本命令即可：
+
+```sh
+pip install -r requirements.txt
+# 示例：Dense 1.8B 全量微调
+bash train_dense.sh 1.8B
+```
+
+##### 多机启动训练
+
+如果要用多台机器启动训练，请先完成 [配置机器间免密 ssh 登录](#配置机器间免密-ssh-登录多机训练) 中的配置，并保证多台机器在一个集群内。
+
+确认依赖已经安装完成（如未安装，请执行`pip install -r requirements.txt`安装），然后在对应的训练脚本中设置 `IP_LIST` 环境变量：
+
+```shell
+export HOST_GPU_NUM=8
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+IP_LIST=${IP_LIST:-"127.0.0.1"}
+```
+
+注意：如果`IP_LIST`环境变量未设置，则将`IP_LIST`替换为IP列表！格式为：
+```
+如果只有一个IP：
+IP_LIST=${ip_1}
+
+如果有多个IP：
+IP_LIST=${ip_1},${ip_2}
+
+```
+
+请将`${ip_1}`和`${ip_2}`替换为真实的IP地址。
+
+然后，在`${ip1}`的机器上，在`train/deepspeed_support/`目录下，执行对应的训练脚本即可。注意第一次启动时可能会看见以下的输出：
+
+```ssh
+The authenticity of host '[ip]:36005 ([ip]:36005)' can't be established.
+ECDSA key fingerprint is xxxxxx.
+ECDSA key fingerprint is MD5:xxxxxx.
+Are you sure you want to continue connecting (yes/no)?
+```
+
+此时输入`yes`即可继续。
+
+##### 关键参数
+
+脚本中的关键参数如下：
+
+- `--deepspeed`: 此参数应当指向一个 deepspeed 的配置文件，`train/deepspeed_support`文件夹下提供了三种 DeepSpeed 的默认配置文件：`ds_zero2_no_offload.json`, `ds_zero3_no_offload.json`, `ds_zero3_offload.json`，这三个配置文件所需显存依次减少
+- `--model_name_or_path`: 要加载的 Hy-MT 的 HF 预训练模型权重，否则无法加载
+- `--tokenizer_name_or_path`: tokenizer 文件夹路径, 否则无法加载
+- `--train_data_file`: 训练文件路径，应该为一个 jsonl 文件
+- `--output_dir`: 输出文件夹，log、tensorboard 和权重都会存储在这个路径下
+- `--per_device_train_batch_size`: 每张卡上的 batch size
+- `--gradient_accumulation_steps`: 梯度累计次数，`per_device_train_batch_size * gradient_accumulation_steps * dp_size`为 global_batch_size
+- `--max_steps`: 训练的总步数
+- `--save_steps`: 每多少个 step 存储一个 checkpoint
+- `--use_lora`: 是否用 lora 训练，同时接收`--lora_rank`，`--lora_alpha`和`--lora_dropout`参数。lora 默认应用于 "q_proj", "k_proj", "v_proj", "o_proj" 四个参数，如果需要改变的话在代码中修改即可。注意：**使用 lora 训练时，只会保存 lora 的权重，而不会保存 base 模型的权重**，如果需要合并 lora 权重，看下面的“Lora 权重合并”一节
+- `--make_moe_param_leaf_module`：当用 zero3 以及 MoE 训练时，将 MoE 模块视作一个 leaf module，即它的参数不进行 zero3 切分，这个选项预计会显著增加显存占用
+- `--gradient_checkpointing`：开启梯度重计算
+- `--train_attention_params_only`: 是否只训练 attention 参数
+- `--learning_rate`: 训练时的最大学习率
+- `--min_lr`: 训练时的最小学习率
+- `--use_flash_attn`: 开启 flash-attention 进行训练加速
+
+**注意：**
+
+- 如果想从一个中途保存的 ckpt 继续训练，而不是加载一个预训练的权重，直接指定`--resume_from_checkpoint`为之前训练保存的 ckpt 路径，不要指定`--model_name_or_path`，这样只会加载权重，而不会加载训练状态
+- 从 ckpt 继续训练时，loss 可能会有微小的偏差，这是由一些非确定性算法带来的随机性，是正常现象。参考：[HuggingFace Transformers Trainer Randomness](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#randomness)
+- 当 `--model_name_or_path` 有效时，所有模型相关的参数都会被忽略
+- 一个 batch 内的样本会通过 padding 对齐 batch 内最长的样本，而每条样本的长度最长为 max_seq_length，超出的部分会被裁剪
+- 如果报出 bias 权重没有 load 的 warning，忽略即可，Hunyuan-Large 中不会用到 bias
+
+##### 显存不足怎么办？
+
+参考：[DeepSpeed Configuration](https://www.deepspeed.ai/docs/config-json/)
+
+可以尝试修改 ds config，去掉这几个参数的 auto 属性，改小试试看：
+
+- `stage3_param_persistence_threshold`
+- `stage3_prefetch_bucket_size`
+- `stage3_max_reuse_distance`
+
+##### Lora 模型合并
+
+保存下来的 lora 权重没法在训练运行时合并到 zero3 模型中，因为 zero3 开启时模型权重会切分到各 dp rank 上。因此如果想把 lora 权重合并到 base 模型上，可以通过离线的方式合并后得到权重文件。执行`merge_lora_weight.sh`即可完成 lora 权重和 base 模型权重的合并，其中的参数有：
+
+- `--base_model_path`：base 模型的权重目录
+- `--adapter_model_path`：lora 权重目录
+- `--output_path`：合并后的权重保存目录
+- `--save_dtype`： 以什么数据格式存储合并后的权重，可选值：fp16，bf16，fp32
+
+#### LLaMA-Factory 训练
+
+如果对 LLaMA-Factory 较为熟悉，可使用 LLaMA-Factory 进行微调。脚本、代码以及配置文件都归档在 `train/llama_factory_support` 目录下。如果没有特别说明，接下来我们提到的文件都是该目录下的文件。
+
+##### 安装
+
+可以通过下载源码 https://github.com/hiyouga/LLaMA-Factory/tree/main ，根据网站的指引进行安装。
+
+##### 训练脚本与配置文件
+
+各模型对应的配置文件和启动脚本如下：
+
+| 模型 | 全量微调配置 | LoRA 微调配置 | 启动脚本 |
+|------|------------|-------------|---------|
+| Hy-MT2-1.8B（Dense） | `hy_dense_1_8b_full_sft.yaml` | `hy_dense_1_8b_lora_sft.yaml` | `bash train_lf_dense.sh` |
+| Hy-MT2-7B（Dense） | `hy_dense_7b_full_sft.yaml` | `hy_dense_7b_lora_sft.yaml` | `YAML_FILE=hy_dense_7b_full_sft.yaml bash train_lf_dense.sh` |
+| Hy-MT2-30B-A3B（MoE） | `hy_v3_full_sft.yaml` | `hy_v3_lora_sft.yaml` | `bash train_lf.sh` |
+
+> **提示**：Dense 模型的启动脚本 `train_lf_dense.sh` 默认使用 `hy_dense_1_8b_full_sft.yaml`，可通过 `YAML_FILE` 环境变量指定其他配置文件。
+
+脚本中的关键参数如下：
+
+**模型相关：**
+
+- `model_name_or_path`: Hy-MT HF 格式预训练模型权重路径
+- `trust_remote_code`: 是否信任远程代码, Hy-MT 需要设置为 `true`
+
+**训练方法：**
+
+- `stage`: 训练阶段, 当前为 `sft`(监督微调)
+- `finetuning_type`: 微调类型, 可选 `full`(全量微调) 或 `lora`(LoRA 微调)
+- `deepspeed`: DeepSpeed 配置文件路径, 全量微调推荐 `ds_zero3_offload.json`, LoRA 微调推荐 `ds_zero2_offload_lora.json`
+
+**LoRA 参数(仅 LoRA 微调时生效)：**
+
+- `lora_rank`: LoRA 秩, 默认 `64`
+- `lora_alpha`: LoRA alpha 系数, 默认 `128`
+- `lora_dropout`: LoRA dropout 比率, 默认 `0.05`
+- `lora_target`: LoRA 应用的目标模块, 默认为 `q_proj,k_proj,v_proj,o_proj`
+
+**数据集：**
+
+- `dataset_dir`: 数据集目录路径
+- `dataset`: 数据集名称, 需要在 `dataset_dir` 下的 `dataset_info.json` 中注册
+- `template`: 对话模板, Hy-MT2-1.8B 使用 `hy_dense_1_8b`, Hy-MT2-7B 使用 `hy_dense_7b`, Hy-MT2-30B-A3B 使用 `hy_v3`
+- `cutoff_len`: 最大序列长度, 超出部分会被截断; 全量微调可设为 `262144`(262K), LoRA 微调建议设为 `8192` 以节省显存
+- `max_samples`: 每个数据集最多使用的样本数
+- `overwrite_cache`: 是否覆盖已缓存的预处理数据集
+
+**输出：**
+
+- `output_dir`: 输出目录, 日志、TensorBoard 和权重都会存储在此路径下
+- `logging_steps`: 每多少步记录一次日志
+- `save_steps`: 每多少步保存一次 checkpoint
+- `plot_loss`: 是否绘制训练 loss 曲线
+- `overwrite_output_dir`: 是否覆盖已有的输出目录
+- `save_only_model`: 是否只保存模型权重(不保存优化器状态等)
+- `report_to`: 日志上报工具, 可选 `none`, `wandb`, `tensorboard`, `swanlab`, `mlflow`
+
+**训练超参数：**
+
+- `per_device_train_batch_size`: 每张卡上的 batch size
+- `gradient_accumulation_steps`: 梯度累积步数, `per_device_train_batch_size * gradient_accumulation_steps * dp_size` 为 global batch size
+- `learning_rate`: 最大学习率, 全量微调推荐 `1.0e-5`, LoRA 微调推荐 `2.0e-4`
+- `num_train_epochs`: 训练轮数
+- `lr_scheduler_type`: 学习率调度器类型, 推荐使用 `cosine_with_min_lr`
+- `lr_scheduler_kwargs.min_lr_rate`: 最小学习率与最大学习率的比值, 例如 `0.1` 表示最小学习率为最大学习率的 10%
+- `warmup_ratio`: 预热阶段占总训练步数的比例
+- `bf16`: 是否使用 BFloat16 混合精度训练
+- `gradient_checkpointing`: 是否开启梯度重计算以节省显存
+- `ddp_timeout`: 分布式训练超时时间(毫秒)
+- `flash_attn`: 注意力实现方式, 推荐 `fa2`(FlashAttention-2), 也可选 `sdpa`; 使用 `fa2` 需要安装 flash-attn 包
+- `resume_from_checkpoint`: 从指定 checkpoint 路径恢复训练, 设为 `null` 表示从头开始训练
+
+##### 启动训练
+
+如需多机训练，请先完成 [配置机器间免密 ssh 登录](#配置机器间免密-ssh-登录多机训练) 中的配置（单机训练可跳过此步骤）。
+
+修改对应启动脚本中开头的以下配置：
+
+```shell
+export HOST_GPU_NUM=8
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+export IP_LIST=${IP_LIST:-"127.0.0.1"}
+```
+
+注意：如果`IP_LIST`环境变量未设置，则将`IP_LIST`替换为IP列表！格式为：
+```
+如果只有一个IP：
+IP_LIST=${ip_1}
+
+如果有多个IP：
+IP_LIST=${ip_1},${ip_2}
+
+```
+
+请将`${ip_1}`和`${ip_2}`替换为真实的IP地址。
+
+然后，在每一台机器上，在`train/llama_factory_support/`目录下执行对应的启动脚本即可。例如：
+
+```sh
+# Dense 1.8B 全量微调
+bash train_lf_dense.sh
+
+# Dense 7B LoRA 微调
+YAML_FILE=hy_dense_7b_lora_sft.yaml bash train_lf_dense.sh
+
+# MoE 30B-A3B 全量微调
+bash train_lf.sh
+```

train/data/example_data.jsonl

@@ -0,0 +1,100 @@
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n实验结果证明了假设的正确性。"}, {"role": "assistant", "content": "The experimental results demonstrate the correctness of the hypothesis."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这篇论文提出了一个新颖的方法。"}, {"role": "assistant", "content": "This paper proposes a novel method."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThis paper proposes a novel method."}, {"role": "assistant", "content": "这篇论文提出了一个新颖的方法。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n谢谢你的帮助，我非常感激。"}, {"role": "assistant", "content": "Thank you for your help, I really appreciate it."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nPlease process this request as soon as possible."}, {"role": "assistant", "content": "请尽快处理这个请求。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nHello! The weather is really nice today."}, {"role": "assistant", "content": "你好！今天天气真不错。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n你最近在忙什么？"}, {"role": "assistant", "content": "What have you been up to lately?"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe global climate change conference was held this week."}, {"role": "assistant", "content": "全球气候变化会议在本周举行。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n请尽快处理这个请求。"}, {"role": "assistant", "content": "Please process this request as soon as possible."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nHow do I get from the airport to the city center?"}, {"role": "assistant", "content": "从机场到市中心怎么走？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe leaders of the two countries held talks and discussed bilateral relations."}, {"role": "assistant", "content": "两国领导人举行了会谈，讨论了双边关系。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n请查收附件中的合同文件。"}, {"role": "assistant", "content": "Please find the contract document attached."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nI have an important meeting tomorrow."}, {"role": "assistant", "content": "我明天有个重要的会议。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nDo you prefer Chinese food or Western food?"}, {"role": "assistant", "content": "你喜欢吃中餐还是西餐？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nResearch has found that regular exercise helps improve mental health."}, {"role": "assistant", "content": "研究发现，定期锻炼有助于改善心理健康。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n学习一门新语言需要时间和耐心。"}, {"role": "assistant", "content": "Learning a new language requires time and patience."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这种药物的副作用是什么？"}, {"role": "assistant", "content": "What are the side effects of this medication?"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe experimental results demonstrate the correctness of the hypothesis."}, {"role": "assistant", "content": "实验结果证明了假设的正确性。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWhat is the landmark building of this city?"}, {"role": "assistant", "content": "这个城市的标志性建筑是什么？"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个API接口需要添加身份验证。"}, {"role": "assistant", "content": "This API endpoint needs to add authentication."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n门票多少钱一张？"}, {"role": "assistant", "content": "How much is one ticket?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n该公司发布了最新款的智能手机。"}, {"role": "assistant", "content": "The company released its latest smartphone model."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这里的当地特色美食是什么？"}, {"role": "assistant", "content": "What are the local specialty foods here?"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nIs the library also open on weekends?"}, {"role": "assistant", "content": "图书馆在周末也开放吗？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nHow much is one ticket?"}, {"role": "assistant", "content": "门票多少钱一张？"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我需要预约医生。"}, {"role": "assistant", "content": "I need to make an appointment with a doctor."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThis program runs too slowly."}, {"role": "assistant", "content": "这个程序的运行速度太慢了。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个城市的标志性建筑是什么？"}, {"role": "assistant", "content": "What is the landmark building of this city?"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThis API endpoint needs to add authentication."}, {"role": "assistant", "content": "这个API接口需要添加身份验证。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个程序的运行速度太慢了。"}, {"role": "assistant", "content": "This program runs too slowly."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n周末你有什么计划吗？"}, {"role": "assistant", "content": "Do you have any plans for the weekend?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n你应该多休息，不要太劳累。"}, {"role": "assistant", "content": "You should rest more and not overwork yourself."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThank you for your help, I really appreciate it."}, {"role": "assistant", "content": "谢谢你的帮助，我非常感激。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nRegarding the previous order, I would like to confirm the delivery time."}, {"role": "assistant", "content": "关于上次的订单，我想确认一下发货时间。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe new expressway will open to traffic next month."}, {"role": "assistant", "content": "新的高速公路将于下个月通车。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个理论是由哪位科学家提出的？"}, {"role": "assistant", "content": "Which scientist proposed this theory?"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWhat are the side effects of this medication?"}, {"role": "assistant", "content": "这种药物的副作用是什么？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nI think this movie is absolutely wonderful."}, {"role": "assistant", "content": "我觉得这部电影非常精彩。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe database server has encountered a failure."}, {"role": "assistant", "content": "数据库服务器出现了故障。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n请将报价单发送给我。"}, {"role": "assistant", "content": "Please send me the quotation."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n请确保所有代码都通过了测试。"}, {"role": "assistant", "content": "Please ensure that all code has passed the tests."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n两国领导人举行了会谈，讨论了双边关系。"}, {"role": "assistant", "content": "The leaders of the two countries held talks and discussed bilateral relations."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我们需要优化算法的性能。"}, {"role": "assistant", "content": "We need to optimize the performance of the algorithm."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个框架支持多种编程语言。"}, {"role": "assistant", "content": "This framework supports multiple programming languages."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n政府宣布了一系列新的经济政策。"}, {"role": "assistant", "content": "The government announced a series of new economic policies."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nOur products have passed international quality certification."}, {"role": "assistant", "content": "我们的产品通过了国际质量认证。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWe will have a meeting at 10 AM next Monday to discuss this project."}, {"role": "assistant", "content": "我们下周一上午10点开会讨论这个项目。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nPlease find the contract document attached."}, {"role": "assistant", "content": "请查收附件中的合同文件。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个问题我不太确定，让我查一下。"}, {"role": "assistant", "content": "I'm not quite sure about this issue, let me check."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我们需要修复这个安全漏洞。"}, {"role": "assistant", "content": "We need to fix this security vulnerability."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nI need to make an appointment with a doctor."}, {"role": "assistant", "content": "我需要预约医生。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n图书馆在周末也开放吗？"}, {"role": "assistant", "content": "Is the library also open on weekends?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我觉得这部电影非常精彩。"}, {"role": "assistant", "content": "I think this movie is absolutely wonderful."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWhat have you been up to lately?"}, {"role": "assistant", "content": "你最近在忙什么？"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我们的产品通过了国际质量认证。"}, {"role": "assistant", "content": "Our products have passed international quality certification."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n系统维护将在今晚凌晨2点进行。"}, {"role": "assistant", "content": "System maintenance will be performed at 2 AM tonight."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nDo you have any plans for the weekend?"}, {"role": "assistant", "content": "周末你有什么计划吗？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe government announced a series of new economic policies."}, {"role": "assistant", "content": "政府宣布了一系列新的经济政策。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n研究发现，定期锻炼有助于改善心理健康。"}, {"role": "assistant", "content": "Research has found that regular exercise helps improve mental health."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nPlease send me the quotation."}, {"role": "assistant", "content": "请将报价单发送给我。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n新的高速公路将于下个月通车。"}, {"role": "assistant", "content": "The new expressway will open to traffic next month."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我明天有个重要的会议。"}, {"role": "assistant", "content": "I have an important meeting tomorrow."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n你喜欢吃中餐还是西餐？"}, {"role": "assistant", "content": "Do you prefer Chinese food or Western food?"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nCould you provide a detailed technical proposal?"}, {"role": "assistant", "content": "能否提供一个详细的技术方案？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWhat are the local specialty foods here?"}, {"role": "assistant", "content": "这里的当地特色美食是什么？"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n科学家在量子计算领域取得了重大突破。"}, {"role": "assistant", "content": "Scientists have made a major breakthrough in the field of quantum computing."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nAre there any good restaurants you would recommend nearby?"}, {"role": "assistant", "content": "这附近有什么好吃的餐厅推荐吗？"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n请在下周五之前提交你的研究报告。"}, {"role": "assistant", "content": "Please submit your research report by next Friday."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThis framework supports multiple programming languages."}, {"role": "assistant", "content": "这个框架支持多种编程语言。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n全球气候变化会议在本周举行。"}, {"role": "assistant", "content": "The global climate change conference was held this week."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n你好！今天天气真不错。"}, {"role": "assistant", "content": "Hello! The weather is really nice today."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nLearning a new language requires time and patience."}, {"role": "assistant", "content": "学习一门新语言需要时间和耐心。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nI'm not quite sure about this issue, let me check."}, {"role": "assistant", "content": "这个问题我不太确定，让我查一下。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nPlease submit your research report by next Friday."}, {"role": "assistant", "content": "请在下周五之前提交你的研究报告。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nCould you do me a favor?"}, {"role": "assistant", "content": "你能帮我一个忙吗？"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWe need to optimize the performance of the algorithm."}, {"role": "assistant", "content": "我们需要优化算法的性能。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n人工智能技术正在改变各行各业。"}, {"role": "assistant", "content": "Artificial intelligence technology is transforming various industries."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThe company released its latest smartphone model."}, {"role": "assistant", "content": "该公司发布了最新款的智能手机。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWe need to fix this security vulnerability."}, {"role": "assistant", "content": "我们需要修复这个安全漏洞。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nI would like to book a double room for two nights."}, {"role": "assistant", "content": "我想预订一间双人房，住两晚。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThank you for your cooperation, and we look forward to more opportunities in the future."}, {"role": "assistant", "content": "感谢贵公司的合作，期待未来更多机会。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这附近有什么好吃的餐厅推荐吗？"}, {"role": "assistant", "content": "Are there any good restaurants you would recommend nearby?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我想预订一间双人房，住两晚。"}, {"role": "assistant", "content": "I would like to book a double room for two nights."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nSystem maintenance will be performed at 2 AM tonight."}, {"role": "assistant", "content": "系统维护将在今晚凌晨2点进行。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nYou should rest more and not overwork yourself."}, {"role": "assistant", "content": "你应该多休息，不要太劳累。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n这个地方真美，我们应该经常来。"}, {"role": "assistant", "content": "This place is so beautiful, we should come here more often."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nThis place is so beautiful, we should come here more often."}, {"role": "assistant", "content": "这个地方真美，我们应该经常来。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nArtificial intelligence technology is transforming various industries."}, {"role": "assistant", "content": "人工智能技术正在改变各行各业。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我们下周一上午10点开会讨论这个项目。"}, {"role": "assistant", "content": "We will have a meeting at 10 AM next Monday to discuss this project."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nScientists have made a major breakthrough in the field of quantum computing."}, {"role": "assistant", "content": "科学家在量子计算领域取得了重大突破。"}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nPlease ensure that all code has passed the tests."}, {"role": "assistant", "content": "请确保所有代码都通过了测试。"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n关于上次的订单，我想确认���下发货时间。"}, {"role": "assistant", "content": "Regarding the previous order, I would like to confirm the delivery time."}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n能否提供一个详细的技术方案？"}, {"role": "assistant", "content": "Could you provide a detailed technical proposal?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n从机场到市中心怎么走？"}, {"role": "assistant", "content": "How do I get from the airport to the city center?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n我需要办理签证才能去那个国家吗？"}, {"role": "assistant", "content": "Do I need a visa to go to that country?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n数据库服务器出现了故障。"}, {"role": "assistant", "content": "The database server has encountered a failure."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nWhich scientist proposed this theory?"}, {"role": "assistant", "content": "这个理论是由哪位科学家提出的？"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n你能帮我一个忙吗？"}, {"role": "assistant", "content": "Could you do me a favor?"}]}
+{"messages": [{"role": "user", "content": "将以下中文翻译为英文，只输出翻译结果，不要额外解释：\n\n感谢贵公司的合作，期待未来更多机会。"}, {"role": "assistant", "content": "Thank you for your cooperation, and we look forward to more opportunities in the future."}]}
+{"messages": [{"role": "user", "content": "Translate the following English text into Chinese, output only the translation result without additional explanation:\n\nDo I need a visa to go to that country?"}, {"role": "assistant", "content": "我需要办理签证才能去那个国家吗？"}]}

train/deepspeed_support/ds_zero2_no_offload.json

@@ -0,0 +1,20 @@
+{
+    "fp16": {
+        "enabled": false
+    },
+    "zero_optimization": {
+        "stage": 2,
+        "allgather_partitions": true,
+        "allgather_bucket_size": 1e8,
+        "overlap_comm": true,
+        "reduce_scatter": true,
+        "reduce_bucket_size": 1e8,
+        "contiguous_gradients": true
+    },
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/deepspeed_support/ds_zero2_offload.json

@@ -0,0 +1,27 @@
+{
+    "fp16": {
+        "enabled": false
+    },
+    "bf16": {
+        "enabled": true
+    },
+    "zero_optimization": {
+        "stage": 2,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "allgather_partitions": true,
+        "allgather_bucket_size": 1e8,
+        "overlap_comm": true,
+        "reduce_scatter": true,
+        "reduce_bucket_size": 1e8,
+        "contiguous_gradients": true
+    },
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/deepspeed_support/ds_zero3_no_offload.json

@@ -0,0 +1,41 @@
+{
+    "fp16": {
+        "enabled": "auto",
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    "bf16": {
+        "enabled": "auto"
+    },
+
+    "zero_optimization": {
+        "stage": 3,
+        "offload_optimizer": {
+            "device": "none",
+            "pin_memory": true
+        },
+        "offload_param": {
+            "device": "none",
+            "pin_memory": true
+        },
+        "overlap_comm": true,
+        "contiguous_gradients": true,
+        "sub_group_size": 1e9,
+        "reduce_bucket_size": "auto",
+        "stage3_prefetch_bucket_size": "auto",
+        "stage3_param_persistence_threshold": "auto",
+        "stage3_max_live_parameters": 1e9,
+        "stage3_max_reuse_distance": 1e9,
+        "stage3_gather_16bit_weights_on_model_save": true
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/deepspeed_support/ds_zero3_offload.json

@@ -0,0 +1,41 @@
+{
+    "fp16": {
+        "enabled": "auto",
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    "bf16": {
+        "enabled": "auto"
+    },
+
+    "zero_optimization": {
+        "stage": 3,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "offload_param": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "overlap_comm": true,
+        "contiguous_gradients": true,
+        "sub_group_size": 1e9,
+        "reduce_bucket_size": "auto",
+        "stage3_prefetch_bucket_size": "auto",
+        "stage3_param_persistence_threshold": "auto",
+        "stage3_max_live_parameters": 1e9,
+        "stage3_max_reuse_distance": 1e9,
+        "stage3_gather_16bit_weights_on_model_save": false
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/deepspeed_support/ds_zero3_offload_no_auto.json

@@ -0,0 +1,36 @@
+{
+    "fp16": {
+        "enabled": false,
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    "bf16": {
+        "enabled": true
+    },
+
+    "zero_optimization": {
+        "stage": 3,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "overlap_comm": true,
+        "contiguous_gradients": true,
+        "sub_group_size": 1e9,
+        "reduce_bucket_size": 1e8,
+        "stage3_prefetch_bucket_size": 1e8,
+        "stage3_param_persistence_threshold": 0,
+        "stage3_max_live_parameters": 1e8,
+        "stage3_max_reuse_distance": 1e8,
+        "stage3_gather_16bit_weights_on_model_save": true
+    },
+
+    "gradient_accumulation_steps": 1,
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_micro_batch_size_per_gpu": 1,
+    "wall_clock_breakdown": false
+}

train/deepspeed_support/merge_lora_weight.py

@@ -0,0 +1,54 @@
+# 导入所需的库
+from transformers import AutoModelForCausalLM  # 用于加载预训练的语言模型
+from peft import LoraConfig, get_peft_model, PeftModel  # 用于处理LoRA权重
+import argparse  # 用于解析命令行参数
+import shutil  # 用于文件操作，如复制
+import os  # 用于文件路径操作
+import torch  # 用于深度学习操作
+
+def main():
+    # 创建参数解析器
+    parser = argparse.ArgumentParser()
+    # 添加命令行参数
+    parser.add_argument("--base_model_path", type=str, required=True, 
+                        help="Path to pretrained model or model identifier from huggingface.co/models")
+    parser.add_argument("--adapter_model_path", type=str, required=True, help="Path to adapter model")
+    parser.add_argument("--output_path", type=str, required=True, help="Path to save the output model")
+    parser.add_argument("--save_dtype", type=str, choices=['bf16', 'fp32', 'fp16'], 
+                        default='fp32', help="In which dtype to save, fp32, bf16 or fp16.")
+    # 解析命令行参数
+    args = parser.parse_args()
+
+    name2dtype = {'bf16': torch.bfloat16, 'fp32': torch.float32, 'fp16': torch.float16}
+    # 加载基座模型
+    model = AutoModelForCausalLM.from_pretrained(
+        args.base_model_path, device_map='cpu', 
+        trust_remote_code=True, torch_dtype=name2dtype[args.save_dtype]
+    )
+    # 在基座模型的基础上加载 adapter 权重
+    model = PeftModel.from_pretrained(model, args.adapter_model_path, trust_remote_code=True)
+    # 融合模型和 adapter
+    model = model.merge_and_unload()
+    # 保存融合后的模型权重
+    model.save_pretrained(args.output_path, safe_serialization=False)
+
+    # Copy tokenizer, config and other non-weight files from base model
+    # Skip model weight files (.safetensors, .bin, .pt) and index files
+    _SKIP_SUFFIXES = ('.safetensors', '.bin', '.pt', '.pth')
+    _SKIP_NAMES = {'model.safetensors.index.json', 'pytorch_model.bin.index.json'}
+
+    for fname in os.listdir(args.base_model_path):
+        src = os.path.join(args.base_model_path, fname)
+        if not os.path.isfile(src):
+            continue
+        if fname in _SKIP_NAMES or fname.endswith(_SKIP_SUFFIXES):
+            continue
+        dst = os.path.join(args.output_path, fname)
+        if not os.path.exists(dst):
+            shutil.copy(src, dst)
+            print(f'Copied {fname}')
+
+    print(f'Merged model weight is saved to {args.output_path}')
+    
+if __name__ == "__main__":
+    main()

train/deepspeed_support/merge_lora_weight.sh

@@ -0,0 +1,6 @@
+SCRIPT_DIR=$(dirname "$0")
+python3 ${SCRIPT_DIR}/merge_lora_weight.py \
+    --base_model_path /xxx/hy_l_train/checkpoint-200 \
+    --adapter_model_path /xxx/runs/hy_l_lora_train/checkpoint-200 \
+    --output_path /xxx/ckpts/merged_hy_lora_weight \
+    --save_dtype bf16

train/deepspeed_support/train.py

@@ -0,0 +1,564 @@
+# Copyright 2024 Tencent Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import os
+import re
+import sys
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import json
+import torch
+import shutil
+import logging
+from dataclasses import dataclass, field
+import deepspeed
+from typing import Optional, Dict
+
+import transformers
+from torch.utils.data import Dataset
+from transformers import Trainer, TrainerCallback
+from peft import LoraConfig, get_peft_model, PeftModel
+from transformers.trainer_utils import PREFIX_CHECKPOINT_DIR
+from transformers.modeling_utils import unwrap_model
+
+
+def print_args(args, name='arguments'):
+    """Print arguments."""
+    if torch.distributed.get_rank() == 0:
+        print(f'------------------------ {name} ------------------------', flush=True)
+        str_list = []
+        for arg in vars(args):
+            dots = '.' * (48 - len(arg))
+            str_list.append('  {} {} {}'.format(arg, dots, getattr(args, arg)))
+        for arg in sorted(str_list, key=lambda x: x.lower()):
+            print(arg, flush=True)
+        print(f'-------------------- end of {name} ---------------------', flush=True)
+
+
+@dataclass
+class ModelArguments:
+    use_flash_attn: bool = field(
+        default=False, 
+        metadata={"help": "Enable FlashAttention-2 for faster training."}
+    )
+    use_lora: bool = field(default=False, metadata={"help": "Enable Lora for faster training."})
+    hidden_size: int = field(default=2048, metadata={"help": "The hidden size of the model."})
+    num_layers: int = field(default=24, metadata={"help": "The number of layers of the model."})
+    num_attention_heads: int = field(default=16, metadata={"help": "The number of attention heads of the model."})
+    intermediate_size: int = field(default=8192, metadata={"help": "The intermediate size of the model."})
+    max_position_embeddings: int = field(
+        default=2048, 
+        metadata={"help": "The maximum sequence length that this model might ever be used with."}
+    )
+    vocab_size: int = field(default=50257, metadata={"help": "The vocabulary size of the model."})
+    type_vocab_size: int = field(default=1, metadata={"help": "The vocabulary size of the model."})
+    layer_norm_eps: float = field(
+        default=1e-5, 
+        metadata={"help": "The epsilon used by the layer normalization layers of the model."}
+    )
+    moe_topk: int = field(default=4, metadata={"help": "The topk for MOE."})
+    num_experts: int = field(default=8, metadata={"help": "The number of experts for MOE."})
+    num_key_value_heads: int = field(default=16, metadata={"help": "The number of key-value heads in GQA."})
+    moe_intermediate_size: int = field(default=1536, metadata={"help": "The intermediate size of each MoE expert."})
+    use_mixed_mlp_moe: bool = field(
+        default=False, 
+        metadata={"help": "Whether to use mixed MoE with shared expert."}
+    )
+    num_shared_expert: int = field(default=1, metadata={"help": "Number of shared experts."})
+    use_qk_norm: bool = field(default=False, metadata={"help": "Whether to use qk norm."})
+    moe_layer_num_skipped: int = field(default=1, metadata={"help": "Number of initial dense layers before MoE layers."})
+    tie_word_embeddings: bool = field(
+        default=True, 
+        metadata={"help": "Whether to tie the word embeddings of the encoder and the decoder."}
+    )
+    lora_rank: int = field(default=64, metadata={"help": "The rank of lora."})
+    lora_alpha: int = field(default=8, metadata={"help": "Lora alpha"})
+    lora_dropout: float = field(default=0.0, metadata={"help": "Lora dropout"})
+    train_attention_params_only: bool = field(default=False, metadata={
+        "help": "Whether to train attention parameters only."}
+    )
+
+
+@dataclass
+class DataArguments:
+    train_data_file: str = field(default=None, metadata={"help": "Path to the training data."})
+    max_seq_length: int = field(
+        default=2048, 
+        metadata={"help": "The max sequence length of the model inputs after tokenization."}
+    )
+    complex_data: Optional[str] = field(default=None)
+    use_dummy_data: bool = field(default=False, metadata={"help": "Use dummy data."})
+
+
+@dataclass
+class TrainingArguments(transformers.TrainingArguments):
+    cache_dir: Optional[str] = field(default=None)
+    optim: str = field(default="adamw_torch")
+    model_max_length: int = field(
+        default=2048,
+        metadata={"help": "Maximum sequence length. Sequences will be right padded (and possibly truncated)."},
+    )
+    tokenizer_name_or_path: Optional[str] = field(default=None)
+    model_name_or_path: Optional[str] = field(default=None)
+    min_lr: float = field(
+        default=0.01, 
+        metadata={"help": "The final learning rate at the end of the decay will be learning_rate * min_lr"}
+    )
+
+
+IGNORE_INDEX = -100
+
+
+class DummyDataset(Dataset):
+    def __init__(self, tokenizer, max_seq_length=512, length=1000):
+        self.tokenizer = tokenizer
+        self.max_seq_length = max_seq_length
+        self.length = length
+    
+    def __len__(self):
+        return self.length
+    
+    def __getitem__(self, index):
+        tokens = torch.randint(0, self.tokenizer.vocab_size, (self.max_seq_length, ))
+        return {'input_ids': tokens, 'labels': tokens}
+
+
+class SFTDataset(Dataset):
+    def __init__(self, data_file, tokenizer, max_seq_length = 2048, prompt_format = 'mplus'):
+        self.tokenizer = tokenizer
+        self.prompt_format = prompt_format
+        self.max_seq_length = max_seq_length
+
+        self.data_list = self.load_data(data_file)
+
+    def __len__(self):
+        return len(self.data_list)
+
+    def load_data(self, data_file):
+        logging.info('Loading data: {}'.format(data_file))
+        with open(data_file, 'r', encoding='utf8') as f:
+            data_list = f.readlines()
+        logging.info("there are {} data in dataset".format(len(data_list)))
+        return data_list
+
+    def encode_data(self, data_dict):
+        model_inputs = {}
+        reasoning_effort = data_dict.get('reasoning_effort', None)
+        if reasoning_effort is None:
+            reasoning_effort = 'no_think'
+        template_output = self.tokenizer.apply_chat_template(data_dict['messages'], tokenize=True, return_dict=False, is_training=True, reasoning_effort=reasoning_effort)
+        if isinstance(template_output, list) and len(template_output) > 0 and isinstance(template_output[0], list):
+            template_output = template_output[0]
+        message_tokens = torch.tensor(template_output, dtype=torch.long)
+
+        # Use new HunYuan tokenizer special tokens
+        assistant_token_id = self.tokenizer.convert_tokens_to_ids('<｜hy_Assistant｜>')
+        eos_token_id = self.tokenizer.convert_tokens_to_ids(self.tokenizer.eos_token)
+        pad_token_id = self.tokenizer.pad_token_id
+
+        # Find assistant reply boundaries: starts at <｜hy_Assistant｜>, ends at eos_token
+        loss_token_begins = (message_tokens == assistant_token_id).nonzero(as_tuple=True)[0].tolist()
+        loss_token_ends = (message_tokens == eos_token_id).nonzero(as_tuple=True)[0].tolist()
+        message_labels = torch.tensor([IGNORE_INDEX] * message_tokens.shape[0])
+        for begin_idx, end_idx in zip(loss_token_begins, loss_token_ends):
+            # Compute loss from the token after <｜hy_Assistant｜> to eos_token (inclusive)
+            message_labels[begin_idx + 1:end_idx + 1] = message_tokens[begin_idx + 1:end_idx + 1]
+        input_ids = message_tokens.to(torch.long)
+        labels = message_labels.to(torch.long)
+
+        input_ids = input_ids[:self.max_seq_length]
+        labels = labels[:self.max_seq_length]
+        attention_mask = [1 if val != pad_token_id else 0 for val in input_ids]
+        model_inputs["input_ids"] = input_ids
+        model_inputs["attention_mask"] = torch.tensor(attention_mask, dtype=torch.bool)
+        model_inputs["labels"] = labels
+
+        return model_inputs
+
+    def __getitem__(self, index):
+        data = self.data_list[index]
+        data = json.loads(data)
+        model_inputs = self.encode_data(data)
+
+        return model_inputs
+
+
+@dataclass
+class DataCollatorForSupervisedDataset(object):
+    """Collate examples for supervised fine-tuning."""
+
+    tokenizer: transformers.PreTrainedTokenizer
+
+    def __call__(self, instances):
+        input_ids = [instance['input_ids'] for instance in instances]
+        labels = [instance['labels'] for instance in instances]
+        pad_token_id = self.tokenizer.pad_token_id
+        input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=True, padding_value=pad_token_id)
+        labels = torch.nn.utils.rnn.pad_sequence(labels, batch_first=True, padding_value=IGNORE_INDEX)
+        return dict(
+            input_ids=input_ids,
+            labels=labels,
+            attention_mask=input_ids.ne(pad_token_id),
+        )
+
+
+def make_supervised_data_module(tokenizer, data_args) -> Dict:
+    """Make dataset and collator for supervised fine-tuning."""
+    if data_args.use_dummy_data:
+        train_dataset = DummyDataset(tokenizer, data_args.max_seq_length)
+    else:
+        train_dataset = SFTDataset(
+            tokenizer=tokenizer, 
+            data_file=data_args.train_data_file, 
+            max_seq_length=data_args.max_seq_length
+        )
+    data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
+    return dict(train_dataset=train_dataset, eval_dataset=None, data_collator=data_collator)
+
+
+# for full model training, change the config.json, copy the model and configuration to support Auto load
+class CustomSaveCallback(TrainerCallback):
+    def on_save(self, args, state, control, **kwargs):
+        if torch.distributed.get_rank() == 0:
+            output_dir = os.path.join(args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{state.global_step}")
+
+            # Copy tokenizer files to checkpoint directory
+            tokenizer_files = [
+                'generation_config.json',
+                'hy.tiktoken',
+                'tokenizer_config.json',
+                'tokenization_hy.py',
+                'tokenizer.json',
+                'special_tokens_map.json',
+                'chat_template.jinja',
+            ]
+            for fname in tokenizer_files:
+                src = os.path.join(args.tokenizer_name_or_path, fname)
+                if os.path.isfile(src):
+                    shutil.copy(src, os.path.join(output_dir, fname))
+
+        return control
+
+
+def train():
+    parser = transformers.HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
+    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
+    print_args(model_args, 'model arguments')
+    print_args(data_args, 'data arguments')
+    print_args(training_args, 'training arguments')
+
+    tokenizer = transformers.AutoTokenizer.from_pretrained(
+        training_args.tokenizer_name_or_path,
+        trust_remote_code = True
+    )
+
+    init_kwargs = {}
+    if model_args.use_flash_attn:
+        init_kwargs["attn_implementation"] = "flash_attention_2"
+        # Workaround: transformers >= 5.x uses importlib.metadata.packages_distributions()
+        # to verify flash-attn package name, which fails when the package is installed under
+        # a custom distribution name (e.g. ptm-flash-attn). Patch the check to skip it.
+        try:
+            from transformers.modeling_flash_attention_utils import FLASH_ATTENTION_COMPATIBILITY_MATRIX
+            _orig_pkg_check = FLASH_ATTENTION_COMPATIBILITY_MATRIX[2]["pkg_availability_check"]
+            FLASH_ATTENTION_COMPATIBILITY_MATRIX[2]["pkg_availability_check"] = lambda *a, **kw: True
+            print("[Patch] Bypassed flash_attn package distribution name check for FA2.")
+        except Exception as e:
+            print(f"[Patch] Could not patch FA2 pkg check (non-fatal): {e}")
+    if training_args.bf16:
+        init_kwargs["dtype"] = torch.bfloat16
+    elif training_args.fp16:
+        init_kwargs["dtype"] = torch.float16
+
+    # Check if model weights exist (not just the directory)
+    _has_weights = (
+        training_args.model_name_or_path is not None
+        and os.path.isdir(training_args.model_name_or_path)
+        and any(
+            os.path.isfile(os.path.join(training_args.model_name_or_path, f))
+            for f in ("model.safetensors", "pytorch_model.bin", "model.safetensors.index.json", "pytorch_model.bin.index.json")
+        )
+    )
+
+    # -----------------------------------------------------------------------
+    # Fix: Rename checkpoint keys so that old-style weight names (e.g.
+    # self_attn.q_norm) are mapped to the current model attribute names
+    # (e.g. self_attn.query_layernorm).  The model's
+    # _fix_state_dict_key_on_load hook is NOT invoked on the DeepSpeed
+    # ZeRO-3 loading path, so we monkey-patch the ZeRO-3 loader instead.
+    # -----------------------------------------------------------------------
+    _CKPT_KEY_RENAMES = [
+        ("mlp.gate.wg.", "mlp.router.gate."),
+    ]
+
+    from transformers.integrations.deepspeed import (
+        _load_state_dict_into_zero3_model as _orig_load_zero3,
+    )
+    import transformers.integrations.deepspeed as _ds_mod
+    import transformers.modeling_utils as _mu_mod
+
+    def _patched_load_zero3(model_to_load, state_dict, load_config=None):
+        new_sd = {}
+        for k, v in state_dict.items():
+            new_k = k
+            for old_sub, new_sub in _CKPT_KEY_RENAMES:
+                if old_sub in new_k:
+                    new_k = new_k.replace(old_sub, new_sub)
+                    break
+            new_sd[new_k] = v
+
+        # Call original ZeRO-3 loader for parameters
+        result = _orig_load_zero3(model_to_load, new_sd, load_config)
+
+        # -------------------------------------------------------------------
+        # Patch: Manually load buffers (e.g. e_score_correction_bias).
+        # ZeRO-3's loader only handles named_parameters, not named_buffers.
+        # -------------------------------------------------------------------
+        buffers_loaded = 0
+        for name, buf in model_to_load.named_buffers():
+            if name in new_sd:
+                src_tensor = new_sd[name]
+                if isinstance(src_tensor, torch.Tensor):
+                    buf.data.copy_(src_tensor.to(buf.dtype))
+                    buffers_loaded += 1
+                    # Remove from unexpected keys if tracked
+                    if isinstance(result, tuple) and len(result) >= 2:
+                        if isinstance(result[1], set):
+                            result[1].discard(name)
+        if buffers_loaded > 0:
+            print(f"[HYV3 Patch] Manually loaded {buffers_loaded} buffers "
+                  f"(e.g. e_score_correction_bias) into model.")
+
+        return result
+
+    _ds_mod._load_state_dict_into_zero3_model = _patched_load_zero3
+    _mu_mod._load_state_dict_into_zero3_model = _patched_load_zero3
+    # -----------------------------------------------------------------------
+
+    # -------------------------------------------------------------------
+    # Patch: Save-time reverse key rename + 3D -> per-expert unfuse.
+    #
+    # When saving checkpoints, the model state_dict uses 3D fused experts
+    # and new naming.  We reverse both for old checkpoint compatibility:
+    #   - mlp.gate.           -> mlp.router.gate.
+    #   - mlp.e_score_correction_bias -> mlp.expert_bias
+    #   - mlp.shared_experts. -> mlp.shared_mlp.
+    #   - experts.gate_up_proj -> experts.{N}.gate_proj.weight + up_proj
+    #   - experts.down_proj    -> experts.{N}.down_proj.weight
+    # -------------------------------------------------------------------
+    _SAVE_KEY_RENAMES = [
+        ("mlp.gate.", "mlp.router.gate."),
+        ("mlp.e_score_correction_bias", "mlp.expert_bias"),
+        ("mlp.shared_experts.", "mlp.shared_mlp."),
+    ]
+    _FUSED_EXPERT_KEY_RE = re.compile(
+        r"^(.*\.mlp\.experts\.)(gate_up_proj|down_proj)$"
+    )
+
+    def _apply_save_reverse_rename_patch():
+        try:
+            from transformers.models.hy_v3.modeling_hy_v3 import HYV3ForCausalLM
+        except ImportError:
+            try:
+                from transformers.hy_v3.modeling_hy_v3 import HYV3ForCausalLM
+            except ImportError:
+                print("[HYV3 Patch] Could not import HYV3ForCausalLM; "
+                      "save reverse rename patch NOT applied.")
+                return
+
+        _orig_save_pretrained = HYV3ForCausalLM.save_pretrained
+
+        def _patched_save_pretrained(self, *args, **kwargs):
+            state_dict = kwargs.get("state_dict", None)
+            if state_dict is not None:
+                reversed_sd = {}
+                for k, v in state_dict.items():
+                    new_k = k
+                    # Apply simple key renames
+                    for new_sub, old_sub in _SAVE_KEY_RENAMES:
+                        if new_sub in new_k:
+                            new_k = new_k.replace(new_sub, old_sub)
+                            break
+
+                    # Check if this is a fused 3D expert key
+                    m = _FUSED_EXPERT_KEY_RE.match(new_k)
+                    if m:
+                        prefix = m.group(1)  # e.g. "model.layers.1.mlp.experts."
+                        proj_type = m.group(2)  # "gate_up_proj" or "down_proj"
+
+                        if proj_type == "gate_up_proj":
+                            # v shape: [num_experts, 2*intermediate, hidden]
+                            num_experts = v.shape[0]
+                            intermediate = v.shape[1] // 2
+                            for i in range(num_experts):
+                                gate = v[i, :intermediate, :]
+                                up = v[i, intermediate:, :]
+                                reversed_sd[f"{prefix}{i}.gate_proj.weight"] = gate
+                                reversed_sd[f"{prefix}{i}.up_proj.weight"] = up
+                        elif proj_type == "down_proj":
+                            # v shape: [num_experts, hidden, intermediate]
+                            num_experts = v.shape[0]
+                            for i in range(num_experts):
+                                reversed_sd[f"{prefix}{i}.down_proj.weight"] = v[i]
+                    else:
+                        reversed_sd[new_k] = v
+
+                kwargs["state_dict"] = reversed_sd
+                print(f"[HYV3 Patch] Reverse-renamed and unfused "
+                      f"{len(state_dict)} -> {len(reversed_sd)} "
+                      f"state_dict keys for old checkpoint compatibility.")
+            return _orig_save_pretrained(self, *args, **kwargs)
+
+        HYV3ForCausalLM.save_pretrained = _patched_save_pretrained
+        print("[HYV3 Patch] Applied: save-time reverse key rename + "
+              "3D -> per-expert unfuse for old ckpt compatibility.")
+
+    _apply_save_reverse_rename_patch()
+    # -------------------------------------------------------------------
+
+    if _has_weights:
+        print(f"Initializing model from local file: {training_args.model_name_or_path}")
+        model = transformers.AutoModelForCausalLM.from_pretrained(
+            training_args.model_name_or_path,
+            trust_remote_code=True,
+            **init_kwargs
+        )
+    else:
+        from transformers import HYV3Config
+        from transformers import HYV3ForCausalLM
+        print(f"Model weights not found at: {training_args.model_name_or_path}, "
+              f"using random initialized HYV3 model instead.")
+        # Use len(tokenizer) to include added special tokens; tokenizer.vocab_size
+        # may only return the base vocabulary size and miss special tokens whose
+        # IDs exceed that range, causing index-out-of-bounds in the embedding layer.
+        config = HYV3Config(
+            vocab_size=len(tokenizer),
+            hidden_size=model_args.hidden_size,
+            intermediate_size=model_args.intermediate_size,
+            max_position_embeddings=training_args.model_max_length,
+            moe_topk=model_args.moe_topk,
+            num_experts=model_args.num_experts,
+            num_attention_heads=model_args.num_attention_heads,
+            num_key_value_heads=model_args.num_key_value_heads,
+            num_hidden_layers=model_args.num_layers,
+            moe_intermediate_size=model_args.moe_intermediate_size,
+            use_mixed_mlp_moe=model_args.use_mixed_mlp_moe,
+            num_shared_expert=model_args.num_shared_expert,
+            use_qk_norm=model_args.use_qk_norm,
+            moe_layer_num_skipped=model_args.moe_layer_num_skipped,
+            tie_word_embeddings=model_args.tie_word_embeddings,
+        )
+        with deepspeed.zero.Init(dtype=init_kwargs.get("torch_dtype", torch.bfloat16), config_dict_or_path=training_args.deepspeed):
+            model = HYV3ForCausalLM(config)
+    
+    if model_args.train_attention_params_only:
+        for name, param in model.named_parameters():
+            if 'self_attn' not in name:
+                param.requires_grad = False
+
+    if model_args.use_lora:
+        # define Lora configuration
+        lora_config = LoraConfig(
+            r=model_args.lora_rank,
+            lora_alpha=model_args.lora_alpha,
+            lora_dropout=model_args.lora_dropout,
+            target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
+            bias="none",
+            task_type="CAUSAL_LM",
+        )
+        model = get_peft_model(model, lora_config)
+
+    data_module = make_supervised_data_module(tokenizer=tokenizer, data_args=data_args)
+    # Tell Trainer not to attempt DataParallel
+    model.is_parallelizable = True
+    model.model_parallel = True
+
+    training_args.lr_scheduler_kwargs = {
+        'min_lr_rate': training_args.min_lr / training_args.learning_rate,
+    }
+
+    # -----------------------------------------------------------------------
+    # Fix: DeepSpeed ZeRO-3 + gradient checkpointing compatibility.
+    #
+    # PyTorch's torch.utils.checkpoint with use_reentrant=False (the default
+    # in transformers) performs strict metadata checks on recomputed tensors
+    # during backward.  Under ZeRO-3, parameters are all-gathered during the
+    # first forward pass (shape=[full_size]) but may be partitioned back
+    # (shape=[0]) when the checkpoint recomputes, causing a CheckpointError.
+    #
+    # Setting use_reentrant=True avoids this strict metadata check.
+    # -----------------------------------------------------------------------
+    if training_args.gradient_checkpointing and training_args.deepspeed:
+        training_args.gradient_checkpointing_kwargs = {"use_reentrant": True}
+
+    trainer = Trainer(
+        model=model, 
+        processing_class=tokenizer, 
+        args=training_args,
+        callbacks=[CustomSaveCallback],
+        **data_module
+    )
+    model.config.use_cache = False
+
+    # -----------------------------------------------------------------------
+    # Monkey-patch: fix dtype mismatch in DeepSpeed ZeRO-3 linear wrapper.
+    #
+    # By this point the DeepSpeed engine has been initialised by the Trainer
+    # and torch.nn.functional.linear has been replaced with
+    # zero3_linear_wrap.  That wrapper does NOT auto-align input/weight
+    # dtypes before the matmul, causing "expected mat1 and mat2 to have the
+    # same dtype" errors in mixed-precision paths (MoE router gate in fp32
+    # with bf16 weights, expert FFN receiving fp32 routing-weighted input
+    # with bf16 weights, etc.).
+    #
+    # We wrap F.linear HERE (after DeepSpeed init) so that:
+    #   1. We are sure to capture the already-replaced function.
+    #   2. The dtype cast happens *outside* the autograd.Function, so
+    #      gradient-checkpointing recompute sees identical tensor metadata.
+    # -----------------------------------------------------------------------
+    import torch.nn.functional as _F
+    _orig_F_linear = _F.linear
+
+    def _dtype_safe_linear(input, weight, bias=None):
+        if input.dtype != weight.dtype:
+            input = input.to(weight.dtype)
+        return _orig_F_linear(input, weight, bias)
+
+    _F.linear = _dtype_safe_linear
+    # -----------------------------------------------------------------------
+
+    trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
+
+
+if __name__ == "__main__":
+    train()

train/deepspeed_support/train.sh

@@ -0,0 +1,120 @@
+#!/bin/bash
+
+NET_TYPE="high"
+export NCCL_DEBUG=WARN
+export NCCL_P2P_LEVEL=NVL
+export NCCL_IB_TIMEOUT=24
+export NCCL_NVLS_ENABLE=0
+export NCCL_MPI_PROFILE_PRIMS_ENABLE=0
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+export TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=3600
+if [[ "${NET_TYPE}" = "low" ]]; then
+    export NCCL_SOCKET_IFNAME=eth1
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_HCA=mlx5_2:1
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+else
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_IB_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+    export NCCL_SOCKET_IFNAME=bond1
+    export UCX_NET_DEVICES=bond1
+    export NCCL_IB_HCA=mlx5_bond_1,mlx5_bond_5,mlx5_bond_3,mlx5_bond_7,mlx5_bond_4,mlx5_bond_8,mlx5_bond_2,mlx5_bond_6
+    export NCCL_COLLNET_ENABLE=0
+    export SHARP_COLL_ENABLE_SAT=0
+    export NCCL_NET_GDR_LEVEL=2
+    export NCCL_IB_QPS_PER_CONNECTION=4
+    export NCCL_IB_TC=160
+    export NCCL_PXN_DISABLE=1
+fi
+
+export HOST_GPU_NUM=8
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+IP_LIST=${IP_LIST:-"127.0.0.1"}
+
+IFS=',' read -ra IP_ARRAY <<< "$IP_LIST"
+export NODES=${#IP_ARRAY[@]}
+export LOCAL_IP=${IP_ARRAY[0]}
+NODE_IP_LIST=""
+for ip in "${IP_ARRAY[@]}"; do
+    if [ -n "$NODE_IP_LIST" ]; then
+        NODE_IP_LIST="${NODE_IP_LIST},"
+    fi
+    NODE_IP_LIST="${NODE_IP_LIST}${ip}:${HOST_GPU_NUM}"
+done
+export NODE_IP_LIST
+export NODE_NUM=$((${NODES} * ${HOST_GPU_NUM}))
+
+SCRIPT_DIR=$(dirname "$0")
+
+model_path=path_to_model_weight
+tokenizer_path=../models
+train_data_file=example_data.jsonl
+
+# ds_config_file=ds_zero2_no_offload.json
+# ds_config_file=ds_zero3_no_offload.json
+ds_config_file=${SCRIPT_DIR}/ds_zero3_offload_no_auto.json
+
+output_path=/root/hf_train_output_full
+
+mkdir -p ${output_path}
+
+current_time=$(date "+%Y.%m.%d-%H.%M.%S")
+log_file=${output_path}/"log_${current_time}.txt"
+
+echo $NODE_IP_LIST > env.txt 2>&1
+sed "s/:/ slots=/g" env.txt | sed "s/,/\n/g" >  "hostfile"
+sed "s/:.//g" env.txt | sed "s/,/\n/g" >  "pssh.hosts"
+export CHIEF_IP=$LOCAL_IP
+
+if [ ${NODES} -gt 1 ]; then
+    HOST_PATH=hostfile
+    DS_ARGS="--hostfile=${HOST_PATH} --master_addr ${CHIEF_IP}"
+else
+    DS_ARGS=""
+fi
+
+echo "NODES: ${NODES}, LOCAL_IP: ${LOCAL_IP}, NODE_IP_LIST: ${NODE_IP_LIST}"
+
+deepspeed ${DS_ARGS} \
+    ${SCRIPT_DIR}/train.py \
+    --do_train \
+    --model_name_or_path ${model_path} \
+    --tokenizer_name_or_path ${tokenizer_path} \
+    --train_data_file ${train_data_file} \
+    --deepspeed ${ds_config_file} \
+    --output_dir ${output_path} \
+    --per_device_train_batch_size 1 \
+    --gradient_accumulation_steps 1 \
+    --gradient_checkpointing \
+    --lr_scheduler_type cosine_with_min_lr \
+    --logging_steps 1 \
+    --max_steps 50 \
+    --save_steps 50 \
+    --learning_rate 1e-5 \
+    --min_lr 1e-6 \
+    --warmup_ratio 0.01 \
+    --save_strategy steps \
+    --bf16 \
+    --hidden_size 2048 \
+    --intermediate_size 6912 \
+    --model_max_length 262144 \
+    --max_seq_length 8192 \
+    --moe_topk 8 \
+    --num_experts 128 \
+    --moe_intermediate_size 768 \
+    --moe_layer_num_skipped 1 \
+    --num_attention_heads 32 \
+    --num_key_value_heads 4 \
+    --num_layers 48 \
+    --use_mixed_mlp_moe \
+    --num_shared_expert 1 \
+    --use_qk_norm | tee ${log_file}

train/deepspeed_support/train_dense.py

@@ -0,0 +1,360 @@
+# Copyright 2024 Tencent Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Training script for HunYuan Dense models (1.8B, 7B).
+
+This script is adapted from the original finetune.py for dense models,
+with improvements from the new training framework (train.py for MoE models).
+
+Key differences from train.py (MoE version):
+  - No MoE-related patches (router dtype fix, expert key rename, etc.)
+  - Supports model_size parameter to handle different tokenizer formats
+  - 7B model uses different special tokens than 1.8B model
+"""
+
+import os
+import sys
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import json
+import torch
+import shutil
+import logging
+from dataclasses import dataclass, field
+from typing import Optional, Dict, Literal
+
+import transformers
+from torch.utils.data import Dataset
+from transformers import Trainer, TrainerCallback
+from peft import LoraConfig, get_peft_model, PeftModel
+from transformers.trainer_utils import PREFIX_CHECKPOINT_DIR
+from transformers.modeling_utils import unwrap_model
+
+
+def print_args(args, name='arguments'):
+    """Print arguments."""
+    if torch.distributed.get_rank() == 0:
+        print(f'------------------------ {name} ------------------------', flush=True)
+        str_list = []
+        for arg in vars(args):
+            dots = '.' * (48 - len(arg))
+            str_list.append('  {} {} {}'.format(arg, dots, getattr(args, arg)))
+        for arg in sorted(str_list, key=lambda x: x.lower()):
+            print(arg, flush=True)
+        print(f'-------------------- end of {name} ---------------------', flush=True)
+
+
+@dataclass
+class ModelArguments:
+    use_flash_attn: bool = field(
+        default=False,
+        metadata={"help": "Enable FlashAttention-2 for faster training."}
+    )
+    use_lora: bool = field(default=False, metadata={"help": "Enable Lora for faster training."})
+    hidden_size: int = field(default=2048, metadata={"help": "The hidden size of the model."})
+    num_layers: int = field(default=32, metadata={"help": "The number of layers of the model."})
+    num_attention_heads: int = field(default=16, metadata={"help": "The number of attention heads of the model."})
+    intermediate_size: int = field(default=6144, metadata={"help": "The intermediate size of the model."})
+    num_key_value_heads: int = field(default=4, metadata={"help": "The number of key-value heads in GQA."})
+    use_qk_norm: bool = field(default=False, metadata={"help": "Whether to use qk norm."})
+    tie_word_embeddings: bool = field(
+        default=True,
+        metadata={"help": "Whether to tie the word embeddings of the encoder and the decoder."}
+    )
+    lora_rank: int = field(default=64, metadata={"help": "The rank of lora."})
+    lora_alpha: int = field(default=128, metadata={"help": "Lora alpha"})
+    lora_dropout: float = field(default=0.0, metadata={"help": "Lora dropout"})
+    train_attention_params_only: bool = field(default=False, metadata={
+        "help": "Whether to train attention parameters only."}
+    )
+
+
+@dataclass
+class DataArguments:
+    model_size: Literal["0.5B", "1.8B", "4B", "7B"] = field(
+        default="1.8B",
+        metadata={"help": "Select the model size from ['0.5B', '1.8B', '4B', '7B']. "
+                  "This affects the tokenizer special tokens used for loss masking."}
+    )
+    train_data_file: str = field(default=None, metadata={"help": "Path to the training data."})
+    max_seq_length: int = field(
+        default=4096,
+        metadata={"help": "The max sequence length of the model inputs after tokenization."}
+    )
+    use_dummy_data: bool = field(default=False, metadata={"help": "Use dummy data."})
+
+
+@dataclass
+class TrainingArguments(transformers.TrainingArguments):
+    cache_dir: Optional[str] = field(default=None)
+    optim: str = field(default="adamw_torch")
+    model_max_length: int = field(
+        default=4096,
+        metadata={"help": "Maximum sequence length. Sequences will be right padded (and possibly truncated)."},
+    )
+    tokenizer_name_or_path: Optional[str] = field(default=None)
+    model_name_or_path: Optional[str] = field(default=None)
+    min_lr: float = field(
+        default=1e-6,
+        metadata={"help": "The minimum learning rate at the end of the cosine decay."}
+    )
+
+
+IGNORE_INDEX = -100
+
+
+class DummyDataset(Dataset):
+    def __init__(self, tokenizer, max_seq_length=512, length=1000):
+        self.tokenizer = tokenizer
+        self.max_seq_length = max_seq_length
+        self.length = length
+
+    def __len__(self):
+        return self.length
+
+    def __getitem__(self, index):
+        tokens = torch.randint(0, self.tokenizer.vocab_size, (self.max_seq_length,))
+        return {'input_ids': tokens, 'labels': tokens}
+
+
+class SFTDataset(Dataset):
+    def __init__(self, data_file, tokenizer, max_seq_length=4096, model_size="1.8B"):
+        self.tokenizer = tokenizer
+        self.max_seq_length = max_seq_length
+        self.model_size = model_size
+        self.data_list = self.load_data(data_file)
+
+    def __len__(self):
+        return len(self.data_list)
+
+    def load_data(self, data_file):
+        logging.info('Loading data: {}'.format(data_file))
+        with open(data_file, 'r', encoding='utf8') as f:
+            data_list = f.readlines()
+        logging.info("there are {} data in dataset".format(len(data_list)))
+        return data_list
+
+    def encode_data(self, data_dict):
+        model_inputs = {}
+        template_output = self.tokenizer.apply_chat_template(
+            data_dict['messages'], tokenize=True, return_dict=False
+        )
+        if isinstance(template_output, list) and len(template_output) > 0 and isinstance(template_output[0], list):
+            template_output = template_output[0]
+        message_tokens = torch.tensor(template_output, dtype=torch.long)
+
+        # Note: The 7B model uses a different vocabulary/special tokens than other models.
+        if self.model_size == "7B":
+            sep_token_id = self.tokenizer.convert_tokens_to_ids('<|extra_0|>')
+            eos_token_id = self.tokenizer.convert_tokens_to_ids('<|eos|>')
+        else:
+            sep_token_id = self.tokenizer.convert_tokens_to_ids('<｜hy_Assistant｜>')
+            eos_token_id = self.tokenizer.convert_tokens_to_ids('<｜hy_place▁holder▁no▁2｜>')
+
+        # Find assistant reply boundaries
+        loss_token_begins = (message_tokens == sep_token_id).nonzero(as_tuple=True)[0].tolist()
+        loss_token_ends = (message_tokens == eos_token_id).nonzero(as_tuple=True)[0].tolist()
+        message_labels = torch.tensor([IGNORE_INDEX] * message_tokens.shape[0])
+        for begin_idx, end_idx in zip(loss_token_begins, loss_token_ends):
+            # Compute loss from sep_token to eos_token (inclusive)
+            message_labels[begin_idx:end_idx + 1] = message_tokens[begin_idx:end_idx + 1]
+
+        input_ids = message_tokens.to(torch.long)
+        labels = message_labels.to(torch.long)
+
+        input_ids = input_ids[:self.max_seq_length]
+        labels = labels[:self.max_seq_length]
+
+        pad_token_id = self.tokenizer.pad_token_id
+        attention_mask = [1 if val != pad_token_id else 0 for val in input_ids]
+        model_inputs["input_ids"] = input_ids
+        model_inputs["attention_mask"] = torch.tensor(attention_mask, dtype=torch.bool)
+        model_inputs["labels"] = labels
+
+        return model_inputs
+
+    def __getitem__(self, index):
+        data = self.data_list[index]
+        data = json.loads(data)
+        model_inputs = self.encode_data(data)
+        return model_inputs
+
+
+@dataclass
+class DataCollatorForSupervisedDataset(object):
+    """Collate examples for supervised fine-tuning."""
+
+    tokenizer: transformers.PreTrainedTokenizer
+
+    def __call__(self, instances):
+        input_ids = [instance['input_ids'] for instance in instances]
+        labels = [instance['labels'] for instance in instances]
+        pad_token_id = self.tokenizer.pad_token_id
+        input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=True, padding_value=pad_token_id)
+        labels = torch.nn.utils.rnn.pad_sequence(labels, batch_first=True, padding_value=IGNORE_INDEX)
+        return dict(
+            input_ids=input_ids,
+            labels=labels,
+            attention_mask=input_ids.ne(pad_token_id),
+        )
+
+
+def make_supervised_data_module(tokenizer, data_args) -> Dict:
+    """Make dataset and collator for supervised fine-tuning."""
+    if data_args.use_dummy_data:
+        train_dataset = DummyDataset(tokenizer, data_args.max_seq_length)
+    else:
+        train_dataset = SFTDataset(
+            tokenizer=tokenizer,
+            data_file=data_args.train_data_file,
+            max_seq_length=data_args.max_seq_length,
+            model_size=data_args.model_size,
+        )
+    data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
+    return dict(train_dataset=train_dataset, eval_dataset=None, data_collator=data_collator)
+
+
+# Copy tokenizer and config files to each checkpoint directory for self-contained inference
+class CustomSaveCallback(TrainerCallback):
+    def on_save(self, args, state, control, **kwargs):
+        if torch.distributed.get_rank() == 0:
+            output_dir = os.path.join(args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{state.global_step}")
+
+            # Copy tokenizer files to checkpoint directory
+            tokenizer_files = [
+                'generation_config.json',
+                'hy.tiktoken',
+                'tokenizer_config.json',
+                'tokenization_hy.py',
+                'tokenizer.json',
+                'special_tokens_map.json',
+                'chat_template.jinja',
+                'config.json',
+            ]
+            src_dir = args.tokenizer_name_or_path or args.model_name_or_path
+            for fname in tokenizer_files:
+                src = os.path.join(src_dir, fname)
+                if os.path.isfile(src):
+                    shutil.copy(src, os.path.join(output_dir, fname))
+
+        return control
+
+
+def train():
+    parser = transformers.HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
+    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
+    print_args(model_args, 'model arguments')
+    print_args(data_args, 'data arguments')
+    print_args(training_args, 'training arguments')
+
+    tokenizer = transformers.AutoTokenizer.from_pretrained(
+        training_args.tokenizer_name_or_path,
+        trust_remote_code=True
+    )
+
+    init_kwargs = {}
+    if model_args.use_flash_attn:
+        init_kwargs["attn_implementation"] = "flash_attention_2"
+    if training_args.bf16:
+        init_kwargs["torch_dtype"] = torch.bfloat16
+    elif training_args.fp16:
+        init_kwargs["torch_dtype"] = torch.float16
+
+    # Load model from pretrained weights
+    if training_args.model_name_or_path is not None and os.path.exists(training_args.model_name_or_path):
+        print(f"Initializing model from local file: {training_args.model_name_or_path}")
+        model = transformers.AutoModelForCausalLM.from_pretrained(
+            training_args.model_name_or_path,
+            trust_remote_code=True,
+            **init_kwargs
+        )
+    else:
+        raise FileNotFoundError(
+            f"Model path {training_args.model_name_or_path} is invalid or does not exist. "
+            f"Dense model training requires pre-trained weights."
+        )
+
+    if model_args.train_attention_params_only:
+        for name, param in model.named_parameters():
+            if 'self_attn' not in name:
+                param.requires_grad = False
+
+    if model_args.use_lora:
+        # Define LoRA configuration
+        lora_config = LoraConfig(
+            r=model_args.lora_rank,
+            lora_alpha=model_args.lora_alpha,
+            lora_dropout=model_args.lora_dropout,
+            target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
+            bias="none",
+            task_type="CAUSAL_LM",
+        )
+        model = get_peft_model(model, lora_config)
+
+    data_module = make_supervised_data_module(tokenizer=tokenizer, data_args=data_args)
+    # Tell Trainer not to attempt DataParallel
+    model.is_parallelizable = True
+    model.model_parallel = True
+
+    training_args.lr_scheduler_kwargs = {
+        'min_lr_rate': training_args.min_lr / training_args.learning_rate,
+    }
+
+    # -----------------------------------------------------------------------
+    # Fix: DeepSpeed ZeRO-3 + gradient checkpointing compatibility.
+    #
+    # PyTorch's torch.utils.checkpoint with use_reentrant=False (the default
+    # in transformers) performs strict metadata checks on recomputed tensors
+    # during backward.  Under ZeRO-3, parameters are all-gathered during the
+    # first forward pass (shape=[full_size]) but may be partitioned back
+    # (shape=[0]) when the checkpoint recomputes, causing a CheckpointError.
+    #
+    # Setting use_reentrant=True avoids this strict metadata check.
+    # -----------------------------------------------------------------------
+    if training_args.gradient_checkpointing and training_args.deepspeed:
+        training_args.gradient_checkpointing_kwargs = {"use_reentrant": True}
+
+    trainer = Trainer(
+        model=model,
+        processing_class=tokenizer,
+        args=training_args,
+        callbacks=[CustomSaveCallback],
+        **data_module
+    )
+    model.config.use_cache = False
+
+    trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
+
+
+if __name__ == "__main__":
+    train()

train/deepspeed_support/train_dense.sh

@@ -0,0 +1,155 @@
+#!/bin/bash
+
+# Unified Dense model full fine-tuning script
+# Supports: 1.8B and 7B dense models
+# Usage: bash train_dense.sh [1.8B|7B]
+#   - 1.8B: 1x GPU (24GB+), DeepSpeed ZeRO-2 (no offload)
+#   - 7B:   2x GPU (80GB+ each), DeepSpeed ZeRO-3 (no offload)
+
+# ============== Model Size Selection ==============
+MODEL_SIZE=${1:-"1.8B"}
+
+if [[ "${MODEL_SIZE}" != "1.8B" && "${MODEL_SIZE}" != "7B" ]]; then
+    echo "Error: MODEL_SIZE must be '1.8B' or '7B', got '${MODEL_SIZE}'"
+    echo "Usage: bash train_dense.sh [1.8B|7B]"
+    exit 1
+fi
+
+# ============== NCCL Configuration ==============
+NET_TYPE="high"
+export NCCL_DEBUG=WARN
+export NCCL_P2P_LEVEL=NVL
+export NCCL_IB_TIMEOUT=24
+export NCCL_NVLS_ENABLE=0
+export NCCL_MPI_PROFILE_PRIMS_ENABLE=0
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+export TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=3600
+if [[ "${NET_TYPE}" = "low" ]]; then
+    export NCCL_SOCKET_IFNAME=eth1
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_HCA=mlx5_2:1
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+else
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_IB_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+    export NCCL_SOCKET_IFNAME=bond1
+    export UCX_NET_DEVICES=bond1
+    export NCCL_IB_HCA=mlx5_bond_1,mlx5_bond_5,mlx5_bond_3,mlx5_bond_7,mlx5_bond_4,mlx5_bond_8,mlx5_bond_2,mlx5_bond_6
+    export NCCL_COLLNET_ENABLE=0
+    export SHARP_COLL_ENABLE_SAT=0
+    export NCCL_NET_GDR_LEVEL=2
+    export NCCL_IB_QPS_PER_CONNECTION=4
+    export NCCL_IB_TC=160
+    export NCCL_PXN_DISABLE=1
+fi
+
+# ============== Model-specific Configuration ==============
+SCRIPT_DIR=$(dirname "$0")
+
+if [[ "${MODEL_SIZE}" == "1.8B" ]]; then
+    export HOST_GPU_NUM=1
+    model_path=path_to_dense_1_8b_model
+    ds_config_file=${SCRIPT_DIR}/ds_zero2_no_offload.json
+    output_path=./dense_1_8b_output
+    HIDDEN_SIZE=2048
+    INTERMEDIATE_SIZE=6144
+    NUM_ATTENTION_HEADS=16
+    NUM_KEY_VALUE_HEADS=4
+    NUM_LAYERS=32
+else
+    export HOST_GPU_NUM=2
+    model_path=path_to_dense_7b_model
+    ds_config_file=${SCRIPT_DIR}/ds_zero3_no_offload.json
+    output_path=./dense_7b_output
+    HIDDEN_SIZE=4096
+    INTERMEDIATE_SIZE=14336
+    NUM_ATTENTION_HEADS=32
+    NUM_KEY_VALUE_HEADS=8
+    NUM_LAYERS=32
+fi
+
+tokenizer_path=${model_path}
+train_data_file=../data/example_data.jsonl
+
+# ============== Multi-node Configuration ==============
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+IP_LIST=${IP_LIST:-"127.0.0.1"}
+
+IFS=',' read -ra IP_ARRAY <<< "$IP_LIST"
+export NODES=${#IP_ARRAY[@]}
+export LOCAL_IP=${IP_ARRAY[0]}
+NODE_IP_LIST=""
+for ip in "${IP_ARRAY[@]}"; do
+    if [ -n "$NODE_IP_LIST" ]; then
+        NODE_IP_LIST="${NODE_IP_LIST},"
+    fi
+    NODE_IP_LIST="${NODE_IP_LIST}${ip}:${HOST_GPU_NUM}"
+done
+export NODE_IP_LIST
+export NODE_NUM=$((${NODES} * ${HOST_GPU_NUM}))
+
+# ============== Output & Logging ==============
+mkdir -p ${output_path}
+
+current_time=$(date "+%Y.%m.%d-%H.%M.%S")
+log_file=${output_path}/"log_${current_time}.txt"
+
+echo $NODE_IP_LIST > env.txt 2>&1
+sed "s/:/ slots=/g" env.txt | sed "s/,/\n/g" >  "hostfile"
+sed "s/:.//g" env.txt | sed "s/,/\n/g" >  "pssh.hosts"
+export CHIEF_IP=$LOCAL_IP
+
+if [ ${NODES} -gt 1 ]; then
+    HOST_PATH=hostfile
+    DS_ARGS="--hostfile=${HOST_PATH} --master_addr ${CHIEF_IP}"
+else
+    DS_ARGS=""
+fi
+
+echo "============================================"
+echo "Dense ${MODEL_SIZE} full fine-tuning"
+echo "NODES: ${NODES}, LOCAL_IP: ${LOCAL_IP}, NODE_IP_LIST: ${NODE_IP_LIST}"
+echo "DeepSpeed config: ${ds_config_file}"
+echo "Model path: ${model_path}"
+echo "Output path: ${output_path}"
+echo "============================================"
+
+# ============== Launch Training ==============
+deepspeed ${DS_ARGS} \
+    ${SCRIPT_DIR}/train_dense.py \
+    --do_train \
+    --model_size ${MODEL_SIZE} \
+    --model_name_or_path ${model_path} \
+    --tokenizer_name_or_path ${tokenizer_path} \
+    --train_data_file ${train_data_file} \
+    --deepspeed ${ds_config_file} \
+    --output_dir ${output_path} \
+    --per_device_train_batch_size 1 \
+    --gradient_accumulation_steps 1 \
+    --gradient_checkpointing \
+    --lr_scheduler_type cosine_with_min_lr \
+    --logging_steps 1 \
+    --max_steps 30 \
+    --save_steps 30 \
+    --learning_rate 1e-5 \
+    --min_lr 1e-6 \
+    --warmup_ratio 0.01 \
+    --save_strategy steps \
+    --bf16 \
+    --hidden_size ${HIDDEN_SIZE} \
+    --intermediate_size ${INTERMEDIATE_SIZE} \
+    --num_attention_heads ${NUM_ATTENTION_HEADS} \
+    --num_key_value_heads ${NUM_KEY_VALUE_HEADS} \
+    --num_layers ${NUM_LAYERS} \
+    --model_max_length 4096 \
+    --max_seq_length 4096 \
+    --use_qk_norm | tee ${log_file}

train/deepspeed_support/train_dense_lora.sh

@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Unified Dense model LoRA fine-tuning script
+# Supports: 1.8B and 7B dense models
+# Usage: bash train_dense_lora.sh [1.8B|7B]
+#   - 1.8B: 1x GPU (24GB+), DeepSpeed ZeRO-2 (no offload)
+#   - 7B:   1x GPU (80GB+), DeepSpeed ZeRO-2 (no offload)
+# LoRA greatly reduces memory requirements compared to full fine-tuning.
+
+# ============== Model Size Selection ==============
+MODEL_SIZE=${1:-"1.8B"}
+
+if [[ "${MODEL_SIZE}" != "1.8B" && "${MODEL_SIZE}" != "7B" ]]; then
+    echo "Error: MODEL_SIZE must be '1.8B' or '7B', got '${MODEL_SIZE}'"
+    echo "Usage: bash train_dense_lora.sh [1.8B|7B]"
+    exit 1
+fi
+
+# ============== NCCL Configuration ==============
+NET_TYPE="high"
+export NCCL_DEBUG=WARN
+export NCCL_P2P_LEVEL=NVL
+export NCCL_IB_TIMEOUT=24
+export NCCL_NVLS_ENABLE=0
+export NCCL_MPI_PROFILE_PRIMS_ENABLE=0
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+export TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=3600
+if [[ "${NET_TYPE}" = "low" ]]; then
+    export NCCL_SOCKET_IFNAME=eth1
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_HCA=mlx5_2:1
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+else
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_IB_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+    export NCCL_SOCKET_IFNAME=bond1
+    export UCX_NET_DEVICES=bond1
+    export NCCL_IB_HCA=mlx5_bond_1,mlx5_bond_5,mlx5_bond_3,mlx5_bond_7,mlx5_bond_4,mlx5_bond_8,mlx5_bond_2,mlx5_bond_6
+    export NCCL_COLLNET_ENABLE=0
+    export SHARP_COLL_ENABLE_SAT=0
+    export NCCL_NET_GDR_LEVEL=2
+    export NCCL_IB_QPS_PER_CONNECTION=4
+    export NCCL_IB_TC=160
+    export NCCL_PXN_DISABLE=1
+fi
+
+# ============== Model-specific Configuration ==============
+SCRIPT_DIR=$(dirname "$0")
+
+# LoRA training uses ZeRO-2 (no offload) for both 1.8B and 7B
+# since only adapter parameters are trained, memory usage is much lower
+export HOST_GPU_NUM=1
+ds_config_file=${SCRIPT_DIR}/ds_zero2_no_offload.json
+
+if [[ "${MODEL_SIZE}" == "1.8B" ]]; then
+    model_path=path_to_dense_1_8b_model
+    output_path=dense_1_8b_lora_output
+    HIDDEN_SIZE=2048
+    INTERMEDIATE_SIZE=6144
+    NUM_ATTENTION_HEADS=16
+    NUM_KEY_VALUE_HEADS=4
+    NUM_LAYERS=32
+else
+    model_path=path_to_dense_7b_model
+    output_path=dense_7b_lora_output
+    HIDDEN_SIZE=4096
+    INTERMEDIATE_SIZE=14336
+    NUM_ATTENTION_HEADS=32
+    NUM_KEY_VALUE_HEADS=8
+    NUM_LAYERS=32
+fi
+
+tokenizer_path=${model_path}
+train_data_file=${SCRIPT_DIR}/../data/example_data.jsonl
+
+# ============== Multi-node Configuration ==============
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+IP_LIST=${IP_LIST:-"127.0.0.1"}
+
+IFS=',' read -ra IP_ARRAY <<< "$IP_LIST"
+export NODES=${#IP_ARRAY[@]}
+export LOCAL_IP=${IP_ARRAY[0]}
+NODE_IP_LIST=""
+for ip in "${IP_ARRAY[@]}"; do
+    if [ -n "$NODE_IP_LIST" ]; then
+        NODE_IP_LIST="${NODE_IP_LIST},"
+    fi
+    NODE_IP_LIST="${NODE_IP_LIST}${ip}:${HOST_GPU_NUM}"
+done
+export NODE_IP_LIST
+export NODE_NUM=$((${NODES} * ${HOST_GPU_NUM}))
+
+# ============== Output & Logging ==============
+mkdir -p ${output_path}
+
+current_time=$(date "+%Y.%m.%d-%H.%M.%S")
+log_file=${output_path}/"log_${current_time}.txt"
+
+echo $NODE_IP_LIST > env.txt 2>&1
+sed "s/:/ slots=/g" env.txt | sed "s/,/\n/g" >  "hostfile"
+sed "s/:.//g" env.txt | sed "s/,/\n/g" >  "pssh.hosts"
+export CHIEF_IP=$LOCAL_IP
+
+if [ ${NODES} -gt 1 ]; then
+    HOST_PATH=hostfile
+    DS_ARGS="--hostfile=${HOST_PATH} --master_addr ${CHIEF_IP}"
+else
+    DS_ARGS=""
+fi
+
+echo "============================================"
+echo "Dense ${MODEL_SIZE} LoRA fine-tuning"
+echo "NODES: ${NODES}, LOCAL_IP: ${LOCAL_IP}, NODE_IP_LIST: ${NODE_IP_LIST}"
+echo "DeepSpeed config: ${ds_config_file}"
+echo "Model path: ${model_path}"
+echo "Output path: ${output_path}"
+echo "============================================"
+
+# ============== Launch Training ==============
+deepspeed ${DS_ARGS} \
+    ${SCRIPT_DIR}/train_dense.py \
+    --do_train \
+    --model_size ${MODEL_SIZE} \
+    --model_name_or_path ${model_path} \
+    --tokenizer_name_or_path ${tokenizer_path} \
+    --train_data_file ${train_data_file} \
+    --deepspeed ${ds_config_file} \
+    --output_dir ${output_path} \
+    --per_device_train_batch_size 1 \
+    --gradient_accumulation_steps 1 \
+    --gradient_checkpointing \
+    --lr_scheduler_type cosine_with_min_lr \
+    --logging_steps 1 \
+    --max_steps 30 \
+    --save_steps 30 \
+    --learning_rate 2e-4 \
+    --min_lr 1e-5 \
+    --warmup_ratio 0.01 \
+    --save_strategy steps \
+    --bf16 \
+    --hidden_size ${HIDDEN_SIZE} \
+    --intermediate_size ${INTERMEDIATE_SIZE} \
+    --num_attention_heads ${NUM_ATTENTION_HEADS} \
+    --num_key_value_heads ${NUM_KEY_VALUE_HEADS} \
+    --num_layers ${NUM_LAYERS} \
+    --model_max_length 4096 \
+    --max_seq_length 4096 \
+    --use_qk_norm \
+    --use_lora \
+    --lora_rank 64 \
+    --lora_alpha 128 \
+    --lora_dropout 0.05 | tee ${log_file}

train/deepspeed_support/train_lora.sh

@@ -0,0 +1,125 @@
+#!/bin/bash
+
+NET_TYPE="high"
+export NCCL_DEBUG=WARN
+export NCCL_P2P_LEVEL=NVL
+export NCCL_IB_TIMEOUT=24
+export NCCL_NVLS_ENABLE=0
+export NCCL_MPI_PROFILE_PRIMS_ENABLE=0
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+export TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=3600
+if [[ "${NET_TYPE}" = "low" ]]; then
+    export NCCL_SOCKET_IFNAME=eth1
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_HCA=mlx5_2:1
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+else
+    export NCCL_IB_GID_INDEX=3
+    export NCCL_IB_SL=3
+    export NCCL_CHECK_DISABLE=1
+    export NCCL_P2P_DISABLE=0
+    export NCCL_IB_DISABLE=0
+    export NCCL_LL_THRESHOLD=16384
+    export NCCL_IB_CUDA_SUPPORT=1
+    export NCCL_SOCKET_IFNAME=bond1
+    export UCX_NET_DEVICES=bond1
+    export NCCL_IB_HCA=mlx5_bond_1,mlx5_bond_5,mlx5_bond_3,mlx5_bond_7,mlx5_bond_4,mlx5_bond_8,mlx5_bond_2,mlx5_bond_6
+    export NCCL_COLLNET_ENABLE=0
+    export SHARP_COLL_ENABLE_SAT=0
+    export NCCL_NET_GDR_LEVEL=2
+    export NCCL_IB_QPS_PER_CONNECTION=4
+    export NCCL_IB_TC=160
+    export NCCL_PXN_DISABLE=1
+fi
+
+export HOST_GPU_NUM=8
+# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
+IP_LIST=${IP_LIST:-"127.0.0.1"}
+
+IFS=',' read -ra IP_ARRAY <<< "$IP_LIST"
+export NODES=${#IP_ARRAY[@]}
+export LOCAL_IP=${IP_ARRAY[0]}
+NODE_IP_LIST=""
+for ip in "${IP_ARRAY[@]}"; do
+    if [ -n "$NODE_IP_LIST" ]; then
+        NODE_IP_LIST="${NODE_IP_LIST},"
+    fi
+    NODE_IP_LIST="${NODE_IP_LIST}${ip}:${HOST_GPU_NUM}"
+done
+export NODE_IP_LIST
+export NODE_NUM=$((${NODES} * ${HOST_GPU_NUM}))
+
+SCRIPT_DIR=$(dirname "$0")
+
+model_path=path_to_model_weight
+tokenizer_path=../models
+train_data_file=example_data.jsonl
+
+# ds_config_file=ds_zero2_no_offload.json
+# ds_config_file=ds_zero3_no_offload.json
+# For LoRA, zero2_offload is recommended to save memory
+ds_config_file=${SCRIPT_DIR}/ds_zero2_no_offload.json
+
+output_path=/root/hf_train_output
+
+mkdir -p ${output_path}
+
+current_time=$(date "+%Y.%m.%d-%H.%M.%S")
+log_file=${output_path}/"log_${current_time}.txt"
+
+echo $NODE_IP_LIST > env.txt 2>&1
+sed "s/:/ slots=/g" env.txt | sed "s/,/\n/g" >  "hostfile"
+sed "s/:.//g" env.txt | sed "s/,/\n/g" >  "pssh.hosts"
+export CHIEF_IP=$LOCAL_IP
+
+if [ ${NODES} -gt 1 ]; then
+    HOST_PATH=hostfile
+    DS_ARGS="--hostfile=${HOST_PATH} --master_addr ${CHIEF_IP}"
+else
+    DS_ARGS=""
+fi
+
+echo "NODES: ${NODES}, LOCAL_IP: ${LOCAL_IP}, NODE_IP_LIST: ${NODE_IP_LIST}"
+
+deepspeed ${DS_ARGS} \
+    ${SCRIPT_DIR}/train.py \
+    --do_train \
+    --model_name_or_path ${model_path} \
+    --tokenizer_name_or_path ${tokenizer_path} \
+    --train_data_file ${train_data_file} \
+    --deepspeed ${ds_config_file} \
+    --output_dir ${output_path} \
+    --per_device_train_batch_size 1 \
+    --gradient_accumulation_steps 1 \
+    --gradient_checkpointing \
+    --lr_scheduler_type cosine_with_min_lr \
+    --logging_steps 1 \
+    --max_steps 200 \
+    --save_steps 100 \
+    --learning_rate 1e-5 \
+    --min_lr 1e-6 \
+    --warmup_ratio 0.01 \
+    --save_strategy steps \
+    --bf16 \
+    --use_lora \
+    --lora_rank 64 \
+    --lora_alpha 128 \
+    --lora_dropout 0.1 \
+    --hidden_size 2048 \
+    --intermediate_size 6912 \
+    --model_max_length 8192 \
+    --max_seq_length 8192 \
+    --moe_topk 8 \
+    --num_experts 128 \
+    --moe_intermediate_size 768 \
+    --moe_layer_num_skipped 1 \
+    --num_attention_heads 32 \
+    --num_key_value_heads 4 \
+    --num_layers 48 \
+    --use_mixed_mlp_moe \
+    --num_shared_expert 1 \
+    --use_qk_norm | tee ${log_file}

train/llama_factory_support/convert_zero_to_hf.sh

@@ -0,0 +1,175 @@
+#!/bin/bash
+# 将 DeepSpeed ZeRO 格式的 checkpoint 转换为 HuggingFace 格式
+# 使用 zero_to_fp32.py 转换权重，然后保存为 HF 格式
+
+set -e  # 遇到错误立即退出
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+PROJECT_ROOT="$(dirname "$(dirname "$SCRIPT_DIR")")"
+
+# 激活 conda 环境
+source "$(conda info --base)/etc/profile.d/conda.sh"
+conda activate llama_factory
+
+# 设置环境变量
+export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
+export DISABLE_VERSION_CHECK=1
+export CUDA_VISIBLE_DEVICES=""  # 使用 CPU 进行转换，避免显存不足
+
+CHECKPOINT_DIR="$SCRIPT_DIR/saves/hy_v3/full/sft/checkpoint-39"
+OUTPUT_DIR="$SCRIPT_DIR/saves/hy_v3/full/sft/checkpoint-39/hf_converted"
+TEMP_WEIGHTS_DIR="$OUTPUT_DIR/zero_fp32_output"  # 分片输出目录
+
+echo "=========================================="
+echo "Converting DeepSpeed ZeRO checkpoint to HF format"
+echo "Input:  $CHECKPOINT_DIR"
+echo "Output: $OUTPUT_DIR"
+echo "=========================================="
+
+# 创建输出目录
+mkdir -p "$OUTPUT_DIR"
+rm -rf "$TEMP_WEIGHTS_DIR"
+mkdir -p "$TEMP_WEIGHTS_DIR"
+
+# Step 1: 使用 zero_to_fp32.py 转换权重
+echo ""
+echo "[Step 1/3] Converting weights from ZeRO format to FP32..."
+cd "$CHECKPOINT_DIR"
+python3 zero_to_fp32.py . "$TEMP_WEIGHTS_DIR"
+
+# 检查输出 - zero_to_fp32.py 可能输出单个文件或多个分片
+if [ -d "$TEMP_WEIGHTS_DIR" ] && [ "$(ls -A "$TEMP_WEIGHTS_DIR" 2>/dev/null)" ]; then
+    echo "Weight conversion completed! Output in: $TEMP_WEIGHTS_DIR"
+    echo "Files: $(ls "$TEMP_WEIGHTS_DIR" | wc -l) files"
+else
+    echo "ERROR: Weight conversion failed!"
+    exit 1
+fi
+
+# Step 2: 复制配置文件
+echo ""
+echo "[Step 2/3] Copying config files..."
+BASE_MODEL_DIR="$PROJECT_ROOT/a3b_ckpt"
+
+# 从基座模型复制完整的配置文件（checkpoint 中的 tokenizer_config 不完整）
+cp "$BASE_MODEL_DIR/config.json" "$OUTPUT_DIR/" 2>/dev/null || true
+cp "$BASE_MODEL_DIR/tokenizer_config.json" "$OUTPUT_DIR/" 2>/dev/null || true
+cp "$BASE_MODEL_DIR/tokenizer.json" "$OUTPUT_DIR/" 2>/dev/null || true
+cp "$BASE_MODEL_DIR/special_tokens_map.json" "$OUTPUT_DIR/" 2>/dev/null || true
+cp "$BASE_MODEL_DIR/chat_template.jinja" "$OUTPUT_DIR/" 2>/dev/null || true
+cp "$CHECKPOINT_DIR/generation_config.json" "$OUTPUT_DIR/" 2>/dev/null || true
+
+echo "Config files copied from base model."
+
+# Step 3: 加载权重并保存为 HF 格式
+echo ""
+echo "[Step 3/3] Converting to HuggingFace format..."
+
+cat > /tmp/convert_to_hf.py << 'PYEOF'
+import torch
+import json
+import os
+import sys
+import glob
+sys.path.insert(0, os.environ.get("PROJECT_ROOT", "."))
+
+# 设置目录
+output_dir = os.environ["OUTPUT_DIR"]
+checkpoint_dir = os.environ["CHECKPOINT_DIR"]
+base_model_dir = os.environ["BASE_MODEL_DIR"]
+temp_weights_dir = os.path.join(output_dir, "zero_fp32_output")
+
+# 加载 tokenizer
+from transformers import AutoTokenizer
+print("Loading tokenizer...")
+tokenizer = AutoTokenizer.from_pretrained(base_model_dir, trust_remote_code=True)
+tokenizer.save_pretrained(output_dir)
+
+# 从 base model 加载配置和模型结构
+from transformers import AutoConfig, AutoModelForCausalLM
+print("Loading config...")
+config = AutoConfig.from_pretrained(base_model_dir, trust_remote_code=True)
+
+# 创建模型（从基座模型加载结构和权重，然后用训练后的权重覆盖）
+print("Loading base model...")
+with torch.no_grad():
+    model = AutoModelForCausalLM.from_pretrained(
+        base_model_dir,
+        torch_dtype=torch.bfloat16,
+        trust_remote_code=True,
+    )
+
+# 加载转换后的权重（可能分片）
+print(f"Loading weights from {temp_weights_dir} ...")
+weight_files = sorted(glob.glob(os.path.join(temp_weights_dir, "*.bin")) + 
+                      glob.glob(os.path.join(temp_weights_dir, "*.safetensors")))
+
+if not weight_files:
+    print(f"ERROR: No weight files found in {temp_weights_dir}")
+    sys.exit(1)
+
+print(f"Found {len(weight_files)} weight files")
+state_dict = {}
+for wf in weight_files:
+    print(f"  Loading {wf} ...")
+    if wf.endswith('.safetensors'):
+        from safetensors.torch import load_file
+        state_dict.update(load_file(wf, device="cpu"))
+    else:
+        state_dict.update(torch.load(wf, map_location="cpu"))
+
+# 加载权重到模型
+print("Loading converted weights into model...")
+model_state_dict = model.state_dict()
+filtered_state_dict = {}
+skipped = 0
+matched = 0
+for k, v in state_dict.items():
+    if k in model_state_dict:
+        # 转换 dtype
+        if v.dtype != model_state_dict[k].dtype:
+            v = v.to(model_state_dict[k].dtype)
+        filtered_state_dict[k] = v
+        matched += 1
+    else:
+        skipped += 1
+        if skipped <= 10:  # 只打印前10个跳过的key
+            print(f"  Skipping key: {k}")
+
+print(f"Matched {matched} tensors, skipped {skipped}")
+
+if matched == 0:
+    print("ERROR: No weights matched! Something is wrong with the conversion.")
+    sys.exit(1)
+
+missing, unexpected = model.load_state_dict(filtered_state_dict, strict=False)
+if missing:
+    print(f"WARNING: {len(missing)} keys missing in converted weights (using base model weights)")
+    for k in missing[:10]:
+        print(f"  Missing: {k}")
+    if len(missing) > 10:
+        print(f"  ... and {len(missing) - 10} more")
+
+# 保存为 HF 格式（使用 safetensors，更安全、更快）
+print(f"Saving model to {output_dir}...")
+model.save_pretrained(output_dir, safe_serialization=True)
+print("Done!")
+
+# 清理临时文件
+print("Cleaning up temp files...")
+import shutil
+shutil.rmtree(temp_weights_dir)
+PYEOF
+
+export OUTPUT_DIR="$OUTPUT_DIR"
+export CHECKPOINT_DIR="$CHECKPOINT_DIR"
+export BASE_MODEL_DIR="$PROJECT_ROOT/a3b_ckpt"
+export PROJECT_ROOT="$PROJECT_ROOT"
+
+python3 /tmp/convert_to_hf.py
+
+echo ""
+echo "=========================================="
+echo "Conversion completed!"
+echo "HF format model saved to: $OUTPUT_DIR"
+echo "=========================================="

train/llama_factory_support/dataset_info.json

@@ -0,0 +1,44 @@
+{
+  "hy_v3_demo": {
+    "file_name": "../example_data.jsonl",
+    "formatting": "sharegpt",
+    "columns": {
+      "messages": "messages"
+    },
+    "tags": {
+      "role_tag": "role",
+      "content_tag": "content",
+      "user_tag": "user",
+      "assistant_tag": "assistant",
+      "system_tag": "system"
+    }
+  },
+  "hy_v3_translation": {
+    "file_name": "../data/example_data.jsonl",
+    "formatting": "sharegpt",
+    "columns": {
+      "messages": "messages"
+    },
+    "tags": {
+      "role_tag": "role",
+      "content_tag": "content",
+      "user_tag": "user",
+      "assistant_tag": "assistant",
+      "system_tag": "system"
+    }
+  },
+  "hy_dense_demo": {
+    "file_name": "../data/example_data.jsonl",
+    "formatting": "sharegpt",
+    "columns": {
+      "messages": "messages"
+    },
+    "tags": {
+      "role_tag": "role",
+      "content_tag": "content",
+      "user_tag": "user",
+      "assistant_tag": "assistant",
+      "system_tag": "system"
+    }
+  }
+}

train/llama_factory_support/ds_zero2_offload.json

@@ -0,0 +1,37 @@
+{
+    "fp16": {
+        "enabled": false,
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    "bf16": {
+        "enabled": true
+    },
+
+    "zero_optimization": {
+        "stage": 2,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "offload_param": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "allgather_partitions": true,
+        "allgather_bucket_size": 5e8,
+        "overlap_comm": true,
+        "reduce_scatter": true,
+        "reduce_bucket_size": 5e8,
+        "contiguous_gradients": true
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/llama_factory_support/ds_zero2_offload_lora.json

@@ -0,0 +1,33 @@
+{
+    "fp16": {
+        "enabled": false,
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    "bf16": {
+        "enabled": true
+    },
+
+    "zero_optimization": {
+        "stage": 2,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "allgather_partitions": true,
+        "allgather_bucket_size": 5e8,
+        "overlap_comm": true,
+        "reduce_scatter": true,
+        "reduce_bucket_size": 5e8,
+        "contiguous_gradients": true
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/llama_factory_support/ds_zero3_offload.json

@@ -0,0 +1,36 @@
+{
+    "fp16": {
+        "enabled": false,
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    "bf16": {
+        "enabled": true
+    },
+
+    "zero_optimization": {
+        "stage": 3,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "overlap_comm": true,
+        "contiguous_gradients": true,
+        "sub_group_size": 1e9,
+        "reduce_bucket_size": 1e8,
+        "stage3_prefetch_bucket_size": 1e8,
+        "stage3_param_persistence_threshold": 0,
+        "stage3_max_live_parameters": 1e8,
+        "stage3_max_reuse_distance": 1e8,
+        "stage3_gather_16bit_weights_on_model_save": true
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 10,
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+}

train/llama_factory_support/hy_dense_1_8b_full_sft.yaml

@@ -0,0 +1,50 @@
+### model
+model_name_or_path: path_to_dense_1_8b_model
+trust_remote_code: true
+
+### method
+stage: sft
+do_train: true
+finetuning_type: full
+deepspeed: ds_zero2_offload.json
+
+### dataset
+dataset_dir: .
+dataset: hy_dense_demo
+template: hy_dense_1_8b
+cutoff_len: 4096
+max_samples: 1000
+overwrite_cache: true
+
+### output
+output_dir: saves/hy_dense_1_8b/full/sft
+logging_steps: 1
+save_steps: 30
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 1.0e-5
+num_train_epochs: 1.0
+lr_scheduler_type: cosine_with_min_lr
+lr_scheduler_kwargs:
+  min_lr_rate: 0.1  # min_lr / learning_rate = 1e-6 / 1e-5 = 0.1
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: true
+ddp_timeout: 180000000
+flash_attn: fa2
+resume_from_checkpoint: null
+
+### eval
+# eval_dataset: alpaca_en_demo
+# val_size: 0.1
+# per_device_eval_batch_size: 1
+# eval_strategy: steps
+# eval_steps: 500

train/llama_factory_support/hy_dense_1_8b_lora_sft.yaml

@@ -0,0 +1,56 @@
+### model
+model_name_or_path: path_to_dense_1_8b_model
+trust_remote_code: true
+
+### method
+stage: sft
+do_train: true
+finetuning_type: lora
+deepspeed: ds_zero2_offload_lora.json
+
+### LoRA parameters
+lora_rank: 64
+lora_alpha: 128
+lora_dropout: 0.05
+lora_target: q_proj,k_proj,v_proj,o_proj
+
+### dataset
+dataset_dir: .
+dataset: hy_dense_demo
+template: hy_dense_1_8b
+cutoff_len: 4096
+max_samples: 1000
+overwrite_cache: true
+
+### output
+output_dir: saves/hy_dense_1_8b/lora/sft
+logging_steps: 1
+save_steps: 10
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 2.0e-4
+num_train_epochs: 1.0
+lr_scheduler_type: cosine_with_min_lr
+lr_scheduler_kwargs:
+  min_lr_rate: 0.1  # min_lr / learning_rate = 2e-5 / 2e-4 = 0.1
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: true
+ddp_timeout: 180000000
+flash_attn: fa2
+resume_from_checkpoint: null
+
+### eval
+# eval_dataset: alpaca_en_demo
+# val_size: 0.1
+# per_device_eval_batch_size: 1
+# eval_strategy: steps
+# eval_steps: 500

train/llama_factory_support/hy_dense_7b_full_sft.yaml

@@ -0,0 +1,50 @@
+### model
+model_name_or_path: path_to_dense_7b_model
+trust_remote_code: true
+
+### method
+stage: sft
+do_train: true
+finetuning_type: full
+deepspeed: ds_zero3_offload.json
+
+### dataset
+dataset_dir: .
+dataset: hy_dense_demo
+template: hy_dense_7b
+cutoff_len: 4096
+max_samples: 1000
+overwrite_cache: true
+
+### output
+output_dir: saves/hy_dense_7b/full/sft
+logging_steps: 1
+save_steps: 10
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 1.0e-5
+num_train_epochs: 1.0
+lr_scheduler_type: cosine_with_min_lr
+lr_scheduler_kwargs:
+  min_lr_rate: 0.1  # min_lr / learning_rate = 1e-6 / 1e-5 = 0.1
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: true
+ddp_timeout: 180000000
+flash_attn: fa2
+resume_from_checkpoint: null
+
+### eval
+# eval_dataset: alpaca_en_demo
+# val_size: 0.1
+# per_device_eval_batch_size: 1
+# eval_strategy: steps
+# eval_steps: 500

train/llama_factory_support/hy_dense_7b_lora_sft.yaml

@@ -0,0 +1,56 @@
+### model
+model_name_or_path: path_to_dense_7b_model
+trust_remote_code: true
+
+### method
+stage: sft
+do_train: true
+finetuning_type: lora
+deepspeed: ds_zero2_offload_lora.json
+
+### LoRA parameters
+lora_rank: 64
+lora_alpha: 128
+lora_dropout: 0.05
+lora_target: q_proj,k_proj,v_proj,o_proj
+
+### dataset
+dataset_dir: .
+dataset: hy_dense_demo
+template: hy_dense_7b
+cutoff_len: 4096
+max_samples: 1000
+overwrite_cache: true
+
+### output
+output_dir: saves/hy_dense_7b/lora/sft
+logging_steps: 1
+save_steps: 30
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 2.0e-4
+num_train_epochs: 1.0
+lr_scheduler_type: cosine_with_min_lr
+lr_scheduler_kwargs:
+  min_lr_rate: 0.1  # min_lr / learning_rate = 2e-5 / 2e-4 = 0.1
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: true
+ddp_timeout: 180000000
+flash_attn: fa2
+resume_from_checkpoint: null
+
+### eval
+# eval_dataset: alpaca_en_demo
+# val_size: 0.1
+# per_device_eval_batch_size: 1
+# eval_strategy: steps
+# eval_steps: 500

train/llama_factory_support/hy_dense_template.py

@@ -0,0 +1,82 @@
+"""
+HunYuan Dense model chat template registration for LLaMA Factory.
+
+Registers two templates:
+  - hy_dense_1_8b: for HunYuan Dense 1.8B model (and 0.5B/4B)
+  - hy_dense_7b:   for HunYuan Dense 7B model
+
+Usage:
+    1. Copy this file's register_template blocks into LLaMA Factory's
+       src/llamafactory/data/template.py  (for upstream MR).
+    2. Or import this module before training to register at runtime:
+       import hy_dense_template
+
+Note:
+    The existing LLaMA Factory built-in templates `hunyuan` and `hunyuan_small`
+    have subtle differences from the official chat_template.jinja files shipped
+    with the models. These new templates are designed to match the official
+    jinja templates exactly.
+"""
+
+from llamafactory.data.template import register_template
+from llamafactory.data.formatter import EmptyFormatter, StringFormatter
+
+# ---------------------------------------------------------------------------
+# Dense 1.8B chat template (also applies to 0.5B/4B)
+#
+# Token format (from dense_1_8b_0508/global_step_560/chat_template.jinja):
+#   BOS:        <｜hy_begin▁of▁sentence｜>
+#   System:     {system_content}<｜hy_place▁holder▁no▁3｜>
+#   User:       <｜hy_User｜>{user_content}
+#   Assistant:  <｜hy_Assistant｜>{assistant_content}<｜hy_place▁holder▁no▁2｜>
+#   Stop:       <｜hy_place▁holder▁no▁2｜>
+#
+# Key differences from LF built-in `hunyuan_small`:
+#   - User format: NO trailing <｜hy_place▁holder▁no▁8｜> after user content
+#   - Assistant format: HAS <｜hy_Assistant｜> prefix before assistant content
+#
+# The eos_token in tokenizer_config.json is <｜hy_place▁holder▁no▁2｜>,
+# so we use efficient_eos=True to let LF append it via {eos_token} slot.
+# ---------------------------------------------------------------------------
+
+register_template(
+    name="hy_dense_1_8b",
+    format_user=StringFormatter(slots=["<｜hy_User｜>{{content}}"]),
+    format_assistant=StringFormatter(slots=["<｜hy_Assistant｜>{{content}}", {"eos_token"}]),
+    format_system=StringFormatter(slots=["{{content}}<｜hy_place▁holder▁no▁3｜>"]),
+    format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
+    stop_words=["<｜hy_place▁holder▁no▁2｜>"],
+    efficient_eos=True,
+)
+
+
+# ---------------------------------------------------------------------------
+# Dense 7B chat template
+#
+# Token format (from dense_7b_0509/global_step_560/chat_template.jinja):
+#   BOS:        <|startoftext|>
+#   System:     {system_content}<|extra_4|>
+#   User:       {user_content}<|extra_0|>
+#   Assistant:  {assistant_content}<|eos|>
+#   Stop:       <|eos|>
+#
+# Key differences from LF built-in `hunyuan`:
+#   - Uses {bos_token} and {eos_token} slots for portability
+#   - efficient_eos=True to use tokenizer's eos_token
+#
+# Note on multi-turn: The official jinja adds <|startoftext|> before each
+# user message (except the first one when system is present). LLaMA Factory's
+# format_prefix only adds BOS once at the beginning. For single-turn training
+# this is correct. For multi-turn, there is a minor discrepancy (missing
+# <|startoftext|> before 2nd+ user turns), which is acceptable for fine-tuning.
+# ---------------------------------------------------------------------------
+
+register_template(
+    name="hy_dense_7b",
+    format_user=StringFormatter(slots=["{{content}}<|extra_0|>"]),
+    format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}]),
+    format_system=StringFormatter(slots=["{{content}}<|extra_4|>"]),
+    format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
+    stop_words=["<|eos|>"],
+    efficient_eos=True,
+)

train/llama_factory_support/hy_v3_full_sft.yaml

@@ -0,0 +1,50 @@
+### model
+model_name_or_path: ../hf
+trust_remote_code: true
+
+### method
+stage: sft
+do_train: true
+finetuning_type: full
+deepspeed: ds_zero3_offload.json
+
+### dataset
+dataset_dir: .
+dataset: hy_v3_demo
+template: hy_v3
+cutoff_len: 4096  # HYV3 supports 262k context length
+max_samples: 1000
+overwrite_cache: true
+
+### output
+output_dir: saves/hy_v3/full/sft
+logging_steps: 1
+save_steps: 10
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 1.0e-5
+num_train_epochs: 3.0
+lr_scheduler_type: cosine_with_min_lr
+lr_scheduler_kwargs:
+  min_lr_rate: 0.1  # min_lr / learning_rate = 1e-6 / 1e-5 = 0.1
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: true
+ddp_timeout: 180000000
+flash_attn: fa2
+resume_from_checkpoint: null
+
+### eval
+# eval_dataset: alpaca_en_demo
+# val_size: 0.1
+# per_device_eval_batch_size: 1
+# eval_strategy: steps
+# eval_steps: 500

train/llama_factory_support/hy_v3_lora_sft.yaml

@@ -0,0 +1,56 @@
+### model
+model_name_or_path: ../hf
+trust_remote_code: true
+
+### method
+stage: sft
+do_train: true
+finetuning_type: lora
+deepspeed: ds_zero2_offload_lora.json
+
+### LoRA parameters
+lora_rank: 64
+lora_alpha: 128
+lora_dropout: 0.05
+lora_target: q_proj,k_proj,v_proj,o_proj
+
+### dataset
+dataset_dir: .
+dataset: hy_v3_translation
+template: hy_v3
+cutoff_len: 4096  # Use shorter context for LoRA to save memory; increase if needed
+max_samples: 1000
+overwrite_cache: true
+
+### output
+output_dir: saves/hy_v3/lora/sft
+logging_steps: 10
+save_steps: 500
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 2.0e-4
+num_train_epochs: 3.0
+lr_scheduler_type: cosine_with_min_lr
+lr_scheduler_kwargs:
+  min_lr_rate: 0.1  # min_lr / learning_rate = 2e-5 / 2e-4 = 0.1
+warmup_ratio: 0.1
+bf16: true
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: true
+ddp_timeout: 180000000
+flash_attn: fa2
+resume_from_checkpoint: null
+
+### eval
+# eval_dataset: alpaca_en_demo
+# val_size: 0.1
+# per_device_eval_batch_size: 1
+# eval_strategy: steps
+# eval_steps: 500

train/llama_factory_support/hy_v3_patches.py

@@ -0,0 +1,215 @@
+"""
+HYV3 monkey-patches for LLaMA Factory + DeepSpeed training.
+
+This module applies all necessary runtime patches so that HYV3 (MoE)
+can be trained correctly under LLaMA Factory with DeepSpeed.
+
+Usage:
+    Import this module **before** calling `llamafactory-cli train`:
+
+        import hy_v3_patches          # applies patches on import
+        # ... then start training
+
+    Or add to the LLaMA Factory YAML via a custom entry-point wrapper.
+
+Patches applied:
+    1. (Removed) -- transformers 5.8.1+ has built-in conversion_mapping for
+       hy_v3 that handles key renaming + expert fusing automatically.
+    2. Router forward dtype fix (MoE router gate dtype alignment for ZeRO-3)
+    3. gradient_checkpointing   (use_reentrant=True for ZeRO-3)
+    4. Tokenizer file copy      (CustomSaveCallback)
+    5. (Removed) -- was per-expert ModuleList, now using native 3D Parameters
+    6. (Removed) -- transformers 5.8.1+ has built-in revert_weight_conversion
+       in save_pretrained that handles outer->inner format automatically.
+"""
+
+import os
+import logging
+import shutil
+from typing import Optional
+
+import torch
+import torch.nn.functional as _F
+
+logger = logging.getLogger(__name__)
+
+# ============================================================================
+# Patch 2: Router forward dtype alignment for ZeRO-3
+#
+# The HYV3 MoE HYV3TopKRouter.forward() calls F.linear with .float().
+# Under DeepSpeed ZeRO-3, F.linear is replaced by zero3_linear_wrap which
+# internally does input.matmul(weight.t()) WITHOUT aligning dtypes.
+# When ZeRO-3 stores the gate weight in bf16, the fp32 input causes a
+# dtype mismatch RuntimeError.
+#
+# Fix: monkey-patch HYV3TopKRouter.forward to cast input to
+# self.weight.dtype before F.linear, then cast the output back to float32.
+# ============================================================================
+
+_router_patch_applied = False
+
+def _apply_router_dtype_patch():
+    """Monkey-patch HYV3TopKRouter.forward to align gate input dtype with weight dtype."""
+    global _router_patch_applied
+    if _router_patch_applied:
+        return
+
+    try:
+        from transformers.models.hy_v3.modeling_hy_v3 import HYV3TopKRouter
+    except ImportError:
+        try:
+            from transformers.hy_v3.modeling_hy_v3 import HYV3TopKRouter
+        except ImportError:
+            logger.warning(
+                "Could not import HYV3TopKRouter; "
+                "router dtype patch NOT applied."
+            )
+            return
+
+    def _patched_router_forward(
+        self,
+        hidden_states: torch.Tensor,
+        e_score_correction_bias: torch.Tensor,
+    ) -> tuple:
+        hidden_states = hidden_states.reshape(-1, self.hidden_dim)
+        # Cast input to match weight dtype (bf16 under ZeRO-3)
+        # instead of hard-coding float32, to avoid matmul dtype mismatch.
+        weight_dtype = self.weight.dtype
+        router_logits = _F.linear(hidden_states.to(weight_dtype), self.weight.to(weight_dtype))
+        # Cast back to float32 for numerically stable sigmoid
+        router_logits = router_logits.to(torch.float32)
+        routing_weights = torch.sigmoid(router_logits)
+
+        scores_for_choice = routing_weights + e_score_correction_bias
+        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
+        top_k_weights = routing_weights.gather(1, top_k_index)
+
+        top_k_weights = top_k_weights / (top_k_weights.sum(dim=-1, keepdim=True) + 1e-20)
+        top_k_weights = top_k_weights * self.router_scaling_factor
+
+        return router_logits, top_k_weights, top_k_index
+
+    HYV3TopKRouter.forward = _patched_router_forward
+    _router_patch_applied = True
+    logger.info("HYV3 patch applied: HYV3TopKRouter.forward dtype alignment for ZeRO-3.")
+
+# ============================================================================
+# Patch 3: gradient_checkpointing use_reentrant=True
+#
+# PyTorch's torch.utils.checkpoint with use_reentrant=False (the default
+# in transformers) performs strict metadata checks on recomputed tensors.
+# Under ZeRO-3, parameters are all-gathered during the first forward pass
+# but may be partitioned back when the checkpoint recomputes, causing a
+# CheckpointError.  Setting use_reentrant=True avoids this.
+#
+# This is applied via a Trainer callback that modifies training_args
+# before training starts.
+# ============================================================================
+
+# ============================================================================
+# Patch 4: Tokenizer file copy callback
+#
+# Ensures each checkpoint directory is self-contained for inference by
+# copying all tokenizer-related files from the original tokenizer path.
+# ============================================================================
+
+# Tokenizer files that should be copied to each checkpoint
+_TOKENIZER_FILES = [
+    "generation_config.json",
+    "hy.tiktoken",
+    "tokenizer_config.json",
+    "tokenization_hy.py",
+    "tokenizer.json",
+    "special_tokens_map.json",
+    "chat_template.jinja",
+]
+
+def _copy_tokenizer_to_checkpoint(tokenizer_dir: str, checkpoint_dir: str):
+    """Copy tokenizer files from tokenizer_dir to checkpoint_dir."""
+    for fname in _TOKENIZER_FILES:
+        src = os.path.join(tokenizer_dir, fname)
+        if os.path.isfile(src):
+            shutil.copy(src, os.path.join(checkpoint_dir, fname))
+
+# ============================================================================
+# LLaMA Factory Callback: integrates patches 3, 4 into the training loop
+# ============================================================================
+
+try:
+    from transformers import TrainerCallback
+    from transformers.trainer_utils import PREFIX_CHECKPOINT_DIR
+
+    class HYV3PatchCallback(TrainerCallback):
+        """
+        LLaMA Factory compatible callback that applies HYV3-specific patches.
+
+        Add to your YAML or pass to Trainer:
+            callbacks: [hy_v3_patches.HYV3PatchCallback]
+        """
+
+        def __init__(self, tokenizer_dir: Optional[str] = None):
+            """
+            Args:
+                tokenizer_dir: Path to the original tokenizer directory.
+                    If None, will try to use model_name_or_path from training args.
+            """
+            self._tokenizer_dir = tokenizer_dir
+
+        def on_train_begin(self, args, state, control, **kwargs):
+            # --- Patch 3: gradient_checkpointing use_reentrant ---
+            if getattr(args, "gradient_checkpointing", False) and getattr(args, "deepspeed", None):
+                if not hasattr(args, "gradient_checkpointing_kwargs") or not args.gradient_checkpointing_kwargs:
+                    args.gradient_checkpointing_kwargs = {"use_reentrant": True}
+                elif "use_reentrant" not in args.gradient_checkpointing_kwargs:
+                    args.gradient_checkpointing_kwargs["use_reentrant"] = True
+                logger.info("HYV3 patch applied: gradient_checkpointing use_reentrant=True.")
+
+            return control
+
+        def on_save(self, args, state, control, **kwargs):
+            # --- Patch 4: Copy tokenizer files ---
+            if torch.distributed.is_initialized() and torch.distributed.get_rank() != 0:
+                return control
+
+            checkpoint_dir = os.path.join(
+                args.output_dir,
+                f"{PREFIX_CHECKPOINT_DIR}-{state.global_step}",
+            )
+
+            # Determine tokenizer directory
+            tokenizer_dir = self._tokenizer_dir
+            if tokenizer_dir is None:
+                # Try common locations
+                tokenizer_dir = getattr(args, "tokenizer_name_or_path", None)
+                if tokenizer_dir is None:
+                    tokenizer_dir = getattr(args, "model_name_or_path", None)
+
+            if tokenizer_dir and os.path.isdir(tokenizer_dir):
+                _copy_tokenizer_to_checkpoint(tokenizer_dir, checkpoint_dir)
+                logger.info(
+                    "HYV3: Copied tokenizer files from %s to %s",
+                    tokenizer_dir, checkpoint_dir
+                )
+
+            return control
+
+except ImportError:
+    logger.warning(
+        "transformers not available; HYV3PatchCallback not defined."
+    )
+
+# ============================================================================
+# Auto-apply patches on import
+# ============================================================================
+
+# Patch 2: Router dtype fix
+_apply_router_dtype_patch()
+
+# Patches 3, 4 are applied via HYV3PatchCallback during training.
+# Users should add HYV3PatchCallback to their Trainer callbacks.
+
+logger.info(
+    "HYV3 patches module loaded. Patch 2 (Router dtype fix) applied. "
+    "Remember to add HYV3PatchCallback to your Trainer callbacks "
+    "for gradient_checkpointing and tokenizer copy support."
+)

train/llama_factory_support/hy_v3_template.py

@@ -0,0 +1,46 @@
+"""
+HYV3 chat template registration for LLaMA Factory.
+
+Usage:
+    1. Copy this file's register_template block into LLaMA Factory's
+       src/llamafactory/data/template.py  (for upstream MR).
+    2. Or import this module before training to register at runtime:
+       import hy_v3_template
+"""
+
+from llamafactory.data.template import ReasoningTemplate, register_template
+from llamafactory.data.formatter import EmptyFormatter, StringFormatter
+
+
+# ---------------------------------------------------------------------------
+# HYV3 (MoE, pure text) chat template
+#
+# Token format (from chat_template.jinja & tokenizer_config.json):
+#   BOS:        <｜hy_begin▁of▁sentence｜>
+#   System:    {system_content}                (directly after BOS, no role tag)
+#   User:       <｜hy_User｜>{user_content}
+#   Assistant:  <｜hy_Assistant｜>{assistant_content}<｜hy_eos｜>
+#   EOS:       <｜hy_eos｜>
+#
+# Loss mask: only compute loss on assistant content (including <｜hy_eos｜>).
+#
+# Note: The system message has NO explicit role token -- it is placed right
+# after BOS.  The eos_token is <｜hy_eos｜>.
+#
+# Reasoning: Supports think tags via ReasoningTemplate.
+#   - thought_words: ("<think>", "</think>") matching jinja template
+#   - enable_thinking: set globally via data_args.enable_thinking (default True)
+#   - Training data always includes think tags (empty or with content)
+# ---------------------------------------------------------------------------
+
+register_template(
+    name="hy_v3",
+    template_class=ReasoningTemplate,
+    format_user=StringFormatter(slots=["<｜hy_User｜>{{content}}"]),
+    format_assistant=StringFormatter(slots=["<｜hy_Assistant｜>{{content}}", {"eos_token"}]),
+    format_system=StringFormatter(slots=["{{content}}"]),
+    format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
+    thought_words=("<think>", "</think>"),
+    stop_words=["<｜hy_eos｜>"],
+    efficient_eos=True,
+)

train/llama_factory_support/train_hy_dense.py

@@ -0,0 +1,81 @@
+"""
+LLaMA Factory training entry-point wrapper for HunYuan Dense models.
+
+This script:
+  1. Registers the hy_dense_1_8b and hy_dense_7b chat templates
+  2. Injects a lightweight PatchCallback (tokenizer copy + gradient checkpointing fix)
+  3. Calls run_exp() to start LLaMA Factory training
+
+How it works:
+  - train_lf_dense.sh launches this script via torchrun directly:
+        torchrun ... train_hy_dense.py hy_dense_1_8b_full_sft.yaml
+  - Each torchrun worker executes this script, so all patches are applied
+    in every worker process before training begins.
+  - We call run_exp() directly (not the CLI launcher) to avoid the
+    launcher re-spawning workers and losing our patches.
+
+Note:
+  Dense models do NOT need MoE-specific patches (router dtype fix, expert
+  key rename, etc.). Only the tokenizer copy callback and gradient
+  checkpointing fix are needed.
+
+Usage:
+    # Via launch script (recommended):
+    bash train_lf_dense.sh
+
+    # Direct single-node (1 GPU, 1.8B model):
+    torchrun --nproc_per_node 1 train_hy_dense.py hy_dense_1_8b_full_sft.yaml
+
+    # Direct single-node (2 GPUs, 7B model):
+    torchrun --nproc_per_node 2 train_hy_dense.py hy_dense_7b_full_sft.yaml
+"""
+
+import sys
+import os
+
+# Add current directory to path so templates can be imported
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+# Step 1: Register Dense model templates (must be before training starts)
+import hy_dense_template  # noqa: F401
+
+# Step 2: Import the patch callback (reuse HYV3PatchCallback for tokenizer copy)
+# The MoE router patch will be silently skipped since Dense models don't have
+# HYV3TopKRouter. Only Patch 3 (gradient_checkpointing) and Patch 4 (tokenizer
+# copy) will be effective.
+import hy_v3_patches  # noqa: F401
+
+# Step 3: Inject PatchCallback into LLaMA Factory's training flow
+from llamafactory.train.sft.workflow import run_sft as _orig_run_sft
+
+
+def _patched_run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks=None):
+    """Wrap run_sft to inject HYV3PatchCallback for tokenizer copy."""
+    if callbacks is None:
+        callbacks = []
+
+    # Determine tokenizer directory for the save callback
+    tokenizer_dir = getattr(model_args, "model_name_or_path", None)
+    callbacks.append(hy_v3_patches.HYV3PatchCallback(tokenizer_dir=tokenizer_dir))
+
+    return _orig_run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks=callbacks)
+
+
+# Monkey-patch the SFT workflow
+import llamafactory.train.sft.workflow as _sft_wf
+_sft_wf.run_sft = _patched_run_sft
+
+
+def main():
+    """Entry point: called by torchrun in each worker process.
+
+    Since train_lf_dense.sh launches us via torchrun directly, all patches
+    (template registration, tokenizer copy callback injection) are already
+    applied in this process. We just call run_exp() to start training.
+    """
+    from llamafactory.train.tuner import run_exp
+    run_exp()
+
+
+if __name__ == "__main__":
+    main()

train/llama_factory_support/train_hy_v3.py

@@ -0,0 +1,72 @@
+"""
+LLaMA Factory training entry-point wrapper for HYV3.
+
+This script:
+  1. Registers the hy_v3 chat template
+  2. Applies all HYV3 monkey-patches (checkpoint key rename, dtype fix, etc.)
+  3. Injects HYV3PatchCallback into the training loop
+  4. Calls run_exp() to start LLaMA Factory training
+
+How it works:
+  - train_lf.sh launches this script via torchrun directly:
+        torchrun ... train_hy_v3.py hy_v3_full_sft.yaml
+  - Each torchrun worker executes this script, so all patches are applied
+    in every worker process before training begins.
+  - We call run_exp() directly (not the CLI launcher) to avoid the
+    launcher re-spawning workers and losing our patches.
+
+Usage:
+    # Via launch script (recommended):
+    bash train_lf.sh
+
+    # Direct single-node (8 GPUs):
+    torchrun --nproc_per_node 8 train_hy_v3.py hy_v3_full_sft.yaml
+"""
+
+import sys
+import os
+
+# Add current directory to path so patches can be imported
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+# Step 1: Register HYV3 template (must be before training starts)
+import hy_v3_template  # noqa: F401
+
+# Step 2: Apply checkpoint key rename patch (must be before model loading)
+import hy_v3_patches  # noqa: F401
+
+# Step 3: Inject HYV3PatchCallback into LLaMA Factory's training flow
+from llamafactory.train.sft.workflow import run_sft as _orig_run_sft
+
+
+def _patched_run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks=None):
+    """Wrap run_sft to inject HYV3PatchCallback."""
+    if callbacks is None:
+        callbacks = []
+
+    # Determine tokenizer directory for the save callback
+    tokenizer_dir = getattr(model_args, "model_name_or_path", None)
+    callbacks.append(hy_v3_patches.HYV3PatchCallback(tokenizer_dir=tokenizer_dir))
+
+    return _orig_run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks=callbacks)
+
+
+# Monkey-patch the SFT workflow
+import llamafactory.train.sft.workflow as _sft_wf
+_sft_wf.run_sft = _patched_run_sft
+
+
+def main():
+    """Entry point: called by torchrun in each worker process.
+
+    Since train_lf.sh launches us via torchrun directly, all patches
+    (template registration, checkpoint key rename, SFT callback injection)
+    are already applied in this process.  We just call run_exp() to start
+    training — no need to go through the CLI launcher.
+    """
+    from llamafactory.train.tuner import run_exp
+    run_exp()
+
+
+if __name__ == "__main__":
+    main()