Update README.md
Browse files
README.md
CHANGED
|
@@ -18,10 +18,10 @@ pipeline_tag: text-generation
|
|
| 18 |
|
| 19 |
# 🛡️ HEBATRON: Hebrew-Specialized Mamba2-MoE
|
| 20 |
|
| 21 |
-
HEBATRON is a state-of-the-art, high-performance language model specialized for the Hebrew language
|
| 22 |
|
| 23 |
## 🚀 Model Summary
|
| 24 |
-
HEBATRON is designed to handle the structural and morphological complexities of Hebrew while providing linear scaling for long-context tasks
|
| 25 |
|
| 26 |
---
|
| 27 |
|
|
@@ -29,51 +29,55 @@ HEBATRON is designed to handle the structural and morphological complexities of
|
|
| 29 |
|
| 30 |
| Feature | Specification |
|
| 31 |
| :--- | :--- |
|
| 32 |
-
| **Model Name** | HEBATRON
|
| 33 |
-
| **Architecture** | Hybrid **Mamba2** (SSM) + **Sparse MoE**
|
| 34 |
-
| **Total Parameters** | 30B
|
| 35 |
-
| **Active Parameters** | ~3B per token
|
| 36 |
-
| **Context Window** | 65,536 (64k) tokens
|
| 37 |
-
| **Hardware** | NVIDIA Blackwell (B300) & H200 GPUs
|
| 38 |
-
| **Precision** | FP8 Mixed-Precision
|
| 39 |
|
| 40 |
---
|
| 41 |
|
| 42 |
## 🧬 Training Curriculum
|
| 43 |
-
The model was trained using a three-phase **Curriculum Learning** strategy
|
| 44 |
|
| 45 |
1. **Phase 1: Formal Foundation (75.5B tokens)**
|
| 46 |
-
Focused on high-quality, structured Hebrew (legal, academic, and literary texts) to establish core grammatical rules
|
| 47 |
2. **Phase 2: Colloquial Expansion (3.36B tokens)**
|
| 48 |
-
Integration of social media, forums, and informal web data to handle slang and modern registers
|
| 49 |
3. **Phase 3: Long-Context Extension (20.4B tokens)**
|
| 50 |
-
Fine-tuning on dense, long-form documents to stabilize the 64k context window
|
| 51 |
|
| 52 |
-
> **Alignment:** Supervised Fine-Tuning (SFT) was performed on **2 million samples**, including localized knowledge distillation and the **"Hebrew IFEval"** dataset
|
| 53 |
|
| 54 |
---
|
| 55 |
|
| 56 |
## 📊 Performance Evaluation
|
| 57 |
|
| 58 |
### Hebrew Reasoning Benchmarks
|
| 59 |
-
* **SNLI (Semantic Reasoning):** 91.2% accuracy
|
| 60 |
-
* **Israeli Trivia:** 72.1% (+14pt vs base)
|
| 61 |
-
* **Hebrew Average Reasoning:** 73.8% (Surpassing DictaLM-3.0-Thinking)
|
| 62 |
-
* **GSM8K (Math):** 83.3% accuracy in native Hebrew
|
| 63 |
|
| 64 |
### English Reasoning Benchmarks
|
| 65 |
-
* **Psychometric Psi (EN):** 91.6%
|
| 66 |
-
* **English Reasoning Average:** 86.0%
|
| 67 |
|
| 68 |
---
|
| 69 |
|
| 70 |
## 🎯 Intended Use & Limitations
|
| 71 |
-
* **Intended Use:** Advanced Hebrew document analysis, long-context summarization (legal/technical), and complex bilingual reasoning
|
| 72 |
-
* **Limitations:** Users should verify outputs for factual accuracy as with any Large Language Model
|
| 73 |
|
| 74 |
---
|
| 75 |
|
| 76 |
## 🤝 Credits
|
| 77 |
-
* **Developed by:** PwC Israel & MAFAT
|
| 78 |
-
* **
|
| 79 |
-
* **
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
# 🛡️ HEBATRON: Hebrew-Specialized Mamba2-MoE
|
| 20 |
|
| 21 |
+
HEBATRON is a state-of-the-art, high-performance language model specialized for the Hebrew language. Developed through a collaboration between **PwC Israel** and **MAFAT**, it introduces a unique hybrid architecture combining **Mamba2** and **Mixture-of-Experts (MoE)**.
|
| 22 |
|
| 23 |
## 🚀 Model Summary
|
| 24 |
+
HEBATRON is designed to handle the structural and morphological complexities of Hebrew while providing linear scaling for long-context tasks. It is a localized and enhanced version of the **Nemotron-3-Nano-30B** framework, optimized for native-level reasoning in Hebrew and English.
|
| 25 |
|
| 26 |
---
|
| 27 |
|
|
|
|
| 29 |
|
| 30 |
| Feature | Specification |
|
| 31 |
| :--- | :--- |
|
| 32 |
+
| **Model Name** | HEBATRON |
|
| 33 |
+
| **Architecture** | Hybrid **Mamba2** (SSM) + **Sparse MoE** |
|
| 34 |
+
| **Total Parameters** | 30B |
|
| 35 |
+
| **Active Parameters** | ~3B per token |
|
| 36 |
+
| **Context Window** | 65,536 (64k) tokens |
|
| 37 |
+
| **Hardware** | NVIDIA Blackwell (B300) & H200 GPUs |
|
| 38 |
+
| **Precision** | FP8 Mixed-Precision |
|
| 39 |
|
| 40 |
---
|
| 41 |
|
| 42 |
## 🧬 Training Curriculum
|
| 43 |
+
The model was trained using a three-phase **Curriculum Learning** strategy:
|
| 44 |
|
| 45 |
1. **Phase 1: Formal Foundation (75.5B tokens)**
|
| 46 |
+
Focused on high-quality, structured Hebrew (legal, academic, and literary texts) to establish core grammatical rules.
|
| 47 |
2. **Phase 2: Colloquial Expansion (3.36B tokens)**
|
| 48 |
+
Integration of social media, forums, and informal web data to handle slang and modern registers.
|
| 49 |
3. **Phase 3: Long-Context Extension (20.4B tokens)**
|
| 50 |
+
Fine-tuning on dense, long-form documents to stabilize the 64k context window.
|
| 51 |
|
| 52 |
+
> **Alignment:** Supervised Fine-Tuning (SFT) was performed on **2 million samples**, including localized knowledge distillation and the **"Hebrew IFEval"** dataset.
|
| 53 |
|
| 54 |
---
|
| 55 |
|
| 56 |
## 📊 Performance Evaluation
|
| 57 |
|
| 58 |
### Hebrew Reasoning Benchmarks
|
| 59 |
+
* **SNLI (Semantic Reasoning):** 91.2% accuracy
|
| 60 |
+
* **Israeli Trivia:** 72.1% (+14pt vs base)
|
| 61 |
+
* **Hebrew Average Reasoning:** 73.8% (Surpassing DictaLM-3.0-Thinking)
|
| 62 |
+
* **GSM8K (Math):** 83.3% accuracy in native Hebrew
|
| 63 |
|
| 64 |
### English Reasoning Benchmarks
|
| 65 |
+
* **Psychometric Psi (EN):** 91.6%
|
| 66 |
+
* **English Reasoning Average:** 86.0%
|
| 67 |
|
| 68 |
---
|
| 69 |
|
| 70 |
## 🎯 Intended Use & Limitations
|
| 71 |
+
* **Intended Use:** Advanced Hebrew document analysis, long-context summarization (legal/technical), and complex bilingual reasoning.
|
| 72 |
+
* **Limitations:** Users should verify outputs for factual accuracy as with any Large Language Model.
|
| 73 |
|
| 74 |
---
|
| 75 |
|
| 76 |
## 🤝 Credits
|
| 77 |
+
* **Developed by:** PwC Israel & MAFAT
|
| 78 |
+
* **MAFAT Lead:** Tal Geva [project Lead], Matan Frank
|
| 79 |
+
* **Technical Lead:** Sarel Weinberger (PwC Next)
|
| 80 |
+
* **PwC Israel Team:** Noam Kaiser, Uri Bar Joseph, Smadar Arbatz, Or Levi, Dan Revital, Omer Baruch (PwC Next)
|
| 81 |
+
* **MAFAT Team:** Noam Ordan
|
| 82 |
+
* **Partners:** Amir Nissan Hacohen (Origin.ai)
|
| 83 |
+
* **Research Collaborators:** Shaltiel Shmidman (Dicta)
|