Update README.md
Browse files
README.md
CHANGED
|
@@ -5,7 +5,7 @@ language:
|
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
library_name: transformers
|
| 7 |
---
|
| 8 |
-
# Boldt-
|
| 9 |
|
| 10 |
<img src="logo.png" width="500">
|
| 11 |
|
|
@@ -43,7 +43,7 @@ outputs = model.generate(**inputs, max_new_tokens=64)
|
|
| 43 |
## Evaluation
|
| 44 |
|
| 45 |
We evaluate Boldt-350M on our [modernized German benchmark suite](https://huggingface.co/collections/Boldt/german-llm-benchmarks). It comprises the German subset of [Global MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU) and updated translations of widely used English benchmarks, produced using [Tower+ 72B](https://huggingface.co/Unbabel/Tower-Plus-72B) to address issues we identified in existing German benchmark translations.
|
| 46 |
-
Despite being trained on a substantially smaller amount of data, Boldt-
|
| 47 |
|
| 48 |
| Category | Model | Tokens | MMLU | ARC-C | ARC-E | H-Swag | LAMBADA | OBQA | Avg. |
|
| 49 |
|----------|--------|--------|------|-------|-------|--------|----------|------|------|
|
|
|
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
library_name: transformers
|
| 7 |
---
|
| 8 |
+
# Boldt-350M
|
| 9 |
|
| 10 |
<img src="logo.png" width="500">
|
| 11 |
|
|
|
|
| 43 |
## Evaluation
|
| 44 |
|
| 45 |
We evaluate Boldt-350M on our [modernized German benchmark suite](https://huggingface.co/collections/Boldt/german-llm-benchmarks). It comprises the German subset of [Global MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU) and updated translations of widely used English benchmarks, produced using [Tower+ 72B](https://huggingface.co/Unbabel/Tower-Plus-72B) to address issues we identified in existing German benchmark translations.
|
| 46 |
+
Despite being trained on a substantially smaller amount of data, Boldt-350M outperforms other similarly-sized SLMs capable of German on our evaluation suite. It also performs competitively with larger-sized (around 1B) German and multilingual models.
|
| 47 |
|
| 48 |
| Category | Model | Tokens | MMLU | ARC-C | ARC-E | H-Swag | LAMBADA | OBQA | Avg. |
|
| 49 |
|----------|--------|--------|------|-------|-------|--------|----------|------|------|
|