dann-od commited on
Commit
a08d0c6
·
verified ·
1 Parent(s): 95959d3

Flipped accuracy table for better visualization on HF

Browse files
Files changed (1) hide show
  1. README.md +17 -4
README.md CHANGED
@@ -125,10 +125,23 @@ Latency measured with TensorRT + `trtexec`, GPU compute time only
125
  Evaluated on the sts17 validation split. The quantized model
126
  retains nearly all of the FP32 accuracy with a small tolerance.
127
 
128
- | Model | Spearman ρ | spearman_ar_ar | spearman_default | spearman_en_ar | spearman_en_de | spearman_en_en | spearman_en_tr | spearman_es_en | spearman_es_es | spearman_fr_en | spearman_it_en | spearman_ko_ko | spearman_nl_en |
129
- |---|---|---|---|---|---|---|---|---|---|---|---|---|---|
130
- | `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` FP32 (ours) | 0.8130 | 0.7915 | 0.7970 | 0.8122 | 0.8422 | 0.8687 | 0.7674 | 0.8444 | 0.8556 | 0.7659 | 0.8235 | 0.7703 | 0.8171 |
131
- | **Embedl Paraphrase Multilingual Minilm L12 V2 INT8** | **0.8008** | **0.7906** | **0.7868** | **0.7914** | **0.8215** | **0.8638** | **0.7555** | **0.8300** | **0.8328** | **0.7536** | **0.8148** | **0.7628** | **0.8059** |
 
 
 
 
 
 
 
 
 
 
 
 
 
132
 
133
  ## Creating Your Own Optimized Models
134
 
 
125
  Evaluated on the sts17 validation split. The quantized model
126
  retains nearly all of the FP32 accuracy with a small tolerance.
127
 
128
+ | Metric | FP32 (ours) | **Embedl Paraphrase Multilingual Minilm L12 V2 INT8** | Δ |
129
+ |---|---|---|---|
130
+ | Spearman ρ | 0.8130 | **0.8008** | -0.0122 |
131
+ | ρ (ar-ar) | 0.7915 | **0.7906** | -0.0010 |
132
+ | ρ (default) | 0.7970 | **0.7868** | -0.0102 |
133
+ | ρ (en-ar) | 0.8122 | **0.7914** | -0.0208 |
134
+ | ρ (en-de) | 0.8422 | **0.8215** | -0.0207 |
135
+ | ρ (en-en) | 0.8687 | **0.8638** | -0.0049 |
136
+ | ρ (en-tr) | 0.7674 | **0.7555** | -0.0119 |
137
+ | ρ (es-en) | 0.8444 | **0.8300** | -0.0143 |
138
+ | ρ (es-es) | 0.8556 | **0.8328** | -0.0228 |
139
+ | ρ (fr-en) | 0.7659 | **0.7536** | -0.0123 |
140
+ | ρ (it-en) | 0.8235 | **0.8148** | -0.0087 |
141
+ | ρ (ko-ko) | 0.7703 | **0.7628** | -0.0075 |
142
+ | ρ (nl-en) | 0.8171 | **0.8059** | -0.0112 |
143
+
144
+ FP32 baseline: [`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2).
145
 
146
  ## Creating Your Own Optimized Models
147