Update benchmarks for v2 (joint cross-modal training)

README.md

@@ -67,35 +67,35 @@ All benchmarks run on a single NVIDIA L4 GPU with 5K samples where applicable.
 
 | Direction | TEG-421M (421M) | LCO-3B (4.7B) | Nemotron-3B (4.7B) | ImageBind (1.2B) | EBind |
 |---|---|---|---|---|---|
-| Text → Image R@1 | 0.687 | 0.660 | 0.529 | 0.712 | **0.779** |
-| Image → Text R@1 | 0.624 | 0.564 | 0.299 | 0.736 | **0.783** |
-| Text → Audio R@1 | **0.117** | 0.042 | 0.018 | 0.038 | 0.047 |
-| Audio → Text R@1 | **0.104** | 0.032 | 0.010 | 0.039 | 0.035 |
-| Audio → Image R@1 | **0.059** | 0.027 | 0.016 | 0.023 | 0.027 |
-| Image → Audio R@1 | **0.057** | 0.034 | 0.018 | 0.025 | 0.032 |
+| Text → Image R@1 | 0.672 | 0.660 | 0.529 | 0.712 | **0.779** |
+| Image → Text R@1 | 0.620 | 0.564 | 0.299 | 0.736 | **0.783** |
+| Text → Audio R@1 | **0.113** | 0.042 | 0.018 | 0.038 | 0.047 |
+| Audio → Text R@1 | **0.115** | 0.032 | 0.010 | 0.039 | 0.035 |
+| Audio → Image R@1 | **0.081** | 0.027 | 0.016 | 0.023 | 0.027 |
+| Image → Audio R@1 | **0.083** | 0.034 | 0.018 | 0.025 | 0.032 |
 
-TEG leads all audio cross-modal directions by 2-10x over models that are 3-11x larger. Vision-text trails EBind/ImageBind but uses encoders small enough for edge deployment.
+TEG leads all audio cross-modal directions by 2-10x over models that are 3-11x larger. Image↔Audio improved ~40% over v1 via joint cross-modal training. Vision-text trails EBind/ImageBind but uses encoders small enough for edge deployment.
 
 ### Audio retrieval — AudioCaps & Clotho
 
 | Benchmark | Direction | TEG-421M | LCO-3B | Nemotron-3B | CLAP-Small | CLAP-Large | ImageBind | EBind |
 |---|---|---|---|---|---|---|---|---|
-| AudioCaps | A→T R@1 | 0.156 | 0.250 | 0.050 | **0.425** | 0.420 | 0.116 | 0.225 |
-| AudioCaps | T→A R@1 | 0.145 | 0.215 | 0.075 | **0.315** | 0.280 | 0.080 | 0.219 |
-| Clotho | A→T R@1 | 0.159 | 0.178 | 0.038 | 0.166 | **0.195** | 0.061 | 0.088 |
-| Clotho | T→A R@1 | 0.125 | **0.187** | 0.070 | 0.159 | 0.167 | 0.074 | 0.118 |
+| AudioCaps | A→T R@1 | 0.159 | 0.250 | 0.050 | **0.425** | 0.420 | 0.116 | 0.225 |
+| AudioCaps | T→A R@1 | 0.149 | 0.215 | 0.075 | **0.315** | 0.280 | 0.080 | 0.219 |
+| Clotho | A→T R@1 | 0.168 | 0.178 | 0.038 | 0.166 | **0.195** | 0.061 | 0.088 |
+| Clotho | T→A R@1 | 0.123 | **0.187** | 0.070 | 0.159 | 0.167 | 0.074 | 0.118 |
 
 CLAP models lead on audio-only benchmarks (audio specialists with no image support). Among trimodal models, TEG is competitive with LCO while being 11x smaller.
 
 ### Image-text retrieval — MSCOCO & Flickr30k
 
-| Benchmark | Direction | TEG-421M (421M) | EBind (~200M) | ImageBind (1.2B) | LCO-3B (4.7B) | Nemotron-3B (4.7B) |
+| Benchmark | Direction | TEG-421M (421M) | EBind (1.78B*) | ImageBind (1.2B) | LCO-3B (4.7B) | Nemotron-3B (4.7B) |
 |---|---|---|---|---|---|---|
-| MSCOCO 5K | I→T R@1 | 0.251 | **0.743** | 0.658 | 0.533 | 0.225 |
-| MSCOCO 5K | T→I R@1 | 0.181 | **0.559** | 0.490 | 0.469 | 0.334 |
+| MSCOCO 5K | I→T R@1 | 0.248 | **0.743** | 0.658 | 0.533 | 0.225 |
+| MSCOCO 5K | T→I R@1 | 0.180 | **0.559** | 0.490 | 0.469 | 0.334 |
 | MSCOCO 5K | I→T R@10 | 0.622 | **0.948** | 0.918 | 0.784 | 0.630 |
-| Flickr30k | I→T R@1 | 0.481 | — | — | **0.840** | 0.419 |
-| Flickr30k | T→I R@1 | 0.375 | — | — | **0.765** | 0.563 |
+| Flickr30k | I→T R@1 | 0.498 | — | — | **0.840** | 0.419 |
+| Flickr30k | T→I R@1 | 0.358 | — | — | **0.765** | 0.563 |
 
 TEG's image-text retrieval trades accuracy for edge deployability — MobileNetV4-Medium is ~100x smaller than the ViT-H/ViT-L encoders used by competitors. On MSCOCO, TEG outperforms Nemotron-3B on I→T despite being 11x smaller.
 
@@ -105,8 +105,8 @@ TEG's image-text retrieval trades accuracy for edge deployability — MobileNetV
 |---|---|---|
 | CLAP-Large | 67.8M | **0.905** |
 | LCO-3B | 4.7B | 0.853 |
-| TEG-421M | 421M | 0.829 |
-| EBind | ~200M | 0.770 |
+| TEG-421M | 421M | 0.820 |
+| EBind | 1.78B* | 0.770 |
 | CLAP-Small | 27.5M | 0.751 |
 | Nemotron-3B | 4.7B | 0.727 |
 | ImageBind | 1.2B | 0.664 |
@@ -187,6 +187,8 @@ teg-421m.safetensors     # All components in one file (~1 GB)
 - **Matryoshka weighting**: Higher weight on smaller dimensions (4x at 128-dim) ensures quality at aggressive truncation levels
 - **Edge-first**: Source encoders chosen for edge deployment — MobileNetV4-Medium and EfficientAT mn20 can run on devices like Raspberry Pi 5
 
+*\*EBind's [HuggingFace checkpoint](https://huggingface.co/encord-team/ebind-full) is 8.93M parameters (bridge heads only), but inference requires frozen backbones (SigLIP ViT-L, CLAP HTSAT, text encoder) totaling 1.78B loaded parameters as measured by our benchmark harness.*
+
 ## Limitations
 
 - Audio retrieval lags behind specialist models like CLAP on audio-only benchmarks