Initial release: FormBench TAPT-MNRL model (anonymised for review)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 1024,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": false,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": true,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,123 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-Embedding-0.6B
+library_name: sentence-transformers
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- formbench
+- patent-retrieval
+- chemistry
+- formulations
+- materials-science
+language:
+- en
+pipeline_tag: sentence-similarity
+---
+
+# qwen3-embed-formbench-mnrl
+
+A domain-adapted sentence-transformers model derived from
+[`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) and fine-tuned on the **FormBench**
+retrieval benchmark for formulation chemistry. It maps passages from formulation patents
+into a 1024-dimensional dense vector space and is optimised for within-domain
+retrieval among structurally similar near-miss passages — the central capability targeted
+by FormBench.
+
+This repository hosts an anonymised release for NeurIPS 2026 double-blind review.
+
+## Model details
+
+| Item | Value |
+|---|---|
+| Base model | `Qwen/Qwen3-Embedding-0.6B` (600M params) |
+| Training method | Task-adaptive pre-training (TAPT) via contrastive fine-tuning |
+| Loss | `MultipleNegativesRankingLoss` (in-batch negatives) |
+| Training data | FormBench-Triplets — 44,413 (query, anchor, hard-negative) tuples |
+| Embedding dimension | 1024 |
+| Max sequence length | 8192 (training: 2048) |
+| Precision | bf16 |
+| Learning rate | 1e-5 |
+| Per-GPU batch size | 32 |
+| Epochs | 5 |
+| Hardware | 8× AMD MI250X, DDP |
+
+The training-triplet set is reconstructable from the qrel files in
+[`Formbench-anon/FormBench`](https://huggingface.co/datasets/Formbench-anon/FormBench)
+following the protocol in §3 of the paper.
+
+## Evaluation results
+
+Evaluated on the FormBench test split (n = 5,459 queries) under both corpus variants,
+following the protocol in §4 of the paper. FAISS exact inner-product search at top-k = 100.
+
+### FormBench-Structured (C1) — within-domain near-miss distractors
+
+| Metric | Value |
+|---|---:|
+| Binary nDCG@10 | **0.4085** |
+| MRR (binary qrels) | 0.3613 |
+| Graded nDCG@10 | 0.2374 |
+| R@100 (binary qrels) | 0.8181 |
+| FAISS search latency | 18.8 ms/query |
+
+### FormBench-Random (C0) — random-distractor corpus
+
+| Metric | Value |
+|---|---:|
+| Binary nDCG@10 | **0.4835** |
+| MRR (binary qrels) | 0.4404 |
+| Graded nDCG@10 | 0.2835 |
+| R@100 (binary qrels) | 0.8525 |
+| FAISS search latency | 18.8 ms/query |
+
+For reference: BM25 lexical baseline: binary nDCG@10 = 0.3751 (C1), 0.4665 (C0).
+
+## Usage
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("Formbench-anon/qwen3-embed-formbench-mnrl")
+
+passages = [
+    "An adhesive composition comprising a styrene-acrylate copolymer ...",
+    "A water-based latex paint formulation containing ...",
+]
+queries = [
+    "what wax-seeded latex polymers improve scuff resistance in architectural coatings?",
+]
+
+passage_embeds = model.encode(passages, normalize_embeddings=True)
+query_embeds   = model.encode(queries,  normalize_embeddings=True)
+```
+
+## Intended use
+
+Domain-specific retrieval over formulation patents — adhesives, coatings, lubricants,
+pharmaceuticals, agrochemicals, personal care, food. Particularly suited to
+within-domain near-miss discrimination, where general-purpose embedders have been shown
+to fail.
+
+## Limitations
+
+- Training queries are LLM-generated (Sonnet 4 + Haiku 4.5 quality filter) and may not
+  match real practitioner intent.
+- Coverage limited to USPTO utility patents (1995–2022) in English only.
+- Performance on out-of-domain retrieval is not characterised.
+
+## Citation
+
+```bibtex
+@misc{formbench2026,
+  title  = { {FormBench}: Evaluating Chemical Knowledge Retrieval in Formulation Patents },
+  author = { Anonymous Authors },
+  year   = { 2026 },
+  note   = { Under double-blind review at NeurIPS 2026 Datasets \& Benchmarks Track }
+}
+```
+
+## License
+
+Apache 2.0, inherited from the base model.

_eval_status.json

@@ -0,0 +1,20 @@
+{
+  "c1": {
+    "status": "completed",
+    "queued_by_job": "4376440",
+    "started_at": "2026-04-11T00:59:45.383077",
+    "phase1_job": "4376440",
+    "phase1_completed_at": "2026-04-11T01:13:49.042215",
+    "completed_at": "2026-04-11T04:34:23.512871",
+    "metrics_path": "/lustre/orion/mat721/proj-shared/formbench_ner/experiments/results/qwen3_embed_tapt_mnrl_ckpt-54/c1/metrics.json"
+  },
+  "c0": {
+    "status": "completed",
+    "queued_by_job": "4379903",
+    "started_at": "2026-04-12T09:48:43.351797",
+    "phase1_job": "4379903",
+    "phase1_completed_at": "2026-04-12T10:01:33.986693",
+    "completed_at": "2026-04-12T14:40:11.409233",
+    "metrics_path": "/lustre/orion/mat721/proj-shared/formbench_ner/experiments/results/qwen3_embed_tapt_mnrl_ckpt-54/c0/metrics.json"
+  }
+}

_legacy_eval_status.json

@@ -0,0 +1,10 @@
+{
+  "status": "completed",
+  "corpus": "c1",
+  "queued_by_job": "4376440",
+  "started_at": "2026-04-11T00:59:45.383077",
+  "phase1_job": "4376440",
+  "phase1_completed_at": "2026-04-11T01:13:49.042215",
+  "completed_at": "2026-04-11T04:34:23.512871",
+  "metrics_path": "/lustre/orion/mat721/proj-shared/formbench_ner/experiments/results/qwen3_embed_tapt_mnrl_ckpt-54/c1/metrics.json"
+}

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "Qwen3Model"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151669
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "prompts": {
+    "query": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine",
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.3.0",
+    "transformers": "4.51.3",
+    "pytorch": "2.3.1+rocm5.7"
+  }
+}

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8369e141d19074d1c3eca876eeb5c90ac5e612eb9277ccddd39f96d9686f1e9c
+size 4766475646

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5597a032971bc3df86d2eae1d18d801a2829da4296c0397380ee8be4ef0ddc77
+size 2383208610

rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:816155bc42315af9e2805507030c0b4a206366b4819a634f7e465da88ebf26c0
+size 15920

rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:822f9bdcf710e1d75200390882dea1514ec073d1e9e7eabc0ce02f9edb07b028
+size 15920

rng_state_10.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ce83be66c2bc996fa5e036127abe566fc9455dd2e1327d84101bdfce3fab2a3
+size 15933

rng_state_11.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b2800d95dec1f69bdfce5e60f594f3a1d60f3c768d16a28ce1c485b188bf81c
+size 15933

rng_state_12.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0db4e078ae5f273ec4296044edaa6cb04dec986164d008549246dd3d7aff47f0
+size 15933

rng_state_13.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:114f057807f2ba0a40a5611f7b1ce42224dfb053836a0b91a8944543dced295d
+size 15933

rng_state_14.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77181658cc19b5ededbcd9c5436ba6d05cc22697f317968c740d5ec30374f390
+size 15933

rng_state_15.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:becb9f1edd5859434daeca6cf0e389b26c0a1792e459ab711f36cabd07928ebf
+size 15933

rng_state_16.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21f60e17d6b0ccf814da734e2b8637376185c892d7f06c6996ea39cf2064ab27
+size 15933

rng_state_17.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57c732ec79817ecdbb0ceca1621a02172edc7b675a33cf5237391cdfe1baf2ae
+size 15933

rng_state_18.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3097598eefa62b61b57e45232b1c4a16fdea482c242eb16c3f7fa58904fd90f
+size 15933

rng_state_19.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac147befdf574a59f715f18baefd6685b48ad656e2b58515855e0d15e63241a4
+size 15933

rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a1a432416e644e8353612f55e555b40cc9c3df75e4fab317ea23e415c18314c
+size 15920

rng_state_20.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7acd0c29c93f93674dfb87fc3509b9bf38255474cbcccf634ca28c3fb3edca4a
+size 15933

rng_state_21.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3afa28afd8389364edcd0b6cbada2c88c43b33f6e1f40c0bdb84eb6b3d52b608
+size 15933

rng_state_22.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b0a9aac557483a34e8832dc9f713e887810a86898c6ae3183c49420b4cf0580
+size 15933

rng_state_23.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81b8d96b42226d619ba5978c7aeb166d1e630ff15dffc5e6b70e9a455ce43b40
+size 15933

rng_state_24.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00045e4ed5bf933579393fbee541d3106c0a8033ec4c898d3af3b3556f03f95d
+size 15933

rng_state_25.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c56d9d33d9d92b44b54ffa4f9a0c6d2a02ab50abfbe21f5291e9375be4c4755
+size 15933

rng_state_26.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:099941f477ee268d9b330dba6229e4f9badce45d4e6c42aad5ba707c8b0cfc19
+size 15933

rng_state_27.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:420bb5fe24668d8af3a8a6ab67bc40719c2bac33d16d4f4957942db9c51ed452
+size 15933

rng_state_28.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dee0a5cd14056d7ed728b4aae2b445431f9720cc9590ff60ba7ada9315dd7cb7
+size 15933

rng_state_29.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad4c04773acaf8695940e8110807106298e90dd9c35cdf36f0a9e2ec5ad3d186
+size 15933

rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5db4389e6f71ad614141db203d1dbdf4d50a819376b930d09e2111410be9e510
+size 15920

rng_state_30.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:633b155be7d85233b035027403ea1307cf8275933baa8ecae80567f43669efec
+size 15933

rng_state_31.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e9030c4ce83cc843fe285148728677645b09c8cce30aa7ca85217ba2868e71c
+size 15933

rng_state_32.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2268c2e90c1c8a542e11392725505718f7d8a30d99e494429bc4f1cd74cf5623
+size 15933

rng_state_33.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2db985d05cb2aa2fa46af7c89fd67a9277d6cf60e357edec1cd0954b3700284f
+size 15933

rng_state_34.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4d74989ba957341857866fef5f7c34877a8004b8270363c84ac65b1b6ef54f5
+size 15933

rng_state_35.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82e71de9a074aec5ab77efd65166f0ed1bfd5fbe5823369a8027f63dfce790be
+size 15933

rng_state_36.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b55a3481854d769eb1a6ebc449933ec6ac23f44f9e367b2d9ce179a9340e7f3e
+size 15933

rng_state_37.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:087c65424a415701971419d7121405ae5971c8ca7ee82cff0f39f9a8f88e0bda
+size 15933

rng_state_38.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5749a2acd72c88d199dacb104d76adb49da4654d3b0daa49306e876d6d2bf081
+size 15933

rng_state_39.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bfcccbf5c266a88f581aea34360e0a95fe612a4cb192c5affa2203f559f9687
+size 15933

rng_state_4.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f64ba52019b820b7d4b373b7950a04b274c8f199be5a2239f49a4d718d694e62
+size 15920

rng_state_40.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53f15e651c8230431df86b25131387a298052decb15fa555121fb2bafa02e815
+size 15933

rng_state_41.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65dd3b8375722c427dbf61573e6980efc7433b29b190363aaa19a26572b10dc4
+size 15933

rng_state_42.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09b9c5640bcafbb6f3a9e4652b702944f9842e36c0d66cb1ee9df9dc2a1c7e98
+size 15933