CE fine-tuned epoch 1/3 best_val=0.5370

README.md

@@ -28,25 +28,25 @@ model-index:
       type: ce-val
     metrics:
     - type: accuracy
-      value: 0.6896969696969697
+      value: 0.536969696969697
       name: Accuracy
     - type: accuracy_threshold
-      value: 0.5847750902175903
+      value: 0.4739919900894165
       name: Accuracy Threshold
     - type: f1
-      value: 0.7074707470747076
+      value: 0.6682966585167075
       name: F1
     - type: f1_threshold
-      value: 0.3505881428718567
+      value: 0.44792813062667847
       name: F1 Threshold
     - type: precision
-      value: 0.5630372492836676
+      value: 0.5036855036855037
       name: Precision
     - type: recall
-      value: 0.9515738498789347
+      value: 0.9927360774818402
       name: Recall
     - type: average_precision
-      value: 0.7391973708035351
+      value: 0.5140790363376013
       name: Average Precision
 ---
 
@@ -99,25 +99,25 @@ from sentence_transformers import CrossEncoder
 model = CrossEncoder("cross_encoder_model_id")
 # Get scores for pairs of inputs
 pairs = [
-    ['The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.', 'Almost all scientists acknowledge that the rate of species loss is greater now than at any time in human history, with extinctions occurring at rates hundreds of times higher than background extinction rates.'],
-    ['[S]unspot activity on the surface of our star has dropped to a new low.', 'This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.'],
-    ['More money is dedicated within the Department of Homeland Security to climate change than what\'s spent combating "Islamist terrorists radicalizing over the Internet in the United States of America."', "The center works on the Internet's routing infrastructure (the SPRI program) and Domain Name System (DNSSEC), identity theft and other online criminal activity (ITTC), Internet traffic and networks research (PREDICT datasets and the DETER testbed), Department of Defense and HSARPA exercises (Livewire and Determined Promise), and wireless security in cooperation with Canada."],
-    ['Worst-case global heating scenarios may need to be revised upwards in light of a better understanding of the role of clouds, scientists have said.', 'Climate model projections summarized in the report indicated that during the 21st century the global surface temperature is likely to rise a further 0.3 to 1.7\xa0°C (0.5 to 3.1\xa0°F) in a moderate scenario, or as much as 2.6 to 4.8\xa0°C (4.7 to 8.6\xa0°F) in an extreme scenario, depending on the rate of future greenhouse gas emissions and on climate feedback effects.'],
-    ['Prof Adam Scaife, a climate modelling expert at the UK’s Met Office, said the evidence for a link to shrinking Arctic ice was now good: ‘The consensus points towards that being a real effect.’”', 'Some models of modern climate exhibit Arctic amplification without changes in snow and ice cover.'],
+    ['The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.', 'As a result, the mean annual air temperature at sea level decreases by about 0.4\xa0°C (0.7\xa0°F) per degree of latitude from the equator.'],
+    ["Empirical measurements of the Earth's heat content show the planet is still accumulating heat and global warming is still happening.", 'The global average and combined land and ocean surface temperature, show a warming of 0.85 [0.65 to 1.06] °C, in the period 1880 to 2012, based on multiple independently produced datasets.'],
+    ['Numerous case studies on both regional and global scales have determined that renewable energy, if properly implemented, can provide baseload power.', 'Currently, there are challenges implementing it on a global scale because there is no government with that power.'],
+    ['“Typically, in such an attribution study, scientists will use sets of climate models — one set including the factors that drive human global warming and the other including purely “natural” factors — and see if an event like the one in question is more likely to occur in the first set of models.', 'Attribution of the temperature change to natural or anthropogenic (i.e., human-induced) factors is an important question: see global warming and attribution of recent climate change.'],
+    ['The sun was warming up then, but the sun hasn’t been warming since 1970.', '"Sun\'s Shifts May Cause Global Warming".'],
 ]
 scores = model.predict(pairs)
 print(scores)
-# [0.7715 0.6352 0.7843 0.844  0.4491]
+# [0.4804 0.4864 0.4651 0.4637 0.4819]
 
 # Or rank different texts based on similarity to a single text
 ranks = model.rank(
     'The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.',
     [
-        'Almost all scientists acknowledge that the rate of species loss is greater now than at any time in human history, with extinctions occurring at rates hundreds of times higher than background extinction rates.',
-        'This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.',
-        "The center works on the Internet's routing infrastructure (the SPRI program) and Domain Name System (DNSSEC), identity theft and other online criminal activity (ITTC), Internet traffic and networks research (PREDICT datasets and the DETER testbed), Department of Defense and HSARPA exercises (Livewire and Determined Promise), and wireless security in cooperation with Canada.",
-        'Climate model projections summarized in the report indicated that during the 21st century the global surface temperature is likely to rise a further 0.3 to 1.7\xa0°C (0.5 to 3.1\xa0°F) in a moderate scenario, or as much as 2.6 to 4.8\xa0°C (4.7 to 8.6\xa0°F) in an extreme scenario, depending on the rate of future greenhouse gas emissions and on climate feedback effects.',
-        'Some models of modern climate exhibit Arctic amplification without changes in snow and ice cover.',
+        'As a result, the mean annual air temperature at sea level decreases by about 0.4\xa0°C (0.7\xa0°F) per degree of latitude from the equator.',
+        'The global average and combined land and ocean surface temperature, show a warming of 0.85 [0.65 to 1.06] °C, in the period 1880 to 2012, based on multiple independently produced datasets.',
+        'Currently, there are challenges implementing it on a global scale because there is no government with that power.',
+        'Attribution of the temperature change to natural or anthropogenic (i.e., human-induced) factors is an important question: see global warming and attribution of recent climate change.',
+        '"Sun\'s Shifts May Cause Global Warming".',
     ]
 )
 # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
@@ -158,13 +158,13 @@ You can finetune this model on your own dataset.
 
 | Metric                | Value      |
 |:----------------------|:-----------|
-| accuracy              | 0.6897     |
-| accuracy_threshold    | 0.5848     |
-| f1                    | 0.7075     |
-| f1_threshold          | 0.3506     |
-| precision             | 0.563      |
-| recall                | 0.9516     |
-| **average_precision** | **0.7392** |
+| accuracy              | 0.537      |
+| accuracy_threshold    | 0.474      |
+| f1                    | 0.6683     |
+| f1_threshold          | 0.4479     |
+| precision             | 0.5037     |
+| recall                | 0.9927     |
+| **average_precision** | **0.5141** |
 
 <!--
 ## Bias, Risks and Limitations
@@ -190,13 +190,13 @@ You can finetune this model on your own dataset.
   |         | sentence_0                                                                        | sentence_1                                                                         | label                                                          |
   |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
   | type    | string                                                                            | string                                                                             | float                                                          |
-  | details | <ul><li>min: 7 tokens</li><li>mean: 26.66 tokens</li><li>max: 80 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 31.75 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.52</li><li>max: 1.0</li></ul> |
+  | details | <ul><li>min: 7 tokens</li><li>mean: 26.35 tokens</li><li>max: 66 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 30.52 tokens</li><li>max: 145 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.47</li><li>max: 1.0</li></ul> |
 * Samples:
-  | sentence_0                                                                                                                                                                                                               | sentence_1                                                                                                                                                                                                                                                                                                                                                                                            | label            |
-  |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
-  | <code>The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.</code> | <code>Almost all scientists acknowledge that the rate of species loss is greater now than at any time in human history, with extinctions occurring at rates hundreds of times higher than background extinction rates.</code>                                                                                                                                                                         | <code>0.0</code> |
-  | <code>[S]unspot activity on the surface of our star has dropped to a new low.</code>                                                                                                                                     | <code>This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.</code>                                                                                                                                                                                                                                                        | <code>1.0</code> |
-  | <code>More money is dedicated within the Department of Homeland Security to climate change than what's spent combating "Islamist terrorists radicalizing over the Internet in the United States of America."</code>      | <code>The center works on the Internet's routing infrastructure (the SPRI program) and Domain Name System (DNSSEC), identity theft and other online criminal activity (ITTC), Internet traffic and networks research (PREDICT datasets and the DETER testbed), Department of Defense and HSARPA exercises (Livewire and Determined Promise), and wireless security in cooperation with Canada.</code> | <code>1.0</code> |
+  | sentence_0                                                                                                                                                                                                               | sentence_1                                                                                                                                                                                                | label            |
+  |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
+  | <code>The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.</code> | <code>As a result, the mean annual air temperature at sea level decreases by about 0.4 °C (0.7 °F) per degree of latitude from the equator.</code>                                                        | <code>1.0</code> |
+  | <code>Empirical measurements of the Earth's heat content show the planet is still accumulating heat and global warming is still happening.</code>                                                                        | <code>The global average and combined land and ocean surface temperature, show a warming of 0.85 [0.65 to 1.06] °C, in the period 1880 to 2012, based on multiple independently produced datasets.</code> | <code>1.0</code> |
+  | <code>Numerous case studies on both regional and global scales have determined that renewable energy, if properly implemented, can provide baseload power.</code>                                                        | <code>Currently, there are challenges implementing it on a global scale because there is no government with that power.</code>                                                                            | <code>0.0</code> |
 * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters:
   ```json
   {
@@ -208,8 +208,8 @@ You can finetune this model on your own dataset.
 ### Training Hyperparameters
 #### Non-Default Hyperparameters
 
-- `per_device_train_batch_size`: 4
-- `per_device_eval_batch_size`: 4
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
 - `num_train_epochs`: 1
 - `fp16`: True
 
@@ -218,8 +218,8 @@ You can finetune this model on your own dataset.
 
 - `do_predict`: False
 - `prediction_loss_only`: True
-- `per_device_train_batch_size`: 4
-- `per_device_eval_batch_size`: 4
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
 - `gradient_accumulation_steps`: 1
 - `eval_accumulation_steps`: None
 - `torch_empty_cache_steps`: None
@@ -316,24 +316,13 @@ You can finetune this model on your own dataset.
 </details>
 
 ### Training Logs
-| Epoch  | Step | Training Loss | ce-val_average_precision |
-|:------:|:----:|:-------------:|:------------------------:|
-| 0.2695 | 500  | 0.6958        | -                        |
-| 0.5391 | 1000 | 0.6883        | -                        |
-| 0.8086 | 1500 | 0.6841        | -                        |
-| -1     | -1   | -             | 0.5670                   |
-| 0.2695 | 500  | 0.6741        | -                        |
-| 0.5391 | 1000 | 0.6662        | -                        |
-| 0.8086 | 1500 | 0.6504        | -                        |
-| -1     | -1   | -             | 0.6569                   |
-| 0.2695 | 500  | 0.6129        | -                        |
-| 0.5391 | 1000 | 0.6091        | -                        |
-| 0.8086 | 1500 | 0.5937        | -                        |
-| -1     | -1   | -             | 0.7392                   |
+| Epoch | Step | ce-val_average_precision |
+|:-----:|:----:|:------------------------:|
+| -1    | -1   | 0.5141                   |
 
 
 ### Training Time
-- **Training**: 6.8 minutes
+- **Training**: 2.2 minutes
 
 ### Framework Versions
 - Python: 3.12.13

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a1ca6dcff0430e6f41bdfdbd69ec03387cbbb18662752f5c67a21eaf3d52ef85
+oid sha256:164bb394a54a6ca6a0e682ccee0bf2f31752f5e8c13b12eb17556e5c825dd634
 size 737716172