CE fine-tuned epoch 1/3 best_val=0.5285

README.md

@@ -21,32 +21,32 @@ model-index:
 - name: CrossEncoder based on cross-encoder/ms-marco-MiniLM-L6-v2
   results:
   - task:
-      type: cross-encoder-binary-classification
-      name: Cross Encoder Binary Classification
+      type: cross-encoder-classification
+      name: Cross Encoder Classification
     dataset:
       name: ce val
       type: ce-val
     metrics:
     - type: accuracy
-      value: 0.8448484848484848
+      value: 0.5284848484848484
       name: Accuracy
     - type: accuracy_threshold
-      value: -2.50508451461792
+      value: 3.9093470573425293
       name: Accuracy Threshold
     - type: f1
-      value: 0.8505214368482039
+      value: 0.6677471636952999
       name: F1
     - type: f1_threshold
-      value: -2.667440176010132
+      value: -10.441707611083984
       name: F1 Threshold
     - type: precision
-      value: 0.8155555555555556
+      value: 0.5018270401948843
       name: Precision
     - type: recall
-      value: 0.8886198547215496
+      value: 0.9975786924939467
       name: Recall
     - type: average_precision
-      value: 0.9152025768340708
+      value: 0.5295137880648401
       name: Average Precision
 ---
 
@@ -99,25 +99,25 @@ from sentence_transformers import CrossEncoder
 model = CrossEncoder("cross_encoder_model_id")
 # Get scores for pairs of inputs
 pairs = [
-    ['The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.', 'It was designed by Jung Brannen Associates.'],
-    ['[S]unspot activity on the surface of our star has dropped to a new low.', 'This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.'],
-    ['More money is dedicated within the Department of Homeland Security to climate change than what\'s spent combating "Islamist terrorists radicalizing over the Internet in the United States of America."', 'According to The Washington Post, "Online recruiting has exponentially increased, with Facebook, YouTube and the increasing sophistication of people online".'],
-    ['Worst-case global heating scenarios may need to be revised upwards in light of a better understanding of the role of clouds, scientists have said.', 'Climate change is more accurate scientifically to describe the various effects of greenhouse gases on the world because it includes extreme weather, storms and changes in rainfall patterns, ocean acidification and sea level.".'],
-    ['Prof Adam Scaife, a climate modelling expert at the UK’s Met Office, said the evidence for a link to shrinking Arctic ice was now good: ‘The consensus points towards that being a real effect.’”', 'Category : Ceremonial officers in the United Kingdom'],
+    ['They (Clinton and Obama) have never to my knowledge been involved in legislation nor hearings nor engagement on this issue (climate change).', 'Gore has been involved with environmental issues since 1976, when as a freshman congressman, he held the "first congressional hearings on the climate change, and co-sponsor[ed] hearings on toxic waste and global warming."'],
+    ['This increase is the result of humans emitting more carbon dioxide into the atmosphere and hence more being absorbed into the oceans.', 'Humans have a substantial influence on the rise of sea level because we emit increasing levels of carbon dioxide into the atmosphere through automobile use and industry.'],
+    ["Venus doesn't have a runaway greenhouse effect", "More recent studies have suggested that several billion years ago, Venus's atmosphere was much more like Earth's than it is now and that there were probably substantial quantities of liquid water on the surface, but a runaway greenhouse effect was caused by the evaporation of that original water, which generated a critical level of greenhouse gases in its atmosphere."],
+    ['At four degrees, the deadly European heat wave of 2003, which killed as many as 2,000 people a day, will be a normal summer.', 'For comparison, the 2003 European heat wave killed an estimated 35,000–70,000 people, with temperatures slightly less than in India and Pakistan.'],
+    ['Under the most ambitious scenarios, they found a strong likelihood that Antarctica would remain fairly stable.”', 'Its remains have been found in Africa, Antarctica, Europe, and North America.'],
 ]
 scores = model.predict(pairs)
 print(scores)
-# [ -9.1708   2.7075   3.8609   8.2862 -10.5857]
+# [ 2.0512  5.9885  7.7773  5.7437 -3.7705]
 
 # Or rank different texts based on similarity to a single text
 ranks = model.rank(
-    'The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.',
+    'They (Clinton and Obama) have never to my knowledge been involved in legislation nor hearings nor engagement on this issue (climate change).',
     [
-        'It was designed by Jung Brannen Associates.',
-        'This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.',
-        'According to The Washington Post, "Online recruiting has exponentially increased, with Facebook, YouTube and the increasing sophistication of people online".',
-        'Climate change is more accurate scientifically to describe the various effects of greenhouse gases on the world because it includes extreme weather, storms and changes in rainfall patterns, ocean acidification and sea level.".',
-        'Category : Ceremonial officers in the United Kingdom',
+        'Gore has been involved with environmental issues since 1976, when as a freshman congressman, he held the "first congressional hearings on the climate change, and co-sponsor[ed] hearings on toxic waste and global warming."',
+        'Humans have a substantial influence on the rise of sea level because we emit increasing levels of carbon dioxide into the atmosphere through automobile use and industry.',
+        "More recent studies have suggested that several billion years ago, Venus's atmosphere was much more like Earth's than it is now and that there were probably substantial quantities of liquid water on the surface, but a runaway greenhouse effect was caused by the evaporation of that original water, which generated a critical level of greenhouse gases in its atmosphere.",
+        'For comparison, the 2003 European heat wave killed an estimated 35,000–70,000 people, with temperatures slightly less than in India and Pakistan.',
+        'Its remains have been found in Africa, Antarctica, Europe, and North America.',
     ]
 )
 # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
@@ -151,20 +151,20 @@ You can finetune this model on your own dataset.
 
 ### Metrics
 
-#### Cross Encoder Binary Classification
+#### Cross Encoder Classification
 
 * Dataset: `ce-val`
-* Evaluated with [<code>CEBinaryClassificationEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CEBinaryClassificationEvaluator)
+* Evaluated with [<code>CrossEncoderClassificationEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderClassificationEvaluator)
 
 | Metric                | Value      |
 |:----------------------|:-----------|
-| accuracy              | 0.8448     |
-| accuracy_threshold    | -2.5051    |
-| f1                    | 0.8505     |
-| f1_threshold          | -2.6674    |
-| precision             | 0.8156     |
-| recall                | 0.8886     |
-| **average_precision** | **0.9152** |
+| accuracy              | 0.5285     |
+| accuracy_threshold    | 3.9093     |
+| f1                    | 0.6677     |
+| f1_threshold          | -10.4417   |
+| precision             | 0.5018     |
+| recall                | 0.9976     |
+| **average_precision** | **0.5295** |
 
 <!--
 ## Bias, Risks and Limitations
@@ -190,13 +190,13 @@ You can finetune this model on your own dataset.
   |         | sentence_0                                                                        | sentence_1                                                                         | label                                                          |
   |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
   | type    | string                                                                            | string                                                                             | float                                                          |
-  | details | <ul><li>min: 7 tokens</li><li>mean: 27.57 tokens</li><li>max: 82 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 31.56 tokens</li><li>max: 321 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.52</li><li>max: 1.0</li></ul> |
+  | details | <ul><li>min: 7 tokens</li><li>mean: 27.81 tokens</li><li>max: 82 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 32.86 tokens</li><li>max: 247 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.49</li><li>max: 1.0</li></ul> |
 * Samples:
-  | sentence_0                                                                                                                                                                                                               | sentence_1                                                                                                                                                                 | label            |
-  |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
-  | <code>The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.</code> | <code>It was designed by Jung Brannen Associates.</code>                                                                                                                   | <code>0.0</code> |
-  | <code>[S]unspot activity on the surface of our star has dropped to a new low.</code>                                                                                                                                     | <code>This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.</code>                             | <code>1.0</code> |
-  | <code>More money is dedicated within the Department of Homeland Security to climate change than what's spent combating "Islamist terrorists radicalizing over the Internet in the United States of America."</code>      | <code>According to The Washington Post, "Online recruiting has exponentially increased, with Facebook, YouTube and the increasing sophistication of people online".</code> | <code>1.0</code> |
+  | sentence_0                                                                                                                                                | sentence_1                                                                                                                                                                                                                                                                                                                                                                                     | label            |
+  |:----------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
+  | <code>They (Clinton and Obama) have never to my knowledge been involved in legislation nor hearings nor engagement on this issue (climate change).</code> | <code>Gore has been involved with environmental issues since 1976, when as a freshman congressman, he held the "first congressional hearings on the climate change, and co-sponsor[ed] hearings on toxic waste and global warming."</code>                                                                                                                                                     | <code>1.0</code> |
+  | <code>This increase is the result of humans emitting more carbon dioxide into the atmosphere and hence more being absorbed into the oceans.</code>        | <code>Humans have a substantial influence on the rise of sea level because we emit increasing levels of carbon dioxide into the atmosphere through automobile use and industry.</code>                                                                                                                                                                                                         | <code>0.0</code> |
+  | <code>Venus doesn't have a runaway greenhouse effect</code>                                                                                               | <code>More recent studies have suggested that several billion years ago, Venus's atmosphere was much more like Earth's than it is now and that there were probably substantial quantities of liquid water on the surface, but a runaway greenhouse effect was caused by the evaporation of that original water, which generated a critical level of greenhouse gases in its atmosphere.</code> | <code>0.0</code> |
 * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters:
   ```json
   {
@@ -317,11 +317,11 @@ You can finetune this model on your own dataset.
 ### Training Logs
 | Epoch | Step | ce-val_average_precision |
 |:-----:|:----:|:------------------------:|
-| -1    | -1   | 0.9152                   |
+| -1    | -1   | 0.5295                   |
 
 
 ### Training Time
-- **Training**: 33.2 seconds
+- **Training**: 34.4 seconds
 
 ### Framework Versions
 - Python: 3.12.13

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:12713a6f7d4fe5f00305ca6306aa5ef44c133ae838ac265c8b0c27a3e328d81c
+oid sha256:c50a517d1275313dcf63127bb3c7e7282772be50cbbe9f885894b98627ea0d19
 size 90866404