ColeH0415
/

comp90042-crossencoder-factcheck

@@ -28,25 +28,25 @@ model-index:
       type: ce-val
     metrics:
     - type: accuracy
-      value: 0.5781818181818181
       name: Accuracy
     - type: accuracy_threshold
-      value: 0.5230777859687805
       name: Accuracy Threshold
     - type: f1
-      value: 0.6700942587832047
       name: F1
     - type: f1_threshold
-      value: 0.4445345997810364
       name: F1 Threshold
     - type: precision
-      value: 0.5185676392572944
       name: Precision
     - type: recall
-      value: 0.9467312348668281
       name: Recall
     - type: average_precision
-      value: 0.5669943455221101
       name: Average Precision
 ---
@@ -99,25 +99,25 @@ from sentence_transformers import CrossEncoder
 model = CrossEncoder("cross_encoder_model_id")
 # Get scores for pairs of inputs
 pairs = [
-    ['If every house in Florida had a solar-heated water tank, that would eliminate consumption by 17 percent.', 'Solar water heating (SWH) is the conversion of sunlight into heat for water heating using a solar thermal collector.'],
-    ['Modellers assume carbon dioxide drives climate change', 'They absorb a huge amount of carbon dioxide, combating climate change.'],
-    ['Some, however, bristle at the belief that because floods and storms have always occurred, they should not be linked to climate change”', 'Although some studies have reported an increase in frequency and intensity of extremes in rainfall during the past 40–50 years, their attribution to global warming is not established."'],
-    ['The tax-payer funded National Oceanic and Atmospheric Administration  (NOAA) has become mired in fresh global warming data scandal involving  numbers for the Great Lakes region that substantially ramp up averages."', 'Feds close 600 weather stations amid criticism they\'re situated to report warming".'],
-    ['The acceleration is making some scientists fear that Antarctica’s ice sheet may have entered the early stages of an unstoppable disintegration.', 'Scientists have found that the flow of these ice streams has accelerated in recent years, and suggested that if they were to melt, global sea levels would rise by 1 to 2\xa0m (3\xa0ft 3\xa0in to 6\xa0ft 7\xa0in), destabilising the entire West Antarctic Ice Sheet and perhaps sections of the East Antarctic Ice Sheet.'],
 ]
 scores = model.predict(pairs)
 print(scores)
-# [0.4903 0.4453 0.5899 0.4856 0.5753]
 # Or rank different texts based on similarity to a single text
 ranks = model.rank(
-    'If every house in Florida had a solar-heated water tank, that would eliminate consumption by 17 percent.',
     [
-        'Solar water heating (SWH) is the conversion of sunlight into heat for water heating using a solar thermal collector.',
-        'They absorb a huge amount of carbon dioxide, combating climate change.',
-        'Although some studies have reported an increase in frequency and intensity of extremes in rainfall during the past 40–50 years, their attribution to global warming is not established."',
-        'Feds close 600 weather stations amid criticism they\'re situated to report warming".',
-        'Scientists have found that the flow of these ice streams has accelerated in recent years, and suggested that if they were to melt, global sea levels would rise by 1 to 2\xa0m (3\xa0ft 3\xa0in to 6\xa0ft 7\xa0in), destabilising the entire West Antarctic Ice Sheet and perhaps sections of the East Antarctic Ice Sheet.',
     ]
 )
 # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
@@ -156,15 +156,15 @@ You can finetune this model on your own dataset.
 * Dataset: `ce-val`
 * Evaluated with [<code>CrossEncoderClassificationEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderClassificationEvaluator)
-| Metric                | Value     |
-|:----------------------|:----------|
-| accuracy              | 0.5782    |
-| accuracy_threshold    | 0.5231    |
-| f1                    | 0.6701    |
-| f1_threshold          | 0.4445    |
-| precision             | 0.5186    |
-| recall                | 0.9467    |
-| **average_precision** | **0.567** |
 <!--
 ## Bias, Risks and Limitations
@@ -190,13 +190,13 @@ You can finetune this model on your own dataset.
   |         | sentence_0                                                                        | sentence_1                                                                         | label                                                          |
   |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
   | type    | string                                                                            | string                                                                             | float                                                          |
-  | details | <ul><li>min: 7 tokens</li><li>mean: 25.97 tokens</li><li>max: 80 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 31.89 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.47</li><li>max: 1.0</li></ul> |
 * Samples:
-  | sentence_0                                                                                                                                          | sentence_1                                                                                                                                                                                            | label            |
-  |:----------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
-  | <code>If every house in Florida had a solar-heated water tank, that would eliminate consumption by 17 percent.</code>                               | <code>Solar water heating (SWH) is the conversion of sunlight into heat for water heating using a solar thermal collector.</code>                                                                     | <code>0.0</code> |
-  | <code>Modellers assume carbon dioxide drives climate change</code>                                                                                  | <code>They absorb a huge amount of carbon dioxide, combating climate change.</code>                                                                                                                   | <code>0.0</code> |
-  | <code>Some, however, bristle at the belief that because floods and storms have always occurred, they should not be linked to climate change”</code> | <code>Although some studies have reported an increase in frequency and intensity of extremes in rainfall during the past 40–50 years, their attribution to global warming is not established."</code> | <code>1.0</code> |
 * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters:
   ```json
   {
@@ -322,6 +322,10 @@ You can finetune this model on your own dataset.
 | 0.5391 | 1000 | 0.6883        | -                        |
 | 0.8086 | 1500 | 0.6841        | -                        |
 | -1     | -1   | -             | 0.5670                   |
 ### Training Time

       type: ce-val
     metrics:
     - type: accuracy
+      value: 0.6351515151515151
       name: Accuracy
     - type: accuracy_threshold
+      value: 0.5755879878997803
       name: Accuracy Threshold
     - type: f1
+      value: 0.6981132075471698
       name: F1
     - type: f1_threshold
+      value: 0.41324031352996826
       name: F1 Threshold
     - type: precision
+      value: 0.5405046480743692
       name: Precision
     - type: recall
+      value: 0.9854721549636803
       name: Recall
     - type: average_precision
+      value: 0.6568676757267966
       name: Average Precision
 ---
 model = CrossEncoder("cross_encoder_model_id")
 # Get scores for pairs of inputs
 pairs = [
+    ['The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.', 'Almost all scientists acknowledge that the rate of species loss is greater now than at any time in human history, with extinctions occurring at rates hundreds of times higher than background extinction rates.'],
+    ['[S]unspot activity on the surface of our star has dropped to a new low.', 'This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.'],
+    ['More money is dedicated within the Department of Homeland Security to climate change than what\'s spent combating "Islamist terrorists radicalizing over the Internet in the United States of America."', "The center works on the Internet's routing infrastructure (the SPRI program) and Domain Name System (DNSSEC), identity theft and other online criminal activity (ITTC), Internet traffic and networks research (PREDICT datasets and the DETER testbed), Department of Defense and HSARPA exercises (Livewire and Determined Promise), and wireless security in cooperation with Canada."],
+    ['Worst-case global heating scenarios may need to be revised upwards in light of a better understanding of the role of clouds, scientists have said.', 'Climate model projections summarized in the report indicated that during the 21st century the global surface temperature is likely to rise a further 0.3 to 1.7\xa0°C (0.5 to 3.1\xa0°F) in a moderate scenario, or as much as 2.6 to 4.8\xa0°C (4.7 to 8.6\xa0°F) in an extreme scenario, depending on the rate of future greenhouse gas emissions and on climate feedback effects.'],
+    ['Prof Adam Scaife, a climate modelling expert at the UK’s Met Office, said the evidence for a link to shrinking Arctic ice was now good: ‘The consensus points towards that being a real effect.’”', 'Some models of modern climate exhibit Arctic amplification without changes in snow and ice cover.'],
 ]
 scores = model.predict(pairs)
 print(scores)
+# [0.6498 0.5873 0.6027 0.6833 0.4922]
 # Or rank different texts based on similarity to a single text
 ranks = model.rank(
+    'The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.',
     [
+        'Almost all scientists acknowledge that the rate of species loss is greater now than at any time in human history, with extinctions occurring at rates hundreds of times higher than background extinction rates.',
+        'This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.',
+        "The center works on the Internet's routing infrastructure (the SPRI program) and Domain Name System (DNSSEC), identity theft and other online criminal activity (ITTC), Internet traffic and networks research (PREDICT datasets and the DETER testbed), Department of Defense and HSARPA exercises (Livewire and Determined Promise), and wireless security in cooperation with Canada.",
+        'Climate model projections summarized in the report indicated that during the 21st century the global surface temperature is likely to rise a further 0.3 to 1.7\xa0°C (0.5 to 3.1\xa0°F) in a moderate scenario, or as much as 2.6 to 4.8\xa0°C (4.7 to 8.6\xa0°F) in an extreme scenario, depending on the rate of future greenhouse gas emissions and on climate feedback effects.',
+        'Some models of modern climate exhibit Arctic amplification without changes in snow and ice cover.',
     ]
 )
 # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
 * Dataset: `ce-val`
 * Evaluated with [<code>CrossEncoderClassificationEvaluator</code>](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderClassificationEvaluator)
+| Metric                | Value      |
+|:----------------------|:-----------|
+| accuracy              | 0.6352     |
+| accuracy_threshold    | 0.5756     |
+| f1                    | 0.6981     |
+| f1_threshold          | 0.4132     |
+| precision             | 0.5405     |
+| recall                | 0.9855     |
+| **average_precision** | **0.6569** |
 <!--
 ## Bias, Risks and Limitations
   |         | sentence_0                                                                        | sentence_1                                                                         | label                                                          |
   |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
   | type    | string                                                                            | string                                                                             | float                                                          |
+  | details | <ul><li>min: 7 tokens</li><li>mean: 26.66 tokens</li><li>max: 80 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 31.75 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.52</li><li>max: 1.0</li></ul> |
 * Samples:
+  | sentence_0                                                                                                                                                                                                               | sentence_1                                                                                                                                                                                                                                                                                                                                                                                            | label            |
+  |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
+  | <code>The last time the planet was even four degrees warmer, Peter Brannen points out in The Ends of the World, his new history of the planet’s major extinction events, the oceans were hundreds of feet higher.</code> | <code>Almost all scientists acknowledge that the rate of species loss is greater now than at any time in human history, with extinctions occurring at rates hundreds of times higher than background extinction rates.</code>                                                                                                                                                                         | <code>0.0</code> |
+  | <code>[S]unspot activity on the surface of our star has dropped to a new low.</code>                                                                                                                                     | <code>This surface activity produces starspots, which are regions of strong magnetic fields and lower than normal surface temperatures.</code>                                                                                                                                                                                                                                                        | <code>1.0</code> |
+  | <code>More money is dedicated within the Department of Homeland Security to climate change than what's spent combating "Islamist terrorists radicalizing over the Internet in the United States of America."</code>      | <code>The center works on the Internet's routing infrastructure (the SPRI program) and Domain Name System (DNSSEC), identity theft and other online criminal activity (ITTC), Internet traffic and networks research (PREDICT datasets and the DETER testbed), Department of Defense and HSARPA exercises (Livewire and Determined Promise), and wireless security in cooperation with Canada.</code> | <code>1.0</code> |
 * Loss: [<code>BinaryCrossEntropyLoss</code>](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters:
   ```json
   {
 | 0.5391 | 1000 | 0.6883        | -                        |
 | 0.8086 | 1500 | 0.6841        | -                        |
 | -1     | -1   | -             | 0.5670                   |
+| 0.2695 | 500  | 0.6741        | -                        |
+| 0.5391 | 1000 | 0.6662        | -                        |
+| 0.8086 | 1500 | 0.6504        | -                        |
+| -1     | -1   | -             | 0.6569                   |
 ### Training Time

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17168fd353619c37e372ebd2c91567ea4eb245987472a8976943c375e76b1e71
 size 737716172

 version https://git-lfs.github.com/spec/v1
+oid sha256:2c1672b14f00ee02f13738af219d2591d8cd95fbc93e474a2647541c519f7e23
 size 737716172