This is a text classification model, fully fine-tuned from a ` allenai/scibert_scivocab_uncased`. It re-uses the main BERT model and fits an ordinal regression head on the `[CLS]` token. The model is fine-tuned on the certainty labels collected in [Wurl et al (2024): _Understanding Fine-Grained Distortions in Reports for Scientific Finding_](https://aclanthology.org/2024.findings-acl.369/). The authors originally collect certainty annotations from humans using a 4-point Likert Scale ranging from (1) Uncertain to (4) Certain. Because the resulting datasets suffer from severe class imbalance, we merge the classes (1) Uncertain and (2) Somewhat Uncertain. 



### Dataset Statistics

There are 1330 examples in the training set and 334 in the test set. 
Each example is a sentence long.
Examples are filtered from the [copenlu/spiced](https://huggingface.co/datasets/copenlu/spiced) dataset to exhibit final score greater or equal than 4.

The original base rates are as follows:

| Class | Base Rate in Training set | Base Rate in Test set |
| ----- | ------------------------- | --------------------- |
| 0 - Uncertain | 5.5970 | 7.1856 | 
| 1 - Somewhat Uncertain | 15.2985 |  17.6647 |
| 2 - Somewhat Certain  | 32.3881 | 33.2335 | 
| 3 - Certain | 46.7164 | 41.9162 |

After combining classes 0 and 1, we obtain the base rates below. Note that this mimicks the procedure adopted in the original paper.

| Class | Base Rate in Training set | Base Rate in Test set |
| ----- | ------------------------- | --------------------- |
| 0 - Uncertain | 20.8955 | 24.8503 | 
| 1 - Somewhat Certain  | 32.3881 | 33.2335 | 
| 2 - Certain | 46.7164 | 41.9162 |




### Hyperparameter Optimization

The published model represents one of the 29 models different configurations. The selected model maximizes Quadratic Weighted Kappa (implemented using [cohen_kappa with quadratic weights](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html)), which is better adapted to ordinal problems, such as ordinal scales. Under this metric, a random model would score 0. We adopt this metric as opposed to accuracy or macro F1 to address class imbalances. 

Here is the classification report and test set metrics:

```
17:44:36  INFO        test  loss=0.9565  acc=0.578  QWK=0.5004
17:44:36  INFO      
              precision    recall  f1-score   support

           0       0.58      0.51      0.54        83
           1       0.47      0.46      0.46       111
           2       0.65      0.71      0.68       140

    accuracy                           0.58       334
   macro avg       0.57      0.56      0.56       334
weighted avg       0.57      0.58      0.57       334
```


We conduct a hyperparameter sweep of the following hyperp

- Freeze / Unfreeze 
- LR: 1e-6 through 1e-3
- Batch Size: 16, 32
- Hidden Size Dimensions: 256, 128
- Warmup Ratio: 0.05, 0.1, 0.2, 0.3
- Epochs 30 (with patience)



## Usage 
 
```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("cbelem/scibert-certainty-ordinal", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("cbelem/scibert-certainty-ordinal", trust_remote_code=True)
```