| --- |
| {} |
| --- |
| # ERC Classifiers |
|
|
| This repository contains a model trained for multi-label classification of scientific papers in the ERC (European Research Council) context. The model predicts multiple categories for a paper, such as its research domain or topic, based on the abstract and title. |
|
|
| ## Model Description |
|
|
| The model is based on **SPECTER** (a transformer-based model pre-trained on scientific literature), fine-tuned for **multi-label classification** on a dataset of scientific papers. The model classifies papers into several categories, which are defined by the **ERC categories**. The fine-tuned model is trained to predict these categories given the title and abstract of each paper. |
|
|
| ### Preprocessing |
|
|
| The preprocessing pipeline involves: |
|
|
| 1. **Data Loading**: Papers are loaded from a Parquet file containing the title, abstract, and their respective categories. |
| 2. **Label Cleaning**: Labels (categories) are processed to remove any unnecessary information (like content within parentheses). |
| 3. **Label Encoding**: Categories are transformed into a binary matrix using the **MultiLabelBinarizer** from scikit-learn. Each category corresponds to a column, and the value is `1` if the paper belongs to that category, `0` otherwise. |
| 4. **Statistics and Visualization**: Basic statistics and visualizations, such as label distributions, are generated to help understand the dataset better. |
|
|
| ### Training |
|
|
| The model is fine-tuned on the preprocessed dataset using the following setup: |
|
|
| * **Base Model**: The model uses the `allenai/specter` transformer as the base model for sequence classification. |
| * **Optimizer**: AdamW optimizer with a learning rate of `5e-5` is used. |
| * **Loss Function**: Binary Cross-Entropy with logits (`BCEWithLogitsLoss`) is employed, as the task is multi-label classification. |
| * **Epochs**: The model is trained for **5 epochs** with a batch size of 4. |
| * **Training Data**: The model is trained on a processed dataset stored in `train_ready.parquet`. |
|
|
| ### Evaluation |
|
|
| The model is evaluated using both **single-label** and **multi-label** metrics: |
|
|
| #### Single-Label Evaluation |
|
|
| * **Accuracy**: The accuracy is measured by checking how often the true label appears in the predicted labels. |
| * **Precision, Recall, F1**: These metrics are calculated for each class and averaged for the entire dataset. |
|
|
| #### Multi-Label Evaluation |
|
|
| * **Micro and Macro Metrics**: Precision, recall, and F1 scores are computed using both micro-averaging (overall performance) and macro-averaging (performance per label). |
| * **Label Frequency Plot**: A plot showing the frequency distribution of labels in the test set. |
| * **Top and Bottom F1 Plot**: A plot visualizing the top and bottom labels based on their F1 scores. |
|
|
| ## Dataset |
|
|
| The dataset consists of scientific papers, each with the following columns: |
|
|
| * **title**: The title of the paper. |
| * **abstract**: The abstract of the paper. |
| * **label**: A list of categories (labels) assigned to the paper. |
|
|
| The dataset is preprocessed and stored in a `train_ready.parquet` file. |
|
|
| ## Files |
|
|
| * `config.json`: Model configuration file. |
| * `model.safetensors`: Saved fine-tuned model weights. |
| * `tokenizer.json`: Tokenizer configuration for the fine-tuned model. |
| * `tokenizer_config.json`: Tokenizer settings. |
| * `special_tokens_map.json`: Special tokens used by the tokenizer. |
| * `vocab.txt`: Vocabulary file for the fine-tuned tokenizer. |
|
|
| ## Usage |
|
|
| To use the model, follow these steps: |
|
|
| 1. **Install Dependencies**: |
|
|
| ```bash |
| pip install transformers torch datasets |
| ``` |
|
|
| 2. **Load the Model and Tokenizer**: |
|
|
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| model_name = "SIRIS-Lab/erc-classifiers" |
| |
| # Load fine-tuned model and tokenizer |
| model = AutoModelForSequenceClassification.from_pretrained(model_name) |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| ``` |
|
|
| 3. **Use the Model for Prediction**: |
|
|
| ```python |
| # Example paper title and abstract |
| text = "Example title and abstract of a scientific paper." |
| |
| # Tokenize the input text |
| inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True) |
| |
| # Make predictions |
| with torch.no_grad(): |
| logits = model(**inputs).logits |
| |
| # Apply sigmoid activation to get probabilities |
| probabilities = torch.sigmoid(logits) |
| |
| # Get predicted labels (threshold at 0.5) |
| predicted_labels = (probabilities >= 0.5).long().cpu().numpy() |
| print(predicted_labels) |
| ``` |
|
|
| ## Conclusion |
|
|
| This model provides an efficient solution for classifying scientific papers into multiple categories based on their content. It uses state-of-the-art transformer-based techniques and is fine-tuned on a real-world dataset of ERC-related scientific papers. |
|
|
