Arcoson/distilgpt2-arxiv-csai

DistilGPT-2 Fine-Tuned on arXiv CS/AI Abstracts

A DistilGPT-2 (82M parameters) model fine-tuned on computer science and artificial intelligence paper abstracts from arXiv. The model generates text in the style of academic CS/AI research abstracts.

Model Details

Property	Value
Base Model	distilgpt2
Parameters	81.9M total, 14.2M trainable (17.3%)
Fine-tuning Strategy	Partial freeze — last 2 transformer blocks + LM head
Training Data	ccdv/arxiv-classification (CS/AI subset)
Training Samples	2,000
Max Sequence Length	128 tokens
License	MIT

Training Details

Dataset

The model was fine-tuned on abstracts from the ccdv/arxiv-classification dataset, filtered for computer science and AI categories:

• cs.CV (Computer Vision)
• cs.AI (Artificial Intelligence)
• cs.SY (Systems and Control)
• cs.CE (Computational Engineering)
• cs.PL (Programming Languages)
• cs.IT (Information Theory)
• cs.DS (Data Structures and Algorithms)
• cs.NE (Neural and Evolutionary Computing)

Hyperparameters

Parameter	Value
Learning rate	3e-4
Batch size	8
Max steps	100
Warmup steps	10
Scheduler	Cosine
Weight decay	0.01
Frozen layers	Embedding + blocks 0-3
Trainable layers	Blocks 4-5 + LayerNorm + LM Head

Metrics

Metric	Value
Training loss	3.82
Eval loss	3.48
Perplexity	32.59
Training time	~7 minutes (CPU)

Usage

from transformers import pipeline

generator = pipeline("text-generation", model="Arcoson/distilgpt2-arxiv-csai")

output = generator(
    "arXiv Abstract: We propose a novel approach to",
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
)
print(output[0]["generated_text"])

Or load directly:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Arcoson/distilgpt2-arxiv-csai")
model = AutoModelForCausalLM.from_pretrained("Arcoson/distilgpt2-arxiv-csai")

inputs = tokenizer("arXiv Abstract: In this paper, we study the problem of", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sample Generations

Prompt: "arXiv Abstract: We propose a novel approach to"

Fine-tuned output:

We propose a novel approach to the problem of time-time in the computation model, i.e., and prediction models for (i) continuous intervals which are nonlinear because they cannot be solved with finite random variables or large number functions...

Base DistilGPT-2 output:

We propose a novel approach to the question of whether we can solve this problem by using real-time neural networks in an MRI technique...

The fine-tuned model produces more domain-appropriate language with CS/AI terminology compared to the base model.

Limitations

• Short context: Trained on 128-token sequences, so it works best for generating short abstracts or completions.
• Limited training: Only 100 steps on 2,000 samples — this is a proof-of-concept fine-tune. More training data and steps would improve quality.
• Domain narrow: Outputs are biased toward CS/AI academic language. Not suitable for general-purpose text generation.
• No factual accuracy: The model generates plausible-sounding but not necessarily factually correct research text.

Training Infrastructure

• Hardware: CPU (2 vCPUs, 8GB RAM)
• Framework: HuggingFace Transformers 5.5.0, PyTorch 2.11
• Strategy: Partial layer freezing to enable CPU-feasible training

Citation

If you use this model, please cite the base model and dataset:

@article{sanh2019distilbert,
  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
  journal={arXiv preprint arXiv:1910.01108},
  year={2019}
}