JagritiRawat/IndicBertology

IndicBERT Probing and Fine-tuning Analysis

Extension of the paper: IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages? (Aravapalli et al., 2024)

This repository reproduces the probing experiments from the IndicSentEval paper on IndicBERT (by AI4Bharat) and extends the analysis with two new contributions — stronger MLP probing and end-to-end fine-tuning — to measure IndicBERT's true linguistic capacity beyond what standard probing reveals.

---

Repository Structure

IndicBertology/ │ ├── src/ │ ├── run_indicbert.py # Probing script — reproduces paper's LogReg methodology │ ├── finetune_indicbert.py # Fine-tuning + Frozen MLP comparison (our extension) │ ├── classifier.py # Original repo classifier script │ ├── ada.py # Original repo embedding extraction script │ └── probingData/ # Dataset CSVs (6 languages × 9 tasks) │ ├── hindi/ │ ├── marathi/ │ ├── telugu/ │ ├── malayalam/ │ ├── kannada/ │ └── urdu/ │ ├── utils/ │ └── ssfAPI.py # Parses SSF (Shakti Standard Format) annotated data │ ├── guidelines/ # Project guidelines and SSF format documentation │ ├── results/ │ ├── indicbert_results.json # Probing results — all 12 layers, all tasks │ └── finetuning_comparison.json # Frozen MLP vs Fine-tuned comparison │ └── finetuning_comparison.csv # Same results in CSV format │ ├── requirements.txt └── README.md

---

Our Three Experimental Approaches

Approach 1 — LogisticRegression Probe (Paper's Method, Reproduced)

IndicBERT (frozen) → extract all 12 layer embeddings → LogisticRegression → best layer accuracy

Answers: Which layer best encodes each linguistic property?

Approach 2 — Frozen MLP Probe (Our Extension)

IndicBERT (frozen) → CLS token from final layer → 2-layer MLP (768→256→ReLU→n_classes) → accuracy

Answers: Can a stronger non-linear classifier extract more from the same frozen representations?

Approach 3 — End-to-End Fine-tuning (Our Extension)

IndicBERT (trainable) + MLP head → trained end-to-end with differential LRs → accuracy

Answers: What is IndicBERT's true maximum linguistic capacity for each task?

---

Probing Tasks

Probing Task	Category	Number of Classes
Sentence Length (SentLen)	Surface	8
Bigram Shift (BShift)	Syntactic	2
Tree Depth (TreeDepth)	Syntactic	5
Subject Number (SubjNum)	Semantic	2
Object Number (ObjNum)	Semantic	2
Word Content	Semantic	var
Gender	Semantic	4
Number	Semantic	3
Person	Semantic	7

---

Examples of Each Class (Hindi)

Subject Number

Type	Example
sg	वैश्विक रूप से यह संक्रमण उन जगहों पर अधिक आम है जहां पर रोग - प्रतिरोधकता कम है ।
pl	सामान्य लोगों में से आधे लोगों में ये छोटे जीव नींद के दौरान प्रवेश करते हैं ।

Object Number

Type	Example
sg	उस वक्त के अंगरेज शासक लार्ड डलहौजी के नाम पर इसका नाम ' डलहौजी ' रख दिया गया ।
pl	अगस्त के बाद गर्म कपड़े साथ रखें ।

Bigram Shift

Type	Example
0 (original)	पंजपुला एक सैरगाह है ।
1 (shifted)	गर्मियों में ठंडी सड़क को सब लुभाती है ।

Gender

Type	Example
any	पंजपुला एक सैरगाह है ।
m	पहले चश्मे में इतना पानी होता था कि 7 धाराएँ बनती थीं ।
f	आज चश्मे में से एक धारा निकलती है ।

Number

Type	Example
sg	डलहौजी में भीड़भाड़ के स्थान पर शांत माहौल है ।
pl	लंबी छुट्टियाँ गुजारने वाले एकांतपसंद लोग डलहौजी बड़ी संख्या में आते हैं ।
any	सामान्य स्वेटर व शाल हर मौसम में चाहिए ।

Person

Type	Example
1	हम पूरी तरह तैयार होकर नहीं जाते ।
1h	हम क्या सबसे झगड़ा करती फिरती हैं का ?
2	चना दाल को दो घंटे पहले पानी में भिगो दें ।
2h	अगस्त के बाद गर्म कपड़े साथ रखें ।
3	पंजपुला एक सैरगाह है ।
3h	सुभाष चंद्र बोस कुछ दिन यहाँ आ कर रहे थे ।
any	कभी पंजपुला से धर्मशाला की ओर एक पैदल मार्ग जाता था ।

---

Languages Covered

Language	Code	Sentences	Family	Script
Hindi	hi	12,292	Indo-Aryan	Devanagari
Marathi	mr	12,029	Indo-Aryan	Devanagari
Telugu	te	3,222	Dravidian	Telugu
Malayalam	ml	7,667	Dravidian	Malayalam
Kannada	kn	9,806	Dravidian	Kannada
Urdu	ur	2,363	Indo-Aryan	Nastaliq

---

Perturbations (From Original Paper)

The original paper applies 13 text perturbations to test model robustness. These are defined in the src/ folder.

Perturbation Type	Description
Append Irrelevant	Append an irrelevant sentence
DropAll	Drop all content words
DropAllNouns	Drop all noun tokens
DropAllVerbs	Drop all verb tokens
DropFirst	Drop first word
DropLast	Drop last word
DropFirstLast	Drop first and last word
DropRandomNoun	Drop a random noun
DropRandomVerb	Drop a random verb
KeepBoth	Keep only subject and object
KeepOnlyNoun	Keep only nouns
KeepOnlyVerb	Keep only verbs
Shuffle	Randomly shuffle all words

---

Word Content — Mid-Frequency Words per Language

Language	Vocabulary Size
Hindi	781
Marathi	626
Kannada	416
Malayalam	188
Urdu	153
Telugu	34

---

Key Results

Probing Results — Best Layer Accuracy (LogisticRegression, Paper Method)

Task	Hindi	Marathi	Telugu	Malayalam	Kannada	Urdu
SentLen	63.1%	25.1%	—	30.6%	—	18.2%
SubjNum	67.5%	71.3%	73.2%	76.0%	79.7%	89.3%
ObjNum	70.1%	88.2%	64.7%	79.1%	80.9%	86.1%
TreeDepth	36.3%	22.9%	—	21.8%	27.9%	20.3%
BShift	71.3%	73.2%	75.2%	71.5%	72.8%	74.0%
Gender	51.4%	41.5%	45.9%	—	78.6%	46.1%
Number	60.5%	76.4%	55.8%	—	76.3%	67.2%
Person	66.2%	88.2%	68.5%	—	90.7%	72.8%

Fine-tuning Comparison — Telugu (Frozen MLP vs Fine-tuned)

Task	Frozen MLP	Fine-tuned	Δ
SentLen	82.6%	96.9%	+14.3%
SubjNum	71.9%	83.0%	+11.1%
ObjNum	66.4%	66.4%	0.0%
TreeDepth	55.8%	59.7%	+3.9%
BShift	71.8%	78.6%	+6.8%
Gender	47.9%	82.4%	+34.5%
Number	51.6%	89.8%	+38.1%
Person	69.3%	88.2%	+18.9%

---

Key Findings

1. Probing underestimates surface task capacity — SentLen accuracy in Urdu goes from 18.2% (LogReg probing) to 84.1% (fine-tuned), a +66% gain. IndicBERT has strong surface encoding capacity that probing alone cannot reveal.

2. Telugu morphology shows the largest fine-tuning gains — Number: +38.1%, Gender: +34.5%. Frozen representations fail to capture Telugu morphological features that the model can actively learn.

3. TreeDepth remains hard regardless of approach — Even with full fine-tuning, tree depth peaks at ~59.7% (Telugu), indicating a fundamental limitation in IndicBERT's syntactic depth encoding.

4. Kannada Person is already near-perfectly encoded — Frozen MLP: 89.6%, Fine-tuned: 90.8% (Δ = +1.2%). Pretraining naturally encodes Kannada person morphology near-perfectly.

5. Delta (Δ) analysis is our new contribution — Large Δ means probing underestimates model capacity. Zero Δ means frozen representations are already optimal. Negative Δ means fine-tuning overfit on small data.

---

Setup

1. Clone the repository

bash git clone https://github.com/YOUR_USERNAME/IndicBertology.git cd IndicBertology

2. Install dependencies

bash pip install -r requirements.txt

3. Download IndicBERT

The model is gated on HuggingFace. Request access at https://huggingface.co/ai4bharat/indic-bert, then place the model files at ./indic-bert-local/indic-bert/

The folder should contain:

indic-bert-local/indic-bert/ ├── config.json ├── pytorch_model.bin ├── spiece.model └── spiece.vocab

---

Running

Probing experiments — reproduces the paper's methodology

bash python src/run_indicbert.py

Extracts embeddings from all 12 IndicBERT layers, trains LogisticRegression on each, reports best-layer accuracy. Results saved to results/indicbert_results.json

Fine-tuning comparison — our extension

bash python src/finetune_indicbert.py

Trains frozen MLP probe and fine-tunes IndicBERT end-to-end for each task. Results saved to results/finetuning_comparison.json and results/finetuning_comparison.csv

---

Implementation Notes

• keep_accents=True in tokenizer — critical for preserving Indic vowel matras (diacritics)
• CLS token (index 0 of last hidden state) used as sentence representation
• Train / Dev / Test split: 70% / 10% / 20% with stratified sampling
• Early stopping with patience=3 on validation accuracy
• Fine-tuning uses differential learning rates: 2e-5 for IndicBERT, 1e-3 for classifier head
• Fresh IndicBERT weights loaded for each task to prevent cross-task interference
• MPS (Apple Silicon GPU) used for acceleration

---

Requirements

torch>=1.13.0 transformers>=4.30.0 sentencepiece>=0.1.99 numpy>=1.23.0 scikit-learn>=1.2.0 pandas>=1.5.0

---

About IndicBERT

IndicBERT is an ALBERT-based multilingual model created by AI4Bharat (IIT Madras) and pretrained on ~9 billion tokens across 12 Indian languages using Masked Language Modeling (MLM) and Sentence Order Prediction (SOP). It has 12 transformer layers with a hidden dimension of 768.

• Model: ai4bharat/indic-bert
• Architecture: ALBERT-base
• Hidden size: 768
• Layers: 12
• Vocabulary: 200,000 SentencePiece tokens

---

Original Paper and Codebase

This repository builds on the original IndicBertology codebase:

• Paper: IndicSentEval — Aravapalli et al., 2024. arXiv: 2410.02611
• Original repo: https://github.com/aforakhilesh/IndicBertology

bibtex @article{aravapalli2024indicsenteval, title = {IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?}, author = {Aravapalli, Akhilesh and others}, journal = {arXiv preprint arXiv:2410.02611}, year = {2024}, url = {https://arxiv.org/abs/2410.02611} }

---

Acknowledgements

• AI4Bharat for the IndicBERT model and IndicCorp dataset
• Authors of IndicSentEval for the probing dataset and methodology