README.md

---
language:
- und
license: cc-by-4.0
tags:
- indus-script
- ancient-scripts
- archaeology
- nlp
- text-generation
- sequence-modeling
- grammar-analysis
- undeciphered-script
library_name: transformers
pipeline_tag: text-generation
---

# Indus Script Models

Trained models for validating, predicting, and generating sequences in the undeciphered
Indus Valley Script (2600–1900 BCE). Built on 3,310 real archaeological inscriptions.

---

## Quick Start (3 steps)

```bash
# Step 1 — Clone the repo
git clone https://huggingface.co/hellosindh/indus-script-models
cd indus-script-models

# Step 2 — Install dependencies
pip install torch transformers

# Step 3 — Run the demo
python inference.py --task demo
```

---

## What you can do

### 1. Validate a sequence
Is this inscription grammatically valid?

```bash
python inference.py --task validate --sequence "T638 T177 T420 T122"
```

Output:
```
Sequence  : T638 T177 T420 T122
BERT      : 0.9650
N-gram    : 0.8930
ELECTRA   : 0.9410
Ensemble  : 0.9410
Verdict   : VALID (>=85%)
```

### 2. Predict a masked sign
What sign most likely fills the missing position?

```bash
python inference.py --task predict --sequence "T638 [MASK] T420 T122"
```

Output:
```
Position 1 predictions:
  T177    18.3%
  T243    12.1%
  T653     9.4%
  T684     7.2%
  T650     5.8%
```

### 3. Generate new sequences

```bash
# Generate 10 sequences (default threshold 85%)
python inference.py --task generate --count 10

# More variety, less strict
python inference.py --task generate --count 20 --threshold 0.78

# High quality only
python inference.py --task generate --count 5 --threshold 0.92
```

### 4. Score any sequence

```bash
python inference.py --task score --sequence "T604 T123 T609"
```

---

## Generating more diverse or longer sequences

Open `inference.py` and find the `task_generate` function. Change the temperature list:

**More random — forces rare signs to appear:**
```python
# Change this line:
temps = [0.85, 0.90, 1.00, 1.10]
# To:
temps = [1.10, 1.20, 1.30, 1.40]
```

**Longer sequences:**
Find the `generate()` method inside `load_nanogpt()` and change `max_len`:
```python
# Default (avg 7 signs):
def generate(self, temperature=0.85, top_k=40, max_len=15):

# For longer sequences:
def generate(self, temperature=0.85, top_k=40, max_len=25):

# For shorter sequences:
def generate(self, temperature=0.85, top_k=40, max_len=6):
```

---

## Pros and cons of tuning

| Setting | Effect | Good for | Watch out for |
|---|---|---|---|
| Temperature 0.7–0.8 | Very focused, repeats common signs | High quality outputs | Low diversity |
| Temperature 0.9–1.0 | Balanced — default | General use | Nothing |
| Temperature 1.1–1.3 | More variety, rare signs appear | Exploring vocabulary | Some unusual sequences |
| Temperature above 1.4 | Very random | Stress testing | Most sequences fail quality gate |
| Threshold 0.85 | Strict — default | Publication quality | Slower generation |
| Threshold 0.75 | Relaxed | Larger datasets | Lower average quality |
| Threshold 0.92 | Very strict | Highest confidence only | Very few sequences pass |
| max_len 6 | Short sequences | Matching real length distribution | Misses complex patterns |
| max_len 20+ | Long sequences | Complex grammar patterns | Not representative of real seals |

---

## Displaying Indus glyphs

Sequences use sign IDs like T638, T177. To see actual glyphs:

1. Search for **indus-brahmi-font** and download it
2. The `glyphs` field in output shows the rendered glyph characters
3. Open `data/id_to_glyph.json` to see the full sign to character mapping
4. If want to see mapping with T, open `data/indus_tokenizer/indus_id_map.json`

Without the font installed, glyphs show as boxes or question marks.
The sign IDs (T638, T177 etc.) always work regardless of font.

---

## Repo structure

```
indus-script-models/
├── inference.py              run this for all tasks
├── indus_ngram.py            required by ngram_model.pkl — do not move
├── README.md
├── models/
│   ├── nanogpt_indus.pt      NanoGPT generator (153K params, PPL 13.3)
│   ├── ngram_model.pkl       N-gram RTL model (88.2% pairwise accuracy)
│   ├── mlm/                  TinyBERT masked language model (val loss 2.06)
│   ├── cls/                  TinyBERT classifier (89.0% test accuracy)
│   ├── electra/              ELECTRA discriminator (95.1% token accuracy)
│   └── deberta/              DeBERTa discriminator (87.1% test accuracy)
└── data/
    ├── id_to_glyph.json      641 sign ID to glyph character mappings
    └── indus_tokenizer/      custom tokenizer for Indus Script
```

---

## How the pipeline works

**Stage 1 — Train on 3,310 real inscriptions:**

Four models trained independently, each learning a different aspect of grammar:

- **TinyBERT MLM** — learns which sign can fill a masked position in a sequence
- **TinyBERT Classifier** — learns to tell valid sequences from corrupted ones
- **N-gram RTL** — learns right-to-left transition probabilities between signs
- **ELECTRA** — learns token-level discrimination between real and fake signs
- **NanoGPT** — learns to generate new sequences from scratch

**Stage 2 — Generate and filter:**

NanoGPT generates candidate sequences in RTL order, then flips them to LTR.
Each candidate is scored by three models: BERT (50%) + N-gram (25%) + ELECTRA (25%).
Only sequences scoring 85% or higher are kept as valid synthetic sequences.
Sequences that exactly match real inscriptions are separated as seal reproductions.
Result: 5,000 novel sequences with 752 exact seal matches as validation evidence.

**Stage 3 — Retrain on combined data:**

The 5,000 synthetic sequences were combined with 3,310 real sequences (8,310 total).
All models were retrained on the larger dataset. Results improved significantly:

| Model | Before | After |
|---|---|---|
| TinyBERT accuracy | 78.4% | 89.0% |
| NanoGPT perplexity | 32.5 | 13.3 |
| DeBERTa accuracy | 80.5% | 87.1% |

The final 5,000 sequences in the dataset were generated with these retrained models.

---

## Key findings

- **RTL reading confirmed** — right-to-left has 12% stronger grammatical structure than LTR
- **Grammar proven** — entropy chain H1 to H2 to H3 = 6.03 to 3.41 to 2.39 bits (language-like decay)
- **Zipf law confirmed** — R squared = 0.968, language-like token distribution
- **752 seal reproductions** — model independently reproduced real archaeological inscriptions
- **Sign roles discovered:**
  - PREFIX signs at reading end: T638, T604, T406, T496
  - SUFFIX signs at reading start: T123, T122, T701, T741
  - CORE signs in the middle: T101, T268, T177, T243

---

## Known limitations

**DeBERTa calibration issue:**
DeBERTa scores near-zero for all sequences due to confidence calibration failure.
It is logged in output but excluded from the quality gate.
BERT, N-gram, and ELECTRA handle all scoring.

**Vocabulary coverage:**
Only about 26% of the 641 known Indus signs appear reliably in generated sequences.
475 signs appear 10 times or fewer in the real corpus — too rare for the model to learn.
This is a property of the archaeological record, not a model bug.
No synthetic corpus can reliably generate signs that barely exist in the training data.

**Short sequences:**
The model rarely generates length-2 sequences even though they are common in real inscriptions.
If you need shorter outputs, set `max_len=4` in the generate function.

---

## Dataset

The 5,000 synthetic sequences with full scores and sign index are available at:

[hellosindh/indus-script-synthetic](https://huggingface.co/datasets/hellosindh/indus-script-synthetic)