README.md

---
license: apache-2.0
language:
- en
- hi
- mr
- ta
- te
- kn
- ml
- bn
- pa
- gu
- or
pipeline_tag: text-generation
library_name: transformers
---

# Nandi-Mini-150M

## Introduction

Nandi-Mini-150M is a compact, efficient multilingual language model designed for strong performance in resource-constrained environments. It is pre-trained from scratch on **525 billion tokens** and supports **English and 10 Indic languages**.

We do not employ any benchmaxing tricks; the model is designed to be genuinely strong and highly effective for fine-tuning on downstream tasks. 

Nandi-Mini-150M focuses on maximizing performance per parameter through architectural efficiency rather than scale. It is optimized for edge devices, on-prem deployments, and low-latency applications, making it ideal for resource-constrained environments.
Nandi-Mini-150M brings the following key features:

- Strong **multilingual capability** across English and Indic languages
- Efficient design enabling **high performance at small scale (150M parameters)**
- Reduced memory footprint using **factorized embeddings**
- Better parameter efficiency through **layer sharing**

## 📝 Upcoming Releases & Roadmap

We’re just getting started with the Nandi series 🚀

- **Nandi-Mini-150M (Base)** — *Available now*  
- **Nandi-Mini-150M (Instruct)** — *Available now*   
- **Nandi-Mini-500M (Base + Instruct)** — Pre-Training Going On
- **Nandi-Mini-1B (Base + Instruct)** — Pre-Training Going On

We are actively working on expanding the Nandi family to cover a wider range of use cases—from lightweight edge deployments to more capable instruction-tuned systems.

📢 **Blogs & technical deep-dives coming soon**, where we’ll share:
- Architecture decisions and design trade-offs  
- Training insights and dataset composition  
- Benchmarks and real-world applications  

Stay tuned!

**This repo contains the base Nandi-Mini-150M model**, which has the following features:

- Type: Causal Language Model
- Training Stage: Pretraining (from scratch)
- Architecture: Transformer decoder with RoPE, RMSNorm, SwiGLU, GQA, tied embeddings, **factorize embeddings**
- Number of Layers: 16*2 [Layer Sharing, effective layer =32]
- Context Length: 2,048 tokens
- Vocabulary Size: 131,072

## 🌍 Supported Languages

The model is trained on English and a diverse set of Indic languages, including:

- Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia

## Benchmark Results

## 📊 Benchmark Comparison (~150M Class)

| Model Name         | Parameters | Tokens(B) | HellaSwag | Winogrande | GPQA  | MMLU | GSM8K | HumanEval | Average |
|------------------|---------------|------------------|----------|------------|------|------|-------|-----------|---------|
| Mobile-LLM-125M  | 125           | 1000             | 38.90    | 53.10  | -    | -    | -     | -         | -       |
| SmolLM-135M-Base | 135           | 600              | 42.66| 53.03  | 25.44| 25.30| 1.36  | 0.00      | 24.63   |
| SmolLM2-135M-Base| 135           | 2000             | 43.13| 53.27  | 22.09| 24.09| 1.74  | 0.00      | 24.05   |
| **Nandi-Mini-150M-Base** | **150**     | **500**          | 37.20    | 52.32      | **28.57** | **28.86** | **2.58** | **4.27** | **25.63** |


## 📊 Model Benchmark Comparison With Slightly Bigger Models (350M–600M Class)

| Model Name            | Parameters | Tokens(B) | HellaSwag | Winogrande | GPQA  | MMLU | GSM8K | HumanEval | Average |
|---------------------|---------------|------------------|----------|------------|------|------|-------|-----------|---------|
| Mobile-LLM-360M     | 350           | 1000             | 49.60    | 56.59      | -    | -    | -     | -         | -       |
| Qwen-2-0.5-Base     | 500           | 12000            | 49.01    | 57.69      | 27.23| 44.06| 10.61 | 22.56     | 35.19   |
| Qwen2.5-0.5B-Base   | 500           | 18000            | 52.16    | 56.82      | 24.10| 47.41| 4.77  | 29.87     | 35.86   |
| Qwen3-0.6B-Base     | 600           | 36000            | 53.77    | 59.19      | 30.80| 50.34| 15.31 | 28.04     | 39.58   |
| SmolLM-360M-Base    | 360           | 600              | 53.33    | 57.22      | 21.20| 24.92| 2.19  | 1.21      | 26.68   |
| SmolLM2-360M-Base   | 360           | 4000            | 56.30    | 59.19      | 25.22| 25.55| 2.88  | 0.00      | 28.19   |
| **Nandi-Mini-150M-Base** | **150**       | 500          | 37.20| 52.32  | 28.57 | 28.86 | 2.58 | 4.27 | 25.63 |

### Note
Mobile-LLM model checkpoints are not publicly available; their results are reported directly from the original paper. All other models have been evaluated using `lm-eval` under a consistent setup. Human-Eval & GSM8K have been evaluated using Greedy-decoding for now for all models.

## Performance onf Finetuned Tasks

#### CrossSum-Hindi (CHRF) Results
We finetuned our model and other open source models on [Google's IndicGenBench](https://github.com/google-research-datasets/indic-gen-bench/) Crossum-Hindi. Nandi-mini-150M was able to outperform other models.

| Base Model              | Before Finetune | After Finetune |
|------------------------|-----------------|----------------|
| Qwen-2-0.5-Base        | 0.09            | 4.22           |
| Qwen2.5-0.5B-Base      | 0.43            | 4.18           |
| SmolLM-135M-Base       | 0.09            | 2.55           |
| SmolLM-360M-Base       | 0.09            | 2.99           |
| SmolLM2-135M-Base      | 0.09            | 2.67           |
| SmolLM2-360M-Base      | 0.12            | 3.51           |
| Nandi-mini-150M        | 0.10            | **4.37**       |


## Tokenization Fertility Score across Languages

| Language  | SmolLM3-3B | Qwen3-0.6B-Base | Sarvam-1 | Nandi-Mini-150M |
|-----------|------------|-----------------|----------|------------------|
| English   | 1.17       | 1.16            | 1.32     | **1.18**             |
| Bengali   | 8.66       | 7.51            | 1.55     | **1.44**             |
| Gujarati  | 10.47      | 9.37            | 1.55     | **1.53**             |
| Hindi     | 2.71       | 5.14            | **1.25**     | 1.32             |
| Kannada   | 16.43      | 12.96           | 2.10     | **1.90**             |
| Malayalam | 17.77      | 14.56           | 2.49     | **2.05**             |
| Marathi   | 3.73       | 6.70            | 1.55     | **1.55**             |
| Oriya     | 19.07      | 15.75           |**2.18**     | 2.68             |
| Punjabi   | 9.23       | 8.66            | 1.47     | **1.42**             |
| Tamil     | 13.56      | 10.93           | 2.06     | **2.05**             |
| Telugu    | 15.40      | 13.38           | 2.09     | **1.77**             |
| Assamese  | 9.26       | 8.13            | 4.31     | **1.51**             |



## 🚀 Usage

```python
!pip install transformers=='5.4.0'

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "FrontiersMind/Nandi-mini-150M"

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.bfloat16
).to(device).eval()


prompt = """
The night was quiet and the streets were empty. 
A single light flickered in the distance. Someone was walking slowly, carrying a small bag. Suddenly,
"""
model_inputs = tokenizer([prompt], return_tensors="pt").to(model.device)

outputs = model.generate(
    **model_inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.3,
    top_k=20,
    repetition_penalty=1.1,
    top_p=0.95
)

response = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True,
)

print(response)
```



## 📬 Feedback & Suggestions

We’d love to hear your thoughts, feedback, and ideas!

- **Discord**: https://discord.gg/ZGdjCdRt
- **Email:** support@frontiersmind.ai
- **Official Website** https://www.frontiersmind.ai/
- **LinkedIn:** https://www.linkedin.com/company/frontiersmind/
- **X (Twitter):** https://x.com/FrontiersMind