---
language:
- pl
- en
license: apache-2.0
base_model: answerdotai/ModernBERT-base
tags:
- chunking
- semantic-segmentation
- token-classification
- modernbert
- nlp
- rag
pipeline_tag: token-classification
datasets:
- wikimedia/wikipedia
---

# ModernBERT Chunker Base 🚀

This model is a fine-tuned version of **ModernBERT-base**, specialized in **semantic boundary detection**. It is designed to be used with the [fine-chunker](https://github.com/JerzyCode/fine-chunker) library for high-quality text segmentation in RAG applications.

## Model Highlights
- **Context Length**: 8192 tokens (full ModernBERT capacity).
- **Architecture**: ModernBERT-base + Deep Classification Head (Linear-ReLU-Dropout-Linear).
- **Training Strategy**: Sequential packing of full Wikipedia articles with weighted Cross-Entropy.
- **Languages**: Bilingual support for **Polish** and **English**.

## Usage

The easiest way to use this model is through the official library:

```python
from fine_chunker import Chunker

# Load the model (runs optimally on CUDA or CPU)
chunker = Chunker.from_pretrained(device="cpu", use_onnx=True)

text = "Your long multi-topic document..."
chunks = chunker.chunk(text)

for chunk in chunks:
    print(f"Index: {chunk.index} | Content: {chunk.content[:100]}...")
```

## Training Details

### Dataset
The model was trained on **Wikipedia (20231101 version)** for both Polish and English.
- **Preprocessing**: Full articles were cleaned of wiki-noise (references, external links, metadata). Additionally, 40% of chunk starts were replaced by a lowercase letter, and 40% of the last dots in chunks were removed.
- **Ground Truth**: Segmentation was based on natural paragraph boundaries (`\n\n`) found in well-structured Wikipedia articles.
- **Packing**: Multiple articles were packed into single `8192` token sequences to maximize training efficiency.

### Training Configuration
- **Hardware**: 4x NVIDIA A100-SXM4-40GB.
- **Duration**: 1 day, 6 hours, 1 minute.
- **Precision**: `bfloat16` with Flash Attention 2.
- **Epochs**: 1
- **Optimization**:
    - **Loss Function**: Weighted Cross-Entropy (`[1.0, 7.0]`) to address boundary sparsity.
    - **Gradient Accumulation**: 8 steps.
    - **Dropout**: 0.1.

### Architecture Details
Unlike standard token classifiers that use a single linear layer, this model uses a **deep classification head**:
1. `Linear(hidden_size, hidden_size)`
2. `ReLU`
3. `Dropout(0.1)`
4. `Linear(hidden_size, 2)` (Boundary vs. Non-boundary)

This allows the model to learn more complex semantic cues for segmentation.

## Intended Use
- **RAG Pipelines**: Generating semantic chunks that preserve context better than fixed-size splitting.
- **Long Document Analysis**: Segmenting reports, legal documents, or books into logical chapters/sections.
- **Pre-processing for LLMs**: Ensuring input fragments are semantically complete.

## Limitations & Future Work

- **Training Data Focus**: The current version was trained exclusively on **Wikipedia datasets** (English and Polish). While it excels at structured, informative prose, it hasn't been exposed to noisy data, conversational text, or specific journalistic styles (news).
- **Base Model Version**: This is a general-purpose base model. While it performs excellently on standard structured text, specialized domains (e.g., legal contracts, medical records, or minified code) might require additional fine-tuning for optimal boundary detection.
- **Logical Structure**: Performance is best on documents with clear paragraph breaks and logical flow, similar to the encyclopedic style of its training data.
- **Niche Domains**: If you're working with datasets far removed from Wikipedia's structure, feel free to reach out or share your feedback—we're looking into domain-specific refinements.

## Evaluation
Status: Under Development > Systematic evaluation of the model's performance across different domains and languages is currently in progress.


## Author
Developed by **Jerzy Boksa**.
Contact: devjerzy@gmail.com
GitHub: [fine-chunker](https://github.com/JerzyCode/fine-chunker)

## Acknowledgements
This model was trained using the infrastructure provided by **Cyfronet** (Academic Computer Centre Cyfronet AGH) as part of a educational grant.



## Citation

If you use this model or the `fine-chunker` library in your research or project, please cite it as follows:

```bibtex
@misc{boksa2026modernbertchunker,
  author = {Jerzy Boksa},
  title = {ModernBERT Chunker Base: Specialized Semantic Boundary Detection for RAG},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/jboksa/modbert-chunker-base}}
}