sewoong/korean-neural-sparse-encoder-base-klue-large

Korean Neural Sparse Encoder (klue/roberta-large)

A Korean sparse retrieval model trained with SPLADE-max on klue/roberta-large (337M params).

This model generates sparse lexical representations for Korean text, enabling efficient keyword-expanded retrieval via OpenSearch neural_sparse queries.

Model Details

Property	Value
Base Model	klue/roberta-large
Parameters	337M (24 layers, 1024 hidden, 32K vocab)
Training Method	SPLADE-max (log(1+ReLU) + max pooling)
Loss	InfoNCE + FLOPS regularization
Training Data	4.84M Korean triplets (46 shards)
Epochs	25
Effective Batch Size	2048 (128/GPU x 2 grad_accum x 8 GPUs)
Hardware	8x NVIDIA B200 (183GB VRAM)
Training Time	13.4 hours

Benchmark Results (Recall@1)

Benchmark	Queries	BM25	V33-base (149M)	This Model (337M)	Delta vs V33
Ko-StrategyQA	592	53.7%	62.2%	64.2%	+2.0pp
MIRACL-ko	213	44.1%	61.0%	70.4%	+9.4pp
Mr.TyDi-ko	421	55.6%	72.7%	76.2%	+3.6pp

Full Metrics

Benchmark	R@1	R@5	R@10	MRR@10	nz_q	nz_d
Ko-StrategyQA	64.2%	82.4%	85.3%	72.1%	37	71
MIRACL-ko	70.4%	89.7%	96.2%	79.1%	35	77
Mr.TyDi-ko	76.2%	92.6%	96.0%	83.4%	35	75

Usage

With Transformers (PyTorch)

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_name = "sewoong/korean-neural-sparse-encoder-base-klue-large"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
model.eval()

text = "서울 맛집 추천해주세요"
inputs = tokenizer(text, return_tensors="pt", max_length=256, truncation=True, padding=True)

with torch.no_grad():
    logits = model(**inputs).logits
    sparse = torch.log1p(torch.relu(logits))
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    sparse_vec = (sparse * mask).max(dim=1).values.squeeze()

# Top activated tokens
top_indices = torch.nonzero(sparse_vec).squeeze()
top_values = sparse_vec[top_indices]
sorted_idx = top_values.argsort(descending=True)

for idx in sorted_idx[:20]:
    token_id = top_indices[idx].item()
    weight = top_values[idx].item()
    token = tokenizer.decode([token_id])
    print(f"  {token}: {weight:.3f}")

With OpenSearch

PUT /my-index
{
  "settings": {
    "index": {
      "default_pipeline": "sparse-pipeline"
    }
  },
  "mappings": {
    "properties": {
      "text": { "type": "text" },
      "sparse_embedding": { "type": "rank_features" }
    }
  }
}

Training Configuration

model:
  name: "klue/roberta-large"
  dropout: 0.1
loss:
  lambda_q: 0.01
  lambda_d: 0.003
  temperature: 1.0
  flops_warmup_steps: 20000
  lambda_initial_ratio: 0.1
data:
  batch_size: 128
  query_max_length: 64
  doc_max_length: 256
training:
  num_epochs: 25
  learning_rate: 5.0e-5
  weight_decay: 0.01
  warmup_ratio: 0.06
  gradient_accumulation_steps: 2

Related Models

• sewoong/korean-neural-sparse-encoder - V33 base model (149M, skt/A.X-Encoder-base)
• opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1 - OpenSearch official multilingual sparse model

License

Apache License 2.0