README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - rubirlm
  - causal-lm
  - base-model
  - text-generation
  - 1b
  - moe
datasets:
  - HuggingFaceFW/fineweb
  - HuggingFaceH4/ultrachat_200k
pipeline_tag: text-generation

RubiRLM-1B-Base

RubiRLM-1B-Base is a 1B-parameter base language model released by DevHunterAI.

Model size: 1B parameters

Training datasets: FineWeb, UltraChat-200k

Model type: Base / pretrained language model

Important: This release is a base model. It can be used for prompt-based generation and experimental chat-style interaction, but it is not an instruction-tuned chat assistant.

Architecture

RubiRLM 1B uses a recursive language modeling architecture with recurrent state flow, Mixture-of-Experts routing, and conditional block execution.

Key Features

• 1B parameters
• Recursive Language Model (RLM) architecture
• 10 recursive blocks
• d_model = 1024
• 16 attention heads
• max sequence length = 2048
• 6 recursive reasoning steps
• Mixture-of-Experts: 32 experts, top-1 routing
• Layer skip router for conditional execution
• Packed execution support
• Tied token embedding and LM head

Training Data

This model was trained using a mixture of:

• FineWeb
• UltraChat-200k

Intended Usage

This model is intended for:

• base language modeling research
• continued pretraining
• experimental prompt-based generation
• architecture experimentation around recursive and MoE-based language models

Not Intended As

This release should not be treated as:

• a fully aligned assistant
• a safety-tuned production chatbot
• an instruction-following model with guaranteed conversational quality

Loading

Because this repository includes custom model code, loading may require trust_remote_code=True depending on your workflow.

Files

• pytorch_model.bin: exported RubiRLM weights
• training_checkpoint.pt: original training checkpoint
• config.json: Hugging Face-facing config
• rubirlm_config.json: full RubiRLM architecture config
• RubiRLM.py: model implementation
• xqs_moe.py, xqs_stack.py, x_quantum_sparse_ops.py, rubi_train_stack.py: supporting code

Notes

The exported weights were produced from the final training checkpoint and packaged for Hugging Face publication.