ai-sage/GigaChat3.1-702B-A36B-GGUF

GigaChat 3.1 Ultra

GigaChat 3.1 Ultra is the flagship instruct model of the GigaChat family. It is a large-scale Mixture-of-Experts (MoE) model with 702B total parameters and 36B active parameters, designed for multilingual assistant workloads, reasoning, code, tool use, and large-cluster deployment.

For high-performance inference, an fp8 version of the model is available - GigaChat3.1-702B-A36B.

bf16 version is also avaliable - GigaChat3.1-702B-A36B-bf16.

More details can be found in the Habr article.

Model architecture

GigaChat 3.1 Ultra uses a custom MoE architecture with the following key components.

Mixture-of-Experts (MoE)

The model has 702B total parameters with 36B active parameters at inference time. This allows it to scale model capacity aggressively while keeping the active compute budget much lower than that of an equally large dense model.

Multi-head Latent Attention (MLA)

Instead of standard multi-head attention, the model uses MLA, which compresses the KV cache into a latent representation. This reduces memory usage and improves inference throughput, especially in long-context settings.

Multi-Token Prediction (MTP)

The model is trained with MTP, which allows it to predict multiple tokens per forward pass. In production systems, this can be used with speculative or parallel decoding techniques to improve throughput.

Training data

The base GigaChat 3 training corpus spans 10 languages and includes books, academic material, code datasets, and mathematics datasets. All data goes through deduplication, language filtering, and automatic quality checks based on heuristics and classifiers.

Synthetic data remains a major contributor to quality. Across the broader training corpus, we used approximately 5.5 trillion synthetic tokens, including:

• question-answer data generated from source texts,
• reverse-prompt chains for structured data generation,
• model-authored notes embedded inside texts,
• millions of synthetic tasks with solutions in mathematics and olympiad-style programming,
• synthetic tests for code and reasoning tasks.

For the 3.1 release, we made major data improvements:

• Hard-domain expansion at Stage 1.5: stronger coverage of mathematics, finance, physics, engineering, biology, chemistry, and medicine.
• Stricter quality validation: our internal Revisor pipeline was extended with stronger checks for Markdown, LaTeX, and answer-format correctness.
• LLM-judge validation: SFT and DPO data is validated with judges selected for the task type and response structure.
• On-policy DPO data: preference pairs were generated from preview-model behavior, making them better aligned with real model failure modes.
• Better product-oriented data: we expanded data for search-and-citation scenarios, file-aware code interpretation, personalization, and agentic dialogues with executable tool calls.
• Improved answer style: we also revised formatting and writing guidelines to improve readability, correctness, and overall response quality.

Post-training improvements

DPO in native FP8

Unlike the preview release, GigaChat 3.1 Ultra includes a full DPO stage. This stage was redesigned for the MoE setup and trained in native FP8, not just quantized after training.

Important changes include:

• MTP heads trained during DPO for better consistency between main-model predictions and MTP predictions,
• weighted gamma with exponential decay over long sequences,
• stronger tuning of batch size and DPO contribution,
• better robustness against loop-inducing failure modes.

In our experiments, native FP8 DPO not only recovered the quality that could be lost with post-training FP8 quantization, but in some cases even exceeded the BF16 result while using substantially less memory.

Faster post-training

We also optimized the SFT pipeline with a combination of sequence packing, dynamic sequence parallelism, and additional pipeline optimizations. This reduced training cost significantly and improved GPU utilization, especially on long-context workloads.

Benchmark Results

Domain	Metric	GigaChat-2-Max	GigaChat-3-Ultra-Preview	GigaChat-3.1-Ultra	DeepSeek V3-0324	Qwen3-235B-A22B (Non-Thinking)
General Knowledge	MMLU RU	0.7999	0.7914	0.8267	0.8392	0.7953
General Knowledge	RUBQ	0.7473	0.7634	0.7986	0.7871	0.6577
General Knowledge	MEPA	0.6630	0.6830	0.7130	0.6770	-
General Knowledge	MMLU PRO	0.6660	0.7280	0.7668	0.7610	0.7370
General Knowledge	MMLU EN	0.8600	0.8430	0.8422	0.8820	0.8610
General Knowledge	BBH	0.5070	-	0.7027	-	0.6530
General Knowledge	SuperGPQA	-	0.4120	0.4892	0.4665	0.4406
Math	T-Math	0.1299	0.1450	0.2961	0.1450	0.2477
Math	Math 500	0.7160	0.7840	0.8920	0.8760	0.8600
Math	AIME	0.0833	0.1333	0.3333	0.2667	0.3500
Math	GPQA Five Shot	0.4400	0.4220	0.4597	0.4980	0.4690
Coding	HumanEval	0.8598	0.9024	0.9085	0.9329	0.9268
Total	Mean	0.6021	0.6115	0.6764	0.6482	0.6398

Arena Results

Arena	GigaChat-2-Max	GigaChat-3-Ultra-Preview	GigaChat-3.1-Ultra	DeepSeek V3-0324
Arena Hard Logs V3	64.9	50.5	90.2	80.1
Validator SBS Pollux	54.4	40.1	83.3	74.5
RU LLM Arena	55.4	44.9	70.9	72.1
Arena Hard RU	61.7	39.0	82.1	70.7
Average	59.1	43.6	81.625	74.4

Usage Example

llama-cpp version>=8495 (58c81f7e8)

cmake version>=3.28.3

Build llama-cpp

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON # build for CUDA
cmake --build build --config Release --target llama-server -j 8

Start the server

# Specify the full path to your model.gguf
export MODEL_PATH="/path/to/your/model.gguf"

./build/bin/llama-server \
    -m $MODEL_PATH \
    -np 1 \
    -cb \
    -ctk q8_0 \
    -ctv q8_0 \
    -fa on \
    --n-gpu-layers 20 \ # number of layers on GPU
    --ctx-size 32768 \
    --port 8080 \
    --host 0.0.0.0 \
    --jinja

Request example

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "tool_choice": "none",
    "messages": [
      {
        "role": "user",
        "content": "Докажи теорему о неподвижной точке"
      }
    ],
    "max_tokens": 1000,
    "temperature": 0
  }'

Request example with function call

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "temperature": 0,
  "messages": [
    {
      "role": "user",
      "content": "Какая сейчас погода в Москве?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Получить информацию о текущей погоде в указанном городе.",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {
              "type": "string",
              "description": "Название города (например, Москва, Казань)."
            }
          },
          "required": ["city"]
        }
      }
    }
  ]
}'