ubergarm

/

DeepSeek-V3-0324-GGUF

like 35

@SFPLM

Glad you got it going and can now begin to optimzie for your exact system. There is a lot of info out thee already so check the discussions on ik_llama.cpp github and other threads here.

First off compile like this especially when using 2x 3090TI. The SCHED_MAX_COPIES will make it easier to understand VRAM allocation and increase batch sizes for more PP. The FORCE_BF16 may increase speed on 3090's and also prevent numerical issues specific to DeepSeek MLA.

cmake -B ./build -DGGML_CUDA=ON -DGGML_BLAS=OFF -DGGML_SCHED_MAX_COPIES=1 -DGGML_CUDA_IQK_FORCE_BF16=1 -DGGML_CUDA_F16=ON
cmake --build ./build --config Release -j $(nproc)

Next, how is your NUMA configured? I assume this is a single socket system? Make sure you have BIOS configured to show only a single numa node e.. numactl --hardware shows only one. As no llama.cpp fork has optimizations here for this and you don't have enough RAM to try ktransformers USE_NUMA=1 which doubles RAM usage (one per CPU socket numa node).

If you can't get a single NUMA node let me know and you can use some numactl stuff to glue them togther to get the best you can.

Regarding your command, try this, I will assume you are using 2x 3090TI which will give more VRAM to offload additional layers

./build/bin/llama-server \
    --model /mnt/.../ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf \
    --alias ubergarm/DeepSeek-R1-V3-0324-IQ4_K_R4 \
    --ctx-size 32768 \
    -ctk q8_0 \
    -mla 3 -fa \
    -amb 512 \
    -fmoe \
    --temp 0.3 \
    --min-p 0.05 \
    --n-gpu-layers 63 \
    -ot "blk\.(3|4)\.ffn_.*=CUDA0" \
    -ot "blk\.(5|6)\.ffn_.*=CUDA1" \
    -ot exps=CPU \
    -ub 2048 -b 2048 \
    --parallel 1 \
    --threads 56 \
    --host 127.0.0.1 \
    --port 8080

Adjust number of additional ffn layers offloaded to CUDA0 and CUDA1 based on if you OOM or not. Basically crank it up until you OOM and dial back by one. Increasing batch sizes takes a little more VRAM but gives a lot more PP.

When you want to benchmark speeds, I advise to use llama-sweep-bench. Simply replace llama-server with llama-sweep-bench and add --warmup-batch and it should ignore alias/host/port fine.

Finally use the most recent version of ik_llama.cpp as PP just got a boost for the _R4 quants running on CUDA like u are doing.

TG will be limited by your RAM bandwidth and in my experience can be below theoretical maximum on intel rigs even when mlc (intel memory latency checker) bandwidth reads higher.

Cheers and keep us all posted!

@ubergarm

Glad to hear and appreciate the advice. After doing a bit of tampering I seem to cannot get it to load more than 1 expert layer to each GPU. I have tried ts 23,24 and ctv (v cache) to q8 and i just cant.

For example

Tensor blk.3.ffn_norm.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_gate_inp.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_gate_exps.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_down_exps.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_up_exps.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_gate_shexp.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_down_shexp.weight buffer type overriden to CUDA0
Tensor blk.3.ffn_up_shexp.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_norm.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_gate_inp.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_gate_exps.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_down_exps.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_up_exps.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_gate_shexp.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_down_shexp.weight buffer type overriden to CUDA0
Tensor blk.4.ffn_up_shexp.weight buffer type overriden to CUDA0
Tensor blk.5.ffn_norm.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_gate_inp.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_gate_exps.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_down_exps.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_up_exps.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_gate_shexp.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_down_shexp.weight buffer type overriden to CUDA1
Tensor blk.5.ffn_up_shexp.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_norm.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_gate_inp.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_gate_exps.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_down_exps.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_up_exps.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_gate_shexp.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_down_shexp.weight buffer type overriden to CUDA1
Tensor blk.6.ffn_up_shexp.weight buffer type overriden to CUDA1
Tensor blk.7.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.7.ffn_down_exps.weight buffer type overriden to CPU
... We know what it looks like 
Tensor blk.60.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.60.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.60.ffn_up_exps.weight buffer type overriden to CPU
llm_load_tensors: offloading 61 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 62/62 layers to GPU
llm_load_tensors:        CPU buffer size =  9218.28 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 31988.10 MiB
llm_load_tensors:        CPU buffer size =   938.98 MiB
llm_load_tensors:      CUDA0 buffer size = 21945.34 MiB
llm_load_tensors:      CUDA1 buffer size = 21782.68 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 32768
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 2048
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn   = 3
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe  = 1
llama_new_context_with_model: ser        = -1, 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =   592.89 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   573.76 MiB
llama_new_context_with_model: KV self size  = 1166.62 MiB, c^KV (q8_0): 1166.62 MiB, kv^T: not used
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=1)
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4472.01 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 4689240064
llama_new_context_with_model: failed to allocate compute buffers

The above also happens if i attempt to add 2 layers. I also tried 3 to CUDA0 and 4 + 5 into CUDA1, CUDA1 OOMs.

The one I have tried so far that works and is fastest so far is:

./build/bin/llama-sweep-bench \
    --model /mnt/.../models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf \
    --alias ubergarm/DeepSeek-R1-V3-0324-IQ4_K_R4 \
    --ctx-size 32768 \
    -ctk q8_0 \
    -mla 3 -fa \
    -amb 512 \
    -fmoe \
    --temp 0.3 \
    --min-p 0.05 \
    -ts 23,24 \
    --n-gpu-layers 63 \
    -ot "blk\.(3)\.ffn_.*=CUDA0" \
    -ot "blk\.(4)\.ffn_.*=CUDA1" \
    -ot exps=CPU \
    -ub 2048 -b 2048 \
    -ser 6,1 \
    --parallel 1 \
    --threads 56

gets me:

llm_load_tensors: offloading 61 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 62/62 layers to GPU
llm_load_tensors:        CPU buffer size = 22726.83 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 40525.67 MiB
llm_load_tensors:        CPU buffer size = 31988.10 MiB
llm_load_tensors:        CPU buffer size =   938.98 MiB
llm_load_tensors:      CUDA0 buffer size = 15500.99 MiB
llm_load_tensors:      CUDA1 buffer size = 15235.03 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 32768
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 2048
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn   = 3
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe  = 1
llama_new_context_with_model: ser        = 6, 1
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =   592.89 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   573.76 MiB
llama_new_context_with_model: KV self size  = 1166.62 MiB, c^KV (q8_0): 1166.62 MiB, kv^T: not used
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=1)
llama_new_context_with_model:      CUDA0 compute buffer size =  4472.01 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =  3560.02 MiB
llama_new_context_with_model:  CUDA_Host comput```e buffer size =   312.02 MiB
llama_new_context_with_model: graph nodes  = 8245
llama_new_context_with_model: graph splits = 149

main: n_kv_max = 32768, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, n_gpu_layers = 63, n_threads = 56, n_threads_batch = 56

The 6 experts instead of 8 does bring it up.

8 Experts I think goes like

| 2048 | 512 | 0 | 43.768 | 46.79 | 58.812 | 8.71 |
| 2048 | 512 | 2048 | 43.994 | 46.55 | 60.076 | 8.52 |
| 2048 | 512 | 4096 | 44.731 | 45.79 | 59.162 | 8.65 |
| 2048 | 512 | 6144 | 45.756 | 44.76 | 61.601 | 8.31 |
| 2048 | 512 | 8192 | 44.844 | 45.67 | 62.035 | 8.25 |
| 2048 | 512 | 10240 | 44.996 | 45.52 | 62.541 | 8.19 |
| 2048 | 512 | 12288 | 45.741 | 44.77 | 64.000 | 8.00 |
| 2048 | 512 | 14336 | 45.896 | 44.62 | 65.038 | 7.87 |
| 2048 | 512 | 16384 | 48.384 | 42.33 | 70.296 | 7.28 |
| 2048 | 512 | 18432 | 53.666 | 38.16 | 66.915 | 7.65 |
| 2048 | 512 | 20480 | 46.520 | 44.02 | 66.358 | 7.72 |
| 2048 | 512 | 22528 | 47.653 | 42.98 | 67.443 | 7.59 |
| 2048 | 512 | 24576 | 63.603 | 32.20 | 72.567 | 7.06 |
| 2048 | 512 | 26624 | 51.910 | 39.45 | 74.521 | 6.87 |
| 2048 | 512 | 28672 | 49.094 | 41.72 | 71.302 | 7.18 |
| 2048 | 512 | 30720 | 49.411 | 41.45 | 72.487 | 7.06 |

1. Is it time to look in to overclocking or squeezing ram speed?
2. Or is it the 3090s fault (I am on EVGA 3090 FTW3 not 3090 Ti if that makes any difference) being a 3090? Or is it that I have it plugged into the monitor (my mobo is Asus W790 Sage SE)
3. EDIT: It seems testing 4096 batch, no experts in GPU, goes around 90-75 PP ts and around 8-9.3 t/s
   I may also look in 1x3090 argument setup since as of this moment my own PC could use a GPU for my own stuff. My guess is though for more PP and max response length I cant offload any to 3090. Darn I want to load more layers but why 3090 why

main: n_kv_max = 32768, n_batch = 8192, n_ubatch = 8192, flash_attn = 1, n_gpu_layers = 63, n_threads = 56, n_threads_batch = 56

Also tried this config . when I use llama server, when I do a test prompt of my own gets 46 PP 10.55 Gen, and a follow up 35 PP 10 Gen. Not sure if this is in line with the bench
--ctx-size 32768
-ctk q8_0
-mla 3 -fa
-amb 512
-fmoe
--temp 0.3
--min-p 0.05
-ts 23,24
-ot "blk.(3).ffn_.*=CUDA1"
--n-gpu-layers 63
-ot exps=CPU
-ub 8192 -b 8192
-ser 6,1
--parallel 1
--threads 56

Looks like you have very similar config as me, Asus Pro WS w790 and looking at 56 thread count I think you have QYFS?

@mtcl I saw some of your videos which was a source of inspiration.

I think I have some similarities with your build. Intel Xeon 8480 ES - QYFS 56 Core, Asus w790 Sage SE, 512GB DDR5 (I got 5600 but I only did a bit of bios to try to get it up but it sets it for 4800 for me; do you know if we can push speeds higher?).

But I only have 2 3090s in hand as of now (I will eventually try to get something better, but until my "situation improves" I will see if I can make it work with 3090s).

I think that 4800 lock is on the processor being engineering sample. And noni haven't been able to overclock it beyond 4800 either. But it's ok for now I think.

I know ever usecase is different and I shouldn't generalize it, but I've noticed that qwen3-235b outperforms my expectations and works very well in my usecase. I run it at 128k context length and it's able to do 400 pp tk/s. Note that you have to use a special 128k variant of the model for that. If you're ok with 40k token limit then @ubergarm 's model is the best with its iq3 quant. Very fast and very stable.

2x 3090s is a very potent setup! I would say stay with them or buy a couple more 3090s instead of getting a 4090 or 5090.

And I'm glad you're coming over from the channel :) this ik_llama community is the best!

@mtcl

Have you been able to test Deepseek with using just 1 RTX Pro 6000? I want to know if PP and TG speeds will improve if I did decide to "change the GPU situation"... I saw you had like 12 tps to 15 tps using either 1 4090, a mix of 4090/5090, or 2 RTX Pros. What do you think were the reasons you could get this fast i think 15 tps is super nice.

As for now I need to give back a 3090 to my own PC so I will look into 1x 3090 performance for now.

Ok now I have another issue (should I take this thread into another place?) need some more help would be appreciated.

I had to drop back down to 1x3090. I also did some simple BIOS goofing around to tweak some memory and also some commands to tell it to "go faster" and I get 10.5 tps for 0 context yet 8 experts.

CUDA_VISIBLE_DEVICES="0," \
./build/bin/llama-server \
    --model /mnt/.../models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf \
    --alias ubergarm/DeepSeek-R1-V3-0324-IQ4_K_R4 \
    --ctx-size 32768 \
    -ctk q8_0 \
    -mla 3 -fa \
    -amb 512 \
    -fmoe \
    --temp 0.3 \
    --min-p 0.05 \
    --n-gpu-layers 63 \
    --override-tensor exps=CPU \
    --parallel 1 \
    --threads 56 \
    --host ... \
    --port ...

But when I use OpenWebUI with Searxng Web Search or the stock RAG system... Every time I change chat threads or do new input I have to wait extremely long for the KV removal as if KV is taking forever.

INFO [            update_slots] kv cache rm [p0, end) | tid="127145341607936" timestamp=1751280242 id_slot=0 id_task=480 p0=2251
INFO [            update_slots] kv cache rm [p0, end) | tid="127145341607936" timestamp=1751280267 id_slot=0 id_task=480 p0=4299
INFO [            update_slots] kv cache rm [p0, end) | tid="127145341607936" timestamp=1751280294 id_slot=0 id_task=480 p0=6347
INFO [            update_slots] kv cache rm [p0, end) | tid="127145341607936" timestamp=1751280326 id_slot=0 id_task=480 p0=8395
INFO [            update_slots] kv cache rm [p0, end) | tid="127145341607936" timestamp=1751280358 id_slot=0 id_task=480 p0=10443
INFO [            update_slots] kv cache rm [p0, end) | tid="127145341607936" timestamp=1751280392 id_slot=0 id_task=480 p0=12491

Each of these lines takes around 12-15 seconds to appear that's far too slow. Is this PP issues again?

I know exactly what you are talking about. Look up what a task model is in open web UI. And switch the task model to a different model. Your task model has to be a public model exposed by either lama.cpp or Ollama. So I have olama along with llama.cpp running. And because I have switched my task model to be llama 3.2 it barely uses any V ram and all the tag generation chat name generation web search and all that happens using that task model from Ollama. The setting for task model is hidden inside of open web UI interface settings I think. I'm driving right now but when I reach home in about an hour and a half I can let you know unless you have figured it out by yourself by then.

Ah okay. I believe I added the task models now but it still has this slow KV rm when I add a document or web search aka I believe its adding a lot to context. is that a bit different?

For example, my opener prompt with websearch on is "Search what is the cost of Hamburgers in [name-burger-place-we-all-know]" and it gets a few sites as results then the dreaded kv rm series happens.
If i respond with websearch off aka normal prompt it responds faster as usual. but if i do another search query prompt it does the slow stuff... typically when I need to inject context RAG/search

@SFPLM

p0=12491 means that you are sending a fairly large context prompt and the server has processed 12491 tokens into the kv-cache. You can enable prompt caching (forget if it is on by default) but that will only help for growing the same conversation and adding new stuff to the end of it.

Seems like your OpenWebUI is likely doing some kind of web search/scraping and then sending it to the big LLM for summarizing or something. As @mctl suggests you might want to have two different LLMs loaded for different purposes e.g.:

1. A very small long-context (no-thinking) model for fast text summarizing like maybe Phi4, Polaris-4B, Gemma ?? (I dunno do your research here).
2. Your big DeepSeek-R1-0528 to take in the processed text from above to give the final answer.

More complex "agentic" workflows will typically require at least a couple or more models/re-rankers/embedding models being run simultaneously. You can just start multiple copies of ik_llama.cppeach running on a different port to support multiple LLM endpoints and configure openwebui.

I don't use OpenWebUI but there are some other settings to like disable tags and autocomplete stuff which could be useful if using a big model e.g. my old script which is probably out of date now. No need for ollama.

#!/usr/bin/env bash

source venv/bin/activate

# IT DOES NOT HONOR HOST and PORT ENV VAR SO PASS IT MANUALLY
# https://docs.openwebui.com/getting-started/env-configuration/#port

export DATA_DIR="$(pwd)/data"
export ENABLE_OLLAMA_API=False
export ENABLE_OPENAI_API=True
export OPENAI_API_KEY="none"
export OPENAI_API_BASE_URL="http://127.0.0.1:8080/v1"
#export DEFAULT_MODELS="openai/foo/bar"
export WEBUI_AUTH=True
export DEFAULT_USER_ROLE="admin"
export HOST=127.0.0.1
export PORT=3000

export ENABLE_TAGS_GENERATION=False
export ENABLE_AUTOCOMPLETE_GENERATION=False

open-webui serve \
  --host $HOST \
  --port $PORT

I am having a great success with these quants on EPYC 7282 + 256GB DDR4 3200MHz RAM (8 channels) + 4x RTX 3090 with the latest ik_llama.cpp (4622fad) - thank you for making these!
Getting very usable ~7t/s for indivudal non-batched requests, with some acceptable degradation as the KV increases:

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|  2048 |    512 |      0 |   31.896 |    64.21 |   68.622 |     7.46 |
|  2048 |    512 |   2048 |   31.820 |    64.36 |   68.388 |     7.49 |
|  2048 |    512 |   4096 |   32.039 |    63.92 |   69.754 |     7.34 |
|  2048 |    512 |   6144 |   32.232 |    63.54 |   71.181 |     7.19 |
|  2048 |    512 |   8192 |   32.445 |    63.12 |   72.344 |     7.08 |

Parameters that work well with my setup:

./llama-server --model model/DeepSeek-V3-0324-IQ2_K_R4/DeepSeek-V3-0324-IQ2_K_R4-00001-of-00005.gguf \
--threads 32 \
--temp 0.3 \
--min_p 0.05 \
--ctx-size 32768 \
-ts 24,24,24,24 \ 
-ngl 63 \
-ctk q8_0 \
-fmoe \
-mla 3 \
-fa \
-amb 512 \
-ub 2048 \
-b 2048 \
-ot exps=CPU \
-ot "blk\.(3|4|5)\.ffn_.*=CUDA0" \
-ot "blk\.(6|7|8)\.ffn_.*=CUDA1" \
-ot "blk\.(9|10|11)\.ffn_.*=CUDA2" \
-ot "blk\.(12|13|14)\.ffn_.*=CUDA3"

Memory allocation:

llm_load_tensors:        CPU buffer size = 42930.78 MiB
llm_load_tensors:        CPU buffer size = 46857.06 MiB
llm_load_tensors:        CPU buffer size = 46857.06 MiB
llm_load_tensors:        CPU buffer size = 43869.77 MiB
llm_load_tensors:        CPU buffer size =   938.98 MiB
llm_load_tensors:      CUDA0 buffer size = 15721.59 MiB
llm_load_tensors:      CUDA1 buffer size = 15033.14 MiB
llm_load_tensors:      CUDA2 buffer size = 15291.42 MiB
llm_load_tensors:      CUDA3 buffer size = 15713.87 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 32768
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 2048
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn   = 3
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe  = 1
llama_new_context_with_model: ser        = -1, 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =   306.01 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   286.88 MiB
llama_kv_cache_init:      CUDA2 KV buffer size =   306.01 MiB
llama_kv_cache_init:      CUDA3 KV buffer size =   267.76 MiB
llama_new_context_with_model: KV self size  = 1166.62 MiB, c^KV (q8_0): 1166.62 MiB, kv^T: not used
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.99 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=1)
llama_new_context_with_model:      CUDA0 compute buffer size =  3588.01 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =  3560.01 MiB
llama_new_context_with_model:      CUDA2 compute buffer size =  3560.01 MiB
llama_new_context_with_model:      CUDA3 compute buffer size =  3560.02 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   312.02 MiB
llama_new_context_with_model: graph nodes  = 8245
llama_new_context_with_model: graph splits = 178

This is on Ubuntu 22.04, NVIDIA drivers 570.133.20, ik_llama.cpp compiled against CUDA Toolkit 12.8

Just finished uploading the a new IQ3_KS based quant for https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF

• IQ3_KS 281.463 GiB (3.598 BPW)

I might add a smaller IQ2_KS as well, maybe a small IQ1_S for the 128GB RAM club, and if interest possibly a slightly larger as well.

@ubergarm
If you do have spare time I would be interested in IQ4 myself for Chimera 2. I personally have not tried Chimera 1 or 2 yet but at a glance may be a sidegrade for something that thinks 'just enough' (I personally prefer nonreasoning over reasoning).