ubergarm

/

Qwen3.5-397B-A17B-GGUF

like 34

Instead of writing the output gradually, it suddenly outputs everything at once after a long delay.
It does not consistently produce stable responses.

Instead of writing the output gradually, it suddenly outputs everything at once after a long delay.
It does not consistently produce stable responses.

Interesting findings, and you have run a lot of models!

I wonder if this could be improved with a different client or system prompt even to encourage it to work "step by step" or "iterate on the design testing each piece as you go" ?

I could make a bigger quant but not sure it would do any better...

Retesting:

llama_init_from_model: n_ctx = 190208
llama_init_from_model: n_batch = 8096
llama_init_from_model: n_ubatch = 8096
llama_init_from_model: flash_attn = 1
llama_init_from_model: attn_max_b = 8096
llama_init_from_model: fused_moe = 1
llama_init_from_model: grouped er = 1
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad = 1
llama_init_from_model: rope_cache = 0
llama_init_from_model: graph_reuse = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: v_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 0
llama_init_from_model: reduce_type = f16
llama_init_from_model: sched_async = 0
llama_init_from_model: ser = -1, 0
llama_init_from_model: freq_base = 10000000.0
llama_init_from_model: freq_scale = 1
llama_kv_cache_init: CUDA0 KV buffer size = 5758.83 MiB
llama_init_from_model: KV self size = 5572.50 MiB, K (f16): 2786.25 MiB, V (f16): 2786.25 MiB
llama_init_from_model: CUDA_Host output buffer size = 0.95 MiB
llama_init_from_model: CUDA0 compute buffer size = 7795.56 MiB
llama_init_from_model: CUDA_Host compute buffer size = 3063.89 MiB
llama_init_from_model: graph nodes = 3920
llama_init_from_model: graph splits = 122
llama_init_from_model: enabling only_active_experts scheduling

main: n_kv_max = 190208, n_batch = 8096, n_ubatch = 8096, flash_attn = 1, n_gpu_layers = 99, n_threads = 101, n_threads_batch = 101