bartowski

/

arcee-ai_Trinity-Large-Thinking-GGUF

like 11

(base) gopi@gopi-X99:~/llama.cpp/build$ ./bin/llama-server
--model "/home/gopi/deepresearch-ui/model/arcee-ai_Trinity-Large-Thinking-Q4_0-00001-of-00006.gguf"
-ngl 99
--ctx-size 200000
--n-cpu-moe 63
--tensor-split 1,1
--threads 28
--threads-batch 28
-np 1
--host 0.0.0.0
--jinja
--port 8080
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 23817 MiB):
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes, VRAM: 11907 MiB
Device 1: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes, VRAM: 11909 MiB
system info: n_threads = 28, n_threads_batch = 28, total_threads = 56

system_info: n_threads = 28 (n_threads_batch = 28) / 56 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

Running without SSL
init: using 55 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/home/gopi/deepresearch-ui/model/arcee-ai_Trinity-Large-Thinking-Q4_0-00001-of-00006.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 3060): 11907 total, 10753 used, 744 free vs. target of 1024
llama_params_fit_impl: - CUDA1 (NVIDIA GeForce RTX 3060): 11909 total, 9427 used, 2294 free vs. target of 1024
llama_params_fit_impl: projected to use 20181 MiB of device memory vs. 23221 MiB of free device memory
llama_params_fit: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
llama_params_fit: fitting params to free memory took 1.43 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3060) (0000:02:00.0) - 11498 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3060) (0000:03:00.0) - 11722 MiB free
llama_model_loader: additional 5 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 51 key-value pairs and 1113 tensors from /home/gopi/deepresearch-ui/model/arcee-ai_Trinity-Large-Thinking-Q4_0-00001-of-00006.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = afmoe
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_p f32 = 0.800000
llama_model_loader: - kv 3: general.sampling.temp f32 = 0.800000
llama_model_loader: - kv 4: general.name str = Trinity
llama_model_loader: - kv 5: general.size_label str = 256x8.8B
llama_model_loader: - kv 6: afmoe.block_count u32 = 60
llama_model_loader: - kv 7: afmoe.context_length u32 = 262144
llama_model_loader: - kv 8: afmoe.embedding_length u32 = 3072
llama_model_loader: - kv 9: afmoe.feed_forward_length u32 = 12288
llama_model_loader: - kv 10: afmoe.attention.head_count u32 = 48
llama_model_loader: - kv 11: afmoe.attention.head_count_kv u32 = 8
llama_model_loader: - kv 12: afmoe.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 13: afmoe.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 14: afmoe.expert_count u32 = 256
llama_model_loader: - kv 15: afmoe.expert_used_count u32 = 4
llama_model_loader: - kv 16: afmoe.expert_group_count u32 = 1
llama_model_loader: - kv 17: afmoe.expert_group_used_count u32 = 1
llama_model_loader: - kv 18: afmoe.expert_gating_func u32 = 2
llama_model_loader: - kv 19: afmoe.attention.key_length u32 = 128
llama_model_loader: - kv 20: afmoe.attention.value_length u32 = 128
llama_model_loader: - kv 21: afmoe.vocab_size u32 = 200192
llama_model_loader: - kv 22: afmoe.rope.dimension_count u32 = 128
llama_model_loader: - kv 23: afmoe.expert_shared_count u32 = 1
llama_model_loader: - kv 24: afmoe.expert_feed_forward_length u32 = 3072
llama_model_loader: - kv 25: afmoe.leading_dense_block_count u32 = 6
llama_model_loader: - kv 26: afmoe.expert_weights_norm bool = true
llama_model_loader: - kv 27: afmoe.expert_weights_scale f32 = 2.448000
llama_model_loader: - kv 28: afmoe.attention.sliding_window u32 = 4096
llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 30: tokenizer.ggml.pre str = afmoe
llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,200192] = ["<|begin_of_text|>", "<|end_of_text|...
llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,200192] = [3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, ...
llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,199712] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 3
llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 12
llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 38: tokenizer.ggml.add_sep_token bool = false
llama_model_loader: - kv 39: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 40: tokenizer.chat_template str = <|begin_of_text|>{%- macro render_ext...
llama_model_loader: - kv 41: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 42: general.quantization_version u32 = 2
llama_model_loader: - kv 43: general.file_type u32 = 2
llama_model_loader: - kv 44: quantize.imatrix.file str = /models_out/Trinity-Large-Thinking-GG...
llama_model_loader: - kv 45: quantize.imatrix.dataset str = /training_data/calibration_datav5.txt
llama_model_loader: - kv 46: quantize.imatrix.entries_count u32 = 696
llama_model_loader: - kv 47: quantize.imatrix.chunks_count u32 = 785
llama_model_loader: - kv 48: split.no u16 = 0
llama_model_loader: - kv 49: split.tensors.count i32 = 1113
llama_model_loader: - kv 50: split.count u16 = 6
llama_model_loader: - type f32: 469 tensors
llama_model_loader: - type q4_0: 319 tensors
llama_model_loader: - type q4_1: 12 tensors
llama_model_loader: - type q5_0: 84 tensors
llama_model_loader: - type q8_0: 198 tensors
llama_model_loader: - type q6_K: 31 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_0
print_info: file size = 210.72 GiB (4.54 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 1 ('<|end_of_text|>')
load: - 3 ('<|im_end|>')
load: special tokens cache size = 32
load: token to piece cache size = 1.2899 MB
print_info: arch = afmoe
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 3072
print_info: n_embd_inp = 3072
print_info: n_layer = 60
print_info: n_head = 48
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 4096
print_info: is_swa_any = 1
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 6
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 12288
print_info: n_expert = 256
print_info: n_expert_used = 4
print_info: n_expert_groups = 1
print_info: n_group_used = 1
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: freq_base_swa = 10000.0
print_info: freq_scale_swa = 1
print_info: n_embd_head_k_swa = 128
print_info: n_embd_head_v_swa = 128
print_info: n_rot_swa = 128
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: model type = ?B
print_info: model params = 398.64 B
print_info: general.name = Trinity
print_info: vocab type = BPE
print_info: n_vocab = 200192
print_info: n_merges = 199712
print_info: BOS token = 0 '<|begin_of_text|>'
print_info: EOS token = 3 '<|im_end|>'
print_info: EOT token = 1 '<|end_of_text|>'
print_info: PAD token = 12 '<|pad|>'
print_info: LF token = 230 'Ċ'
print_info: EOG token = 1 '<|end_of_text|>'
print_info: EOG token = 3 '<|im_end|>'
print_info: max token length = 126
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
load_tensors: offloading output layer to GPU
load_tensors: offloading 59 repeating layers to GPU
load_tensors: offloaded 61/61 layers to GPU
load_tensors: CPU_Mapped model buffer size = 37413.00 MiB
load_tensors: CPU_Mapped model buffer size = 36857.55 MiB
load_tensors: CPU_Mapped model buffer size = 36857.55 MiB
load_tensors: CPU_Mapped model buffer size = 36857.55 MiB
load_tensors: CPU_Mapped model buffer size = 36862.19 MiB
load_tensors: CPU_Mapped model buffer size = 30325.71 MiB
load_tensors: CUDA0 model buffer size = 2248.93 MiB
load_tensors: CUDA1 model buffer size = 2384.96 MiB
....................................................................................................
common_init_result: added <|end_of_text|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 200192
llama_context: n_ctx_seq = 200192
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = false
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (200192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 0.76 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 200192 cells
llama_kv_cache: CUDA0 KV buffer size = 5474.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 6256.00 MiB
llama_kv_cache: size = 11730.00 MiB (200192 cells, 15 layers, 1/1 seqs), K (f16): 5865.00 MiB, V (f16): 5865.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating SWA KV cache, size = 4608 cells
llama_kv_cache: CUDA0 KV buffer size = 432.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 378.00 MiB
llama_kv_cache: size = 810.00 MiB ( 4608 cells, 45 layers, 1/1 seqs), K (f16): 405.00 MiB, V (f16): 405.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve: CUDA0 compute buffer size = 2598.50 MiB
sched_reserve: CUDA1 compute buffer size = 409.00 MiB
sched_reserve: CUDA_Host compute buffer size = 412.02 MiB
sched_reserve: graph nodes = 3957
sched_reserve: graph splits = 194 (with bs=512), 111 (with bs=1)
sched_reserve: reserve took 277.97 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv load_model: initializing slots, n_slots = 1
no implementations specified for speculative decoding
slot load_model: id 0 | task -1 | speculative decoding context not initialized
slot load_model: id 0 | task -1 | new slot, n_ctx = 200192
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use --cache-ram 0 to disable the prompt cache
srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
init: chat template, example_format: '<|begin_of_text|><|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv update_slots: all slots are idle
srv log_server_r: done request: GET / 127.0.0.1 200
srv log_server_r: done request: GET /bundle.css 127.0.0.1 200
srv log_server_r: done request: GET /bundle.js 127.0.0.1 200
srv log_server_r: done request: HEAD /cors-proxy 127.0.0.1 404
srv params_from_: Chat format: peg-native
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 200192, n_keep = 0, task.n_tokens = 12
slot update_slots: id 0 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot update_slots: id 0 | task 0 | prompt processing progress, n_tokens = 8, batch.n_tokens = 8, progress = 0.666667
slot update_slots: id 0 | task 0 | n_tokens = 8, memory_seq_rm [8, end)
reasoning-budget: activated, budget=2147483647 tokens
slot init_sampler: id 0 | task 0 | init sampler, took 0.01 ms, tokens: text = 12, total = 12
slot update_slots: id 0 | task 0 | prompt processing done, n_tokens = 12, batch.n_tokens = 4
reasoning-budget: deactivated (natural end)
slot print_timing: id 0 | task 0 |
prompt eval time = 909.64 ms / 12 tokens ( 75.80 ms per token, 13.19 tokens per second)
eval time = 23974.48 ms / 199 tokens ( 120.47 ms per token, 8.30 tokens per second)
total time = 24884.11 ms / 211 tokens
slot release: id 0 | task 0 | stop processing: n_tokens = 210, truncated = 0
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-native
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 22322803448
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 210, total state size = 49.225 MiB
srv load: - looking for better prompt, base f_keep = 0.048, sim = 0.042
srv update: - cache state: 1 prompts, 49.225 MiB (limits: 8192.000 MiB, 200192 tokens, 200192 est)
srv update: - prompt 0x5b040bb30230: 210 tokens, checkpoints: 0, 49.225 MiB
srv get_availabl: prompt cache update took 104.55 ms
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 0 | task 201 | processing task, is_child = 0
slot update_slots: id 0 | task 201 | new prompt, n_ctx_slot = 200192, n_keep = 0, task.n_tokens = 237
slot update_slots: id 0 | task 201 | n_past = 10, slot.prompt.tokens.size() = 210, seq_id = 0, pos_min = 0, n_swa = 4096
slot update_slots: id 0 | task 201 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id 0 | task 201 | n_tokens = 0, memory_seq_rm [0, end)
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot update_slots: id 0 | task 201 | prompt processing progress, n_tokens = 233, batch.n_tokens = 233, progress = 0.983122
slot update_slots: id 0 | task 201 | n_tokens = 233, memory_seq_rm [233, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
slot init_sampler: id 0 | task 201 | init sampler, took 0.08 ms, tokens: text = 237, total = 237
slot update_slots: id 0 | task 201 | prompt processing done, n_tokens = 237, batch.n_tokens = 4
slot update_slots: id 0 | task 201 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 232, n_tokens = 233, size = 40.961 MiB)
slot print_timing: id 0 | task 201 |
prompt eval time = 27818.87 ms / 237 tokens ( 117.38 ms per token, 8.52 tokens per second)
eval time = 146600.14 ms / 1266 tokens ( 115.80 ms per token, 8.64 tokens per second)
total time = 174419.00 ms / 1503 tokens
slot release: id 0 | task 201 | stop processing: n_tokens = 1502, truncated = 0
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-native
slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.158
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 1502, total state size = 352.067 MiB
srv load: - looking for better prompt, base f_keep = 0.158, sim = 1.000
srv update: - cache state: 2 prompts, 442.253 MiB (limits: 8192.000 MiB, 200192 tokens, 200192 est)
srv update: - prompt 0x5b040bb30230: 210 tokens, checkpoints: 0, 49.225 MiB
srv update: - prompt 0x5b040b8c0cb0: 1502 tokens, checkpoints: 1, 393.028 MiB
srv get_availabl: prompt cache update took 446.33 ms
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 0 | task 1469 | processing task, is_child = 0
slot update_slots: id 0 | task 1469 | new prompt, n_ctx_slot = 200192, n_keep = 0, task.n_tokens = 237
slot update_slots: id 0 | task 1469 | n_past = 237, slot.prompt.tokens.size() = 1502, seq_id = 0, pos_min = 0, n_swa = 4096
slot update_slots: id 0 | task 1469 | Checking checkpoint with [0, 232] against 0...
slot update_slots: id 0 | task 1469 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id 0 | task 1469 | erased invalidated context checkpoint (pos_min = 0, pos_max = 232, n_tokens = 233, n_swa = 4096, pos_next = 0, size = 40.961 MiB)
slot update_slots: id 0 | task 1469 | n_tokens = 0, memory_seq_rm [0, end)
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot update_slots: id 0 | task 1469 | prompt processing progress, n_tokens = 233, batch.n_tokens = 233, progress = 0.983122
slot update_slots: id 0 | task 1469 | n_tokens = 233, memory_seq_rm [233, end)
reasoning-budget: activated, budget=2147483647 tokens
reasoning-budget: deactivated (natural end)
slot init_sampler: id 0 | task 1469 | init sampler, took 0.08 ms, tokens: text = 237, total = 237
slot update_slots: id 0 | task 1469 | prompt processing done, n_tokens = 237, batch.n_tokens = 4
slot update_slots: id 0 | task 1469 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 232, n_tokens = 233, size = 40.961 MiB)