Error on inputting lots of prompts
#2
by dwaynedu - opened
I've been trying to make this model (cpatonn's AWQ-4bit quantized version) work on my environment:
GPU: H20 96GB*1
vLLM: docker-based vllm/vllm-openai:0.10.2
Front end: Open WebUI 0.6.26
I have observed that inputting long prompts may cause vLLM to crash:
(APIServer pid=1) INFO: 10.109.70.65:47040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:50234 - "GET /health HTTP/1.1" 200 OK
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.2) with config: model='/models/cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', speculative_config=SpeculativeConfig(method='qwen3_next_mtp', model='/models/cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', num_spec_tokens=2), tokenizer='/models/cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwen3-next-80b-instruct, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/50dc8b16d4","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":128,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/50dc8b16d4/rank_0_0/eagle_head"},
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['cmpl-88b67eb2a6c445a6bc5e670c31f0a6b0-0', 'cmpl-0fcccbceca5943f29fe1a397fc326de8-0', 'cmpl-735c2016db5e4e0aa524ef4c43660092-0', 'cmpl-5018f0752bcb46ca82b229a72fd72c8f-0', 'cmpl-9c44045eb6b5456097e9ddf2b9a67e8d-0', 'chatcmpl-35f97694f5f54990a9abce9aca0a0819'], resumed_from_preemption=[false, false, false, false, false, false], new_token_ids=[], new_block_ids=[null, null, null, null, null, [[206, 207, 208, 209, 210], [], [], []]], num_computed_tokens=[4088, 3914, 3834, 3584, 3549, 16354]), num_scheduled_tokens={chatcmpl-35f97694f5f54990a9abce9aca0a0819: 2870, cmpl-88b67eb2a6c445a6bc5e670c31f0a6b0-0: 3, cmpl-0fcccbceca5943f29fe1a397fc326de8-0: 3, cmpl-9c44045eb6b5456097e9ddf2b9a67e8d-0: 3, cmpl-5018f0752bcb46ca82b229a72fd72c8f-0: 3, cmpl-735c2016db5e4e0aa524ef4c43660092-0: 3}, total_num_scheduled_tokens=2885, scheduled_spec_decode_tokens={cmpl-5018f0752bcb46ca82b229a72fd72c8f-0: [0, 0], cmpl-0fcccbceca5943f29fe1a397fc326de8-0: [0, 0], cmpl-735c2016db5e4e0aa524ef4c43660092-0: [0, 0], cmpl-88b67eb2a6c445a6bc5e670c31f0a6b0-0: [0, 0], chatcmpl-35f97694f5f54990a9abce9aca0a0819: [0, 0], cmpl-9c44045eb6b5456097e9ddf2b9a67e8d-0: [0, 0]}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 0], finished_req_ids=[], free_encoder_mm_hashes=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=6, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.047728140511645645, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] Traceback (most recent call last):
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 711, in run_engine_core
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] engine_core.run_busy_loop()
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 738, in run_busy_loop
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] self._process_engine_step()
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in _process_engine_step
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 292, in step
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 278, in execute_model_with_error_logging
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] raise err
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 269, in execute_model_with_error_logging
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] return model_fn(scheduler_output)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 93, in execute_model
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] output = self.collective_rpc("execute_model",
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] return func(*args, **kwargs)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] return func(*args, **kwargs)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 436, in execute_model
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] return func(*args, **kwargs)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2136, in execute_model
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] self._draft_token_ids = self.propose_draft_token_ids(
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2234, in propose_draft_token_ids
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] next_token_id = req_state.get_token_id(seq_len)
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_input_batch.py", line 69, in get_token_id
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] return self.output_token_ids[idx - self.num_prompt_tokens]
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) ERROR 09-13 20:41:04 [core.py:720] IndexError: list index out of range
(EngineCore_DP0 pid=229) Process EngineCore_DP0:
(EngineCore_DP0 pid=229) Traceback (most recent call last):
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] AsyncLLM output_handler failed.
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] Traceback (most recent call last):
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in output_handler
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] outputs = await engine_core.get_output_async()
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 845, in get_output_async
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] raise self._format_exception(outputs) from None
(APIServer pid=1) ERROR 09-13 20:41:04 [async_llm.py:485] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(EngineCore_DP0 pid=229) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=229) self.run()
(EngineCore_DP0 pid=229) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=229) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 722, in run_engine_core
(EngineCore_DP0 pid=229) raise e
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 711, in run_engine_core
(EngineCore_DP0 pid=229) engine_core.run_busy_loop()
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 738, in run_busy_loop
(EngineCore_DP0 pid=229) self._process_engine_step()
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in _process_engine_step
(EngineCore_DP0 pid=229) outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 292, in step
(EngineCore_DP0 pid=229) model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 278, in execute_model_with_error_logging
(EngineCore_DP0 pid=229) raise err
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 269, in execute_model_with_error_logging
(EngineCore_DP0 pid=229) return model_fn(scheduler_output)
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 93, in execute_model
(EngineCore_DP0 pid=229) output = self.collective_rpc("execute_model",
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=229) answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=229) return func(*args, **kwargs)
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=229) return func(*args, **kwargs)
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 436, in execute_model
(EngineCore_DP0 pid=229) output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=229) return func(*args, **kwargs)
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2136, in execute_model
(EngineCore_DP0 pid=229) self._draft_token_ids = self.propose_draft_token_ids(
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2234, in propose_draft_token_ids
(EngineCore_DP0 pid=229) next_token_id = req_state.get_token_id(seq_len)
(EngineCore_DP0 pid=229) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_input_batch.py", line 69, in get_token_id
(EngineCore_DP0 pid=229) return self.output_token_ids[idx - self.num_prompt_tokens]
(EngineCore_DP0 pid=229) ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=229) IndexError: list index out of range
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] Error in chat completion stream generator.
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] Traceback (most recent call last):
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 562, in chat_completion_stream_generator
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for res in result_generator:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 392, in generate
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] out = q.get_nowait() or await q.get()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 59, in get
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise output
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 339, in completion_stream_generator
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for prompt_idx, res in result_generator:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 774, in merge_async_iterators
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for item in iterators[0]:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 392, in generate
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] out = q.get_nowait() or await q.get()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 59, in get
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise output
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 339, in completion_stream_generator
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for prompt_idx, res in result_generator:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 774, in merge_async_iterators
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for item in iterators[0]:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 392, in generate
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] out = q.get_nowait() or await q.get()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 59, in get
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise output
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 339, in completion_stream_generator
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for prompt_idx, res in result_generator:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 774, in merge_async_iterators
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for item in iterators[0]:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 392, in generate
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] out = q.get_nowait() or await q.get()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 59, in get
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise output
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 339, in completion_stream_generator
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for prompt_idx, res in result_generator:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 774, in merge_async_iterators
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for item in iterators[0]:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 392, in generate
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] out = q.get_nowait() or await q.get()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 59, in get
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise output
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 339, in completion_stream_generator
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for prompt_idx, res in result_generator:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 774, in merge_async_iterators
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] async for item in iterators[0]:
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 392, in generate
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] out = q.get_nowait() or await q.get()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 59, in get
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise output
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in output_handler
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] outputs = await engine_core.get_output_async()
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 845, in get_output_async
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] raise self._format_exception(outputs) from None
(APIServer pid=1) ERROR 09-13 20:41:04 [serving_chat.py:1136] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[rank0]:[W913 20:41:04.213239660 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) INFO: Shutting down
(APIServer pid=1) INFO: Waiting for application shutdown.
(APIServer pid=1) INFO: Application shutdown complete.
(APIServer pid=1) INFO: Finished server process [1]