We tried to run on A100X2 80GB, but failed.
Same, failed with nightly sglang and vllm. I got the vllm version to work but it spouted gibberish.
use llama.cpp
· Sign up or log in to comment
