Fix for onnxruntime-gpu slow first run
#5
by ehrrh - opened
With this model if you set providers=["CUDAExecutionProvider", "CPUExecutionProvider"] to use the gpu with onnx it spends a long time using the GPU for something when you try to do the first inference before finally quickly tagging everything, happens with both onnxruntime-gpu 1.18.1 and 1.19.0; if the number of images you want to tag is small it ends up being faster to use CPU because of the long wait time for the first image on GPU, why is that?
ehrrh changed discussion status to closed
ehrrh changed discussion status to open
Ok, found the actual fix: https://github.com/microsoft/onnxruntime/issues/19838
cuda_options = {
"cudnn_conv_algo_search": "HEURISTIC" # Set the cuDNN convolution algorithm search to HEURISTIC
}
model = rt.InferenceSession(model_path, providers=[('CUDAExecutionProvider', cuda_options), 'CPUExecutionProvider'])
ehrrh changed discussion title from Bug with onnxruntime-gpu to Fix onnxruntime-gpu slow first run
ehrrh changed discussion title from Fix onnxruntime-gpu slow first run to Fix for onnxruntime-gpu slow first run
Keeping this open for awareness. I'm assuming you reported this upstream too?
SmilingWolf changed discussion status to closed
SmilingWolf changed discussion status to open