Spaces:
Running
running ONNX models on ZeroGPU
My space (https://huggingface.co/spaces/rizavelioglu/fashionfail) successfully runs locally on my GPU but does not work on ZeroGPU when pushed to HF-spaces.
Has anyone tried running ONNX models on ZeroGPU, maybe it requires a special initialization?
Currently this error is thrown when executed on ZeroGPU:
[E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
[W:onnxruntime:Default, onnxruntime_pybind_state.cc:861 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirementsto ensure all dependencies are met.
Actually yes, ONNX can run on ZeroGPU but it won't be very efficient (slower than PyTorch to go from CPU to GPU). You have to first load the model with CPUExecutionProvider and then inside @spaces.GPU, change the execution provider to CUDAExecutionProvider
I had the same problem. I had to use find and mv/cp in subprocess for all onnx and cuda files for it to work, based on trial and error. I loaded the model inside the @spaces.gpu decorator too