Spaces:
Running on Zero
GPU Task abort
this space now often getting error :GPU task abort
Yes, too often, especially when it's getting expensive because the guidance scale is set higher than 1 and the video is longer than 6 seconds.
this space now often getting error :GPU task abort
This has started happening more frequently since the update applied to the space by r3gm 12 days ago. I have noted that the space does recognize human based images and works with them more fluidly now without issue, however, I'm wondering if the GPU is aborting because the processor gets confused because of the task provided to it; or if there is a memory leak with the update. You can usually tell when it's going to do it, because it's processing is generally a tad bit slower and takes longer to get to 33%, 50%, 67%, and then usually aborts at 83%.
It can do so with normal speed processing as well, and that generally happens when there is high demand (It's usual max use from what I've seen is 10 H200s for the space at once, only exceeding to 12 once), and sometimes, it will rarely give you priority, watching the 10 fall to 9 for a literal split second and then back upto 10 over and over almost as if a bot is quick striking the generate button. When it acts like this is generally when you finally do get a GPU, it will process to rendering post processing normal speed, and then throw out the 'GPU Task Aborted' error at the last second, as if whatever is spamming the usage of the H200s is taking priority over your GPU that is processing, and that should not happen.
I was slapped with the 'Need 35s, 0s Remains, Try again in 0:00:00' error earlier despite me having 2.2 minutes left to use so that's an issue in itself.
If the 'GPU task aborted' error occurs, you should be refunded your time, since nothing was technically finished, and that is something that the infrastructure of this site should work on.
So one of the things I noted is that the results from this space aren't linking properly to the Zero GPU usage meter on the site visually, which is why I'm getting the 35s vs 0 left try again in 0:00:00, as the results are using between 98 to 112 seconds each time, but the meter that measures ZeroGPU minute use is showing a used amount lower that what it should be showing, while properly logging the actual seconds use, giving a glitched ZeroGPU meter (Showing 1.5 to 2.2 out of 4 minutes used), even though each video from what I made (running at 2 seconds) uses around 1 minute 30 seconds each time.