I was adding a progress display to a video-generation dashboard when one job stopped behaving. Image-to-video at 640 pixels wide and 161 frames has a well-measured cost on the remote GPU machine: a median of 4.7 minutes across 41 runs. This job passed an hour.
What the scheduler did
The scheduler gives each item a per-job timeout of about 29.7 minutes. When the clock ran out it wrote the job down as failed and submitted the next one. But the remote server hadn't been told to stop; it was still working on the first job. The next item just joined the queue behind it. A timeout in the scheduler is a statement about the scheduler's patience, not about the remote work, and without an interrupt, work stacks up on a machine that is already struggling.
Looking at the GPU
The GPU reported 100% utilisation, 43 W of a 180 W power limit, and 15.6 of 16.3 GB of memory in use. High utilisation at roughly a quarter of the power budget, with memory nearly full, doesn't look like one job running flat out.
Then I cleared our queue entirely, deleting the waiting job, interrupting the running one. The GPU stayed at 100% with 13–15 GB in use. Something else was using it. A second render instance on the same machine had a client submitting a new job about every 35 seconds, and those were not ours.
Where I was wrong first
Partway through, I revised the “normal” duration. Another job had taken 14.4 minutes, so I wrote that the usual time was 14.4 minutes, and that the first jobs' 4–5 minutes were the exception. I had it backwards. That figure came from one job that had itself been slowed, against 41 runs that said otherwise. The original 4–5 minutes stands; I corrected the correction.
What I'm claiming
The other client is the most likely cause of the slowdown. I have not proven it. Proof would mean stopping that client, running one job, and seeing whether the time returns to about 4.7 minutes. I didn't stop it and re-measure, so it sits in the notes as a probable cause with the test still open. Next to it are the decisions that don't depend on being right about the cause: whether the scheduler should interrupt remote work when its own timeout fires, and where a queued item should be held when its target machine is busy.
A progress bar that says it is guessing
The feature I'd started on has a limit of its own. This version of the render server doesn't report step-by-step progress over its plain HTTP interface; that only comes through a websocket held by the session that submitted the job. So the bar shows elapsed time against past measurements for that job type. It is an estimate, and the interface says so on the bar itself.