The file that waited forever: a worker that retired one instant too early

What happened

Simple SFTP Client is the Windows desktop tool for uploading and downloading files over SFTP. It runs transfers through a queue drained by a background worker. To save resources the worker does not stay alive between batches: when the queue empties it retires itself, and adding more work starts a fresh one. Once in a while a file added at exactly the wrong moment would sit in the queue and never transfer. No error, no failure marked, just a file left pending with nothing to drain it.

The artifact

This bug throws nothing, so there is no error line to quote. It shows only as a stranded item, and the clearest statement of the mechanism is the code comment left where it was fixed:

a file enqueued in the instant this worker finds the queue empty would see a still-alive worker (this one) and _ensure_worker would skip starting one, yet this worker has already left the loop, and the file would wait forever.

The fix commit says the same in four words: “Fix worker restart race stranding queued files.”

Root cause

Two pieces of code disagreed about whether a worker was running, because they read that fact a fraction of a second apart. Adding work starts a worker only if one is not already alive; the worker, on claiming nothing from the queue, decides it is done and exits. When a file arrives in the instant between the worker’s “queue is empty” reading and its actual exit, the add sees the still-live worker and starts none, the worker then finishes exiting, and the file is left in a queue with no worker. Nothing looks again until the next add, which may never come. The two sides read “is a worker alive” and “should I retire” without a shared lock, so each saw a stale answer at the one instant it mattered.

The fix

Put the start decision and the stop decision under the same lock, and have the worker re-check the queue while holding it. Before retiring, the worker takes the lock _ensure_worker starts workers with, looks at the queue once more, and stops only if nothing is still waiting:

with self._worker_lock:
    if self.queue.waiting() == 0 or paused:
        if me in self._workers:
            self._workers.remove(me)
        # ... last one out clears the shared state

Sharing the lock means starting and stopping can no longer interleave: either the worker retires and the next add starts a fresh one, or the add wins and the worker sees the new file on its re-check and keeps going. Two tests pin this down, one running a second batch after a full drain and one injecting a file at the exact empty-queue instant by wrapping the claim call. The check counts items still waiting, not total pending, since pending also counts an item being transferred by another worker and would wrongly keep a worker alive.

Evidence boundary

This is about one stop-and-start race on the worker that drains the queue, not a claim that the queue is free of every timing bug. The fix guarantees a file in the queue has a worker to drain it. It does not change what happens to a transfer that fails on its own, which is marked failed and shown in the queue rather than left silent.

Source ledger