The retry loop that sleeps until Thursday

The bug report says the daemon "sometimes hangs after a network blip, but only on machines that have been up a while, and only after a reboot of the switch." That is not three conditions. It is one condition described three ways: the clock got stepped.

The shape of it

A retry loop written the obvious way:

deadline = time.time() + timeout
while time.time() < deadline:
    if attempt():
        return
    time.sleep(backoff)

time.time() is the wall clock. It is not required to move forward at one second per second, and it is not required to move forward at all. When the time daemon decides the local clock is badly wrong — which is exactly what happens after a long network partition, or on a VM whose host suspended — it steps the clock rather than slewing it. Step it backwards by an hour and that loop now runs for an hour and a timeout. Step it forward and the loop exits immediately, having made one attempt.

Use the clock that only goes forward

Every platform has a monotonic clock for precisely this. It has no relationship to civil time, it is unaffected by NTP and by timezone changes, and it is the only correct source for a duration:

deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
    ...

The rule is short enough to memorise: wall clock for timestamps you will show a human or store in a database, monotonic for every interval, timeout, deadline, and rate limit. If you are subtracting two times to get a duration, you want monotonic.

The same bug in shell

SECONDS in bash is derived from the wall clock and has the same defect. So does anything built on date +%s. On Linux the monotonic source from shell is:

awk '{print int($1)}' /proc/uptime

Not elegant, but it does not go backwards, which is the only property being asked of it.

Why it survives testing

Nobody steps the clock in CI. The failure needs a real machine, a real disruption, and a time daemon deciding to correct a real drift — and by the time it happens the logs have rotated. It is worth injecting deliberately: timedatectl set-time on a scratch VM while the loop runs will reproduce it in one attempt.