One-minute frequency disturbances from synchronization in power systems

The corresponding concept in gears is to avoid meshing two gears unless the number of teeth are co-prime, because this will cause uneven wear patterns. If the tooth counts are coprime, then each tooth will contact every tooth on the other gear the same number of times, effectively "wear leveling" the teeth.

In this instance, one fix is to fire the timer every 59.XXX seconds, so it will naturally drift out of phase with the minute hand of the clock.

If work needs to be done exactly every 60 seconds, this is fixed by having a WAIT(RAND(60e6)) microseconds during the first timer initialization, so if 10,000 units start simultaneously they don't try all at once. This is good practice in most (all?) systems. Hint: test power and infra outage/reconnect, which can unintentionally sync timers.

This is why many software will do automated updates every 17 hours: 17 is coprime with 24.

Classic "thundering herd" mitigation tactics.

Edit: Tempting to blame IoT, but (since this is news) I don't think anyone knew there was something to fix here. We have the tools however.

I'm skeptical that this one minute jitter period is solely attributable to one minute timers in power inverter equipment. It's likely the net scale effect of all cron-like scheduling of actions in everything with a microprocessor. All the long-term scheduling that is done with minute precision (i.e. seconds field is implicitly :00) regardless of its individual period. Like if one device is scheduling at x:17 and x:47 every hour, another one is doing x:06, x:09, x:12, x:15, ... and numerous other combinations, at scale, the common divisor will be one minute. If wall clock time is used, all the devices that are synchronized to a network time will have cron-like scheduling events firing at xx:xx:00 for all values of x.

This type of scheduling should be randomized at the seconds level, perhaps. Unfortunately, it would require a massively coordinated effort across numerous operating system and application stacks.

Maybe too many things are started from crontab. Public NTP servers observe large spikes in traffic on minute marks due to admins putting ntpdate in crontab instead of running a proper NTP service.

I know my older APC UPSes do weekly self tests, but I guess newer equipment does more frequent self testing? And of course they're all internet-connected meaning clocks are within a few hundred ms of each other.

It also made me think about how many GitHub Actions I have that kick off on Sunday morning at a specific time. I try to vary the time/day for those jobs but I'm guessing there are enough things that kick off at specific time intervals to make an impact at the top of an hour, or at :15, :30, :45...

I'm not sure I agree with the conclusion that this is merely caused by UPS system control logic. That is an extreme amount of coordination across a lot of makes, models and configurations.

The FFT indicates this is a periodic signal, not a once a day or week thing. It's happening continuously all day every day. I don't know of any UPS systems that perform tests with this kind of regularity.

In any case, I refuse to believe an external sync signal is causing this. The more likely explanation is that the active elements are pulling each other to a common mode due to being physically coupled on the same grid. This is like how clocks on the same beam will synchronize.

https://en.wikipedia.org/wiki/Injection_locking

Potentially related phenomenon: Across multiple locations >200km apart in France, I've personally noticed that mains powered lights reliably and visibly pulse a few times at the beginning of every hour, and half-hour throughout the day. From memory, it's roughly one or two 1-2s long sequences of pulsed dimming, at around 5-10 pulses/second. The pulse frequency doesn't seem to change, and it starts and stops abruptly. Does anyone know what this could be?

Cool, I thought grid stability attacks would rely on large numbers of IoT devices but it looks like computing infrastructure is the better target.

Whether it's by buying and using a trillion tokens strategically or by inserting synchronized malicious jobs it should be possible to find some resonant frequencies in the line frequency control system. By loading and unloading multi-megawatts of draw you could get the local supply throttling up and down enough desynchronize from the larger grid. If you could get real time measurements of the line frequency you could use that to tune your malicious load and do the whole 'pushing a kid at the peak of their swing' thing.

To mitigate you would have to put dummy loads or some kind of minute-scale power storage onsite or nearby to compute loads.

Van Jacobsen TCP synchronisation at queue points in the Internet. Led to protocol changes to de-cohere with a small random backoff.

A very interesting article, but it is disappointingly vague about what these UPS self tests are, and exactly how they cause disruption to the power grid. Figure 5b shows that their building's UPS has 20-amp spikes exactly every minute, which is a pretty big spike. My guess is that a UPS performs a self test by switching from line power to battery power. If most of the UPS units in a country do that exactly every minute, synchronized by NNTP [edit: NTP], yeah I can see that it would mess up the grid having all that load dumped at once. (Although the observed frequency disturbance of 1.5 mHz is extremely small, 3/1000 of a percent.) Now that the problem is identified, it should be trivial for UPS manufacturers to eliminate it.

(By the way, the article is in Nature Energy, not Nature which the URL suggests.)

[flagged]