The better the autopilot the worse the pilot

I think this premise is flawed or, at best, too narrow. A system is just a logical grouping of items that perform a function. Sometimes that function can be to reduce cognitive burden, but it doesn't have to be. A "vision system" like what humans use does not reduce attention, but increases/enables it, while a autonomic nervous system can reduce attention. The ability to increase/reduce attention is not the central principle of a system.

You know I noticed this... I lived in a country where people obey traffic laws, and in a country where they very much don't.

I witnessed many more traffic accidents in the country where people are used to relying on the traffic lights to tell them if it's safe or not.

Whereas in the other country, everyone correctly assumes that the other drivers are completely insane, and so they stay vigilant.

No they don't, they tell you and other vehicles to stop. You would fail your driving test if you depend only on the traffic lights and don't bother to verify it is safe to pass yourself.

Someone learning to fly may be described as paying careful attention: to every little sound, vibration, and sensation. A common tactic by student pilots is overcontrolling the aircraft, e.g., large sudden changes rather than smooth pressures from flying with a light touch.

Automation requires active, intentional attention particularly when flying in clouds. What are my instruments telling me? Are they all telling the same story? Have any failed? Which ones?

A significant part of flight training and testing emphasizes the ability to divide attention between multiple competing needs, being able to correctly prioritize them, and responding promptly and safely in order of priority.

>The argument for automation is that it frees up cognitive bandwidth. Fewer routine decisions means more headroom to think carefully about the ones that matter.

So if the expectation is that the human pilot is expected to pay attention to mitigate the dangerous edge cases "that matter", there is a contradiction: the tool that promises to free up the bandwidth for that attention creates a complacency that prevents that attention from being applied.

In other words, it makes the normal situations safer but the abnormal situations more dangerous.

If the automation is for the easy/routine stuff, then no. The automation doesn't work in exactly the most safety-critical situations, and then the human operator is thrust into fixing the situation without the full context.

even if the average rate goes up tor net benefit, are organizations prepared for increased carastrophic failures?

5min to learn. A week of normal driving to get not bad. I wouldn't say it's hard.

In aviation, commercial pilots have very strict and extensive training and monitoring and as a result are generally able to utilize automation effectively while keeping up their manual skills. There are very rarely CFIT incidents in major commercial airlines.

The opposite is true in general aviation (small private Cessnas, etc), where it’s extremely common for pilots to buy more plane than they can handle and then rely on automation to bridge their skill gap. CFIT is much more common in general aviation, along with incorrect actions in response to real system failures that should have been recoverable. Automation complacency regularly kills in general aviation.

A key thing to notice is that automation isn’t outright prohibited in either commercial or general aviation, but there are distinct regulatory frameworks based on potential impact.

We accept looser rules for general aviation because the failures are societally less severe and because the population is much larger so effective training and enforcement would be significantly harder. In commercial airliners where failures are catastrophic, we have much stricter policies and require training and testing regularly to avoid automation complacency.

Will we start to see this practice in software? Probably, but only if/when the societal cost of NOT doing it becomes more clear. We regulated aviation because crashing planes are obviously bad. We license structural engineers because collapsing bridges are obviously bad. Will automation-induced software failures hit a similar tipping point?