Human

On June 1, 2009, Air France Flight 447 fell out of the sky over the Atlantic. The Airbus A330 had been flying itself for hours when ice crystals blocked its speed sensors and the autopilot disconnected. Three pilots were in the cockpit. None of them could fly the plane.

The investigation revealed something unsettling. The pilots hadn't forgotten how to fly in any obvious sense. They'd been trained. They'd passed their checks. But the automation had been so reliable for so long that the skill had become theoretical. When the moment arrived that demanded actual airmanship, the distance between knowing and doing was 228 lives wide.

Lisanne Bainbridge named this the “irony of automation” in 1983, twenty-six years before Flight 447 proved her right. The more reliable you make the system, the less prepared the human becomes to intervene. You need the human most precisely when the automation fails, and that is precisely when the human is least practiced.

There is a difference between a tool that removes difficulty and a tool that teaches you about it. A hand plane reveals the grain of the wood. A power sander hides it. Both produce a smooth surface. Only one produces a craftsman.

Richard Sennett writes in “The Craftsman” that the resistance of material is not an obstacle to mastery but the medium through which mastery develops. The potter's hands learn from the clay's refusal to cooperate. The surgeon's hands learn from tissue that doesn't behave as the textbook promised. Remove the resistance and you remove the education.

Software tools face the same choice. A tool can abstract away complexity so completely that the developer never encounters it, or it can make complexity legible so the developer learns to navigate it. The first approach is faster on day one. The second produces someone who can still think on day five hundred.

Simone Weil wrote that attention is the rarest form of generosity. She meant it in a spiritual sense, but there is a mechanical truth embedded in it: attention is finite, it is expensive, and it is easily displaced.

An omniscient system creates the conditions for inattention. If the tool sees everything, tracks everything, remembers everything, the human has no reason to look. Once you stop looking you stop noticing. Once you stop noticing you lose the ability to notice. The degradation is invisible precisely because the faculty that would detect it is the one being lost.

A partial observer has a different effect. When the system is known to be incomplete, the human remains engaged by necessity. You were in the meeting it missed. You read the thread it never saw. The act of filling gaps, correcting assumptions, supplying context: this is judgment being exercised, not merely invoked. The difference matters more than it appears.

There is an old principle in medicine: use it or lose it. Muscles atrophy. Skills decay. Oliver Sacks wrote about patients who lost capacities not from injury but from disuse, the brain quietly reallocating resources away from faculties that stopped being called upon.

Agency works the same way. Every time a system makes a decision you could have made, the circuit that would have fired doesn't. One decision is nothing. A thousand is a habit. Ten thousand is a dependency. The tool didn't take your agency. You stopped using it, and it left.

The question for anyone building tools is not “how much can we automate?” It is the harder, less comfortable question: how do we keep the human in a condition to think?