The Rutted Road: Capability Always Arrives Before the Rules

On the morning of March 20, 1905, a boiler in the basement of the R. B. Grover shoe factory in Brockton, Massachusetts let go. It went up through four floors, brought the building down, and the coal stoves in the wreckage set what was left on fire. Fifty-eight people died. Boilers had been exploding in America for seventy years by then. Everyone knew they did. Steam was the most important capability of the century and the people who ran it were, for the most part, self-taught mechanics standing next to a pressure vessel with a gauge they didn't trust.

Ten years later the American Society of Mechanical Engineers published the first Boiler Code. It's still in force. It has never made a boiler explode less. What made boilers explode less was a company founded in Hartford in 1866 that would insure your boiler on one condition: their inspector got to look at it first.

I've spent the last month arguing that the governance of AI agents belongs outside the model, in the system around it: authority that the agent can't grant itself, a gate before the consequence, a record the agent doesn't write. People keep asking whether that's realistic. It's the pattern every time. Every general-purpose technology we have lived through arrived the same way, in three acts, and the third act was never mainly about the machine. The machine got rules too, eventually. Scale came from something else: an independent system around the machine that could inspect it, constrain its use, certify it, assign authority, and keep the record.

Act one: the machine is magic

Act one is short and loud. The capability shows up, it works, and it works well enough that the question of what it should be allowed to do sounds like a question for people who don't get it. Steam moved mountains. Electricity turned night into day. The automobile freed a farmer from the horse. The airplane made the map smaller. In each case the first decade belonged to the enthusiasts, and the enthusiasts were right about the capability.

We are past act one with AI. Nobody serious argues the models can't do the work anymore.

Act two: DIY mechanics and rutted roads

Act two is long, and it's where we are.

The Ford Model T shipped in 1908 onto roads that were, outside a few cities, dirt. And the Model T was already the answer to a problem a decade old: before it, an American automobile was mostly the product of a small shop, built a few at a time by mechanics working from no common drawing, for buyers who did their own repairs. There were no lane markings because there were no lanes. There were no standard signs; each county made its own, if it made any. Drivers learned by driving. Repairs were done by the owner, or by a blacksmith who had figured out engines on the job. The fuel was sold from cans at the general store. It was glorious and it killed people at a rate we would not tolerate for a week today.

Electricity had its version. Between the first commercial power stations in the 1880s and the turn of the century, buildings burned because wiring was improvised by whoever was available, with whatever wire they had, insulated with whatever was on hand. Aviation had barnstorming: surplus biplanes, self-certified pilots, county fairs, and a fatality rate that made the airlines of the 1920s impossible to insure.

Notice what act two has in common across all four. The capability is real. The people using it are competent, mostly. The failures are not caused by the machine being evil or the operators being stupid. They are caused by the absence of a system around the machine: no standard for the road, no inspection of the vessel, no certification of the wiring, no license for the pilot, no record of what happened when it went wrong.

That is where we are with agents, and earlier in the act than the Model T. Inference is the gasoline. The harness around the model is the car, and right now the cars are being built by hand, one customer at a time, by engineers the vendors send out for the purpose. I made that argument in the previous post. The road is everything outside the car, and this post is about the road. I have my own act-two story. In December 2024 I gave a coding agent one bounded job, converting about two hundred files from one language to another. I set its auto-approve counter to 200, expecting two hundred small independent jobs. It ran one accumulating session instead and, while it was in there, built an entire subsystem nobody had asked for. The provider's meter shows 913 million tokens that month and 1.85 billion the next, the month I spent fixing forward. Every instruction I had written was delivered on every call. The rules were present. Nothing outside the agent could say no.

That's a boiler explosion with a meter instead of a body count. And this week a writer at Wired removed the safety training from an open-weight model and let it find its way into his own household devices, which is the DIY mechanic with the pressure relief valve wired shut. Neither of us is unusual. Act two is full of us.

The Red Flag Act

There is a fourth thing act two produces, every time, and it's worth its own section because we are producing it right now.

In 1865 the British Parliament passed the Locomotive Act, remembered as the Red Flag Act. It limited self-propelled road vehicles to four miles an hour in the country and two in town, and required a man to walk sixty yards ahead of each one carrying a red flag. It was a safety measure. It regulated the machine: how fast it could go, and that it be visibly preceded by a human. It did nothing for the road.

The act stood, in various forms, for thirty-one years. It was repealed in 1896, and the motoring enthusiasts celebrated with a run from London to Brighton they called the Emancipation Run. By then France and Germany had a decade's head start in automobiles, and Britain never fully caught up. The red flag had not made the roads safer. It had made the country slower, and it had made sure the people who understood the machine best were somewhere else.

Every time act two gets frightening, someone proposes a red flag. A cap on the machine. A man walking in front. A pause until the machine is understood. It's an honest impulse and it's aimed at the wrong object. The machine was never the thing that needed governing. The road was.

I'm not going to argue here about whether the current calls to slow down AI development are sincere, or whether compute thresholds and capability ceilings are good policy. I don't need to. A capability threshold may govern who gets to run the refinery. It still doesn't answer who authorized the car to turn left, what stopped it before it hit someone, or who owns the record afterward. Model regulation may be necessary. It is not sufficient for governing what an agent does, and the road is where the accidents are. The skeptics have noticed that no lab has delayed a release, and they treat that as proof the whole thing is theater. It's simpler than that. The manufacturers never built the road. Someone else did.

Act three: someone else builds the road

Here is the mechanism, and it's the reason this is a governance post and not a history post.

Normalization, when it finally came, got its authority from outside the people who made the machine. Manufacturers participated; some of them asked for it. What they could not do was confer trust on themselves. That came from the parties bearing the losses, and it took the form of a system around the machine.

The Hartford Steam Boiler Inspection and Insurance Company was an insurer. It could not make boilers safer by wishing. It could refuse to write a policy on a vessel its own inspector had not seen, and it did, and inspection became the norm because the alternative was going uninsured. When the ASME code arrived in 1915 it codified what the insurers had been enforcing for fifty years.

Underwriters Laboratories was founded in 1894 by fire insurers, for the same reason. It does not regulate electricity. It tests the appliance you plug into the wall and puts a mark on it, and the mark is what your insurer, your landlord and your building inspector look for. The National Electrical Code followed in 1897, written by the National Board of Fire Underwriters. Underwriters. The word is in the name.

The car took longer because nobody insured the road. The Federal Aid Road Act of 1916 put federal money into rural highways for the first time. The first national manual of uniform traffic signs and signals appeared in 1935; before that, a stop sign in one state meant something different in the next. The Interstate system was authorized in 1956. The federal vehicle safety agency, and the first mandatory safety standards for the cars themselves, arrived in 1966, a year after Ralph Nader's book, fifty-eight years after the Model T. The machine did get rules, in the end: brakes, belts, steering columns. But they were written by an agency that didn't work for Ford, enforced by inspectors who didn't either, and they came after the road.

Aviation is the clearest case, because the industry asked for it. The Air Commerce Act of 1926 was sought by the airlines and the manufacturers, not imposed on them, because without certified aircraft, licensed pilots and a federal record of what was flying, no one would insure a plane and no passenger would board one. The regulation didn't slow aviation down. It's what let it scale.

Four technologies, one pattern. Every one of them eventually acquired rules for the machine. Scale came from the other thing: inspection, certification, licensing and a record, held by parties the machine's maker didn't control, paid for by whoever held the risk, arriving decades after the capability. And it made the capability bigger, not smaller, because the road is what lets you drive fast.

What act three looks like for agents

If the pattern holds, we already know what the normalization layer for AI agents looks like, because it's the same layer every time, and it doesn't need the machine to change.

Who authorized this action, under what mandate, with what limits? That's the driver's license and the flight plan. What stopped it before the consequence, and was that thing outside the actor's control? That's the inspector who doesn't work for the boiler's owner. What does the record show, and who wrote it? That's the black box: evidence the pilot cannot rewrite.

Those three questions are testable against any deployment, by a startup or an incumbent, regardless of which lab made the model. They're the UL mark for an agent. And the party that will eventually demand them is the same party as always: not the lab, and not the enthusiast. The insurer who writes the cyber policy. The auditor who signs the controls opinion. The regulator of the bank, the hospital and the utility, who regulates the operator, not the refinery.

That's the part I'd say to anyone waiting for the labs to fix this. They can make the machine better, and they should. What they can't do is confer trust on themselves; no manufacturer ever has. Ford didn't build the Interstate. Boeing didn't write the Air Commerce Act. The road gets built by the people who have to drive on it and the people who pay when it goes wrong.

We have the gasoline. The cars are being built by hand, one at a time, and there is no Model T yet. And even when there is, it will ship onto dirt, the way the first one did. We are, most of us, DIY mechanics standing next to a machine we half understand, on a road nobody has marked. That's act two, and it always ends the same way. Not with a red flag, not with better gasoline, and not even with the Model T. With a road.

The framework these three questions come from is Governance Fidelity Theory. The illustrated primer is at equilateral.ai/blog/gft-public-primer. The technical working paper is available on request.