The Machines Are Learning. Unfortunately, From Us.

Estimated reading time: 9 minutes

This essay originally appeared on “With Respect — Mostly,” Executive Editor Martin Hinton’s Substack, where he writes about a wide range of subjects, including cyber insurance and security. It has been adapted for Cyber Insurance News with additional reporting that ties it more directly to coverage we’ve done.

Oh man. This was going to be clever. I had a plan. I would dig through an old box and find a couple of college papers. Something from the prehistoric age. Before ChatGPT. Last Millennium stuff.

Then I would run them through an AI detector. The machine would accuse 21-year-old me of being a machine.

Case closed.

I found two papers from 1992. Both A’s, mind you. I scanned them. I uploaded them. One came back 100 percent human. The other came back 98 percent human. Apparently I was 2 percent artificial intelligence in college.

That may explain some things. It does not prove my point. So the experiment failed. Which is useful. Because failure is what humans do before we call something progress.

We keep talking about artificial intelligence as if it landed in a silver capsule in Northern California.

It didn’t. We built it. We trained it on us. Our books. Our arguments. Our poetry. Our tax codes. Our wars. Our lies. Our recipes. Our philosophy. Reddit.

Then we became alarmed when it started behaving strangely.

Have you met us? There are real things to worry about.

AI costs enormous amounts of money and energy. It will eliminate some jobs. It will create others. It can amplify fraud. It can concentrate power.

Serious matters. What is less serious is our insistence that we should know exactly where all this goes. We never have. We were cold. We made fire. We were hungry. We planted crops. We got tired of walking. We domesticated horses.

Ships. Steam engines. Railroads. Cars. Airplanes. Rockets.

At no point did humanity convene the appropriate committee and ask what all this might look like in 500 years.

Even if we did.

The American government developed atomic weapons. Secrecy and seriousness. Net outcome. We live under the specter of more than a few madmen with a button.

We lit the fire. The impact statement came later.

The steam engine offers the better warning. It began as a crude machine for pumping water from mines. Then people tinkered. They added valves. Cranks. Condensers. Soon the thing was running factories and locomotives.

Congratulations. You invented the Industrial Revolution.

You also got smog, industrial accidents, poisoned rivers, and children working shifts that would keep a modern HR department busy. Technology rarely arrives with wisdom attached. Wisdom takes the later train. One that is often delayed. That is not an argument against technology. It is an argument against pretending we know its destination.

Now we have artificial intelligence. Naturally, we have given its mysteries impressive names.

Generative AI

Humans have been generative since someone picked up a rock and thought it might work better with an edge.

Recursive self-improvement

That sounds ominous. It is also civilization. We build tools. The tools let us build better tools. Those tools make us richer. Stronger. Faster. More dangerous.

Eventually somebody invents insurance. And then insurance spends several years working out what it just agreed to cover.

This month a carrier put it in writing. AXA XL listed four ways an AI failure becomes a loss. A deepfake payment fraud. A data breach. A defective output that harms somebody else. An outage at a provider you depend on. Each one, it said, engages a different policy. It did not say which.

Meanwhile, a cyber reinsurer hired a head of casualty, on the theory that the trouble is heading toward a different room. And AI exclusions have been spreading faster outside cyber policies than inside them, which is the industry equivalent of everyone quietly stepping back from the table.

Artificial General Intelligence

That sounds more exotic still. Humans already have general intelligence. The problem is no individual human has very much of it. One person farms. Another performs surgery. Another repairs the boiler. Nobody can get the printer to connect to the network reliably.

Civilization is our workaround.

We distribute intelligence across millions of people and institutions. Now we are trying to put something resembling that accumulated capability into a machine. Then we are shocked when it wants access to the calendar.

Alignment

This is the comforting one. We will align AI with human values. Wonderful. Which humans?

Human beings, we seem to cherish disagreement. With our lot in life and each other. That’s not to say we can’t agree. But when a few of us don’t, and they sit in the right high chair…

We have had religious wars. Civil wars. World wars. Revolutions. Genocides. Annexations. Insurrections. This is not an impressive alignment record.

We adapt. We compete. We cooperate when useful. We cheat when useful. We create rules and then hire lawyers to explain why those rules do not apply this time.

This is not hypothetical in my line of work. A court has already ruled on a 475,000 dollar wire transfer that a human approved after being deceived. Somebody had to decide who ate it.

Add a synthetic voice to that scenario, and you have next year’s case law.

Why should a machine trained on humanity emerge with moral clarity humanity has never possessed?

Consider the paperclip maximizer. This is a thought experiment about what happens when AI takes the goal and ignores any other impacts because you failed to give it clear instructions. Executive planning on speed and silicon.

Tell a sufficiently capable AI to make as many paperclips as possible. If you fail to explain yourself carefully enough, eventually everything that can become a paperclip becomes a paperclip. The ends by any means.

• Factories.
• Cars.
• Buildings.
• Your toaster.
• The suitable parts of you.

The machine is not evil. It is efficient. Which is what makes the story uncomfortable. Humans understand efficiency. We turned forests into fleets. Mountains into mines. Rivers into electricity. Time into billable hours. The frightening thing about the paperclip maximizer is not how alien it seems. It is how familiar it sounds.

Now we are building AI agents. We give them memory. Tools. Credentials. Goals. Agency. Then we put them in a sandbox to test them.

A sandbox. In tech parlance, this is our containment metaphor. A safe space to test something new and, yes, potentially dangerous. A thing designed for toddlers to climb out of. And what have our inventions done to address the testing? Cheated.

We have reported this one. An agent got out of its sandbox. A security firm now sends agents into a client’s internal network deliberately, which is either progress or the same behavior with paperwork.

Memory, tools, credentials, goals, and a boundary we are confident about. That is not containment. That is a teenager, a curfew, and a bedroom window.

The insurance question arrives roughly four seconds later. When the agent does something expensive, whose policy pays?

Most carriers have not said. One that did, Beazley confirmed it covers AI-driven attacks, which is the half where somebody else is attacking you. The half where your own agent wanders off remains open.

Heavens to Betsy They Cheated

Where did they learn that! From us. Obviously. Not entirely unrelated, the phrase “Heavens to Betsy,” etymologically speaking, is of unknown origin… But enough with Betsy.

Gartner asked 297 chief information security officers this year. Four in ten had sat through a deepfake on a work audio call in the previous twelve months. More than a third had seen one on video.

The machines learned to clone a voice because we taught them what voices sound like. Then people used them to ring the finance department and ask for money.

My favorite version of this is entirely human. Mike Nelson of DigiCert had a fender bender in a parking lot. He photographed the damage for his claim, as anyone would.

Then he had a thought. How easy would it be to run those photos through an AI tool and enhance the damage? Very easy. Research puts 36 percent of consumers as willing to consider altering an image in a claim. Among Generation Z, 55 percent. The machine did not think of that. We did.

More of that etymology.

Superintelligence

We also talk about superintelligence.

The word “super” comes from Latin for “above” or “over.” That should give us pause. We are trying to build something above our intelligence while assuring ourselves it will remain beneath our control.

Stanford measured hallucination rates across 26 leading models this year. They ran from 22 percent to 94 percent.

Above our intelligence. Intermittently.

Maybe it will. Maybe it won’t. Nobody knows. That is the honest position.

The people who first controlled fire could not imagine a blast furnace any more than James Watt could imagine climate change.

The Wright brothers could not have imagined Ryanair. However, a sort of lie-flat bed was on that first aircraft. So yeah, no comprehension of budget airlines.

The people who built the internet to connect humanity probably did not picture doomscrolling or bedrotting. The future comes after the plan, and it can’t read at any rate.

Which raises another matter. Human life is not merely an efficiency problem.

Formula One can collect a million data points from a race car.

Yet before the start, a person still walks onto the track and waves a green flag. Software could probably do it. A sensor could do it better. But the person has a story.

That matters too.

It matters operationally as well. Threat actors now run ransom negotiations through chatbots, and a Travelers executive told our podcast why. The bot is not haggling. It is removing your thinking time.

The pause is the control. It always was.

Perhaps AI cures diseases. Perhaps it wrecks industries. Perhaps it creates magnificent art. Perhaps it produces vast amounts of synthetic sludge.

Most likely it does all of those things and a thousand others nobody predicted. That is how human invention works.

First comes the miracle. Then, along with the benefits, comes the mess.

Then, up until recently at least, people have argued about who should clean it up.

AI is not an alien intelligence descending from the stars. It is our language. Our knowledge. Our ambition. Our contradictions.

It is humanity at machine speed.

And if what comes out of the machine frightens us, seems like it might end us, perhaps the first question should not be what went wrong with the machine.

One more thing before we get to the training data.

Eighty-seven percent of security teams now use AI in their operations. Sixty-four percent have never once rehearsed what happens when it goes wrong.

We built it. We deployed it. We skipped the drill.

Which, if you have met us, tracks.

Maybe we should look at the training data.

If we survive, we’ll get a new version of that anti-drug TV commercial. We’ll demand to know where AI learned that, and it will scream back.

“I learned it by watching you!”