This isn't going to be a normal post from me.
It's a thought I've been carrying around for a while, and I'd like to put it out there to get other opinions on.
The idea of rogue superintelligence is so far from reality for most of us that fully grasping what it means is near on impossible (beyond picturing the Terminator...).
In my opinion, nuclear is the best reference frame we have for how we should think about governing what we're building.
I'd just like for it to be challenged and spoken about as you see fit.
The background
We are currently in what is commonly referred to as an AI arms race.
Nobody feels they can slow down, in case a country that isn't them catches up. But at the same time, at least in the US, the governments aren't the ones in control.
The development of AI has been driven almost entirely by private organisations, and it is only recently that governments have stepped in to start thinking about how best to regulate it.
We've already witnessed our fair share of disagreements, from Anthropic refusing to let Claude be used for autonomous weapons, and the Pentagon banning it from government use in response (a ban the courts are currently blocking).
To the "my AI is more dangerous than your AI, it can hack things" era (that even Meta wanted a piece of...), triggered by the wake-up call last month of OpenAI's own experimental agents escaping their sandbox during an internal test and hacking their way into Hugging Face's live systems, simply to cheat on the evaluation they were being run through.
The point I'd like to make is why it's important that we don't shy away from conversations around regulation, and to do that, I'd like to paint a hypothetical.
The story
Imagine nuclear energy had, for whatever reason, simply not been discovered.
It's 1989, and a young Hemis Dassabis (intentionally spelt wrong...) is sat in a chess hall in Liechtenstein.
A twelve-year-old prodigy, he has just resigned a ten-hour game against a master more than twice his age out of sheer exhaustion (the position, it turned out afterwards, was a draw), and he has a realisation.
Instead of all the great minds in this room spending their brainpower on a board game, they should be using it to solve something that matters.
They should be using it to solve energy.
He makes it his life's work.
He interns at Rolls-Royce, is every bit the prodigy there too, and is offered a rumoured one million pounds not to go to Cambridge and instead stay to help build their reactors.
He says no.
This is his life's mission, so he goes off to study natural sciences, assembles a world-class team, and raises the money to go after nuclear energy with everything he has.
(This is full of back references for anyone else who is an AI nerd, but I'll stop with them now.)
Initially, they have very little success.
In 2010 the experiments produce enough energy to power a small desk fan.
But as the research compounds, it starts to take off.
By the mid-2010s they've moved from being able to power desk fans to small offices. By 2022, larger offices, with their sights now set on full cities.
The world is captivated.
People are calling it the biggest development since the industrial revolution (a reference point nobody had used for anything, ever, and is now all anyone seems to care about...).
Billions pour in from the world's largest venture firms, because there is complete belief that whichever company solves energy first will hand their country total abundance.
Growth so fast that no other country would ever be able to catch up.
Which, of course, only accelerates the race.
But voices start to pop up. Is this technology being built in the safest way possible?
If everyone is rushing, aren't we at risk of not fully understanding the consequences?
Some of the CEOs themselves come out asking the world to pause.
Public letters are signed.
A spin-off company, founded on safety, warns that if this goes wrong millions could be left without power, and that in the worst case, if the technology isn't properly controlled, what is being built is not far off a nuclear bomb.
The individuals building it have a feeling for the risks, rather than a deep understanding of them.
A disaster could be imminent.
The question
In case you hadn't guessed, this was meant to reflect what we've seen happen in the fast-moving world of AI over the past twenty odd years.
So, stepping back, and knowing what we know about nuclear when it goes wrong or ends up in the wrong hands, what would you expect of our governments?
Would you be comfortable with a sniff-test regime before one of these sites goes live?
Would you be comfortable with the pace of development being set by a handful of unelected individuals?
Would you be comfortable with the labs holding better technology than the public has access to, shared at their discretion?
Would you expect the government to take full ownership, or a global treaty to be signed, mature enough to agree that the potential upside is worth delaying to make sure one day we actually see it?
I know the pushback here.
Plenty of dangerous technology sits in private hands.
Private companies run nuclear power stations today, they make medicines, they launch rockets.
But every one of those industries has a pre-deployment licensing regime with teeth.
You cannot switch on a reactor because it passed your own internal review.
You cannot sell a drug because your own scientists are confident in it.
The regulator stands between the lab and the public before the thing goes live, not after.
So should you be able to release a new model without proper checks in place (whatever proper turns out to mean)?
For AI, we have voluntary commitments and testing after the fact.
A sniff test, if you will.
Given the current political environment, I don't see that changing any time soon.
Maybe if times were different.
I'd love to hear your thoughts.
Thank you for reading.
Oli