Who Do We Trust?

It is the year 2000 and I’m with a group touring Italy. When we get to Florence, our bus stops outside a central tourist area. We have a couple of hours to “explore” and as we get off the bus, a woman checks her tour book for a specific, lesser known, attraction near the Florence Cathedral. Our tour guide chimes in with directions on how to get there and the woman responds firmly - “The book says…”. The guide replies, “trust the guide, not the guide book.” In 2026, I’m left wondering whether AI is the book or the guide.
Recently, there’s been a flood of stories, podcasts, and posts about AI safety. The leaders of OpenAI and Anthropic have called for a slowdown of frontier model progress and for some form of regulation. Safety researchers inside and outside of these companies have said AI poses a literal existential threat to humanity with some placing that threat at 10% over the next decade. Jensen Huang went on the Ezra Klein show and essentially said it’s an engineering problem - if the model isn’t safe then the frontier labs should not release it. As an engineer myself, this answer resonated with me. When Ezra pushed him on details, I found Jensen patronizing in his response. It felt to me, like Jensen was implying it was a big boy problem, just leave it to the experts. They know how to solve hard problems. Jensen said net new regulation is not needed, that existing regulation is sufficient. Bill Gates came on a separate episode and said AI has recently passed certain thresholds and now must be managed and regulated. His idea is that every frontier model would be required to have a kind of “control program” that could prevent it or those who use it from doing harm. I found the idea of a control program interesting for its nostalgia. I found AI recently passing safety thresholds slightly disingenuous, as a way to explain why he was surfacing the concerns now. These concerns have been known to those in the field for a long time. Ezra has also had on other experts that separately pushed the “it is just a technology problem” or on the other end of the spectrum “we have/will lose control to bad actors including the models themselves”. Recursive self improvement (RSI) has become a boogeyman.
I come back to this question: do you trust the book or the guide?
I drive a modern electric vehicle, a China built NIO sedan. It has a lot of technology built in. If we need directions, music, seat adjustment, parking, and even driving my wife can ask me to do it, or ask the AI device (NOMI) in the car. She asks NOMI every time. I am the book. AI is the guide.
Bill Gates’ idea of a control program is an interesting one and brings back memories of early operating system development. To greatly oversimplify, without a control program, applications would be responsible for their own I/O and their own scheduling. It would be (and was in the very early days) a disaster with crashes and disruptions. A control program provided common services while also protecting the applications from themselves. The frontier models today protect users from themselves (and the rest of us) by safely handling harmful requests such as how to make a bioweapon, etc. There is no common control layer or consistent set of regulations across frontier models and it is possible to use open weight models without any constraints at all. I think Gates’s idea is worthy of pursuit, at least as part of a defense in depth strategy.
Satya Nadella, who succeeded Bill Gates and Steve Ballmer as Microsoft CEO, had a more concrete and realistic proposal than either Jensen Huang or Bill Gates.
When I read it, I was reminded of proven principles I learned from my time at Microsoft.
- Least Privilege: A service (AI) should only have authorization/access rights to resources it needs to perform a task, and no more. This has the added benefit of making the system owner think through what resources are really required.
- Zero Trust: Verify every request and don’t assume a request is trustworthy because it comes from the same network or a known user.
- Assume Breach: Design systems as if bad actors have gotten into the system and you need to limit/recover any damage.
- Defense in Depth: Use multiple layers of defense. One layer doesn’t fully solve the problem but taken together they can prevent attacks. This is where I see the Bill Gates proposal fitting in.
It’s not about whether you trust AI or humans. It’s a question of what you trust them with.
I trust the Italian tour guide to tell me how to get to the historic cafe. I don’t trust him with my room key. I certainly don’t trust him in the room with my wife.