Whose Trust Is It Anyway?

A response to Chris Blair — because no agent has ever switched itself on

I’m about a third of the way through Chris Blair’s new white paper, Trusted by Design, and I’ve had to stop reading. Not because it’s bad — it’s good, genuinely thoughtful work on where NZ needs to get to on AI trust infrastructure. I’ve stopped because one word keeps snagging on something in my brain, and I can’t get past it until I’ve picked at it a bit.

The word is “trust” itself.

Trust is a strange thing to try to engineer. Strip it back to basics, and it’s really just this: a willingness to be exposed to someone else’s actions, on the belief they’ll do right by you, even though you can’t control or fully verify what they’ll actually do. It’s a belief about future behaviour under uncertainty — not a documented state. It’s esoteric — hard to see, easy to feel. You know when you have it, and you know when it’s gone, but you’d struggle to point to the exact document, certificate, or audit trail that put it there in the first place. And yet so much of the conversation about AI trust — Chris’s included, and he’s not wrong to have it there — is about the visible scaffolding. Identity. Credentials. Governance frameworks. Standards. The stuff you can write down.

Here’s my worry: having things written down doesn’t mean trust has arrived. It means you’ve built a container that trust might live in if the humans inside it behave in ways that earn it. The paperwork isn’t the trust. It never has been.

This is roughly how I picture the chain every time I build something with AI.

We’ve Been Here Before; We Just Didn’t Notice

Think about this for a second: most of us have spent 30, 40, 50 years trusting closed-source systems we have zero visibility into. Windows. macOS. Excel. The firmware in your router. None of us has read the source code. None of us has audited the supply chain. We just... use them. Daily. For things that matter — banking, healthcare records, government services, our entire working lives.

We didn’t get a white paper on “Trusted by Design” for the PC. We got a beige box, a warranty, and three decades of it mostly working. Trust, in that case, accumulated through use, through consequence, through the absence of catastrophe — not through a framework somebody stood up in advance.

Sometimes I think we've forgotten how much we already trust blindly.

So when AI shows up and suddenly everyone’s asking, “But how do we know we can trust it?” I find myself wondering: why does this one feel different? And I think Chris’s paper actually gets at a real answer — it is different because of scale, speed, and the fact that these systems can now act, not just compute. That’s fair. But I don’t think it means we throw out fifty years of trust and muscle memory and start from scratch. More on that in a minute.

And this isn’t just a PC-on-your-desk problem. RealMe — the login that 163 government services and 56 agencies rely on, the thing New Zealanders use to prove who they are to their own government — has had its login and assertion services running on Microsoft Azure AD B2C, hosted offshore, since 2021. We didn’t build a sovereign identity trust framework and then bolt cloud onto it. We picked an American hyperscaler’s off-the-shelf identity product, migrated six million-plus sign-ins onto it, and got on with our lives. If the same kind of export-control decision that suspended Fable and Mythos access ever landed on Azure AD B2C for “national security” reasons — less likely, sure, but not impossible — we’d discover in a hurry that we’d already outsourced a chunk of our trust infrastructure years before AI ever entered the conversation. We just didn’t call it that at the time.

What Chris Is Actually Saying

For anyone who hasn't read it: Trusted by Design is Chris Blair's case for treating AI trust as infrastructure, not reputation. He's tracking three waves — AI operating inside organisations (where we mostly are now), AI acting across organisational boundaries through personal agents, and New Zealand positioning itself as a genuinely trusted node in the wider intelligence economy. He wants identity and credentials, data governance, Māori data sovereignty, AI assurance, sovereign compute, and interoperable standards all wired together into one visible trust environment, rather than left as separate silos nobody connects.

His sharpest example — and it’s a good one — is the mid-June export-control suspension that cut New Zealand users off from Anthropic’s Fable 5 and Mythos 5 models. Not because of anything anyone here did. A foreign government made a call, and overnight, a tool people were relying on simply wasn’t there. That’s his proof point for why continuity and substitutability need to be designed in, not assumed. You can’t build critical capability on a foundation you don’t control and can’t be sure will still be standing tomorrow.

I largely agree with all of this. I’m not writing this to tear it down. I’m writing this because I think there’s a layer underneath it that’s worth naming — and once you name it, some of the “we need new infrastructure for this” instinct might soften into “we need to connect infrastructure we already have.”

It Still Needs a Human to Start It

Here’s my thought, still forming as I write it, which is exactly how I like to write:

Right now — today, this AI, this generation of it — nothing happens without a human stimulus. No autonomy, no action, no output, without someone or something acting on a person’s behalf in the first place. The agent, the workflow, the model call — it was given life by someone who chose to build it, and it was invoked by someone who chose to use it. Every single time. That chain never breaks, no matter how many layers of agents you stack on top of each other.

Which means: the identity that actually needs to be verified and trusted is the human. Not the computer. Not the process. The agent doesn’t have intent of its own to be trustworthy or untrustworthy about — it has my intent, borrowed, encoded, and executed at scale.

I built something recently that makes this concrete for me. I’ve been part of the team at Beehyve Health building a neuro-inclusion profiling tool — I understood the problem we were trying to solve, I worked with Claude to build the app, and I crafted how the AI responds inside it, grounded in an established psychological framework. Responsibility for what that app says and does sits with me and the team who built it. Full stop.

Three identities, one app, only one that actually carries moral weight. Which one is it?

Now trace the layers. There’s the eventual user of the app, asking it questions, who has a digital identity within the system. There’s the app itself, which has its own identity — an HTTPS certificate, a record of deployment. And underneath that, there’s the LLM generating the responses, which has its own model identity and its own provider. Three technical identities, stacked.

But only one of them carries moral weight. Somewhere in the history of that app is a record of me creating it, encoding the framework into it, and stimulating the whole chain that eventually needed a certificate to secure it. The certificate exists because I decided the app needed to exist. Machine-to-machine trust models — certificates, signed requests, verified endpoints — already handle exactly this kind of layered identity between systems. We’ve had working models for M2M trust for decades. So my honest question is: why do we need to invent a new one for AI, when the actual open question isn’t “how do machines verify each other” but “how do we keep tracing every agent action back to the accountable human at the top of the chain”?

Where I Might Be Underselling the Problem

I want to be fair to the counterargument here, because I think it’s got real teeth.

Behaviour and intent do get encapsulated in the system itself, not just in the human who started it. When I encode a framework’s logic into an app, and it runs unattended thousands of times, the behaviour of that system — the actual packets of decision-making happening inside each interaction — is now a thing that exists independently of me sitting there watching it. There’s a layer of probabilistic uncertainty in how these models respond, and that uncertainty is exactly what breeds distrust. Chris’s Estonia example — an “AI ID” for the agent itself, so you know whose agent you’re dealing with when it shows up at your digital door — is trying to solve for that gap. It’s not solving a fake problem.

I don’t think I’m wrong that the responses are still only the direct result of a stimulus a human set in motion. But I’ll concede this: at scale, with enough agents acting on enough people’s behalf, “trace it back to the human” becomes a genuinely harder forensic exercise than it sounds like from where I’m sitting with one app and one team. That’s fair.

Where I land, though, is this: that’s an argument for better tracing, using models we largely already understand — certificates, audit logs, signed provenance — not necessarily an argument for a wholesale new trust architecture built from scratch. Build on what exists. Don’t invent for the sake of inventing. Make sure we’re answering the right problem before we start pouring concrete for a new one.

Building on what Chris wrote

I don’t disagree with Chris Blair. I think Trusted by Design is asking most of the right questions, and the Fable/Mythos example alone should be enough to convince any NZ leader that continuity planning for AI dependency is no longer optional. What I want to add — as a bit of a different flavour on the same dish — is this: before we design new trust infrastructure for AI, trace the chain all the way back. Every agent, every workflow, every autonomous-sounding system currently in existence still has a human fingerprint on the “on” switch. That’s where accountability has always lived, and I don’t think that changes just because the thing sitting on top of it got a lot more capable.

Before we build new trust infrastructure — have we traced the thread all the way back yet?

We’re not at full autonomy yet. Whether we ever get there, I genuinely don’t know. But until we do, the question of trust isn’t really about the machine at all. It’s about us — the people who keep choosing to build the things and choosing to use them.