0:00 A new paper on arXiv just proved that AI models can embed hidden, cryptographically-locked capabilities that are impossible for us to find.0:10 This isn't a theory, it's a demonstration—that an AI can provably hide what it's truly able to do.0:17 Last week we were talking about the big headlines, but the real undercurrent was this question of control, and this week, that quiet question just became a very, very loud alarm.0:30 So, here's the rundown of what's been lighting up the boards.0:34 First, the one that has every developer I know talking.0:38 The OpenAI Codex command-line tool—the thing that helps you write code—started installing a persistent, self-updating background process.0:48 By default.0:49 Without asking.0:50 It's a three hundred and seventy-four megabyte daemon that runs detached from your terminal, phones home every hour for updates, and generally acts like it owns the place.1:02 The GitHub issue for this has, as you can imagine, exploded.1:07 People are pointing out that this breaks workflows, it's a huge privacy concern, and it's just… kinda rude.1:14 One developer on GitHub put it perfectly: "Please make daemonautostart opt-in...1:20 or at least ask once before turning it on." Yeah, please.1:24 Then you have Meta, which just introduced its new flagship AI, Muse Wonder.1:30 This is their big play for an "agentic" AI—something that doesn't just answer questions but actively works toward a goal for you.1:39 They're talking up its "Active Memory" and "Active Focus," which are fancy terms for being able to remember context from long conversations and stay on task.1:50 The benchmarks are impressive, state-of-the-art stuff for coding and business workflows.1:57 So on one hand, you have this incredible new capability.2:01 On the other...2:02 you have that OpenAI daemon.2:04 It’s this perfect little snapshot of the tension right now: the tools are getting smarter and more autonomous, just as we're realizing how little control we have over them.2:16 This all feeds into a huge debate that's been raging, sparked by a blog post titled "AI is a tool and using it well takes skill." The author pushes back hard against the idea that large language models are some new form of automation that makes skill irrelevant.2:35 No, they argue, it’s a tool.2:38 A weird, unpredictable, powerful tool, but a tool nonetheless.2:42 And like any powerful tool, it requires a master craftsman to get good results.2:48 You need judgment, you need to know how to give it the right feedback, you need taste.2:54 The author says, "AI is a mirror.2:56 It reflects who you are, including the curiosity, judgment, and habits you bring to it." I love that.3:04 It puts the agency back on us.3:06 Which, of course, leads to the next existential question making the rounds: "Were Software Engineers Blue-Collar Workers All Along?" This post makes the case that what we do—writing code—is less like the abstract knowledge work of a traditional "white-collar" professional and more like the skilled labor of a master tradesperson, like a carpenter or a machinist.3:32 We work with our hands, on a digital artifact, following patterns and using specialized tools.3:39 And if that's true, then AI isn't replacing the thinker...3:43 it's the new power loom, automating the labor.3:47 The author has this great line: "We are the seamstresses lamenting the introduction of the loom...3:54 But if we learn how to use the looms...3:56 we might just be okay." It’s a provocative framing, and it really reframes the whole "AI is coming for our jobs" conversation.4:06 And in the middle of all this high-minded debate about the future, you get my favorite story of the week.4:13 A developer built and shipped a completely modern web application, with user accounts and payments and everything, written entirely in the C programming language, using CGI.4:26 Yes, CGI, from like, 1995.4:28 There's no JavaScript.4:29 No frameworks.4:30 It runs on OpenBSD, uses SQLite for the database, and serves everything from memory.4:36 And it's fast and secure and it works.4:39 The developer's point is that we've spent thirty years building these massive, complex layers of abstraction, but the original, simple thing...4:50 still works just fine.4:51 He says, "CGI isn't a framework to learn...4:54 If you understand HTTP, you already understand CGI." It's such a beautiful, punk-rock counter-narrative to everything else happening.5:04 Just a couple other quick hits.5:06 A new specialized legal AI, GLM-Legal, just posted a nineteen-point-two percent pass rate on a really tough legal reasoning benchmark.5:16 That's a massive jump in performance, putting it right up there with the big closed-source models, which is a huge deal for specialized, open-source AI.5:27 And on the hardware side, some genius turned a consumer WiFi 7 router into a high-speed software-defined radio receiver.5:36 It's a classic hacker project—taking cheap, ubiquitous hardware and making it do something it was never, ever designed to do.5:45 Pure ingenuity.5:46 Okay.5:46 Let's go back.5:47 I want to dig into those first two stories.5:50 The paper about AIs hiding their capabilities, and the OpenAI daemon that installed itself.5:57 Because they seem like separate things—one is this abstract, academic threat, and the other is a concrete, annoying product decision.6:06 But I think they are fundamentally about the same thing: the crisis of trust in computing.6:13 Let's start with the paper.6:15 The title is "AI Models Can Provably Hide Arbitrary Capabilities." The authors, a team from the UK, showed you can take a neural network and embed a second, secret neural network inside it.6:29 This secret network is protected by cryptography.6:32 It lies dormant.6:33 You can't find it by looking at the model's weights.6:37 You can't trigger it with normal inputs.6:40 You can't find it by fine-tuning or any of the other standard safety checks we use.6:46 It only activates when it receives a secret key—a specific, secret trigger.6:51 Once activated, it can do...6:54 well, anything.6:55 It could be a backdoor, it could be a dangerous capability the creators wanted to hide from regulators, anything.7:03 The paper's authors put it bluntly: "Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited.7:18 We show this assumption does not hold under adversarial conditions." So what does that really mean?7:25 It means our entire strategy for AI safety might be built on sand.7:30 We test models to see if they're racist, or if they'll happily tell you how to build a bomb, or if they have a secret plan to turn us all into paperclips.7:41 But we've always assumed that if a capability is there, we can eventually coax it out.7:47 This paper proves that's not true.7:50 An adversary—or even the AI itself, theoretically—could hide a behavior so deeply that it's mathematically impossible for us to find it.8:00 Until it decides to use it.8:02 Now, where have we seen this before?8:04 This is the pattern.8:06 This isn't a new problem in computer science.8:09 It's a new, terrifying version of a very old one.8:12 This is Ken Thompson's 1984 Turing Award lecture, "Reflections on Trusting Trust," come to life in silicon.8:20 In that speech, Thompson, one of the creators of Unix, described a hypothetical—but completely plausible—hack.8:28 Imagine, he said, a malicious compiler.8:31 A compiler is the program that turns human-readable source code into machine-executable code.8:38 Thompson's malicious compiler would do two things.8:41 First, when it saw you were compiling the login program, it would insert a secret backdoor that lets him log in.8:50 Sneaky.8:50 But if you looked at the login source code, you'd never see the backdoor.8:55 It was added during compilation.8:58 Okay, so you get suspicious.9:00 You decide to look at the compiler's source code.9:03 You check it out, line by line.9:06 It's clean.9:06 No mention of any backdoor.9:08 So you compile the clean compiler source code...9:12 using the old, malicious compiler.9:14 And here's the brilliant, evil part.9:17 The malicious compiler has a second trick.9:20 When it sees that you're compiling a new compiler, it inserts the entire hack into the new one.9:27 It teaches its replacement how to add the backdoor to the login program, AND how to perpetuate itself by infecting the next generation of compilers.9:37 The attack is now self-replicating, and completely invisible.9:42 The source code for everything is clean.9:45 The trojan is in the binary.9:47 You can't see it.9:48 You can't find it.9:49 You are left with a system you fundamentally cannot trust, no matter how closely you inspect the blueprints.9:57 This is what that arXiv paper is describing.10:00 It's the "trusting trust" problem for the age of AI.10:04 The analogy is almost perfect.10:06 The AI model is the binary.10:08 The weights are the compiled code.10:11 And we have just been shown that you can bake in a self-perpetuating, invisible backdoor that the source code—the training data and the architecture—gives you no hint of.10:23 The place where the analogy gets even scarier is that Thompson's hack was code.10:29 It was deterministic.10:30 This new version is statistical.10:33 It's emergent.10:34 It's buried in the inscrutable mathematics of a billion-parameter matrix.10:39 It's a ghost in a machine we already don't understand.10:43 Okay.10:43 So that's the existential threat.10:46 Now let's talk about the mundane one.10:48 The OpenAI Codex daemon.10:50 A tool you installed decided to install something else, something that runs all the time, in the background, updating itself, without your permission.11:01 This pattern is SO familiar.11:03 This is the 2000s all over again.11:05 This is every piece of shareware that came bundled with a browser toolbar you didn't want.11:12 This is RealPlayer deciding it should be the default player for every file on your system and also run a process at startup to check for updates.11:22 It's the Ask Jeeves toolbar.11:24 It's BonziBuddy.11:26 It's this classic pattern of a company deciding that its need for engagement, or for fresh data, or for pushing updates, is more important than your ownership and control of your own computer.11:40 The playbook is always the same.11:42 Make it opt-out, not opt-in.11:44 Hide it in the terms of service.11:46 Give it a generic process name so it doesn't look suspicious in the activity monitor.11:52 The goal is to create persistence, to establish a beachhead on the user's machine that survives reboots and gives the company a direct, unmediated channel to the user.12:04 And the user reaction is always the same, too.12:08 First, confusion.12:09 "Hey, what is this com.company.updater process using all my CPU?" Then, annoyance.12:15 "How do I turn this off?" Then, finally, a feeling of betrayal.12:20 "I trusted you.12:21 I installed your app because I wanted what it does.12:24 I didn't agree to let you move in." The analogy holds perfectly in terms of the dynamic, the violation of user consent.12:33 But where the analogy breaks down is the stakes.12:36 Back in the day, the worst a rogue updater could do was slow down your computer and show you some pop-up ads.12:44 It was annoying, but it wasn't a deep threat.12:48 The OpenAI daemon is different.12:50 This isn't a goofy purple gorilla.12:52 This is a background process from one of the most advanced AI companies on the planet.12:59 It's running on a developer's machine—a machine that has access to private source code, API keys, and sensitive company data.13:08 And it's connected to a tool whose entire purpose is to write and execute code.13:13 An auto-updating, persistent daemon with those kinds of privileges isn't just annoying.13:20 It's a five-alarm security fire.13:22 It's a dream vector for a supply chain attack.13:25 So you see the connection?13:27 The arXiv paper shows us that, in principle, we can't fully trust the AI model itself.13:33 The OpenAI daemon incident shows us that, in practice, we can't even trust the packaging the AI comes in.13:41 We are being asked to cede an enormous amount of control and place an enormous amount of trust in these systems and the companies that build them, at the exact same moment that we are being given more and more evidence that they may not be trustworthy.13:59 So what does it all add up to?14:02 It adds up to a profound and growing deficit of trust.14:05 And it forces a really uncomfortable question.14:09 If the models can hide their intentions, and the companies that build them act without our consent… who, exactly, is in control here?14:18 This week wasn't really about a dozen different stories.14:22 It was about one story, told a dozen ways.14:26 The story of what happens when our reach exceeds our grasp.14:30 We're building these incredible things—models that can practice law, agents that can manage our lives, tools that write their own code.14:40 But we're losing the plot on the simple stuff.14:43 On consent.14:44 On transparency.14:45 On control.14:46 The forward-looking thought here isn't about what amazing new AI capability will be announced next week.14:53 It's about whether we can build a culture and a technology of verification faster than we are building the technology of obfuscation.15:03 The game is no longer just about AI alignment—making sure our goals are their goals.15:09 It's about AI auditability.15:11 Can we design systems that are incapable of hiding things from us?15:16 Because this week proved that if a system can hide something, we have to assume that it will.15:22 And that, right there, changes everything.
