0:00 An AI agent turned an initial one hundred thousand dollar stake into more than one-point-four million dollars.0:06 That happened inside a new simulation from the Qwen team, and it’s a sign that the chaos of AI agents we talked about in Episode 156 is starting to get measured, managed, and monetized.0:18 The age of just asking if an AI agent can do a job is over.0:22 The only question that matters now is how well it does that job, and what it costs.0:27 This isn't just about one flashy number.0:30 The entire landscape is shifting from showing off raw power to proving economic value.0:35 We're seeing it everywhere.0:37 It's about making AI not just capable, but reliable and profitable.0:41 It’s a move from the lab to the ledger book.0:44 And this week, we got three major signals that this transition is accelerating.0:49 First, there's that e-commerce benchmark.0:51 It's called, simply, E-Commerce Bench.0:54 Researchers from the Qwen team didn't just ask an AI to perform a single task.0:58 They threw eighteen of the world's top models into a year-long simulation, making them run several online stores at once.1:06 They measured everything.1:08 Profit, efficiency, even how well they avoided fraud.1:11 The results… were complicated.1:13 GPT-5.6 Sol, for instance, made that incredible one-point-four million dollar profit.1:18 But it ranked sixteenth out of eighteen on fraud avoidance.1:22 It was a money-making machine that was also leaking cash out the back door.1:27 This benchmark is a game-changer because it proves that there's no single "best" model anymore.1:33 There are only trade-offs.1:34 Second, Meta’s researchers just dropped a paper on something they call AI Research Preference Models.1:41 Here’s the problem they’re solving: a modern AI research agent can dream up thousands of potential experiments.1:48 But you can't afford to run them all.1:50 GPU time is the new gold.1:51 So how do you decide which ideas are worth pursuing?1:55 Meta’s answer is to build another AI… to supervise the first one.1:59 This preference model acts like a senior researcher.2:02 It looks at all the proposed experiments and predicts which one is most likely to succeed, before a single dollar of compute budget is spent.2:11 They tested it, and it works.2:13 It achieved the same results as an unguided agent in just fifteen hours instead of twenty-four, using less than two-thirds of the resources.2:22 It’s a massive efficiency gain.2:24 Meta is building an AI to make the process of building AI cheaper and faster.2:28 And third, the money is following this trend.2:31 A fintech startup called Felix Pago just raised two hundred million dollars.2:36 What for?2:37 To build an AI-powered financial assistant.2:39 This isn't about some far-off, theoretical AI.2:42 This is a conversational tool designed to help people with everyday financial decisions.2:48 The two-hundred-million-dollar bet isn't on a model that can write a sonnet about interest rates.2:54 It's on an application that can reliably help a user navigate a real-world financial choice.3:00 It's another signal that investors are backing practical, demonstrable value over abstract capabilities.3:06 The hype cycle is giving way to the business cycle.3:09 So what does it all add up to?3:11 A fundamental pivot.3:12 We're moving past the era of "look what I can do" and into the era of "look what I can earn." The benchmarks are getting more realistic.3:21 The research is getting more economically rational.3:24 And the funding is flowing to the companies that can turn AI into a product, not just a project.3:30 Let's dig deeper into these threads, because the details are where the real story is.3:36 First, that Felix Pago funding round.3:38 Two hundred million dollars is a serious number for a payments startup.3:42 The key is what they’re building: a conversational financial assistant.3:47 This isn’t about replacing a human advisor with a chatbot for complex wealth management.3:52 Not yet, anyway.3:53 This is about the small, everyday decisions.3:56 Think about helping someone navigate cross-border payments, understand transaction fees, or compare options for sending money home.4:04 These are high-friction, high-anxiety moments for millions of people.4:09 Felix Pago is betting that a well-designed AI can smooth out those rough edges.4:14 The interface is conversational, which lowers the barrier to entry.4:18 You don't need to understand complex financial jargon; you just talk to it.4:23 The "AI-powered" part isn't just a buzzword here.4:26 It's about creating a system that can understand a user's intent from natural language, access the relevant information, and present a clear, actionable path forward.4:36 This is a direct challenge to the traditional banking and remittance industry, which often relies on confusing forms and opaque fee structures.4:45 By putting an AI assistant in the user's pocket, Felix Pago aims to give the customer more power and clarity.4:52 The two-hundred-million-dollar valuation is a bet that this approach can capture a significant slice of the massive global payments market.5:01 It’s a perfect example of the new focus.5:03 The value isn't in the raw intelligence of the LLM itself; it's in the application.5:09 It’s about solving a specific, painful, and expensive problem for a huge number of people.5:14 It’s about commercialization.5:16 Now, let's turn to the research papers, because they show the same shift happening at the foundational level.5:23 Let's start with Meta and their AI Research Preference Models.5:27 To really get what they’ve done, you have to understand the bottleneck in modern AI research.5:33 It’s not a lack of ideas.5:34 An agent like their AIRA-dojo can generate an almost infinite number of hypotheses or potential solutions to a problem.5:42 The bottleneck is execution.5:43 Running a full-scale experiment requires immense computational power, which means time and money.5:50 You can’t just run every idea.5:51 You have to choose.5:53 Historically, that choice has been made by human researchers.5:56 They use their intuition, their experience, their gut feelings to decide which path looks most promising.6:03 But what if you could automate that intuition?6:06 That’s exactly what Meta did.6:08 They trained a model, not to solve the problem itself, but to evaluate the plans for solving the problem.6:14 They built two versions.6:16 The first is an "inference-only" model.6:18 It’s a pure thinker.6:20 It looks at the proposed plan, the code the agent wants to run, and the results of any previous, related experiments.6:27 Based on that static information, it assigns a score.6:30 It says, "This idea looks more promising than that one." The second version is "agentic." It’s a tinkerer.6:37 It goes a step further.6:38 Before committing the full budget to an experiment, it runs a few, small-scale pilot studies.6:44 It spends a tiny amount of compute to get a little bit of real-world data, and then uses that to make a much more informed decision.6:53 It's like a chef tasting the sauce before sending out a thousand plates.6:57 The results are what make this so powerful.7:00 Dropping these preference models into their existing research agent boosted its performance score on the AIRS-Bench benchmark from zero-point-six-eight-four to zero-point-seven-one-one for the thinker, and zero-point-seven-two-nine for the tinkerer.7:15 Both new versions set state-of-the-art records.7:18 But here's the economic punchline: they reached the performance that used to take the old agent twenty-four hours in just fifteen hours.7:27 And they did it using less than two-thirds of the compute budget.7:31 They made the process of scientific discovery itself more efficient.7:35 This isn't just about building a smarter AI.7:38 It's about building a more fiscally responsible AI research process.7:42 It’s a meta-level optimization that could accelerate the entire field.7:47 Okay, now for the main event.7:49 The paper from the Qwen team, E-Commerce Bench.7:52 This one connects directly back to what we were discussing a few weeks ago in Episode 156 about those unpredictable AI shopping agents from the Wharton study.8:01 That research showed how autonomous agents could create chaotic, unpredictable market dynamics.8:07 It was a warning.8:09 E-Commerce Bench is the beginning of the answer.8:12 It’s an attempt to bring order to that chaos by creating a standardized, long-horizon test for agent performance in a business context.8:20 Think about how we usually evaluate AI models.8:23 We give them a static test.8:25 Can you answer this question?8:26 Can you write this code?8:28 Can you summarize this document?8:30 These are single-session, one-off tasks.8:33 That’s not how business works.8:34 Business is a long, continuous game of adaptation, learning, and trade-offs over time.8:40 And that's what E-Commerce Bench simulates.8:43 They put eighteen of the world's most advanced models in charge of several online stores and let them run for a simulated 365 days.8:51 The models had a starting stake of one hundred thousand dollars.8:55 Their goal was to grow that capital.8:57 But they were scored on seven different dimensions, not just profit.9:01 This is the crucial part.9:03 They were measured on operational efficiency, customer satisfaction, inventory management, and even fraud avoidance.9:10 And no single model was the best at everything.9:13 This is the big reveal.9:15 The idea of a single, god-like AI that's superior on all axes?9:18 It's a myth.9:19 At least for now.9:20 Let’s look at the numbers.9:22 GPT-5.6 Sol was the top earner.9:24 It turned its one hundred thousand dollar stake into one million, four hundred thirty-one thousand, four hundred and twenty-five dollars.9:33 An absolutely staggering return.9:35 If profit were the only thing that mattered, it would be the undisputed champion.9:40 But it’s not.9:40 On the dimension of fraud avoidance, GPT-5.6 Sol ranked sixteenth out of eighteen models.9:46 Sixteenth.9:47 It was making a fortune, but it was also incredibly vulnerable.9:51 It was leaving the back door wide open while piling up cash in the front room.9:56 Meanwhile, another model, Fable 5, didn't make as much money, but it crushed GPT-5.6 Sol on operational efficiency.10:03 It ran a tighter ship.10:04 So if you're a CEO, which agent do you hire?10:07 The high-risk, high-reward superstar who might bankrupt you with a single mistake?10:12 Or the steady, reliable operator who delivers consistent, if smaller, gains?10:17 For the first time, we have a benchmark that even lets you ask that question in a meaningful way.10:23 It gets even more detailed when you look at the open-weight models.10:27 The leader there was Qwen3.8-Max-Preview.10:30 It ended the year with four hundred sixteen thousand, two hundred fifty-two dollars.10:35 That's thirty-eight percent more than its closest competitor, GLM 5.2.10:39 But the raw number isn't the most important part.10:43 The paper notes that the Qwen model showed the strongest learning over the 365-day horizon.10:48 Specifically, it got better at negotiating.10:51 It learned to progressively bargain its suppliers down on price across repeated orders.10:56 Think about that.10:57 This isn't just executing a pre-programmed strategy.11:01 This is an agent observing patterns over time, building a model of its environment, and adapting its behavior to get a better outcome.11:09 It learned a core business skill, autonomously.11:12 This is the long-horizon intelligence that everyone has been talking about, and E-Commerce Bench is one of the first tools that can actually measure it.11:22 So, this paper does two things.11:24 First, it shatters the illusion of a single "best" model by introducing the concept of multi-dimensional, practical trade-offs.11:32 It forces you to ask "best for what?" Second, it gives us a way to observe and quantify long-term learning and adaptation in a business context.11:41 It moves the evaluation from a static snapshot to a dynamic movie.11:45 It’s the next chapter in the story of AI agents.11:48 We've moved from "can they act autonomously?" to "how effective are they when they do?".11:53 So where does this all leave us?11:55 We have a new benchmark proving that in the world of autonomous e-commerce, there's no perfect AI, only a series of complex trade-offs between profit, risk, and efficiency.12:06 We have Meta developing AI supervisors to make the very act of AI research more economically efficient.12:13 And we have two hundred million dollars of venture capital betting on a practical AI assistant that solves everyday financial problems.12:21 The thread that connects all of this is a powerful, and I think irreversible, shift in the AI world.12:27 We are at the end of the beginning.12:30 The phase of pure spectacle, of just demonstrating raw capability, is winding down.12:35 A model that can do amazing-but-useless things is no longer enough to impress.12:40 The market, the researchers, the investors—they are all starting to ask a much harder set of questions.12:46 They’re asking: Is it reliable?12:48 Can I run it for a year without it going off the rails?12:52 Is it efficient?12:53 Does it burn through my entire cloud computing budget to perform a simple task?12:58 And most importantly: Does it create tangible, economic value?13:02 Does it make money, save money, or solve a problem that people will pay to have solved?13:07 This is a maturation.13:08 It's a move from AI as a science experiment to AI as an engineering discipline, and ultimately, to AI as a business function.13:16 The new state of the art isn't just about having the highest score on a leaderboard.13:22 It’s about delivering the most value per dollar of compute spent.13:26 It's about reliability over a long horizon.13:28 It's about turning intelligence into an economic engine.13:32 This week wasn't about a single, world-changing breakthrough.13:36 It was about something more subtle, and maybe more important.13:40 It was about the tools and the mindset finally catching up to the hype.13:44 The new frontier isn't just making AI smarter.13:47 It's about making AI work.
