Magic-Pilled

Investors aren't going to buy this story for much longer

Twin Earth, 2026

Imagine a possible world in which the global pharmaceutical market worked like this: 6-9 months after an American company developed a blockbuster drug, Chinese labs develop a generic that’s just as effective and 10-100x cheaper. Patent protections and import controls don’t prevent this; the drugs are just different enough to skate by. US compounders then deliver the drug to Americans at cost plus, but most US households prefer at-home compounding. They download the recipes for free and only pay material costs. Sure, there are specialized drugs that the Chinese labs can’t replicate. Some patients will still pay American firms top dollar for these. But 95% of what Americans need gets made at home. Health outcomes soar. The people rejoice.

Let’s call this world Magic Pill World. In the actual world, we recognize that Magic Pill World would be unsustainable for American companies. That’s why we have patent protections for drugs. But in Magic Pill World, pharma companies have convinced their investors that — even though they lose their pricing power on new drugs after a few months — it’s still worth funding their R&D because each frontier molecule brings them closer to discovering the Magic Pill.

It’s not clear what they’ll do with the Magic Pill once they discover it. Maybe they’ll sell it. Or maybe they’ll use it on themselves because the magic pills increase their cognitive abilities such that their ingestion can lead to the discovery of increasingly-better Magic Pills in some kind of recursive cycle. While some investors remain skeptical, they’re convinced that at the very least, the US can’t stop climbing Magic Pill Mountain lest the Chinese beat them to the top. (Little do they know, China has no such interest. They just want to undercut the US market for geopolitical reasons; they don’t even believe in Magic Pills.)

And, of course, neither do we — because we generally understand the science behind pharmacological progress. So then why do we want to believe this story when it comes to AI? That’s the puzzle I want to explore in this essay (a shorter version of which I submitted to Dwarkesh’s essay contest).

The labs need you to believe AGI is inevitable

So first, let’s deal with those who’d insist we don’t have to believe this story — that the frontier labs could be profitable even without AGI. OpenAI and Anthropic are delivering a valuable product after all, and growing their revenues quickly. No dispute there. I personally love using Claude, and pay for a pro subscription. I use it at home, have built several productivity tools with it, and use LLMs every day at work. Shit rips!

But it’s also a very expensive product to deliver, with diminishing marginal returns for incremental improvements. I don’t know about you, but I haven’t really noticed much of a difference between Opus 4.6, 4.8, and Fable for the majority of my personal or professional workloads. I find myself defaulting to Sonnet or Opus 4.6 to save tokens. Good enough is good enough.

Most of us don’t need the best math or science capabilities, particularly when we can get domestically-hosted open source inference at pennies on the dollar compared to frontier lab inference. This is why EpochAI doesn’t believe there’s a clear case for frontier lab profitability without assuming they discover radically new capabilities.

And tech leaders are starting to notice. It’s been reported by some VCs that most early-stage startups they’re seeing choose to build on open source over closed source models. Other established companies are also starting to make the switch, with leaders from Uber, Coinbase, Lindy, Cursor, Harvey AI, and others. What felt like just a whisper weeks ago is about to grow into a chorus. The irony is that Sam Altman himself hinted at all this last year when he gestured towards a future in which intelligence would become nearly “too cheap to meter.”

But “too cheap to meter” is surely a strange way for a CEO to describe the service upon which his business is built! It’s less strange if you recall his early pitch to investors in OpenAI: that they would discover AGI, then — I kid you not — ask it how to generate a return. This is why you have to believe in Magic Pills if you’re going to believe the labs have a path to profitability at their target valuations.

So it should come as no surprise that just last week, Anthropic’s co-founder Jack Clark trotted out a warning that they were beginning to see signs of Claude accelerating model development research, which they took as sufficient evidence of self-improvement to merit speculation about a global pause in AI development. The argument itself is bad, but that’s really not important. It’s the move that matters. Moves like confidentially filing your S-1. Or telling your employees you’ll sue them if they try to cash out their equity early by selling shares in the private markets. Or quietly shifting your services up the stack into applications and away from models. These are all signals that they don’t know how to make the magic work.

DeepMind also had one of these magical pitches about their path to profitability — that they’d “solve intelligence, and then use it to solve everything else.” The labs don’t put it in those terms anymore, but the belief still seems implicit in their assumptions about revenue growth: specifically, that it will continue to scale in ways that would support their valuation targets even while the cost of good-enough is dropping precipitously thanks to open source. Given that the supply-demand dynamics for AI are not obviously Jevons-like, the US labs are facing a dilemma. Either they secure a sustainable competitive edge over open source that enables them to command price premia at massive volumes, or they unlock something so powerful that it generates massive returns for them independent of their customer demand. Either way, the underlying model capabilities would have to be very good: like “geniuses in a data center”-good.

But what is that ultimately, if not a belief that they will have achieved general intelligence? The kind that won’t delete production data overnight, but will do — you know — genius things? I’m not arguing that this won’t happen. I just want to point out that arguments for its inevitability usually go by more quickly than they should. We have much more immunity to Magic Pill arguments in pharma because we understand biological science. We don’t yet have that kind of herd immunity towards bad arguments in the philosophy of mind.

But the arguments for it’s inevitability are not very good

The most common argument for inevitability involves appeal to recursive self-improvement: that models will “help with their own training” in ways that compound more fruitful results from run to run. And while it’s true that models can be used to automate certain parts of the model development process, there aren’t good reasons to believe the effects occur in the place they need to for us to get real recursive gains. To get that, models would need to accelerate productive AI research by coming up with fruitful new ideas. Despite Jack Clark’s recent suggestions to the contrary, there’s little evidence that current models do anything like this — with OpenAI quietly admitting as much in their system card for GPT 5.5.

But if RSI isn’t inevitable given current or reasonably foreseeable capabilities, inevitability proponents still have another common line of argument: that because nature made general intelligence work for us, it must be possible to make it work artificially. The thought is that because there are at least some physical systems that are intelligent, it must be physically possible to construct other (perhaps silicon-based) systems that could do the same. Maybe this won’t happen in three years, but surely it will happen sometime in the next… thirty? And with so much money thrown at the problem, surely we should pull in our “timelines” right? After all, if intelligence is just a matter of realizing the right functional relations between inputs and outputs, it must be possible for us to simulate it in non-biological stuff. This is called “functionalism” in the philosophy of mind and is the theoretical foundation for cognitive science and artificial intelligence.

There are many different flavors of functionalism, and I won’t bore you with specific varietals. What matters for the inevitability argument is just one thing — the level of abstraction at which we model the relevant functional relations. First, we can level functional relations at the level of a whole organism or artificially intelligent system — as you might do if you’re narrating in a nature program — or if you’re administering, say, a Turing Test. While these organism/system-level functional relations may truly model intelligent behavior, they do so trivially. They’re not particularly helpful if you’re a scientist or engineer who wants an explanation of how it actually works.

To get there, functionalists look for more micro explanations of a system’s internal goings-on that might be said to “realize” the macro-level intelligent behavior. (‘Realize’ means ‘enable’ if they’re feeling mechanistic, or ‘constitute’ if they’re feeling metaphysical). But functionalists don’t opt for descriptions which are so micro that describing the functional relations becomes an intractably complex tangle. Instead, they locate intelligence at a level between the messy microscopic and the mundane macro — they call this the “software” level. Cognitive capacities like visual perception, memory, action, and reasoning are all modeled at this level — as operations that internal “modules” perform on environmental inputs to “produce” the system’s cognitive outputs.

So far so good. The majority of AI and cognitive science can be described as a way to try and discover and then simulate these modules. The challenge is just that… they might not exist — at least not at the scale or level of robustness needed for the theory to work. It might turn out that the microscopic biological details actually matter a lot more than most functionalists would like — specifically, that the functional relations realized at the level of microbiology might upwardly constrain the functional relations we observe at the level of macrobiological intelligent behavior. The matter might turn out to actually matter.

Metabolism is the paradigmatic example. All life on Earth centers around this act of highly-specific microbiological chemical transformation driven by an organism’s macro-level actions in the world to pursue and ingest nutrients. If intelligence turns out to be more like metabolism, then AI researchers might struggle to replicate the relevant macrobiological functions without replicating the microbiological ones. Put differently, functionalism depends upon the claim that when you’ve fully specified the relevant functional roles between sub-individual stuff needed to enable an agent to do intelligent things, those functional roles will be sufficiently abstract that they could be realized on multiple kinds of hardware — biological and artificial. The worry is just that when you actually spell out the functional roles behind say, bacteria descending a glucose gradient, in full detail, you might find that they essentially involve loopy, precarious, and autopoietic activity all the way down to the very chemical cycles that are constitutive of life — in which case, there’s hardly any room left to realize those functions on different kinds of hardware. The processes imposed by the chemistry propagate upwards to structure the macro-level processes away from which the AI researcher wanted to abstract.

What kind of evidence should we be looking for to assess whether that might be true? Well, consider why — after so many years — Claude and other LLMs still don’t seem to understand what following a rule actually means. If it were true that microbiological functions upwardly constrained macrobiological intelligence, we’d have an easy explanation. Unlike organisms that do follow rules (humans and some domesticated animals), Claude doesn’t understand the stakes involved in failure. Specifically, Claude does not understand the metabolic stakes involved in failure. Humans learn from an early age that rule-following has social stakes, and that those social stakes can be life-threatening: get it wrong, and you might be cast out. Get cast out, and you might not be able to eat. For rule-following organisms, there are deeply-understood connections between obeying an order, social awareness, fear of punishment, and — the gut. For LLMs to actually follow rules, they might need to be embedded in precariously self-producing physical agents that have what philosopher Kathryn Nave calls “a drive to survive.” Wittgenstein too speaks of this. So does Michael Thompson, Francisco Varela, JJ Gibson, Hans Jonas, and my old advisor Alva Noë.

So look. Biology is really fucking complicated. The mere existence of naturally intelligent biological systems doesn’t entail anything about the inevitability or even physical possibility of artificial intelligence, short-term or long-term. You can believe that and still be a fully committed naturalist about intelligence. There are many such people working in the philosophy of mind and cognitive science today.

Again, I’m not arguing that the labs won’t discover AGI, let alone that they can’t. This is all just gestural. My point is merely that the literature on cognition is too heterogeneous to accept one simple story — a story which we’ve largely accepted because we have a blinkered view of what intelligence is. When drug companies want to raise money, their investors do diligence with outside scientists who advise on the likelihood that the drug candidate can clear trials, be scalably distributed, and achieve sustainable advantage over competitors. Do we have enough of this discipline in AI — or have we mostly been Magic-Pilled? With so much on the line, it’s probably time to ask more rigorous questions about the science of intelligence.