The Limits of Next-Token Predictors Have Been Greatly Exaggerated

How an LLM trained on human text can end up smarter than any human that wrote that text. Also (lack of) AI consciousness.

First things first: Claude Fable 5 is back and I’m embarrassed how caught up in the hype I am. It’s just so gobsmackingly good at software development (and math and data analysis and hunting down sources — still not writing). Did I shred my credibility by saying this about the last three models? I get it. They keep getting more useful and less likely to fall on their face or otherwise be worse than useless. I tend to be too forgiving of their foibles, just by being excited about what’s possible. So to try to be more concrete, I’d say that if you could choose between a week of access to Fable and a full-time junior software developer (and you only care about productivity, not the plight of poor junior software developers) then you’d get more done with Fable. From now till July 7 you can use Fable as part of a paid Claude plan. After that you pay by the token. Which is probably cheaper than a full-time software developer, but who knows. (This is why Anthropic is worth almost a trillion dollars despite being founded just five years ago; it’s insane.) As usual, it’s the trajectory we seem to be on that’s more exciting/terrifying than what’s possible today.

As a little trial of how good it is now, I had Fable revamp my old Technological Richter Scale app. I’m not a visual design person but I think we’re passing the point where you can readily tell if an app was vibe-coded. Feel free to embarrass me further by telling me you can absolutely still tell:

The above is also showing my own updated distribution for the possible AI futures we’re facing by the time progress plateaus, whenever that may happen. One change from five months ago¹ is I’m pretty much declaring AI to already be past things like air conditioning and mobile phones at level 7 — once-in-a-decade technologies — and landing at least at level 8, once-in-a-century. I’ve left a handful of percentage points at level 7 in case I have LLM psychosis. It’s at least easy to imagine, as some of you surely are, that my excitement will seem dumb in hindsight. Do coding agents even help as much as compilers? I do think that much is baked in already, even if we don’t see further improvement, which I expect we will.

(And just to emphasize, the 9% in the last column — my p(doom) essentially — is high enough that I view AI risk as humanity’s most urgent problem to solve. That might mean an international treaty to pause AI before we get frog-boiled or before we solve recursive self-improvement and get frog-incinerated.)

Next-token prediction

Mostly today’s AGI Friday is to spell out a general point about what it really means that LLMs are “next-token predictors”. This is to refute the common argument that LLMs being trained on text written by humans means that the smartest of those humans will tend to be a ceiling on how smart the LLM can get. I’ve made this point before in passing but I want to lay out the argument so I can link to it when this confusion arises. I was confused about it myself for a long time.

The key insight is that predicting what comes next in a piece of text is sometimes much harder than generating that text in the first place. As a stylized example², imagine some human doctors diligently writing down case histories for patients with a novel disease. And suppose the doctors have no understanding of this disease. Patients just show up, tell you about their history farming bats or something and present with a cough and later they spike a fever. The doctor just writes all that down. But if you train someone or something to predict that the text “bat farmer presents with a cough and then _____” needs its blank filled in with “spikes a fever”, how could they do that?

Dumber LLMs just vibe their way through. “Cough” and “fever” tend to appear near each other, so just make a vaguely plausible guess. But to get more and more accurate at those predictions, at some point the neural network ends up encoding a world model. It builds a theory of the disease and can predict what will happen next with these patients, or what medicine will help them. Humans merely writing down case histories weren’t that smart. They just wrote down what happened in the world. Much easier!

This isn’t to say that LLMs are smarter than humans. Just that there’s no clear ceiling on how smart they could get. And as Scott Alexander explains in “Next-Token Predictor Is An AI’s Job, Not Its Species”, the task of next-token prediction is highly analogous to a task very much central to the human brain. All the pieces aren’t necessarily there yet for AI, but they’re also not exactly necessarily not there.

In practice, the progression is pretty palpable. A few years ago the most elementary geometric or physical reasoning problems would unmask an LLM as just vibing a response. I tried the following today on Facebook’s old LLM, Llama:

Cut a cube with 6 faces in half; how many faces does each half have?

It gave the vibey answer of 3. A modern LLM will do whatever the LLM equivalent of visualizing the slicing is and give the actual answer. If you object to using words like “understand” for AI, that’s fine, but the AI answers any such question in a way that, if a human answered that way, would prove that the human understood. So the distinction ends up being purely philosophical.

So, consciousness?

Speaking of philosophy, I like Rob Miles’s new lament (it’s a YouTube short) about how annoying the AI consciousness question is. I want to go further than Miles (the extra mile?) and say it’s just entirely premature to consider current AI to have any degree of consciousness. I appreciate the philosophical point that if, say, insects are a fraction of a fraction of a percent conscious then why not AI models? And if there’s even a possibility that the degree is nonzero, that’s interesting and important. I’m taking the hard line that it’s strictly interesting, not important.

Miles adds that it’s noncrazy to expect the degree of AI consciousness to gradually increase over time. But that means the first people to freak out that AI might be showing signs of consciousness will be like the “say ‘I’m alive’” meme:

Miles points out that, by training LLMs on the entirety of the internet, it’s in retrospect wholly unsurprising that they will role-play conscious characters. I often hear people propose that someone do some actual academic research on this. Like painstakingly purge all mentions of consciousness from an AI’s training data and then see if you can get that AI to aver it has any kind of subjective experience. Of course that would be prohibitively expensive and, realistically, I think a positive result would just make me suspect that the concept managed to sneak through in cryptic form. I guess I’m weirdly closed-minded about this. Mostly I think it’s something to worry about after a lot more research on mechanistic interpretability and other AI alignment work that prioritizes keeping AI from spinning out of human control.

Fifty-Two Friday Flashback

A year ago I was writing about AGI analogies. AGI is like an asteroid with clueless astronomers who can’t actually tell if it’s on a collision course. The pooh-poohers may get lucky but their pretension of certainty is wrong and reckless.

Or AGI is like an earthquake that’s overdue. It’s coming “soon” for some definition of “soon” we have a terrible handle on. But we’re plausibly compressing — via the trillions we’re spending — how long it would’ve taken. Intuitively 25-50 years might’ve seemed reasonable at the pace we were going before Wall Street and the hyperscalers lost their minds over this. I called it plausible that we’re speedrunning that progress into 2.5-5 years. So 1.5-4 years now if you wanted to hold me to that. The upper end of that I could still call just barely plausible.

Next I analogized³ intelligence to the visible spectrum of light. As in, what seems to us humans like a wide range might not be. AI could shoot past us:

Side update from last year: The golems are no longer stymied by the vertically compressed horizontal rainbow I wanted for that graph.

I followed up the next week with my argument for why this argument isn’t spoiled by the fact that intelligence isn’t actually one-dimensional.

My last AGI analogy was Y2K. If you’re not old enough to know what that is, consider pandemics and how there are two possible times you can react to an exponential: too early and too late. It’s called the Preparedness Paradox. Fine point, but I don’t like the way I characterized AGI risk a year ago. I’m working on a future AGI Friday in which I’ll talk about at least four levels of risk.

In the news roundup last year, I quoted Scott Alexander on using the golems for research: “an unprecedented combination of brilliant and mendacious; too useful to avoid but too unstable to fully trust.” Still very true, but the brilliance/mendacity ratio just keeps steadily improving.

The news a year ago was full of cases of golems driving people literally crazy. I called it mostly but not entirely silly. Nailed it.

Finally, I was already, in mid-2025, impressed by how much the Overton window had shifted in terms of taking the extinction risk from superintelligence seriously. I’d say the shifting is continuing apace. The government involvement in deciding which AI models are safe is part of that shift. I don’t think the dangers of Mythos and Fable are very similar to the dangers of a future superintelligence, but just appreciating that AI can be dangerous at all is a good step.

Random Roundup

• The Wall Street Journal wrote a highly misleading article about how China has caught up to the US on AI. I’ll let Zvi Mowshowitz explain how false that is.

• Speaking of AI and prediction, the latest Astral Codex Ten says the AI superforecasters are here. His vision of where this is headed still sounds a bit sci-fi to me right now but, helpfully, he describes a test we can apply 1-2 years from now. Namely, if AI forecasters don’t shoot past the best human superforecasters in that timeframe, Scott will concede he’s wrong. And, more importantly, that the “AI as Normal Technology (AINT)” people are more likely to be right in general (at least in the near-term). I think the AINTers are very wrong to be as certain as they are but they could turn out to be right so I’m anxious to revisit this question in 1 and then 2 years and see how the probabilities have shifted. People who bet against extending lines on graphs have a terrible track record but I’m struggling to go higher than a 15% chance that Scott Alexander is right that AI superforecasters will soon be wildly superhuman.

• Grant Sanderson of 3Blue1Brown had a fascinating conversation on the future of AI and math with Dwarkesh Patel. Grant points out three ways AI can solve an open math problem: grinding, connecting, and constructing. Grinding is like if it comes up with a thousand-page proof that technically works but doesn’t help humans understand anything. Connecting is like what happened with the unit distance problem: the AI found a way to apply an idea from a different subfield. And constructing would be if the AI creates some whole new mathematical edifice as a way to attack the problem. That last one seems like the kind of thing that would imply AGI, but of course that’s what we always think. (Eventually it will be true though.)

1. I was going back and forth back then. I had 16% probability on level 7 but just the previous month I declared AI to be solidly between levels 7 and 8 and said I was “confident” we’d end up beyond 7. Maybe I’ve been consistent if you squint hard enough.

2. Thanks to Yudkowsky and Soares for coming up with it.

3. This one I stole from Christopher Moravec.