Goalpost-Moving vs The Simplest Sufficient Condition

Updating in light of evidence

Welcome to all the new subscribers sent by conservative blogger and venture capitalist Nic Carter! (And huge thanks to him for the endorsement and praise for my review of Leopold Aschenbrenner’s book predicting imminent AGI, though I have a list of quibbles.) I was worried I was in a bit of a rationalist echo chamber so this should help with that. If you’re new, AGI Friday’s shtick is Massive Uncertainty: AI progress could play out in wildly different ways. Deriding AI as a “blurry JPEG of the web” or fancy autocorrect started out correct but at this point is willful denial of reality. At the other extreme, plenty of the hype wildly outpaces current reality. The bull case is that current AI exhibits “sparks” of AGI that may or may not be about to catch fire. In between, the commentary on AI is rife with fallacies, like the Safe Uncertainty fallacy or the “Fidel Castro will never die” fallacy. Today let’s talk about a common theme: moving goalposts.

A recurring pattern for decades is that we identify candidate problems that we expect to be AGI-complete, meaning that solving the problem will only be possible with AGI. Surely if a computer is smart enough to beat grandmasters at chess, that will be AGI, right? (See Moravec’s paradox for how/why this is so counterintuitive.) Then we solve these problems and say “oops, just kidding, that didn’t require AGI after all.” Is that moving the goalposts? I think there’s a key distinction. The goalpost for AGI is whether it can do any keyboard-and-mouse-mediated task that any human can do. That’s the standard definition, factoring out robotics, and doesn’t change. It’s the candidate problems that change. Yet it remains valuable to keep identifying those candidate sufficient conditions for AGI.

I myself was sure that “explain any joke” would be such a sufficient condition. Turned out it wasn’t! I’m now anxiously waiting to see if “solve any math problem any human can solve” is AGI-complete. Replace “any human” with me personally and we’re practically there. Just today, Epoch AI credibly claimed that most of the impossibly hard FrontierMath benchmark is almost surely going to fall in 2026. But current AI, last I checked, falls on its face trying to do simple tasks like navigating car rental websites. So, more and more, I expect we’ll get superhuman math problem-solving without that qualifying as AGI. Again, not goalpost-moving, just being wrong about what suffices for AGI.

(Mildly embarrassing interlude: I was a previous AGI curmudgeon, even betting large sums of money against AI optimists on a ten-year timescale, back in 2008. My line in the sand back then was asking the AI common-sense questions like “what’s bigger, your mom or a french fry?”. In retrospect it’s easy to explain how that alone is short of true understanding. Not moving the goalposts meant admitting I was just wrong and that AGI is plausibly/potentially happening soon-ish.)

The goalpost-moving that drives me crazy is pooh-poohers who ridicule AI for not being able to count the number of r’s in the word strawberry or draw a human hand with the right number of fingers and then, instead of updating their beliefs when AI stops making those mistakes, they just grab the next example.

I used to accuse Gary Marcus of this but more recently he’s impressed me by making real shifts in his opinions and projections in light of the latest AI capabilities. He also endorses setting lines in the sand as a guard against goalpost-moving. On the other hand he’s still giving his blog posts sensationalist titles like “the grand AGI delusion”. Sigh.

And then there’s Marcus’s commentariat. Hoo-boy. The discussion of every AI lab announcement is met with comments like “yawn, wake me up when it can do that with less than 100 watts of power like a FREAKING HUMAN BRAIN”. We could hit AGI running on our phones and Gary Marcus fans will be like “yawn, wake me up when it can do that with CARBON ATOMS”. I’m exaggerating here but every time I look it’s like they’re parodying themselves with their smug, overconfident derision of every new capability, no matter how impressive it is. They mock something AI can’t do, six months later it does it, and they just silently switch to mocking AI’s inability to produce a humanoid robot that can swim the English Channel or something. I’m really grateful that it’s the opposite here, with people excited to be proven wrong.

I think the worst offender, besides Marcus’s commenters, might be Emily Bender. Her claim to fame is introducing the term “stochastic parrots” in a 2021 paper and more recently wrote an infuriatingly tendentious book called AI Con. I think the strongest point in the book, from what I understand (by talking to my unreliable robot friend who does seem to know the whole book inside and out), is that there’s a lot of free-riding on the excitement about AI. Any software that can find an excuse to call itself AI gets more attention. Watch out for snake oil, in other words.

So far so good. The key is to ignore the hypesters who think AGI is already here and ignore the pooh-poohers (like Bender) who think this is all sci-fi silliness. There are things AI can do now that are mind-blowing and we just don’t know yet how much farther it can go. It’s perfectly reasonable to suspect we’ll hit a wall any time now. But to be certain of that is idiocy. We keep making predictions and adjusting our timelines in light of how they play out.

Speaking of failure to update in the face of evidence, a 2020 paper by Bender seems to be the backbone of her thesis in AI Con. It lists example after example of things LLMs can’t do and explains why LLMs will never be able to do them. Most of them are absolutely being done by LLMs in 2025.

I don’t want to be too mean to her. I absolutely believed all of that myself until 2022. What’s shameful is her unwillingness to change her mind.

Random Roundup

• Speaking of rationalist echo chambers, this charitable grant is pretty funny:

  $5K for approximately five thousand novels about AI going well. This one requires some background: critics claim that since AI absorbs text as training data and then predicts its completion, talking about dangerous AI too much might “hyperstition” it into existence. Along with the rest of the AI Futures Project, I wrote a skeptical blog post, which ended by asking — if this were true, it would be great, right? You could just write a few thousand books about AI behaving well, and alignment would be solved! At the time, I thought I was joking. Enter Aaron, who you may remember from his previous adventures in mad dental science. He and a cofounder have been working on an “AI fiction publishing house” that considers itself state-of-the-art in producing slightly-less-sloplike AI slop than usual. They offered to literally produce several thousand book-length stories about AI behaving well and ushering in utopia, on the off chance that this helps. Our grant will pay for compute. We’re still working on how to get this included in training corpuses. He would appreciate any plot ideas you could give him to use as prompts.

  I can’t decide what to think of it. It’s fun and funny and potentially interesting even if the probability of it impacting AI alignment is exactly zero. And nothing has probability exactly zero. Maybe the counterargument is that if you’re genuinely worried (as I endorse being!) about AI risk, what else could this amount of time and energy be put towards? I guess I’m not one to talk. Their hilarious project seems likely to have more of an impact than this newsletter.

• Christopher Moravec thinks Ford’s BlueCruise has leapfrogged Tesla’s soi disant Full Self-Driving. (PS: Maybe I take that back. He’s comparing to pre-FSD Tesla’s and I suspect he’d be at least as impressed by the latest FSD.)

• Chatbot tip of the day! It can be useful to sanity-check an opinion by running it by your friendly neighborhood robot (I’m back to recommending Claude for normal people, or if you’re not willing to pay) but do not forget what pathetic sycophants they are. So here’s the actual tip: Open two tabs and in one say “I think [XYZ], what do you think?” and in the other tab say “I think [OPPOSITE OF XYZ], what do you think?”. If the two answers jibe, that’s a far better sanity check. (And as a bonus, you develop a better sense of how much to trust it. It’s a pretty crazy superpower, when you think about it. It’s like that Harry Potter scene where the bad guy iterates on corrupting Hermione, obliviating her between each attempt. You don’t know what someone really thinks until you probe them under identical initial conditions in multiple branched timelines. If the someone is a chatbot, you can actually do that.)

• Relatedly, from Eliezer Yudkowsky:

  AI companies be like: As cognition becomes cheaper, what expensive services, that formerly only CEOs and kings could afford, shall we make available to all humankind...? (Thought for 24 seconds.) hey let’s do evil viziers whispering poisonous flattery

Thanks to Gabe Hayos, frabcus in David MacIver’s Discord, and Bethany Soule for discussions that led to this AGI Friday.