Zuckerberg's Meta Confusion

Also skepticism is growing that Tesla has pulled off unsupervised self-driving

Last week we talked about a couple AGI disaster scenarios. Discussion in the comments (and the “drones” channel of the Beeminder Discord) is ongoing and I’m putting together a more realistic and more devastating “scenario 3” and working on understanding the objections to it. I could actually use some help with that, if you could answer this reader poll:

(The “AI doom train” refers to the chain of events and chain of reasoning that leads to a catastrophic conclusion. Answer with what you consider the weakest link in that chain.)

While that simmers, let’s talk about where Mark Zuckerberg thinks this is all heading.

Why care what Zuckerberg thinks? Well, he’s betting the company on AI and is making offers of hundreds of millions of dollars for individual AI researchers, trying to poach them from OpenAI especially. Supposedly one of the offers was over a billion dollars. To hire a single person. I’m old enough to remember when a billion dollars seemed like an extravagant price to acquire a whole company. (It’s how much Facebook paid for Instagram in 2012.)

If this is all a hype-filled bubble, it’s going to burst spectacularly. I’m betting it isn’t. (See also the first bullet item in the in-the-news section below.)

So let’s dig into what Zuckerberg thinks. First and foremost, he sees recursive self-improvement on the horizon. So do many people at the frontier labs. (Whether Metabook counts as a frontier lab yet is debatable.) So, Zuck continues, that implies we’re mere years away from superintelligence (ASI) and that the rest of the 2020s will be decisive in how this plays out. Also a common opinion.

He calls it an “open question what we will direct superintelligence towards”. I would add: or if we’ll be able to direct it at all. He continues with some “AI as normal technology” platitudes. As I mentioned in the in-the-news section last time, Scott Alexander has a beautiful counterargument to this kind of thing on the AI Futures Project blog. Overall, Zuckerberg’s p(doom)¹ is so low that all it warrants is a generic aside about how we’ll need to be rigorous about mitigating risks, before going right back to full-steam-ahead, “as required by building a free society”.

His ultimate point is that everyone can have their own personal ASI. Which sure sounds dangerous even if (massive if) we solve alignment. Where his confusion about all this really goes next-level is here:

Others in the industry believe superintelligence should be directed centrally towards automating all valuable work, and then humanity will live on a dole of its output.

This is infuriatingly disingenuous. If valuable work can be automated, it will be! What do you think people will choose to do with their personal ASIs in a free society? Unless of course ASI spins out of control and gets us all killed before it ever gets into the hands of normal people.

Finally, we see where he’s really going with all this: “Personal devices like glasses that understand our context because they can see what we see, hear what we hear, and interact with us throughout the day will become our primary computing devices.” Ok, great. I'm actually wearing said Faceglasses right now and can tell you that the AI is worse than useless. But maybe it will get better. We’ll see.

PS: I wrote a version of the above in the Beeminder Discord (thanks to Nathan Arthur) and after I did, Zvi Mowshowitz had a similar reaction. My favorite part was this conclusion:

The argument makes perfect sense if you understand that when Mark Zuckerberg says “superintelligence” he means “cool tricks with smart glasses and LLMs and algorithmic feeds,” not actual superintelligence.

In the News

• Yascha Mounk on the Peculiar Persistence of AI Denialists.

• The CEO of Nvidia seems to be more on Zuckerberg’s side.

• Anthropic describes their baby steps toward recursive self-improvement.

• Tesla robotaxi update: They’ve launched in the Bay Area with a human in the driver’s seat. Tesla says this is just while they await regulatory approval, but they may not have actually started the process to get that approval. Skepticism is growing about whether they’ve actually figured out level 4 autonomy:

  Down to a 19% chance today. I keep joking to my family that I’ll have to burn AGI Friday to the ground in shame if Tesla pulls this off, after my “We Regret To Inform You There Will Be No Tesla Robotaxis This Summer” post back in April. At least at this point we should probably agree it isn’t obvious? Also fueling the skepticism is the revelation in Tesla’s Q2 earnings call that the whole robotaxi project had logged 7,000 autonomous miles after a month of operation. With 10-20 robotaxis, that works out to 12-23 miles per car per day, on average. So arguably closer to a controlled demo than a rapidly scaling launch. But, again, huge uncertainty.

1. For anyone just tuning in, “p(doom)” means the probability that AGI leads to catastrophe for humanity. Even things like the earth becoming uninhabitable for biological life. See last week’s (and next week’s) AGI Friday for more.