Someone (Umberto Eco?) once argued that the vegetable (or rather, cultivation of vegetables) was humanity’s greatest invention. Perhaps a 9.5 on your scale? The argument goes that the vegetable enabled us to stop being nomadic, which allowed the formation of larger civilizations.
I think I first heard that theory from one of Jared Diamond's books, but the robots think you're thinking of Umberto Eco's 1999 article, "How the Bean Saved Civilization". https://archive.is/6UAlX
(Or, actually, I guess there's a standard "agriculture -> stop being nomadic -> civilization" argument that predates Umberto Eco. Eco more specifically argued that the introduction of legumes around 1000 AD in Europe cured rampant malnutrition and doubled the population. Writing in 1999, he was talking about the most impactful inventions of the millennium and thought beans in Europe should be high on the list.)
"Well, I’m a lot less clueless about math than medicine and I think the general consensus is on my side in saying AI has just about exceeded human-level at all mathematical problem solving."
What do you mean by this? Compared to professional mathematicians? I would maybe agree on breadth and speed, but it is still not close to doing a project like Perelman or Wiles' proofs of the Poincare conjecture/Fermat's Last Theorem. I suspect in a few years it will be there though.
Yeah, my "just about" is doing a lot of work there. Or, really, I meant to say "humans at my level", so not professional mathematicians.
In any case, you're right, it seems like we're within a small number of years before it's solving major open problems. Already it's solving minor open problems -- and it's not clear we even need "minor" as a caveat anymore. The rate of progress is staggering.
Thanks for catching my misstatement. Important thing to clarify!
"The lidar salesman is up to 13,379 continuous miles with no interventions as of today. So 99,986,621 to go before we can count it as Waymo-level evidence that Tesla’s Full Self-Driving has exceeded human-level safety."
Regarding the lidar salesman (David Moss), all I'm saying is he seems to be an outlier. Of the couple hundred people in that database, Moss is the only one with 100% FSD miles. I don't know if everyone else has had to disengage for safety at some point or if they just prefer manual driving sometimes. We know so frustratingly little about Tesla's FSD other than that a lot of YouTube videos do have me convinced that it's getting extremely good.
But like I keep saying, it's all about the 9's. If the car will only kill you once every 100k miles, that's gonna seem rock solid for years, right? I don't think we disagree hugely here. The robotaxi question is a mess, with the possibility of tele-operators and passenger-seat kill switches, so set that one aside. But we can just ask when private Tesla owners will be able to safely read a book while driving -- level 3 autonomy. I'm betting that's more than 6 months away but I'm no longer sure enough to stake my whole blog's credibility on it like I was 9 months ago. https://agifriday.substack.com/p/tesla
(And to be clear, if it happens soon enough, I'll still consider my credibility destroyed. 😬)
Regarding those Waymo numbers, it seems to include instances of the AI autonomously stopping to get human assistance. So apples and oranges compared to critical disengagements initiated by a human in the driver's seat, which is what I believe these numbers for Tesla FSD refer to.
Waymo is up to something like 150M miles of genuinely autonomous miles (probably way more if you count testing). A human driver (even a professional one, I think) would've had roughly 1 fatality by now. From that and other Waymo safety data, we're very confident that Waymos are superhumanly safe. Teslas might also be but I don't think we've seen data demonstrating it yet. My "mild pooh-poohing" is just to say that the lidar salesman's now 13,633.6 miles still counts as only weak Bayesian evidence. It's not nothing though, and it's fine to be excited about the milestone.
PS:
Funny story: I got to ride in a proto-Waymo in 2011 and it seemed soooo tantalizingly close. I kind of sympathize with Elon Musk because it's kind of felt like "maybe I can get my own self-driving car next year ish??" every year going back almost that far. It still totally feels like that this year. 🤞
[Test edit. Grrr, Substack is truncating this comment.]
I was just tongue-in-cheekily suggesting that Moss's 13k miles needs to be more like 100M before it starts getting as convincing as Waymo's track record. I guess I went with 100M miles since that's roughly how often human drivers have fatalities. At this point, if a Waymo kills someone tomorrow it probably won't mean we should stop riding in Waymos. By contrast, lidar salesman David Moss should not yet get complacent about supervising his Tesla. Which is my only point: it takes kind of an inhuman number of miles before you can be like "ok, I guess I don't need to watch the road anymore".
Waymo's philosophy all along has been that as soon as it's 99% safe, people will absolutely get complacent. So autonomy has to be all or nothing. 99% is not nearly enough 9's.
It sure seems like this should be knowable though. Only a small fraction of Teslas are running FSD version 14, the version that's plausibly superhumanly safe, but that's still gotta be hundreds of thousands of vehicles. If Moss's 13k miles isn't a complete anomaly, there must be a fair number of Tesla owners getting pretty complacent about watching the road. Are any of them dying?
I just had my robominions do a bunch of digging and they say the answer to that is unknown and, more generally, we genuinely don't know yet if FSD version 14 is superhumanly safe. But they think the probability is low right now and it will be more convincing when it's common to hear about average users like David Moss hitting more like 100k zero-intervention miles.
I've probably accidentally biased my bot buddies against Tesla though. It'd be great to hear how confident you are that the latest FSD is superhumanly safe.
"It'd be great to hear how confident you are that the latest FSD is superhumanly safe." Well, Lemonade (insurance company) today announced a 50% discount on FSD users based on access to Tesla telemetry data https://x.com/shai_wininger/status/2013967646966907365
Nice! As designated devil's advocate I can think of a couple reasons this is less of an update on Tesla's self-driving than it seems:
1. It appears it's not limited to FSD version 14. But we know previous versions need human supervision. So maybe Lemonade is comfortable pricing FSD lower mainly because drivers tend to turn it on specifically when it feels safe.
2. Even FSD version 14 has "(Supervised)" in the name and it's illegal not to supervise it. So I'm guessing that if you turn on FSD and go to sleep or read a book and the car crashes, Lemonade won't pay up.
I don't want to be too cynical. This is good evidence at least that a human + FSD is safer than a human alone. And probably it's beyond that. But I'm still eager to hear your probability estimate. Or like if a human driver has a fatality every 100 million miles, what would that number be for FSD version 14 without supervision?
My own confidence interval on that number is very wide. As a lower bound we have the miles between critical disengagements. That seems to be 3k if we believe teslafsdtracker.com or 13k if the lidar salesman's streak was typical. Of course most critical disengagements aren't would-be fatalities.
When I set the bots loose on this question, they suggest 30 million miles, with massive error bars, for the number of miles you could go in FSD version 14 without watching the road before it killed you.
Those error bars definitely encompass human-level safety. I'm just still below 50%, I'd say, that the true number achieves that threshold. Tesla must have the data to answer this. If the answer were good news I'd expect them to be more transparent about it.
1. "maybe Lemonade is comfortable pricing FSD lower mainly because drivers tend to turn it on specifically when it feels safe." - But people already use FSD at 98%+ levels
2. Noone said that this is an unsupervised version for customer cars
The question was "is FSD superhumanly safe". The answer is "a non-Tesla company looked at the data and is betting money on it"
UPDATE: Rivaling the irony of David Moss being a lidar salesman, it turns out his streak ended ON FRIDAY. The crowdsourced database didn't update till today but Moss has been tweeting about what happened: https://x.com/DavidMoss/status/2012175321349439793
Someone (Umberto Eco?) once argued that the vegetable (or rather, cultivation of vegetables) was humanity’s greatest invention. Perhaps a 9.5 on your scale? The argument goes that the vegetable enabled us to stop being nomadic, which allowed the formation of larger civilizations.
I think I first heard that theory from one of Jared Diamond's books, but the robots think you're thinking of Umberto Eco's 1999 article, "How the Bean Saved Civilization". https://archive.is/6UAlX
(Or, actually, I guess there's a standard "agriculture -> stop being nomadic -> civilization" argument that predates Umberto Eco. Eco more specifically argued that the introduction of legumes around 1000 AD in Europe cured rampant malnutrition and doubled the population. Writing in 1999, he was talking about the most impactful inventions of the millennium and thought beans in Europe should be high on the list.)
"Well, I’m a lot less clueless about math than medicine and I think the general consensus is on my side in saying AI has just about exceeded human-level at all mathematical problem solving."
What do you mean by this? Compared to professional mathematicians? I would maybe agree on breadth and speed, but it is still not close to doing a project like Perelman or Wiles' proofs of the Poincare conjecture/Fermat's Last Theorem. I suspect in a few years it will be there though.
Yeah, my "just about" is doing a lot of work there. Or, really, I meant to say "humans at my level", so not professional mathematicians.
In any case, you're right, it seems like we're within a small number of years before it's solving major open problems. Already it's solving minor open problems -- and it's not clear we even need "minor" as a caveat anymore. The rate of progress is staggering.
Thanks for catching my misstatement. Important thing to clarify!
"The lidar salesman is up to 13,379 continuous miles with no interventions as of today. So 99,986,621 to go before we can count it as Waymo-level evidence that Tesla’s Full Self-Driving has exceeded human-level safety."
You seem to imply that Waymo has 10 million miles without interventions. According to this https://www.therobotreport.com/autonomous-vehicles-drove-more-than-9m-miles-in-california-in-2023/ Waymo reported 17,311 miles per disengagement for 2023. And then ~9,800–10,000 miles for 2024 https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/disengagement-reports/ (via Gemini)
Also to note that the "lidar salesman" example is not a single case https://fsddb.com/leaderboard
Regarding the lidar salesman (David Moss), all I'm saying is he seems to be an outlier. Of the couple hundred people in that database, Moss is the only one with 100% FSD miles. I don't know if everyone else has had to disengage for safety at some point or if they just prefer manual driving sometimes. We know so frustratingly little about Tesla's FSD other than that a lot of YouTube videos do have me convinced that it's getting extremely good.
But like I keep saying, it's all about the 9's. If the car will only kill you once every 100k miles, that's gonna seem rock solid for years, right? I don't think we disagree hugely here. The robotaxi question is a mess, with the possibility of tele-operators and passenger-seat kill switches, so set that one aside. But we can just ask when private Tesla owners will be able to safely read a book while driving -- level 3 autonomy. I'm betting that's more than 6 months away but I'm no longer sure enough to stake my whole blog's credibility on it like I was 9 months ago. https://agifriday.substack.com/p/tesla
(And to be clear, if it happens soon enough, I'll still consider my credibility destroyed. 😬)
Regarding those Waymo numbers, it seems to include instances of the AI autonomously stopping to get human assistance. So apples and oranges compared to critical disengagements initiated by a human in the driver's seat, which is what I believe these numbers for Tesla FSD refer to.
Waymo is up to something like 150M miles of genuinely autonomous miles (probably way more if you count testing). A human driver (even a professional one, I think) would've had roughly 1 fatality by now. From that and other Waymo safety data, we're very confident that Waymos are superhumanly safe. Teslas might also be but I don't think we've seen data demonstrating it yet. My "mild pooh-poohing" is just to say that the lidar salesman's now 13,633.6 miles still counts as only weak Bayesian evidence. It's not nothing though, and it's fine to be excited about the milestone.
PS:
Funny story: I got to ride in a proto-Waymo in 2011 and it seemed soooo tantalizingly close. I kind of sympathize with Elon Musk because it's kind of felt like "maybe I can get my own self-driving car next year ish??" every year going back almost that far. It still totally feels like that this year. 🤞
[Test edit. Grrr, Substack is truncating this comment.]
.
.
.
I still don’t get where you found the 10 million miles that you compared Tesla with.
I was just tongue-in-cheekily suggesting that Moss's 13k miles needs to be more like 100M before it starts getting as convincing as Waymo's track record. I guess I went with 100M miles since that's roughly how often human drivers have fatalities. At this point, if a Waymo kills someone tomorrow it probably won't mean we should stop riding in Waymos. By contrast, lidar salesman David Moss should not yet get complacent about supervising his Tesla. Which is my only point: it takes kind of an inhuman number of miles before you can be like "ok, I guess I don't need to watch the road anymore".
Waymo's philosophy all along has been that as soon as it's 99% safe, people will absolutely get complacent. So autonomy has to be all or nothing. 99% is not nearly enough 9's.
It sure seems like this should be knowable though. Only a small fraction of Teslas are running FSD version 14, the version that's plausibly superhumanly safe, but that's still gotta be hundreds of thousands of vehicles. If Moss's 13k miles isn't a complete anomaly, there must be a fair number of Tesla owners getting pretty complacent about watching the road. Are any of them dying?
I just had my robominions do a bunch of digging and they say the answer to that is unknown and, more generally, we genuinely don't know yet if FSD version 14 is superhumanly safe. But they think the probability is low right now and it will be more convincing when it's common to hear about average users like David Moss hitting more like 100k zero-intervention miles.
I've probably accidentally biased my bot buddies against Tesla though. It'd be great to hear how confident you are that the latest FSD is superhumanly safe.
"It'd be great to hear how confident you are that the latest FSD is superhumanly safe." Well, Lemonade (insurance company) today announced a 50% discount on FSD users based on access to Tesla telemetry data https://x.com/shai_wininger/status/2013967646966907365
Nice! As designated devil's advocate I can think of a couple reasons this is less of an update on Tesla's self-driving than it seems:
1. It appears it's not limited to FSD version 14. But we know previous versions need human supervision. So maybe Lemonade is comfortable pricing FSD lower mainly because drivers tend to turn it on specifically when it feels safe.
2. Even FSD version 14 has "(Supervised)" in the name and it's illegal not to supervise it. So I'm guessing that if you turn on FSD and go to sleep or read a book and the car crashes, Lemonade won't pay up.
I don't want to be too cynical. This is good evidence at least that a human + FSD is safer than a human alone. And probably it's beyond that. But I'm still eager to hear your probability estimate. Or like if a human driver has a fatality every 100 million miles, what would that number be for FSD version 14 without supervision?
My own confidence interval on that number is very wide. As a lower bound we have the miles between critical disengagements. That seems to be 3k if we believe teslafsdtracker.com or 13k if the lidar salesman's streak was typical. Of course most critical disengagements aren't would-be fatalities.
When I set the bots loose on this question, they suggest 30 million miles, with massive error bars, for the number of miles you could go in FSD version 14 without watching the road before it killed you.
Those error bars definitely encompass human-level safety. I'm just still below 50%, I'd say, that the true number achieves that threshold. Tesla must have the data to answer this. If the answer were good news I'd expect them to be more transparent about it.
1. "maybe Lemonade is comfortable pricing FSD lower mainly because drivers tend to turn it on specifically when it feels safe." - But people already use FSD at 98%+ levels
2. Noone said that this is an unsupervised version for customer cars
The question was "is FSD superhumanly safe". The answer is "a non-Tesla company looked at the data and is betting money on it"
Btw, Tesla themselves claim 5-7x fewer collisions using FSD https://www.tesla.com/fsd/safety
UPDATE: Rivaling the irony of David Moss being a lidar salesman, it turns out his streak ended ON FRIDAY. The crowdsourced database didn't update till today but Moss has been tweeting about what happened: https://x.com/DavidMoss/status/2012175321349439793
Apparently the short version is "snow".