PS, I think it's important to set the stage this way before I start spouting more opinionated opinions. My thinking is that it's very easy to cry wolf and lose credibility. Scott Alexander has a beautiful post about this: https://www.astralcodexten.com/p/against-the-generalized-anti-caution
So I want to emphasize that the reason to freak out about AGI is the uncertainty. The inability to rule out all the catastrophic scenarios.
PS, I think it's important to set the stage this way before I start spouting more opinionated opinions. My thinking is that it's very easy to cry wolf and lose credibility. Scott Alexander has a beautiful post about this: https://www.astralcodexten.com/p/against-the-generalized-anti-caution
So I want to emphasize that the reason to freak out about AGI is the uncertainty. The inability to rule out all the catastrophic scenarios.