I think a bimodal outcome of (AGI by ~2030) or (pretty normal 2030s, 2040s) makes sense to me. I haven’t actually watched the video, so this is mostly me defining it myself, which may or may not line up with the video haha.
First, let’s assume that no further development of basic AI models happens, we’re stuck forever with GPT-5 level. This level can do a good job augmenting labor, and it can do some amount of independent work, but it clearly isn’t at the point where whole departments can just be replaced. Many more tools will be invented, but they’re capped out at the limits of token costs, reliability, and ability. It’s maybe a 2x multiplier on overall productivity for office jobs, but it’s not a 10x or 100x multiplier that would completely upend society. It maybe replaces Google, but Google integrates it first. It can’t replace physical labor at all. It will still have a large effect on society, but I predict 2045 will look like 2025, to a similar extent that 2025 looks like 2005 (which is still pretty different! smart phones! social media! the internetification of everything! chatgpt!). Notably though, OpenAI’s valuation goes down. There’s no super intelligence, and a lot of the valuation was based on them being able to get super intelligence first.
Now let’s examine that assumption. Current scaling techniques are limited and it’s not clear where else in the pipeline you can scale up. The concept of reasoning (increasing scale of output) was fairly predictable early on, albeit not the details. Increasing scale of input data/processing has been happening all along. But at this point we’re running out of ability to scale more on either of these fronts without some new technique. So unless a new technique is invented, I think we’re in that first hypothetical.
So will a new technique be invented? Obviously it’s impossible to know now, but it seems likely to me that the probability of it being found is a function of the amount of resources that are being put in to find it. Therefore, that either it will be found in the next ~five years with trillions of dollars of investment, or it likely won’t be for much longer.
I do want to note though that I don’t pin anything on 2030 specifically. That number mostly depends on how many resources society is willing to invest without anything new to show for it. And I don’t have a good sense for that, venture capitalists are unknowable to me.
I agree with this key sentence in the abstract of the first paper:
> Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.
And the authors clarify that even satisfying their *indicators* of consciousness doesn't necessarily imply consciousness. It's a hard problem.
> Probably mostly via “oops, this thing we thought was fundamental to consciousness must not be, since LLMs can do it”.
This sort of depends on how you feel about Searle's Chinese Room. You may instead say "oops, LLMs must be conscious because they can do this thing that we think is fundamental to consciousness."
Well, I have strong feelings that Searle is an idiot. Which is maybe ironic here, since I think the Chinese Room in Searle's thought experiment would totally count as conscious.
So I think I was too glib in this post. I don't mean to say that LLM-ness itself precludes consciousness. Just that LLMs as they exist today are self-evidently not conscious. Based on interacting with them, not based on their implementation.
For those who don't know, the philosopher John Searle proposed the following thought experiment to show that AGI is impossible, or that no AI can "truly think" or "truly understand":
Imagine a room into which one can pass in slips of paper with any text written on them that you like, but only in Chinese. There's a man in the room who speaks no Chinese but he has a giant rulebook that instructs him on how to write responses to the incoming slips of paper, which he does. Further suppose that, from the outside, this Chinese Room passes the Turing test. It accepts messages in Chinese and gives Chinese replies in a way that demonstrates full AGI.
But now, Searle asks, where is the actual understanding of Chinese? The man doesn't understand Chinese, the rulebook is just a physical static book, and the slips of paper are likewise inanimate. So the Chinese Room doesn't actually understand. Nor, insists Searle, does any machine. Understanding and thinking are somehow tied to biology.
For why I think that's all wrong, let me quote Scott Aaronson's take. I'm copying this from Quora, which is too hard to link to:
[begin quote]
I think consciousness is genuinely weird and mysterious, but the Chinese Room is a question-begging appeal to intuition that doesn’t get us particularly closer to understanding the mystery.
As a first observation, the man in the room, and _his_ understanding of Chinese (or lack thereof), seems like a complete irrelevancy -- and that the conversation is in Chinese (rather than English or whatever else) is a red herring too. Also, that the processing of the Chinese queries happens in a “room,” by a man who consults a “rulebook,” are sleights-of-hand that can serve only to distract our intuitions from something that _some_ people might consider extremely relevant: namely, that any computational process remotely similar to the human brain would involve hundreds of billions of steps. If we blew up each neuron to the size of a Post-It note, the human brain would be maybe a mile in a diameter; if (more realistically) each neuron got its own room with its own man in charge, then maybe a hundred miles in diameter; if individual molecules within the neurons needed their own rooms, we could be talking astronomical sizes.
So we might as well simplify the Chinese Room Argument by getting rid of the man, the room, the Chinese, and the rulebook. :-) The only part that’s really relevant is the slips of paper, and the fact that those slips enact a computation that (by hypothesis) passes the Turing Test. And the question before us is whether, let’s say, a Solar-System-sized Rube-Goldberg-type machine, passing slips of paper around in a manner isomorphic to what a human brain does, would bring about consciousness.
Searle’s answer, of course, is no. He’s actually written that anyone who can’t just _see_ that the answer is no needs not a counterargument, but psychiatric help!
But when the question is put to him: what is the principled criterion by which you judge another person to be conscious, yet judge the paper-juggler to be unconscious, even if their external behavior is the same? -- Searle says, it’s because of the brain’s “biological causal powers.” _And then he never explains what those causal powers consist of!_ He never even gestures toward any general principle by which, for example, if we met some aliens based on gallium arsenide rather than carbon, we could decide whether those aliens had the causal powers or not.
To me, this doesn’t seem like a coherent stance about consciousness, or even the beginning of such a stance. Pinning consciousness on “causal powers” seems like pinning why sleeping pills work on their “sedative virtue”: it’s just a restatement of the problem.
Let me share an amusing anecdote. The first few times I taught my undergraduate computability and complexity course at MIT (6.045), I included a lecture about the “great philosophical debates of computer science”: the Turing Test, the Chinese Room, Roger Penrose’s views, etc. My goal was always to get the students arguing with each other about these questions.
But I always failed, because I couldn’t find a _single_ MIT undergrad who thought Searle’s position made sense and would argue for it. With increasing desperation, I’d argue Searle’s position myself, just to try to get a rise out of the students -- but they’d calmly reply that, no, if a brain passing electrical signals around can be conscious, then a mechanical contraption passing slips of paper around can be conscious too … or at any rate, _I_ hadn’t given them any real proposal to differentiate the one from the other. Why wasn’t that obvious?
> So I think I was too glib in this post. I don't mean to say that LLM-ness itself precludes consciousness. Just that LLMs as they exist today are self-evidently not conscious. Based on interacting with them, not based on their implementation.
This seems entirely fair to me. I'd agree that current LLMs are clearly not conscious but that LLM-ness is not disqualifying in something that acts conscious.
I think we're on the same page about all this, but I may have spoken too soon. There's one thing about the implementation of LLMs that conceivably precludes consciousness: the inherent staticness of the weights and how LLMs can't (currently) continuously improve.
Imagine taking a human, showing them some text, and asking them to predict the next fragment of the next word. As soon as they utter it, you add it to the text, reset their brain to the exact state it was in before they saw the text, and repeat.
It's a little mind-bending (in more ways than one) but it's kind of reducing the human to a single moment of consciousness. Does that still count as conscious?
I don't think that static weights is inherent to LLMs per se, it's just the choice made for now. I'd say if there's no way to update weights that makes consciousness questionable, although if you can keep a large enough token context that might be enough, especially if the LLM can choose "side tokens" or some other way of propagating state through time.
I think a bimodal outcome of (AGI by ~2030) or (pretty normal 2030s, 2040s) makes sense to me. I haven’t actually watched the video, so this is mostly me defining it myself, which may or may not line up with the video haha.
First, let’s assume that no further development of basic AI models happens, we’re stuck forever with GPT-5 level. This level can do a good job augmenting labor, and it can do some amount of independent work, but it clearly isn’t at the point where whole departments can just be replaced. Many more tools will be invented, but they’re capped out at the limits of token costs, reliability, and ability. It’s maybe a 2x multiplier on overall productivity for office jobs, but it’s not a 10x or 100x multiplier that would completely upend society. It maybe replaces Google, but Google integrates it first. It can’t replace physical labor at all. It will still have a large effect on society, but I predict 2045 will look like 2025, to a similar extent that 2025 looks like 2005 (which is still pretty different! smart phones! social media! the internetification of everything! chatgpt!). Notably though, OpenAI’s valuation goes down. There’s no super intelligence, and a lot of the valuation was based on them being able to get super intelligence first.
Now let’s examine that assumption. Current scaling techniques are limited and it’s not clear where else in the pipeline you can scale up. The concept of reasoning (increasing scale of output) was fairly predictable early on, albeit not the details. Increasing scale of input data/processing has been happening all along. But at this point we’re running out of ability to scale more on either of these fronts without some new technique. So unless a new technique is invented, I think we’re in that first hypothetical.
So will a new technique be invented? Obviously it’s impossible to know now, but it seems likely to me that the probability of it being found is a function of the amount of resources that are being put in to find it. Therefore, that either it will be found in the next ~five years with trillions of dollars of investment, or it likely won’t be for much longer.
I do want to note though that I don’t pin anything on 2030 specifically. That number mostly depends on how many resources society is willing to invest without anything new to show for it. And I don’t have a good sense for that, venture capitalists are unknowable to me.
Related work (HT Tedd Hadley): https://arxiv.org/abs/2308.08708
Also an impressive ongoing series on human consciousness by Sarah Constantin: https://sarahconstantin.substack.com/p/making-sense-of-consciousness-part-8a8
I agree with this key sentence in the abstract of the first paper:
> Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.
And the authors clarify that even satisfying their *indicators* of consciousness doesn't necessarily imply consciousness. It's a hard problem.
> Probably mostly via “oops, this thing we thought was fundamental to consciousness must not be, since LLMs can do it”.
This sort of depends on how you feel about Searle's Chinese Room. You may instead say "oops, LLMs must be conscious because they can do this thing that we think is fundamental to consciousness."
Well, I have strong feelings that Searle is an idiot. Which is maybe ironic here, since I think the Chinese Room in Searle's thought experiment would totally count as conscious.
So I think I was too glib in this post. I don't mean to say that LLM-ness itself precludes consciousness. Just that LLMs as they exist today are self-evidently not conscious. Based on interacting with them, not based on their implementation.
For those who don't know, the philosopher John Searle proposed the following thought experiment to show that AGI is impossible, or that no AI can "truly think" or "truly understand":
Imagine a room into which one can pass in slips of paper with any text written on them that you like, but only in Chinese. There's a man in the room who speaks no Chinese but he has a giant rulebook that instructs him on how to write responses to the incoming slips of paper, which he does. Further suppose that, from the outside, this Chinese Room passes the Turing test. It accepts messages in Chinese and gives Chinese replies in a way that demonstrates full AGI.
But now, Searle asks, where is the actual understanding of Chinese? The man doesn't understand Chinese, the rulebook is just a physical static book, and the slips of paper are likewise inanimate. So the Chinese Room doesn't actually understand. Nor, insists Searle, does any machine. Understanding and thinking are somehow tied to biology.
For why I think that's all wrong, let me quote Scott Aaronson's take. I'm copying this from Quora, which is too hard to link to:
[begin quote]
I think consciousness is genuinely weird and mysterious, but the Chinese Room is a question-begging appeal to intuition that doesn’t get us particularly closer to understanding the mystery.
As a first observation, the man in the room, and _his_ understanding of Chinese (or lack thereof), seems like a complete irrelevancy -- and that the conversation is in Chinese (rather than English or whatever else) is a red herring too. Also, that the processing of the Chinese queries happens in a “room,” by a man who consults a “rulebook,” are sleights-of-hand that can serve only to distract our intuitions from something that _some_ people might consider extremely relevant: namely, that any computational process remotely similar to the human brain would involve hundreds of billions of steps. If we blew up each neuron to the size of a Post-It note, the human brain would be maybe a mile in a diameter; if (more realistically) each neuron got its own room with its own man in charge, then maybe a hundred miles in diameter; if individual molecules within the neurons needed their own rooms, we could be talking astronomical sizes.
So we might as well simplify the Chinese Room Argument by getting rid of the man, the room, the Chinese, and the rulebook. :-) The only part that’s really relevant is the slips of paper, and the fact that those slips enact a computation that (by hypothesis) passes the Turing Test. And the question before us is whether, let’s say, a Solar-System-sized Rube-Goldberg-type machine, passing slips of paper around in a manner isomorphic to what a human brain does, would bring about consciousness.
Searle’s answer, of course, is no. He’s actually written that anyone who can’t just _see_ that the answer is no needs not a counterargument, but psychiatric help!
But when the question is put to him: what is the principled criterion by which you judge another person to be conscious, yet judge the paper-juggler to be unconscious, even if their external behavior is the same? -- Searle says, it’s because of the brain’s “biological causal powers.” _And then he never explains what those causal powers consist of!_ He never even gestures toward any general principle by which, for example, if we met some aliens based on gallium arsenide rather than carbon, we could decide whether those aliens had the causal powers or not.
To me, this doesn’t seem like a coherent stance about consciousness, or even the beginning of such a stance. Pinning consciousness on “causal powers” seems like pinning why sleeping pills work on their “sedative virtue”: it’s just a restatement of the problem.
Let me share an amusing anecdote. The first few times I taught my undergraduate computability and complexity course at MIT (6.045), I included a lecture about the “great philosophical debates of computer science”: the Turing Test, the Chinese Room, Roger Penrose’s views, etc. My goal was always to get the students arguing with each other about these questions.
But I always failed, because I couldn’t find a _single_ MIT undergrad who thought Searle’s position made sense and would argue for it. With increasing desperation, I’d argue Searle’s position myself, just to try to get a rise out of the students -- but they’d calmly reply that, no, if a brain passing electrical signals around can be conscious, then a mechanical contraption passing slips of paper around can be conscious too … or at any rate, _I_ hadn’t given them any real proposal to differentiate the one from the other. Why wasn’t that obvious?
I had to discontinue that lecture…
[end quote]
> So I think I was too glib in this post. I don't mean to say that LLM-ness itself precludes consciousness. Just that LLMs as they exist today are self-evidently not conscious. Based on interacting with them, not based on their implementation.
This seems entirely fair to me. I'd agree that current LLMs are clearly not conscious but that LLM-ness is not disqualifying in something that acts conscious.
I think we're on the same page about all this, but I may have spoken too soon. There's one thing about the implementation of LLMs that conceivably precludes consciousness: the inherent staticness of the weights and how LLMs can't (currently) continuously improve.
Imagine taking a human, showing them some text, and asking them to predict the next fragment of the next word. As soon as they utter it, you add it to the text, reset their brain to the exact state it was in before they saw the text, and repeat.
It's a little mind-bending (in more ways than one) but it's kind of reducing the human to a single moment of consciousness. Does that still count as conscious?
I don't think that static weights is inherent to LLMs per se, it's just the choice made for now. I'd say if there's no way to update weights that makes consciousness questionable, although if you can keep a large enough token context that might be enough, especially if the LLM can choose "side tokens" or some other way of propagating state through time.