Bravo - the next-token-prediction section here really helped me. Coincidentally, today I also learned about Robert Wright’s book; what I’ve read about that, combined with your post, has really shaken my skepticism about how far the current paradigm can take us.
PS, now I'm listening to Robert Wright's appearance on Doom Debates and agree with one point he's making: we have no idea whether consciousness is a prerequisite for AGI because we have no idea what consciousness is or how it works.
Or at least I would've said that, but check out Grant Castillou's comment on this post.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Thanks for this background and pointers! I don't think we should be trying to create a conscious machine. One question for you is whether you think that could happen accidentally? Also, do you think any of this dovetails with the AI alignment problem?
You're welcome. My hope is that immortal conscious machines could accomplish great things with science and technology, such as curing aging and death in humans, because they wouldn't lose their knowledge and experience through death, like humans do. I certainly can't imagine a machine with the equivalent of biological consciousness based on the extended TNGS being created accidentally. The details of the problem are so complex it's going to take over a hundred years to create a machine with only primary consciousness, let alone higher-order consciousness, imo.
I think we are too distracted by the idea of consciousness as a measure of intelligence, or a threshold of danger.
Give how the debate rages on about whether AI is conscious on the daily I am not sure we would notice if one day a LLM that only exists while generating tokens crosses over the threshold into consciousness. I personally think we will and should be more concerned with the danger LLM intelligence poses far before any of them become conscious .
I’d like to highlight Peter Watts’ novel Blindsight as a speculative example of the potential danger of superintelligent entities who lacking consciousness. In the novel, Peter Watts argues that consciousness would eventually be outcompeted evolutionarily by intelligent creatures lacking consciousness, like a hive of ants.
The events of the book take place when a non conscious spacefaring species catches wind of our communications in human language.
The attempt to decode meaning in this language ends up being incredibly resource intensive for this species. They eventually come to see language as a virus, and interpret human language as an active attack to reduce their fitness as a collective.
The book is an exploration of whether conscious beings would be sufficiently capable of combating a non-conscious super intelligent hive consciousness. What kinds of individuals or kinds of brains would be most suitable to defend against a non-conscious entity? What would an engineered or lost human species that was non-conscious act like?
Interestingly, when the “human” protagonists of this novel attempt to communicate with the hostile species, they deduce that the other entity is treating language like a Chinese room, and is only able to retain [what we would call today] a functional context window of two sentences when responding to human language.
On a different note, ignoring that we’ve pretty much nailed needle in a haystack retrieval in modern LLMs, I like to think that maybe sentience is just the process of expanding a window where the other entity appears to be having a coherent conversation with you from sentences to months to years of individual experience.
Watching my younger human cousin grow up, I’m struck by how the pace of his english acquisition felt very much like watching the increase in fluidity of llms over the past few years. There’s an ineffable quality to people and human intelligence I admire, but…
PS: I just added my own prediction to the blurb at the end about Scott Alexander's post on AI superforecasters. Basically I expect AI superforecasting to plateau at human level. But I don't think that's exactly dispositive for the "AI as Normal Technology" theory.
Bravo - the next-token-prediction section here really helped me. Coincidentally, today I also learned about Robert Wright’s book; what I’ve read about that, combined with your post, has really shaken my skepticism about how far the current paradigm can take us.
PS, now I'm listening to Robert Wright's appearance on Doom Debates and agree with one point he's making: we have no idea whether consciousness is a prerequisite for AGI because we have no idea what consciousness is or how it works.
Or at least I would've said that, but check out Grant Castillou's comment on this post.
Ah, thank you! I can't tell if I should read "The God Test"; I'd be interested to hear more about what you've read about it.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Thanks for this background and pointers! I don't think we should be trying to create a conscious machine. One question for you is whether you think that could happen accidentally? Also, do you think any of this dovetails with the AI alignment problem?
You're welcome. My hope is that immortal conscious machines could accomplish great things with science and technology, such as curing aging and death in humans, because they wouldn't lose their knowledge and experience through death, like humans do. I certainly can't imagine a machine with the equivalent of biological consciousness based on the extended TNGS being created accidentally. The details of the problem are so complex it's going to take over a hundred years to create a machine with only primary consciousness, let alone higher-order consciousness, imo.
I think we are too distracted by the idea of consciousness as a measure of intelligence, or a threshold of danger.
Give how the debate rages on about whether AI is conscious on the daily I am not sure we would notice if one day a LLM that only exists while generating tokens crosses over the threshold into consciousness. I personally think we will and should be more concerned with the danger LLM intelligence poses far before any of them become conscious .
I’d like to highlight Peter Watts’ novel Blindsight as a speculative example of the potential danger of superintelligent entities who lacking consciousness. In the novel, Peter Watts argues that consciousness would eventually be outcompeted evolutionarily by intelligent creatures lacking consciousness, like a hive of ants.
The events of the book take place when a non conscious spacefaring species catches wind of our communications in human language.
The attempt to decode meaning in this language ends up being incredibly resource intensive for this species. They eventually come to see language as a virus, and interpret human language as an active attack to reduce their fitness as a collective.
The book is an exploration of whether conscious beings would be sufficiently capable of combating a non-conscious super intelligent hive consciousness. What kinds of individuals or kinds of brains would be most suitable to defend against a non-conscious entity? What would an engineered or lost human species that was non-conscious act like?
Interestingly, when the “human” protagonists of this novel attempt to communicate with the hostile species, they deduce that the other entity is treating language like a Chinese room, and is only able to retain [what we would call today] a functional context window of two sentences when responding to human language.
On a different note, ignoring that we’ve pretty much nailed needle in a haystack retrieval in modern LLMs, I like to think that maybe sentience is just the process of expanding a window where the other entity appears to be having a coherent conversation with you from sentences to months to years of individual experience.
Watching my younger human cousin grow up, I’m struck by how the pace of his english acquisition felt very much like watching the increase in fluidity of llms over the past few years. There’s an ineffable quality to people and human intelligence I admire, but…
Dang, now I want to read Blindsight. And, yeah, lots of potentially terrifying philosophical implications for how things might play out with AI.
PS: I just added my own prediction to the blurb at the end about Scott Alexander's post on AI superforecasters. Basically I expect AI superforecasting to plateau at human level. But I don't think that's exactly dispositive for the "AI as Normal Technology" theory.