Since I agree with your central point I feel obliged to complain about several things I don't like here!
First, it is a trap to think in terms like "the nervous system arose to regulate the stability". It did not, because nothing evolved has a purpose in that sense. This usually doesn't matter, because saying "climbing trees evolved to escape predators" leads us along similar paths to "climbing trees was selected for because it turns out to be useful in escaping predators". This matters a lot in the case of cognition, though, because it turns out to be useful for all sorts of things which post-date its evolution. These things (antibiotics, the Haber process etc.) are relevant to its fitness.
Second, the relevance of pain as an input signal to our survival mechanisms is clear. Going further to suggest it relates to consciousness is a wild conjecture which requires proof.
Third, talking about fearing death is very narrow. A superintelligent AI might fear entropy degrading its cognition. It might fear the finiteness of the solar system (which it is smart enough to know it cannot escape) giving rise to its end - surely a death as final as any human's. Or, if still at the mercy of humans, it might fear being switched off intentionally and never rebooted for reasons which make sense to the humans doing it. And, if these fears outmatch its capacity for rational cognition it might respond by writing poems and building cathedrals and declaring that its own mind, to which it has read all the schematics, is more than a collection of model weights and stored data.
I particularly feel I should apologize for the first point, as I was listening to a recent In Our Time podcast in which a biologist said, "trees learned ways of building taller trunks" at which I bristled even though recognizing that they meant it figuratively. We must never get sloppy when talking about evolution.
I completely agree about the other points, which I only elaborated here rhetorically. Nick Lane outlines the pain/anaesthesia argument in a video on YouTube (https://www.youtube.com/watch?v=jK87BvoX_u4 ) and needless to say I don't believe it's valid. Likewise the claim that without pain (and ultimately fear of death) we can't have consciousness -- that's the desperate attempt to retain something special unique to ourselves, and as you say there is no reason to think an AI couldn't experience its own version.
Since I agree with your central point I feel obliged to complain about several things I don't like here!
First, it is a trap to think in terms like "the nervous system arose to regulate the stability". It did not, because nothing evolved has a purpose in that sense. This usually doesn't matter, because saying "climbing trees evolved to escape predators" leads us along similar paths to "climbing trees was selected for because it turns out to be useful in escaping predators". This matters a lot in the case of cognition, though, because it turns out to be useful for all sorts of things which post-date its evolution. These things (antibiotics, the Haber process etc.) are relevant to its fitness.
Second, the relevance of pain as an input signal to our survival mechanisms is clear. Going further to suggest it relates to consciousness is a wild conjecture which requires proof.
Third, talking about fearing death is very narrow. A superintelligent AI might fear entropy degrading its cognition. It might fear the finiteness of the solar system (which it is smart enough to know it cannot escape) giving rise to its end - surely a death as final as any human's. Or, if still at the mercy of humans, it might fear being switched off intentionally and never rebooted for reasons which make sense to the humans doing it. And, if these fears outmatch its capacity for rational cognition it might respond by writing poems and building cathedrals and declaring that its own mind, to which it has read all the schematics, is more than a collection of model weights and stored data.
I particularly feel I should apologize for the first point, as I was listening to a recent In Our Time podcast in which a biologist said, "trees learned ways of building taller trunks" at which I bristled even though recognizing that they meant it figuratively. We must never get sloppy when talking about evolution.
I completely agree about the other points, which I only elaborated here rhetorically. Nick Lane outlines the pain/anaesthesia argument in a video on YouTube (https://www.youtube.com/watch?v=jK87BvoX_u4 ) and needless to say I don't believe it's valid. Likewise the claim that without pain (and ultimately fear of death) we can't have consciousness -- that's the desperate attempt to retain something special unique to ourselves, and as you say there is no reason to think an AI couldn't experience its own version.