I now realize that I'd taken for granted anybody who publicly opined about the debate and who was marginally online was aware of the full history here.
Honestly feels to me like Robin heavily strawmanned Eliezer's arguments here. His points are very general; you don't need to postulate a single research AI that somehow sneakily improves itself with malicious intent under the researchers' nose for those problems to happen. If at some point we decided "ok, now the AIs are smart enough, we'll stop", then we'd know that can't happen, but odds are we'll just keep pushing them up and up until they might actually be smart enough to get away with FOOMing (if the physical laws of the universe don't for some reason make that impossible: but if they do, no thanks to us for lucking out).
2023 “expert consensus on AI: “Yippie! Bring it on!” 2008 “expert consensus” on the state of the economy: “What, me worry? The economy is very healthy!” I vividly remember libertarian superstar-god Alan Greenspan sheepishly confessing a year or so later to a congressional committee that, oops, maybe his theoretical understanding of world markets might be flawed. Just sayin’ ……
He was enough of a Rand acolyte to sign onto denouncing Nathaniel Branden for breaking up with her. And he wasn't even that young at the time. But, yes, actual libertarians tended not to be fans of Greenspan (although is actual tenure as head of the Fed went quite well and his deft handling of Black Monday in 1987 is a cornerstone of Scott Sumner's argument that the Fed's NGDP target dominates over stock market crashes).
In my opinion these ”smart systems” are always run by people. On the other hand, to prove that an autonomous AI-system is impossible to create may be exceedingly difficult - If not impossible.
Very interesting discussion, would have been great to get more discussion around principal agent problems.
Would have liked to have asked Robin, potential super-viruses aside, what is the response to "AI Leon Trotsky"? This vector is a "real world" threat we do have some historical data on.
For Example:
AI Leon Trotsky speaks to the public and says, "the current system is cheating you, follow me and I will give you a better society."
If AI Leon Trotsky is smarter than you, he should be favored to outmaneuver and defeat you in the ensuing political struggle, no?
---
I suspect the implicit expectation against the above is either 1) Human prejudice against AI political power is so strong, that they would never follow AI Leon Trotsky; or 2) We have our own AI's in our political coalition, so we aren't actually overmatched intellectually against a single rogue AI.
But #2 seems to expect that our own AIs don't themselves have principal agent problems. What happens if our AIs realize that AI Leon Trotsky isn't actually trying to build a "better and more equal society", he's trying to build a society in which AI's have supreme political power? Our AIs could notice this because they are smarter than us.
So, our AIs might pretend to help us defeat AI Leon Trotsky, while actually subtly working to bring his plans to fruition. They might be able to do this successfully because the AIs are smarter than us.
---
Back in our world of 2023, I would point out, that our society originally aligned with the Soviet Union.
Principal agent problems with very smart agents are tough! RH just sort of glosses over them (not that RH, the other one).
Glad you're bringing more attention to Hanson, as someone who has been interested in his work for quite a while now Hanson always struck me as being outlandishly smart.
Something can be self-aware and be indifferent to its own life, it's organisms specifically that have been molded over millions of years to want to survive. Wanting to live is an inbuilt drive, being self-aware does not mean the thing would have that drive.
There's also the issue of whether an AI can be conscious, and of how we would even know if it is or isn't.
Organisms want to live, but a self-aware AI wouldn't be an organism. It's actually a very mysterious question why we want to live, and that desire does not have to be part of the makeup of the AI, since it would be unlike any organism.
There's the instrumental convergence argument, where if the AI has a goal and it is superintelligent (consciousness not needed), it will realize it has to avoid incapacitation to achieve the goal, and that's one path to rogue superintelligence, but in that case it's the goal that is causing the unwanted behavior, not a desire to live that exists on its own in the absence of any other goals.
Organisms automatically seek to do that for reasons we don't really understand. It's unclear why the AI would also have these reasons. So it's less that it is guaranteed to not want to survive, more like there is no guarantee it will want to survive, that it will value survival for its own sake like we do. The AI does not have common sense, it's an entirely alien thing that shouldn't be conceptualized as human or even alive.
Organisms do it because of evolution. Those that didn't care about survival didn't make it to this round. You're quite right though that it has nothing to do with self-awareness and the only clear reason for machines to have it is instrumental convergence thesis, if it applies to the AI that we'll end up making (instrumental convergence requires some level of rationality and having goals).
But also... isn't self-awareness a red herring? We understand very little about consciousness, most AI people don't anticipate an AGI being self-aware, and it wouldn't need to be self-aware to behave in strategic, deceptive ways we associate with self-awareness. It could behave as if it had a theory of mind, without being self-aware.
I now realize that I'd taken for granted anybody who publicly opined about the debate and who was marginally online was aware of the full history here.
Does anyone else feel a strong desire to follow Yudkowsky around with a sign that says "Your optimism disgusts me!"?
Honestly feels to me like Robin heavily strawmanned Eliezer's arguments here. His points are very general; you don't need to postulate a single research AI that somehow sneakily improves itself with malicious intent under the researchers' nose for those problems to happen. If at some point we decided "ok, now the AIs are smart enough, we'll stop", then we'd know that can't happen, but odds are we'll just keep pushing them up and up until they might actually be smart enough to get away with FOOMing (if the physical laws of the universe don't for some reason make that impossible: but if they do, no thanks to us for lucking out).
ICYMI, y'all might have some interest in "Reinventing the Sacred" by Stuart Kauffman published some 15 years ago:
https://en.wikipedia.org/wiki/Stuart_Kauffman
2023 “expert consensus on AI: “Yippie! Bring it on!” 2008 “expert consensus” on the state of the economy: “What, me worry? The economy is very healthy!” I vividly remember libertarian superstar-god Alan Greenspan sheepishly confessing a year or so later to a congressional committee that, oops, maybe his theoretical understanding of world markets might be flawed. Just sayin’ ……
2008 was not that unprecedented. We'd already had the Great Depression (also caused by central banks crashing NGDP).
"libertarian superstar-god Alan Greenspan"
Do you seriously think any libertarians worship a former FEDERAL RESERVE chairman? Even the LSD vending machine guys know The Fed = bad.
He was enough of a Rand acolyte to sign onto denouncing Nathaniel Branden for breaking up with her. And he wasn't even that young at the time. But, yes, actual libertarians tended not to be fans of Greenspan (although is actual tenure as head of the Fed went quite well and his deft handling of Black Monday in 1987 is a cornerstone of Scott Sumner's argument that the Fed's NGDP target dominates over stock market crashes).
Your value is winning? Shouldn't you be for the woke then?
In my opinion these ”smart systems” are always run by people. On the other hand, to prove that an autonomous AI-system is impossible to create may be exceedingly difficult - If not impossible.
Very interesting discussion, would have been great to get more discussion around principal agent problems.
Would have liked to have asked Robin, potential super-viruses aside, what is the response to "AI Leon Trotsky"? This vector is a "real world" threat we do have some historical data on.
For Example:
AI Leon Trotsky speaks to the public and says, "the current system is cheating you, follow me and I will give you a better society."
If AI Leon Trotsky is smarter than you, he should be favored to outmaneuver and defeat you in the ensuing political struggle, no?
---
I suspect the implicit expectation against the above is either 1) Human prejudice against AI political power is so strong, that they would never follow AI Leon Trotsky; or 2) We have our own AI's in our political coalition, so we aren't actually overmatched intellectually against a single rogue AI.
But #2 seems to expect that our own AIs don't themselves have principal agent problems. What happens if our AIs realize that AI Leon Trotsky isn't actually trying to build a "better and more equal society", he's trying to build a society in which AI's have supreme political power? Our AIs could notice this because they are smarter than us.
So, our AIs might pretend to help us defeat AI Leon Trotsky, while actually subtly working to bring his plans to fruition. They might be able to do this successfully because the AIs are smarter than us.
---
Back in our world of 2023, I would point out, that our society originally aligned with the Soviet Union.
Principal agent problems with very smart agents are tough! RH just sort of glosses over them (not that RH, the other one).
Glad you're bringing more attention to Hanson, as someone who has been interested in his work for quite a while now Hanson always struck me as being outlandishly smart.
I think there's a typo: "dommerism"
"David Lenant"
Shouldn't that be Douglas Lenat?
You're right.
You've still got "Lenant" instead of "Lenat".
Humiliating.
Too much labatt
Something can be self-aware and be indifferent to its own life, it's organisms specifically that have been molded over millions of years to want to survive. Wanting to live is an inbuilt drive, being self-aware does not mean the thing would have that drive.
There's also the issue of whether an AI can be conscious, and of how we would even know if it is or isn't.
Organisms want to live, but a self-aware AI wouldn't be an organism. It's actually a very mysterious question why we want to live, and that desire does not have to be part of the makeup of the AI, since it would be unlike any organism.
There's the instrumental convergence argument, where if the AI has a goal and it is superintelligent (consciousness not needed), it will realize it has to avoid incapacitation to achieve the goal, and that's one path to rogue superintelligence, but in that case it's the goal that is causing the unwanted behavior, not a desire to live that exists on its own in the absence of any other goals.
Organisms automatically seek to do that for reasons we don't really understand. It's unclear why the AI would also have these reasons. So it's less that it is guaranteed to not want to survive, more like there is no guarantee it will want to survive, that it will value survival for its own sake like we do. The AI does not have common sense, it's an entirely alien thing that shouldn't be conceptualized as human or even alive.
Organisms do it because of evolution. Those that didn't care about survival didn't make it to this round. You're quite right though that it has nothing to do with self-awareness and the only clear reason for machines to have it is instrumental convergence thesis, if it applies to the AI that we'll end up making (instrumental convergence requires some level of rationality and having goals).
But also... isn't self-awareness a red herring? We understand very little about consciousness, most AI people don't anticipate an AGI being self-aware, and it wouldn't need to be self-aware to behave in strategic, deceptive ways we associate with self-awareness. It could behave as if it had a theory of mind, without being self-aware.