Robin Hanson Says You're Going to Live

On the nature of intelligence, knowledge, the abstraction hierarchy, consciousness, and the future of humanity

Like many people, I was taken aback by Eliezer Yudkowsky’s recent appearance on the Bankless podcast. I find Yudkowsky’s arrogant doomerism to be quite charming, and think it explains why he’s been so successful in spreading his ideas. Whenever I listen to him, I sense a feeling of pure exasperation, in the sense of yes, we’re all going to die, it’s going to happen very soon, and why are you bothering to make me explain this again when you’re literally about to be wiped off this earth and everyone else will too you idiot. His recent transition to telling us that there’s basically nothing that can be done at this point and we should die with dignity makes his ideas all the more appealing to me, especially since I’m such a huge fan of legalized euthanasia.

Recently, however, it came to my attention that Yudkowsky has been involved in a long-running debate with Robin Hanson about all of this that started a decade and a half ago. The individual articles were gathered and conveniently made available in Kindle format. Here are some of the main pieces authored by Hanson over the years on the AI alignment problem.

• “Prefer Law to Values” (October 10, 2009)

• “The Betterness Explosion” (June 21, 2011)

• “Foom Debate, Again” (February 8, 2013)

• “How Lumpy AI Services?” (February 14, 2019)

• “Agency Failure AI Apocalypse?” (April 10, 2019)

• “Foom Update” (May 6, 2022)

• “Why Not Wait?” (June 30, 2022)

Upon discovering this work, my heuristic of “trust smart-seeming arrogant doomers” collided with my heuristic of “trust Robin Hanson.”

So I decided I’d try to have Yudkowsky and Hanson both on to talk about it. Unfortunately, I wasn’t able to get Yudkowsky, but Robin agreed to come back on the CSPI podcast, this time to reassure us that it is highly unlikely that a fast AI takeoff ends the world. He gives the probability of this happening as less than 1%. You can listen to the podcast or find a link to the video here. A lightly edited transcript of our conversation is below.

I find talking to Robin to be a unique intellectual experience. I’ve rarely come across someone who can speak off-the-cuff about ideas at such a high level of abstraction while remaining so firmly grounded in logic and empirical reality. More than once, I’ve thought that he was going off in a direction where he wasn’t really answering my question, but I would follow along and there was always a payoff in the sense that he was actually developing an interesting and relevant point that was directly related to what I was asking. None of this is common, and it gives me the impression of being in the presence of an extremely rare form of intelligence, which is ironic given the topic of our conversation here.

As you can see, just because Robin has doubts about “foom” doesn’t mean that he brings only good news to humanity. In fact, he predicts that artificial life will eventually conquer the earth, and we will still have an “alignment problem,” in the sense that the future rulers of the planet are highly unlikely to be very similar to us. Regardless, I still consider Hanson much more optimistic than Yudkowsky, since he’s not guaranteeing anything happening within our own lifetime, and the machines that are coming seem like they will have something in common with us at least for a while until we lose control. They’ll be our deformed and misshapen descendants, not a completely alien force we suddenly conjured into being and made our heads explode before we had the slightest inclination of what was going on.

Near the end, Robin talks about his theory of the sacred, and why he thinks my computer has subjective experiences.

As someone who has been thinking about this topic for a much shorter period of time than Robin has, I wasn’t afraid to ask what some might consider dumb, basic questions. For example, what do the AI alignment people do all day? And why can’t their program just say “don’t make us regret creating you?” This I think benefits others who are new to AI alignment issues, along with people who have gotten their ideas overwhelmingly from the doomer side.

Overall, I came away from this conversation invigorated by the exchange, and much more optimistic about the possibility of my children reaching adulthood. I’ll continue to try to find doomers to talk to so they can maybe change my mind.

---

Priors Based on Past Experience

Richard: Hi, everyone. Welcome to the podcast. I’m here with Robin Hanson today, and he’s here to talk about AI, and specifically the alignment problem. I started looking into this issue seriously over the last few months, and I was really actually surprised. I didn’t know there were smart critics of what I call AI doomerism. I recently discovered Steven Pinker had a long exchange with Scott Aaronson about this, and then I am ashamed to say I didn’t discover the great Hanson-Eliezer debates from about a decade, a decade and a half ago, until very recently.

And so, I’ve been reading your stuff, Robin, and in one of the essays, you say some people looked at the AI alignment problem, thought it wasn’t such a big deal, and they moved on and didn’t keep writing about it. And the people who did think it was a big deal sort of became obsessed with it, and they’re the people we hear from all the time. So what do you think? You have a lot of objections, I think, to doomerism, but what’s the heart of it? What’s the heart of the problem with the idea that there’s going to be something that’s so smart, it makes us feel like ants, and that basically it can do whatever it wants and we can’t help to control it and foresee it, and we’re all going to be at its mercy and potentially die. What’s the argument against this?

Robin: I think you’re right that there’s this very common phenomenon whereby most people have some sort of default views about the world and history and the future, and then some smaller groups come to a contrary view, that is a view that on the face of it would seem unlikely from some broad considerations. And then they develop a lot of detailed discussion of that, and then they try to engage that with the larger world, and then what they get usually is a big imbalance of attention, in the sense that they...

Think of 9/11 truthers or something, right? They’re going to talk about this building and this piece of evidence and this time and this testimony or something, and people on the outside are just going to be talking mainly about the very idea of this thing, and is this at all plausible? And the insiders are going to be upset that they don’t engage all their specific things, and they introduce terminology and concepts and things like that, and they have meetings or they invite each other to talk a lot. And the outsiders are just at a different level of, does this really make much sense at all?

And then, when the insiders are trying to get more attention and the outsiders, some of them will engage. There’ll be a difference between some very high profile people who just give very dismissive comments, versus lower status people who might look at their stuff in more detail, and they’re just going to be much more interested in engaging that first group than the second because the fact that somebody high profile even discussed them is something worthy of note. And then, the fact that this person was very dismissive and doesn’t really know much of their details, in their mind supports their view that they’re right and the other people are just neglecting them.

So here, the key thing to notice is just, on the face of it, they’re postulating something that seems a priori pretty unlikely relative to a background of the past and other sorts of things. That would be the crux of the main response, is to say, look, what are you proposing here, and how would that look if we had seen it in the past? How unusual would that be?

As you know, the world is really big and the world has been growing, but relatively steadily and slowly for a long time. And for a long time, basically any one innovation anywhere found in the world made usually a modest difference to a particular local region or industry, and the net effect on the world has been pretty small, and it’s been the net effect of lots of innovations like that made up the world changing, and that any one organization in the world is typically a small part of the entire world, and anything that happens in that one organization doesn’t affect the whole world that much. It affects the people involved in that organization, and the world progress is more the net effect of some organizations and ventures growing and shrinking, etc., changing, selecting over time, right? So, that’s history.

Okay, and so, in that scenario, you would be really surprised to see one particular venture in one place, that on a global scale was hardly even noticed, doesn’t even show up in the catalogs or something, and then on a short time scale compared to how fast the world grows, that one thing suddenly grows so fast that in a very short time, it takes over the world faster than anybody can even notice it to react to oppose it.

And not only that, during that period of very rapid growth, its priorities radically change, that it starts out with a certain sort of behavior and priorities and it seems that its priorities are consistent with what it usually does, and then during that very fast evolution, its priorities change radically, so that it basically wants something orthogonal, arbitrarily or randomly different. And then, because it takes over the world, it now implements these arbitrary orthogonal priorities. So that’s, compared to history, a pretty unlikely scenario. That would be the most fundamental objection, is you got to overcome my prior here, of “that just doesn’t happen.”

Richard: Okay. So, there’s a prior that the world doesn’t end, because the world usually doesn’t end. Humans are usually not, have never been exterminated, and they’ve never been exterminated sort of by the process, something even close to resembling the process Yudkowsky sort of posits… But still, these ideas, they do overcome a lot of people’s priors, and a lot more than other things that people say that are going to end the world. Most smarter people are not joining apocalyptic cults. I mean, the really smart people in Silicon Valley and other areas of life are not buying into other kinds of doomsday scenarios.

So, people do find something about the logic of this compelling, right? They think that steps are, each sort of step in the logical chain is solid enough on its own that we put them, we could just posit three or four things and we put them together and then we can overcome that prior that something’s not going to come out of nowhere and end the world, right?

These assumptions are, intelligence is a real thing that goes on some kind of scale from bacteria to normal human to Einstein to something else. And then, it’s hard for a lower intelligence to foresee what a higher intelligence is going to do. Even if you’re the programmer, even if you program what the goals are, there’s the idea that if you are highly intelligent, you could probably improve yourself and gain more, that’s a recursive process, you could gain more intelligence. And that’s basically, I mean, that’s basically it, right? So, where in the logical chain do you think, where’s the weakest link?

Robin: I mean, any ordinary corporation is a superintelligence of sorts. That is, it’s much more capable than any individual human. It, of course, can improve itself. Corporations do improve themselves, but they don’t do it very fast. So, we’re positing a very unusual rate of improvement of this one system, not only compared to the past, but compared to the other similar systems in the world at that time. So, it’s not enough merely to posit that. You have to consider why it would be so unusually fast, so terribly unusually fast. I mean, of course, we can talk about cities sometimes grow, and some cities grow faster than others. Firms sometimes grow. But individual cities don’t take over the world because they’re capable of growing, nor do individual firms take over the world, right? So, we’re not just talking about some degree of growth and maybe some variation of growth. We’re talking sort of a crazy extreme scenario of that.

Richard: Yeah. I like the analogy of firms more than cities here because firms do have a goal. We could talk about a firm’s goal. I mean, I think it’d be coherent. Talk about a city’s goal, I think the mayor could have a program, but it’s much less clear that you could talk about that. So, I guess we could take the firm thing.

So, Walmart is a superintelligence, right? It has stores all over the world. It has millions of products that it stocks the shelves with. It has a payroll, it has distribution, it has logistics. It’s involved in local communities. And I think that’s very interesting. Walmart can improve itself, and it has data and it has money, but it’s incremental, and it’s facing off against other forces of society, competitors, right? Other stores, online shopping, things that grab people’s attention.

And so, yeah, that makes sense. I mean, is it because it’s all just, the intelligence in the computer, is it because it’s all hardware? Does that make it different? It’s like, you have a code, and the code gives you a human level, a superhuman level of intelligence. Is it the fact that it’s just code? Does that matter? I mean, is that easier than doing something else in the real world, like Walmart might do?

Robin: In the past and today, the world co-evolved, in the sense that it’s composed of many parts, all of them depending in some way on each other and all of them trying to grow. And whenever we try to improve any one part of it, we’re opportunistically looking for what other parts can most help with that. So yes, some things happen in silicon, and the things in silicon we try to improve, and other things happen out here in real life and we try to improve those. We opportunistically try to use code to improve code when we can. Use people to improve code when we can. Use code to improve warehouses and stockyards when we can. I mean, we’re just trying to use everything to improve everything. But the status quo is that when we do our best to improve things, the rate of growth and change is what we see. That is, we’re presumably improving things as fast as we can. So the question is, where is this sudden, vast scale of rapid improvement coming from? I mean, it has to be we suddenly find some much easier way of doing something really important.

Richard: Yeah. But could you say that, well, it’s our rate of improvement, if you want to measure by economic growth or scientific knowledge or whatever, compared to the rate of improvement among chimpanzee society or ant society or something. It’s exponentially higher or infinitely higher because there is no, besides evolution, putting that outside, sort of a cultural progression.

So, we’re improving a lot faster than these less intelligent beings. Maybe a being that is more intelligent than us will be magnitudes of, basically improving much faster to the same extent that we improve faster than, say, chimpanzees do. And the intelligence, I think, is what gets people. It’s like, this is really, really smart. We’re smart. We have an analogy, humans to ants. Okay, there’ll be something compared to humans that is much smarter than us. And so, why can’t that be a good argument?

Robin: Well, in our world today, we have parts of the world that vary by many parameters, right? There’s rich nations and poor nations, there’s capital-intensive industries and labor-intensive industries. There’s industries where the employees tend to be very well-educated, industries where the employees are less well-educated. And in all of these places, in each one, we’re trying to improve it the best we can.

And when we find some areas of the world where we can improve it faster, we do. That is, investment resources focus more on the areas in the world where you can more rapidly improve things at a lower cost. But the world we see is the net result of that prioritization, where we do our best to find the most promising places to make improvements at the lowest cost and do them.

We already see a variation in intelligence in the world. That is, certainly there are people and places where more smarts is concentrated than in other places, but that’s the result of our efforts to prioritize and invest as best we can. But our simple economic prediction, which seems to be roughly right, which is that on the margin, a dollar spent in each possible area of investment will get about the same risk-adjusted returns, because we have focused our efforts on the most promising things and produced diminishing returns there.

So, it’s just not a good investment. You say, gee, I’m going to allocate my money according to what companies are smarter. I’m going to find some measure of which companies are smarter and put your money in that and have a hedge fund. That’s a strategy. It’s not going to make money compared to the other investment funds.

Richard: Yeah. Bringing up the idea that we already have superhuman intelligence, I think, is very interesting. When you brought up firms, that made me think. You can talk about the market. Although again, the goal thing, it makes it a little bit harder. But the firm is, I think, just such a good analogy because it’s a big thing. It’s way smarter. I mean, Walmart is much smarter than any human being could be. And we could talk about it having goals, and we could talk about it having influence in the world, and I think that analogy is really good. I wonder, markets are also superintelligent, but markets don’t have goals. They’re sort of… a market is an abstract way of talking about the aggregation of the goals of various actors. Is there something else like firms, that we could... Countries, maybe?

Robin: Once you allow a distinction between big things in our world that are useful that have goals and that don’t, and that you say we have big chunks of the world that are useful and that don’t have goals, then you might say, well, if you’re worried about AI with goals, just don’t use the versions with goals. Just use the versions without goals, that is.

Richard: [laughs] Yes, that’s a good point.

Robin: Pretty much all the actual useful computer systems out there are not general agents with sort of general goals about their whole future and all the actions they can take in their world. They tend to be specialized tools for specialized purposes, and they’re focused on doing those specialized things. So, that seems to be a reasonable path forward to the future, to the extent you’re worried about AI as having goals. But the example of firms says we can also have agents with goals, and superintelligent agents with goals exist and seem to be reasonably well aligned with this in our world.

The Genie Problem

Richard: Yeah, that’s right. And then, as far as the goal thing, I’ve been a little bit confused on this point, from the perspective of the people who are worried about AI. I was watching Eliezer Yudkowsky on a podcast recently. It got a lot of attention online. I forget what the podcast is called. And actually, I wanted him to debate you on this, but he didn’t get back to me, unfortunately. So, hopefully one day I’ll be able to talk to him.

He said something like, we can’t solve the alignment problem because we have no idea how to give a computer a goal of “make this strawberry, down to the very molecule, an exact copy of, make another strawberry that’s an exact copy, molecule by molecule, of this strawberry” and not destroy the world.

And I was confused by this, and I don’t know if there’s something I don’t know about computer programming, but it sounds to me like if you could give the instructions, “make a strawberry molecule by molecule,” that’s all it takes and the computer could figure out what to do with that, why can’t you say, “don’t destroy the world,” or “don’t do this in a way that will make the human creator of the program regret it”? Why is it, why aren’t things like this considered realistic? I should probably ask him, but you’re familiar with their arguments. So, why can’t that be a solution?

Robin: In our world, whenever you give an employee or an associate some assignment, you give them a bunch of implicit context to that. You say, here’s the resources available to you to achieve that assignment, and this is the scope in which you’re allowed to act, and you know that there’s the rest of the world out there that you should try not to mess with, and you know there’s boundaries where you should stay within those boundaries and try to achieve your assignment within those boundaries. And you will, typically.

Now, the claim is that if you give a very powerful agent who has full scope of everything, it can do anything it wants in the universe, and it’s willing to do anything it wants in the universe, and you’ve given it some sort of abstract goal like make a copy of a strawberry, then unless you constrain it very carefully, the claim is, well, it’ll achieve that purpose in its most cost-effective way, and maybe as a side effect, destroy vast swaths of the rest of the universe in order to achieve this one particular stated goal.

So, the question is then, can you make clear to it all the other things it’s not supposed to do in order to do this one thing? Now again, as limited agents in our world, that’s sort of implicit. You give me an assignment and I know that I can’t change the universe. I know that I can only do a limited range of things, and I’ll try to achieve your purposes with my limited resources, but I will know I have all these limits. But he’s postulating this other sort of creature, vast and powerful and really without limits on its capabilities.

And so, by assumption, it’s basically there’s only one of these things. If there were a million of them, certainly any one of them would face the limits of all the rest of the million, and it would have to figure out a way to make the strawberry while not pissing off all the other million AIs who are similarly capable. But if you just imagine one of them, why, it might go do arbitrary things in the pursuit of this goal you gave it.

Richard: Yeah, but why do they think that’s plausible? Okay, it’s intelligent. They’re positing it can do a lot. It can kill humanity, it can destroy the world. It could do X, Y, Z. And you could program it, and there’s some unforeseen consequence of what you program the goal. But why can’t part of the utility function, the goal, be like, don’t make us regret having created you as a species? It has a theory of mind, right? It can understand humanity well enough to manipulate humanity. It can understand humanity well enough to, in these scenarios, control nations and get them to do all kinds of things. Why... What’s stopping... What’s the answer as to why we can’t just tell it, “don’t make us regret this,” basically?

Robin: Well, I mean, obviously one, you have to ask what happens if we interpret these things very literally. That’s often...

Richard: “Don’t make us regret this.” Okay, it kills us and we will never regret this. [laughs]

Robin: Yes. That will achieve that, or it just rearranges your brain so you no longer regret anything ever?

Richard: I see.

Robin: You do have to be careful about what you wish for in these cases. But I mean, the scenario is you’ve got this genie, sort of Aladdin’s lamp sort of scenario. You’ve got this genie who will do what you say, perhaps, but not in the way you intended, and it’s very powerful and you have to be careful about what you say. But in a world where there’s this genie who can do anything and there aren’t other genies who constrain it, and it’s trying, perhaps, to misunderstand you, then you have to be careful.

Richard: But usually the genie thing is, it’s like, you give a sentence, like “make me a ham sandwich,” and it makes you into a ham sandwich or something. We can write... Okay, I think if we got together, we can write 10 paragraphs or even a book sort of explaining what we mean by... And judges do this, too. Judges, they’ll have a law, and it won’t be exact, but they’ll say, okay, we sort of understand the context. We understand what the purpose of the law is. Why can’t we say, be our ideal, the median human’s ideal of a wise jurist when there is doubt, and then write a 300-page book about all the things it should and shouldn’t do. Could that be a solution?

Robin: I think what we’re talking about is related to the distinction between planning and, say, reactive adaptation. So if you have a housekeeper, say, or an assistant, you can work with them to get them to understand roughly what you want, and then you can give them very brief instructions and they can reliably give you roughly what you want because you can have an adaptation. If you see, the first time you give them an assignment and they do something off, you can say, no, that’s not what I meant, and then you can instruct them, and then with the process of feedback that way, you can get to a place where you can roughly guess the kind of assignments they can do, and they can figure out roughly what you meant by them and things work well.

But it would be different if you had to write down these instructions for this new housekeeper and that was it. You would never get a chance to correct them, and when you saw the misunderstandings, it was too late to do anything. So, that’s what they’re worried about with this rapid growth scenario, that it’ll happen so fast that as you start to see it go wrong, you won’t have a chance to stop it or correct it. You have to specify everything ahead of time, and you have to sort of imagine all the scenarios the future universe could be in ahead of time and have some specification that covers all those possibilities before you have much of an idea of what it’s going to want, or what it might be able to do.

What Working on AI Risk Means

Richard: Yeah. And what are sort of the dominant... Because I was listening to Yudkowsky’s podcast, and he seemed completely hopeless. Did you listen to this podcast? He was very, it was very dark.

Robin: No, I didn’t, but I’ve heard him many times before.

Richard: [laughs] Okay, so you’ve gotten your fill of that, your debates were very long. But he’s updated over time. He’s become sort of more pessimistic now. I saw he had an essay, just have death with dignity, and ChatGPT and all these other things are sort of scaring him, so he’s saying a lot of the same arguments but now more convinced he’s right and more pessimistic about our ability to do anything about it.

And so, from what I was listening to recently, he basically thinks there’s been no progress on this. What are these people, what are the AI alignment people exactly doing? Can you sort of, do you have any idea? Are they thinking of the instructions you give to the superintelligence that will not give you this genie problem? Or are they like... Well, what exactly is their solution? Can you explain to me more like what working on AI alignment actually means?

Robin: The analogy to the problem would be, imagine that in the year 1500, you had some foresight to imagine that in the 20th century there would be vast corporations and tanks and planes and nuclear weapons, and you were trying to give advice about how people in that distant future should manage their problems with geopolitical strategy and war and weapons and the modern economy and cars, right?

The problem is, they would hardly know anything about these things. And so, they would struggle to find abstractions, even to talk in terms of what could apply to that future. And then, you might have some specific things in your world that you think is sort of like that future world, like a windmill, and you say, well, a windmill is kind of like a machine we expect to see in the future.

And so, you would be doing two basic things. First, you would just be struggling with at a very abstract level to come up with abstract formulations and descriptions that would be roughly applying to this future era, and then reason about those abstractions. Or you would take the most concrete things around you that you think are analogous to that and focus on, well, how do we control a windmill? What happens if somebody fights with a windmill? If there’s an organization that owns a windmill, what do we do about that?

And you’d be practicing through those sorts of concrete variations in order to prepare for the coming future world. So that’s in essence what the AI risk people have been doing, because we know so little about what future AI would be like, in terms of its organization or priorities or structures or constraints. They either just talk at the very abstract level of an agent, and a general agent with general algorithms, and just talk about how could you specify for a general agent with general algorithms the kind of features or constraints you wanted to give it. That’s one approach.

And the other approach is to take the most recent versions of the future systems they’re worried about, say large language models or reinforcement learning models, and say, “Well, if the future AIs of concern were of this sort of model, what would we do to control them?”

And that’s what they’ve been doing. That’s very abstract stuff, it’s made relatively limited progress as you might expect. [laughter] It’s just really hard to do much at that level.

And the more specific ones, they can catalog the kinds of problems that occur and fix them, but they aren’t very big problems, so they don’t feel like they’re actually solving the fundamental problems, because the big problem they have about a rapidly self-improving thing, that’s not showing up in these concrete systems in front of them, and the things that people complain about in the concrete systems in front of them, that they’re too racist or whatever, or they’re offending people, and then there’s a lot of people trying to figure out how to regulate AI, so they don’t do that. And you know, you can either focus on those, or try to find analogous problems of just saying “Hey, any time one of these systems doesn’t do something we expect, that’s like a failure of control, and let’s just try to get on all of those problems.”

What if They’re Right?

Richard: Yeah, and their argument, you in the debate, you guys call it foom, the takeoff scenario. And I guess their argument is, “We have no choice. It’s going to be a millisecond between the time it develops a 200 IQ, and the time it develops a 5,000 IQ, and then develops a 1 million IQ, and then our heads will explode.” They’re saying this is by necessity. All we can do is sort of make these analogies, right?

Robin: Right, because they don’t know much details about the systems that will be a problem.

Richard: And so if you were convinced that they were, let’s say, right about most of these things, about the possibility of foom, would your view be there’s no point in worrying about it, because it’s hopeless just to waste time, and we just have to wait for it to happen, because there’s no way to align this thing without any experience, or knowing what it’s going to look like?

Robin: I mean it depends on when in the process, I guess. So if you get closer to seeing more what the system might be like... So there’s two main actions you can do here. One is you could increase resources toward their alignment efforts. And the other is you could try to slow down progress in AI and related fields.

Those have very different consequences. But the first thing is relatively cheap. You might say, “Hey, compared to the size of the world, it doesn’t cost that much to throw a lot more resources into these sort of attempts. So why not?”

Trying to slow down AI progress in the world would have pretty big consequences. And I think we’re in a world where we have, in fact, over the last half century or so, sort of slowed down progress in a lot of areas that people were scared of. And you can imagine AI being another one of those areas.

You can also imagine it not, that seems to be more of an open question, but the question is just how bad do you think it is? An analogy would be with nuclear power, the more you thought that there was just going to be a really huge nuclear accident if we allowed people to make nuclear power plants, and it was going to blow up half the world, then you might say just, “Nope, no power plants, nothing at all. Just don’t allow it. We’re just going to stay with coal or whatever,” and just not allow it, or put vast resources into trying to study how to do safe nuclear power plants.

Richard: And the impact of that is not encouraging. We basically just have no more nuclear power plants. But if you thought foom was a real thing, you might take that hit. Yeah, and so...

Robin: Again, the key thing is there’s this key unusual part of the scenario, that is if you lay out the scenario, you say which of these parts looks the most different from prior experience? It’s this postulate of this sudden acceleration, and very, very large acceleration.

So we have a history of innovation, that is we’ve seen a distribution of the size of innovations in the past. Most innovation is lots of small things. There’s a distribution, a few of them are relatively big, but none of them are that huge.

I would say the largest innovations ever were in essence the innovations underlying the arrival of humans, farming, and industry, because they allowed the world economy to accelerate in its growth, but they didn’t allow one tiny part of the world to accelerate in its growth. They allowed the whole world to accelerate in its growth. So we’re postulating something of that magnitude or even larger, but concentrated in one very tiny system.

That’s the kind of scenario we’re postulating, when this one tiny system finds this one thing that allows it to grow vastly faster than everything else. And I’ve got to say, don’t we need a prior probability on this compared to our data set of experience? And if it’s low enough, shouldn’t we think it’s pretty unlikely?

Intelligence and “Betterness”

Richard: Yeah. So there’s a related essay you wrote, you’re saying this is unlikely and we could just say based on past experience, but the other one, I think one of your better essays that explains why this is unlikely from just a reasoning perspective rather than looking at history.

“The Betterness Explosion”… I love this essay because it was very short and it’s actually very funny. I just want to read it, because I actually laughed while reading it, and I’m a little bit sick, so this might be hard for me, but it’s worth it. This is from 2011.

We all want the things around us to be better. Yet today billions struggle year after year to make just a few things a bit better. But what if our meager success was because we just didn’t have the right grand unified theory of betterness? What if someone someday discovered the basics of such a theory? Well then this person might use his basic betterness theory to make himself better in health, wealth, sexiness, organization, work ethic, etc. More important, that might help him make his betterness theory even better.

After several iterations this better person might have a much better betterness theory. Then he might quickly make everything around him much better. Not just better looking hair, better jokes, or better sleep. He might start a better business, and get better at getting investors to invest, customers to buy, and employees to work. Or he might focus on making better investments. Or he might run for office and get better at getting elected, and then make his city or nation run better. Or he might create a better weapon, revolution, or army, to conquer any who oppose him.

Via such a “betterness explosion,” one way or another this better person might, if so inclined, soon own, rule, or conquer the world. Which seems to make it very important that the first person who discovers the first good theory of betterness be a very nice generous person who will treat the rest of us well. Right?

Okay, so this is really funny. And obviously “betterness,” you’re just using “betterness” sort of as a joke to make this look ridiculous, because this is basically what they’re doing with intelligence. They have this thing called intelligence, and it’s going to make them better in every single way, and they’re going to somehow take over the world. And so is the argument here that intelligence is multifaceted?

And when I talk to people about this, the argument that they say is, “Look, that may be true to an extent, but there is a sense in which we say a human is more intelligent than an ant, right?” Any question you give it, a human will be better able at reasoning through it.

Why can’t there be something with a superintelligence that is just at a different level? Or do you think that, or alternatively, I don’t know enough about how programming works, but why can’t you build various modules and just put them together, and then that would be a superhuman kind of intelligence? Is there anything to say for the argument that this is just not like betterness, that there is something sort of more solid here that we could hold onto?

Robin: So the issue is the kind of meaning behind various abstractions we use. So abstractions are powerful. We use abstractions to organize the world, and abstractions embody similarities between the things out there. And we care about our abstractions, and which pool we use.

But for some abstractions, they well summarize our ambitions and our hopes, but they don’t necessarily correspond to a thing out there, where there’s a knob on them you can turn and change things. So it’s important to distinguish which of our abstractions correspond to things that we can have more direct influence over, and which abstractions are just abstractions about our view of the world and our desires about the world. So that’s the key distinction here. We could talk about a good world and a happy world and a nice world, but there isn’t a knob in the world to turn out and make the world nicer in some sense.

A nice world is the nice world to you [laugh]. Nice things happen to you, but that’s very local to you. There isn’t sort of a parameter, a knob out there in the world you can turn and just make the world nicer for everybody, because “nice” isn’t exactly a concept that’s describing the world, it’s describing your reaction to it.

So “better” can be more seen as that way, we might think, “Well yeah, ‘better’ is describing us, what we see as better.” And it’s an important abstraction to have, in the sense that we need to evaluate stuff that happens around us, and we need to consider which things we prefer, but we don’t think of the world out there as better or not, that is, we’re not looking for the place in the world where there’s a “better” knob, and turn up the better knob, and then everything gets better in the world. That’s not how we think about the world.

Now intelligence, the question is what kind of an abstraction is that? So at one level, intelligence is just literally the ability to do many kinds of mental tasks. So then we might say Walmart is intelligent, because it can do many kinds of tasks, because it has a hundred thousand employees, and many of them are very smart and capable. And so Walmart is intelligent, because it can do many things, and the United States is even smarter, because look at all the things the United States can do.

And so then we might say yes, there is a parameter out there that this is describing, but it’s just sort of a general capacity descriptor, like the general wealth and capability of a firm or a country, is just this parameter. Then it lets it do many things and then yes, a more higher capacity entity out there can just do more many things. And sure then it could do more mental tasks better, and we’ll call it more intelligent.

Now when we think about these things in the world, they are out there. I think, look, how did they change? How could you improve one? And that’s where the topic of innovation comes from. We say innovation is how things improve or change, is one of the ways.

Of course, things can just improve or change by accumulating more capital and resources. And so we have a whole story of economic growth, whereby we understand which things can change how. And in that story we tend to have a relatively limited number of high level abstract parameters that describe things. For a firm or a nation, there’s just basically wealth, but there’s maybe physical mass, and energy, and various abstractions.

And we say, how could you improve such a thing? How could we make Walmart richer? How could we make the US better? We know about these aggregates, we try to increase the population of the US. We could try to make economic growth better. We could try to make overall efficiency better, then we can talk at that level about making them better in those ways, because those are the kinds of abstractions we have, that make sense of describing those systems. And so if we look at a person in their life, we say, well, they start out young and ignorant, but they have potential. And then we have some ways we describe how they can improve with time. They might get more experience, they might get more knowledgeable, they might get more refined, they might have more connections. Those are the kind of abstractions that make sense to describe an individual, and describe how they might improve over time.

And if you wanted to talk about who to select for a role, you’d be using those kinds of abstractions. You’re going to talk about how to improve somebody, you’d be asking how could you improve any one of those parameters?

And now we have this parameter intelligence, and the question is, what’s that? How does that fit in with all these other parameters? We don’t usually use intelligence, say, as a measure of a country or a measure of a firm. We use wealth or other parameters. If it’s equivalent, then fine. If it’s something separate then we want to go, “Well, what is that exactly?”

For an individual, we have this measure of intelligence for an individual in the sense that there’s a correlation across mental tasks and which ones they can do better. And then the question is, what’s the cause of that correlation? One theory is that some people’s brains just trigger faster, and if I got a brain that triggers faster, it can just think faster and then overall it can do more.

There are other theories, but there are ways to cash out, what is it that makes one person smarter than another? Maybe they just have a bigger brain. That’s one of the stories, a brain that triggers faster. Maybe a brain with certain modules that are more emphasized than others. Then that’s a story of the particular features of that brain, that makes it be able to do many more tasks.

If you just say, why don’t you tell a person to make themselves smarter? Why don’t you tell Walmart to make itself richer? Then you have to ask what’s the causal process by which that parameter can influence itself? And usually that’s pretty hard.

It’s hard for a firm to make itself richer just because it wants. It can if it tries. But we know there’s all these limitations. You say, “Hey, person, make yourself smarter.” Then we know, well, they could learn more, they could practice more. Maybe they could try to be more rational. But there’s a limited range of things.

Now we’re postulating, there’s this AI and it just says, ‘I want to be smarter,’ and then it does it, and it does it really fast. And you go, “How did that work? The rest of us can’t seem to do that. The rest of us find it pretty hard to improve these major parameters about us that we care a lot about.”

Richard: “Well it thinks a million times faster than you,” that’s what they will say it could do. It could scan the internet, that it could do mathematical calculations, and it can digest all the literature in the universe, and all that is not that too far off. So why isn’t it just this intelligence makes it more intelligent?

Robin: Because that’s not our standard model of economic growth. So Walmart has 100,000 employees, say. That doesn’t mean it can grow 100,000 times faster. We have to say what’s our best model of growth, how growth works, and what are the key parameters of growth? And the rate at which computers run isn’t really a central parameter in that analysis. So that’s sort of imagining there’s an algorithm you could run, and when you run that algorithm, then at the end of it you’re smarter. So if you can just run it faster, you’ll be smarter faster. But is there such an algorithm?

The Knowledge Hierarchy and Innovation

Richard: Yeah. And what does smarter mean? Do we even know? Is this concept even...

Robin: So here’s a different way to think about it. I got this from Douglas Lenat long ago with his famous Eurisko system and AM system and AI. His idea was that there’s an abstraction hierarchy of concepts that we use in mathematics and elsewhere. There are some very high level abstract concepts, and then there are a lot more specific concepts. And that when we learn things, what we learn sits somewhere in that abstraction hierarchy.

We either learn something about something specific, or we learn something about something abstract. When we learn something that where our knowledge more naturally sits at a high level in the abstraction hierarchy, then that’s going to have a wider scope of application.

When you learn about energy in general, for example, you learn a lot more than when you learn about coal in particular, or about a particular kind of coal in a particular plant.

So if you run a coal plant, in order to run it, you’ll use knowledge of various kinds of levels of abstraction. You’ll know about where the building is sited, and where the pipes come in, that’s very specific to that plant.

You’ll know about the particular people who work there, and their schedules and their inclinations. And then you might know general things about thermodynamics or the physics, energy conservation. The point is that knowledge in general sits somewhere in the abstraction hierarchy, and the observation Lenant had, which I think is true, is that the vast majority of useful knowledge is pretty far down in the abstraction hierarchy.

It’s mostly specific. Most of the things we learn are relatively specific, and relatively few are general, but of course the general things count for more. So if we do an integral, the median innovation is pretty far down, but the median weighted innovation by its impact will be higher up.

But even so, it’s not that high up. So if you learn something in general about intelligence, that would be a very high level thing. That would be something that was true about a very wide range of applications. So very basic decision theory, say, or very basic algorithm facts. Those would be very high level things you know about, that apply to a very wide range of things. And that’s more learning very general things.

And when you learn very general things, that does improve your ability to do a wide range of things. So now the question is, what’s the distribution of knowledge in this hierarchy? And can you, by putting more effort into the abstract things, make it happen faster? Or do you just get random draws of insight from all across the hierarchy, and then you might say a system that’s learning is just trying to get more knowledge, but it’s going to go at a rate that’s determined by some growth equation? And then the question is, what’s the fundamental growth equation of trying to collect more knowledge?

And so this is something I think economists know in terms of economic growth. That is, how do we learn to innovate, what sort of processes tend to produce innovation, and what’s the most cost-effective way to do that, and where does it tend to have the most?

But merely because you have a computer that can run fast, that doesn’t mean it’s going to do super innovation, because innovation is not just running some algorithms, it’s also interacting with the world, and trying things out, and getting ideas from elsewhere. So in our world today, we try as best as we can to innovate, but if we had twice as many researchers, we wouldn’t necessarily have twice as much economic growth. That’s a key thing to notice. Therefore, making researchers run twice as fast would not double economic growth.

Richard: Yeah. So that’s interesting. If I understand your point, it’s that this thing might be very smart and it might be very good at high level reasoning, but if it wants to improve the world, it’s going to need this specific information that it might or might not have access to.

So for example, if it wants to manipulate human beings, for example, I mean this is one doomsday scenario, it just manipulates you through your email, and gets you to release some kind of virus or something. That would require probably knowledge that you just can’t get from reading a bunch of journals about human psychology, and then just looking at a person’s search history. Maybe a less intelligent human being would be better able to manipulate a human being, than a superintelligence would, right? Is that sort of the idea? This is just one example of like you need things at the very specific level in order to be able to control the world?

Robin: So for the last few decades, many companies have been excited about AI and have said, “Come use AI, help our firm.” And when AI researchers or applicant experts try to go look for places they could apply AI, one of their main heuristics is, “Where do you have a lot of data?” And so AI’s been the most successful in things like games where you can generate the data automatically from the definition of the rules of the game. Or maybe in something like biochemistry, where you can simulate the biochemistry, or where you’ve got the entire dataset of the internet of text, where you can try to predict the next text. When you can just get a lot of data, well then you can use machine learning to predict things about that data. But most firms out there, when they’ve tried to apply AI, they’ve realized they just don’t have very much data that they can feed into.

And in fact, most of them are better off just doing something like a linear regression, because modern machine learning techniques just need more data than they have. And that’s just the sad fact about most people in the world trying to apply AI, is they just don’t seem to have enough data nearby to hand to an AI learning algorithm to actually do much.

So in order for future computer systems to gain power in the world, they’re going to have to either find data of somebody who already knows the world, and just use data on their behavior or their thoughts in order to learn that. Or they have to go interact with the world, in order to learn how the world works. And the world is slow, and it takes a lot of time to interact with the world, to find out what works.

Richard: Yeah, that’s funny. When you say they just go where the data is, it reminds me a lot of social science. I mean, we develop theories based on whether there’s a new dataset, and whether it actually matters for what we care about it or not, people don’t care about as much.

I remember during covid-19, a lot of people would say where the science says this, or the science says that. And the science was never relevant to covid-19. Or it was you’d often find these experts would come in and say the health experts believe that we should lie to people because X, Y, Z, and the science says this, and the science of persuasion.

There’s nothing directly analogous that’s even been done. But because they have data on something in the universe, they say, whatever this peer-reviewed paper, whatever this data said about this one thing must apply to our new situation.

Robin: Academia and science in general, including medicine, tries to give the impression they have vast datasets on everything you want, and they’re doing all these careful regressions. And if you ask them about anything, they’ll have expert knowledge to say, and that is true to some degree on some limited range of things, where they have a lot of data.

But then there’s all the things between their data and their theories, where they’re kind of interpolating and speculating, and they aren’t so honest about how they don’t know those things quite as well. Most medical treatments do not have a randomized experiment that tests their efficacy. There are a few, but most of them don’t.

And most variations that doctors use haven’t been tried out in that way, yet the idea of a randomized trial is this gold standard that we say, “Look, you can trust medicine because they do randomized experiments.” Except mostly they don’t.

In the same way for social science and mostly the rest of the world, most of the experts you trust in most of things in the world are interpolating from some places where they have a lot of data, to the places they don’t.

Richard: Yeah. The treatments, don’t all medical treatments, aren’t the new ones, don’t they all have to go through randomized control trials for the FDA, or no?

Robin: Only drugs. Not most treatments.

Richard: So if the doctor tells you...

Robin: Most treatments aren’t drugs, and of course, once you get a drug approved, it can be used for many other things, which…

Richard: Off-label use.

Robin: Exactly, which don’t have randomized trials.

The History of Economic Growth

Richard: Yeah. Interesting. So it’s just about getting the hurdle of over the drug use. And then you have this use, thing which could be used for anything, which is sort of funny.

Is this idea about the level of abstraction of knowledge is this, would you say, and I’m guessing you’re going to say it is, but is it backed up by the history of economic growth? From the industrial revolution, what portion of growth…

Robin: So say you take the history of locomotives, you can graph on the history of locomotives their speed or their energy efficiency, and you can see that the graph is relatively steady, but with some jumps once in a while. And that’s surely showing you the distribution of the sizes of innovations. It’s showing you that most years the improvement was small. That’s because the sum total of all the innovations, the improvement was small.

That’s because the sum total of all the innovations that year added up to a relatively small change, and then in other years it was bigger mainly because maybe there was one or two especially big lumps of improvement. And that’s just the nature of pretty much all the technical systems you can see, even solar cells or whatever. You’ll see that they tend to have relatively steady improvements in their abilities because it’s mostly lots of little things. And the jitteriness of them when they jerk a little, that’s the sign of a bigger thing. And by that, you can see that most innovation is composed of many small things and that big things are fewer. And of course, big things are going to be higher up in an abstraction hierarchy. They’re going to typically cover a wider range of aspects.

Richard: So you have something like, say the history of medicine, you have something like the germ theory of disease. So something like that would be a high level of abstraction, and that was a big deal that led to many changes. You have the concept of vaccination. And you’re right, I guess what you’d say is that these things matter, but then there’s so few of these things that if you took the small things, like a specific drug works in this specific situation or this surgical method works in this, there would be something... Well, I don’t know if that’s right. Wouldn’t you rather just know the germ theory of disease, or would you rather have a surgeon with all the specific knowledge about the best surgical techniques? I think I’d rather know about the germ theory of disease. Well…

Robin: School is organized usually around teaching people the abstractions. Schools mostly don’t teach you all the details, and so schools are emphasizing the value of abstraction. So we’re giving the students the impression that abstraction is really valuable and it is most everything. And then of course, students leave school and they start to try to do jobs and they quickly realize they hardly know anything. Their abstractions have hardly prepared them, and they mostly need to learn on the job. That’s the basic nature of people when they leave school and have to do things.

But still learning the abstractions is a good way to sort out the good students from the less good students often, so the pretense is okay there because you have to use something to sort them out. A lot of the history of technology is where specific technologies were developed, and then the abstraction came later. So the example of the steam engine, people figured out how to make a steam engine and then they invented thermodynamics to explain the steam engine. And then using thermodynamics, they could more easily invent variations on the steam engine. But quite often, abstractions come to rationalize and make sense of things that already work for reasons you didn’t understand before.

Richard: So was the germ theory of disease... So vaccines I think were like this. I think we had some concept of, I’ve read a few articles on this, we had some concept of vaccination and we knew that if you infect someone with a disease before we knew anything about the immune system or if they get the disease, they’ll... Even, I think it was India, they went from India actually to Britain, so there was a folk knowledge about some disease, I forget which one, that made its way. And then they learned about the immune system eventually and they had that.

What about what the germ theory of disease? Do you know the history well enough? Did they notice that if you washed your hands things got better, or did they have to do experiments and figure out the theory first?

Robin: Most of these abstract things, you actually need a fair number of more concrete things to make them work. So the most abstract things we know, even, say a steam engine, you can’t really make a steam engine just knowing thermodynamics. You’re also going to need to know some material science about which materials melt at what temperatures, how to make them, etc. So in fact, we often had abstractions long before we had the other parts to make them useful. So people have often looked in the past and blamed people, saying, “Hey, those ancient Greeks, looks like they had the basics of the steam engine and why didn’t they make the Industrial Revolution?” And you might say, “Well, they understood that steam had a force here, but they didn’t have all the other parts you would need to make the system work.”

And so we often don’t give enough credit to the other more concrete things necessary to make abstraction. So for example, take the cell phone, you might think what a great invention to invent the cell phone, but people could forecast long before the cell phone that the chips they had at the time and the costs of communication were just way too high to make a cell phone work, and they had to wait until the cost came down. And then there was a point where somebody said, “Let’s take a shot at the cell phone.” And it was less about the idea of a cell phone and more about, well, can we get enough towers and can we get enough chips and can we make a go of this? And then it’s about a business plan to make a go of something, and not the abstract idea.

Nanotech and the AI Singularity

Richard: And so here, I guess to circle back to the AI, what we’re saying is we’re imagining it being very good at abstraction, but even with its abstraction, it’s going to have to know specific things. Do you know anything about nanotechnology? I don’t know anything about nanotechnology, but when I read these doomer scenarios, and I’ve wanted to read more on this, but I haven’t had a chance, it’s like they’re going to build nano something or other that’s going to come kill us all. And I know nanotechnology means something having to do with being very small, but I don’t know much of what that means. So why do they seem so confident that nanotechnology is a way for computers to come get us?

Robin: Well, my friend Eric Drexler years ago wrote a book called Engines of Creation, and then he wrote a later book called Nanosystems and some other systems, and he basically argued persuasively that it would be possible to make a technology of manufacturing and devices that was based on machines where each atom was placed exactly where you wanted it to be. And that wasn’t true then, and it still isn’t true now. We don’t have a manufacturing industry where that’s a general usual capacity. It will be possible to make manufacturing machines that do that, and once you can do that, you can make more of them and then they can make many devices, and then we could have powerful abilities to use such devices and he could calculate just how faster computers could be or other sorts of cheap other devices could be if you could put every atom where you wanted.

And it’s possible, the standard chemistry and quantum mechanics, etc., to actually make computer models of these devices and how they would work and show how effective they would be, but we just aren’t really at the point of being able to cheaply actually put each atom where we want it. And so it’s an envisioned future technology where you could just do a lot more things. Now, obviously if you were the first person to have nanotechnology, then you would have a huge advantage over other people in the world and making devices, computers or weapons or other sorts of things. It would be a really breakthrough technology. I think if they postulate foom, they also want to postulate something like this, a big leap forward in capabilities that would then allow a system to have a big powerful advantage.

And then you’d have to postulate that in order to make one of these manufacturing devices, where you know can actually put the atoms exactly where you want cheap enough, and then do it at scale, if you thought, well, computing power is the limiting factor, the reason we can’t do that now is you just haven’t computed it well enough. And so they think the smart computer, it could figure out how to better compute the simulation of these nano machines and the nano factory. And then this AI, if it was smart enough to figure out how to assemble to start creating a nano factory, then it could be the first one with nano factories and then it could be the first one with nano weapons created in the nano factories, and then it has this big technological advantage.

Richard: I see. So the nanotechnology intrigues them because the idea is the input you would need to create an output is just so small. Is that the idea? So a computer, it just needs a few atoms and that’s all it would take?

Robin: Or that it’s just computing, that would be the answer. So the idea would be in order to make a nano factory and a nano machine, we don’t need to experiment a lot and try stuff out in chemical labs and write lots of papers to each other. We just need clever enough calculation to figure out the device, and then we send the right instructions to an ordinary factory now of some sort that puts atoms, and then we can make the right sort of thing. And then ta-da, we have take off.

Richard: Is that any more plausible than, say, just figuring out from first principles how to build a car factory or something? Is there any reason to think that’s more likely?

Robin: It just better fits into the scenario of a very sudden, fast takeover. If you figured out how to make this nano factory, then it probably could grow very quickly and it probably could have a very rapid impact by comparison with most other things in the world. So if your AI is postulated, it has to have a new design for a tanker or a missile or something. And it’s slow to make tank factories and missile factories, you’ll have to clear some ground or buy something up and send new shipments to it and do all sorts of things. Doing real things in the world can take years, as we know, and so if your scenario is this AI takes over the world in a few months or a week, you need a scenario with only fast things in it.

Richard: Because the nanotechnology, the process of building is faster, it requires…

Robin: Because they’re all really tiny.

Richard: Yeah. The process of building is faster, and it requires fewer... Is there a theoretical reason to believe it requires fewer just inputs like mass or matter than, say, a truck company would? Or is it possible that they figure it out and actually you need a nuclear power plant or something? You know what I mean?

Robin: No, yeah, we’re very sure just simple chemistry would be enough. So you just have to be clever about arranging the atoms, and then everything goes well. But you have to be very clever.

Richard: I see. Okay. So I see, that’s very interesting. So potentially, what we’re calling the nano factory, it wouldn’t look like a factory. It would perhaps just look like…

Robin: A shoebox.

Richard: A shoebox. Just got to move those atoms around. That’s very interesting. Is nanotechnology seen as something like cell phones, where there isn’t any conceptual hurdle we need to get over? I guess what I’m asking, is there reason to believe that just before we do it, that you could actually control exactly where the atoms are? That this is possible with the laws of physics?

Robin: So what Eric Drexler did an excellent job of is persuading you that there was no fundamental limitation that would prevent us. I think…

Richard: I see.

Robin: Okay, but there’s a getting from here to there problem.

Richard: Yes.

Robin: That is, if you had a little tiny nano factory that could put each atom where it wanted, then you could send it instructions to make machines, which had each atom where they wanted, and then you could quickly make millions and billions of these little machines, which had each atom where you want. But you don’t even have the first nano factory to put each atom where you want, and you’re stuck. You can’t make the first item that would make the rest.

So that’s where we’ve been for many decades. Eric Drexler wrote his book in the mid-1980s, so it’s now almost 40 years and it looks like it may be another 40 years at least until this is actually achieved. But if you think only a smart enough machine could figure it out, then you think, “Aha, you see, it’ll then make the first nano factory because it’s so much more clever than us.” So now the question is, what does it take to get to be clever about nano factories? Is all it takes just being able to think abstractly about your simulation of molecules, or do you actually have to do real experimentation and try lots of things, in which case it would take a lot longer?

Richard: Yeah. Wouldn’t it be a smart strategy for a superintelligence to just wait for humans to do all that work and then just steal their research and then build the factories, take over the world?

Robin: Sure, but under that scenario, the superintelligence has to sit around for decades waiting.

Richard: It could be very patient.

Robin: So they’re trying construct a scenario where the superintelligence takes over the world in a week, basically.

Richard: Well, it takes over a week... It could be here right now, but it’s just so smart. It knows the only way I could kill everyone is with nanotechnology. So whatever, it’s got a long time horizon. It says I’m going to wait 50 years and watch humans do the work.

Robin: We’re postulating that this superintelligence is the result under some organization. Some organization has this computer program they’re running, and an organization that has a computer program running is running different variations on it, looking at its code, seeing what happens when it does things. Most organizations with a computer system are monitoring it and testing it to see what it’s like and what it’s doing.

So right off the bat, we have a problem for imagining that there’s this system out there that is a computer system that some organization is using to schedule cab rides or whatever, and it’s got this whole other line of thought in its head about its plans to take over the world that the people who run the system never see somehow. It’s encoded in some strange thing, and this system is sitting there biding its time waiting to figure out how to take over the world while it pretends to just schedule cabs. The question is, where does all this code sit exactly? And how do the programmers who set up the system have no idea that this code is there running?

Richard: Yeah. Okay, forget about it waiting. It doesn’t matter for the scenario. Imagine we build nanotechnologies first and then we get the superintelligence. And then it has the goal, and now it can... I guess in a world where nanotechnology is fully mature, is the idea that it could build anything with a pretty minimal amount of effort and it would…

Robin: The whole advantage of the nanotechnology in the near-term scenario is that it would be the first with nanotechnology, and therefore it would have a huge advantage. In a world where everybody has nanotechnology, it using nanotechnology doesn’t give it much of an advantage. It doesn’t take over the world that way.

Richard: I see. Well, maybe because it equalized, because we humans have more just bulk massive stuff and we have distributed... We have firms and we have governments and ways to coordinate and logistical systems and all this stuff. And right now, maybe it’s in 2043, the superintelligence wouldn’t be a fair fight. And I just learned what nanotechnology is five minutes ago from listening to you, so I could be saying stuff that doesn’t make any sense, but once nanotechnology is mature, perhaps the argument would be that basically it evens the score because all you need is a little bit of matter, and now the superintelligence could be the deciding factor in the humans versus machines conflict.

Robin: Well, then you have to postulate that it can design much better nano machines, you see.

Richard: I see.

Robin: And if everybody has nano machines then it already has... We have nano detectors and we’re out watching for other people’s nano machines and we’re defending against other people’s nano weapons. And then in a world where everybody has this nano stuff, the AI can only get an advantage if it’s going to have better versions of those things. So then it has to find, then you have to say, “Well, it’s going to use its extra cleverness to figure out much better versions.” Of course, you could do that today.

Some people have said, “Well, there’s a stock market place where you could make money if you’re clever.” And so why an AI, it wouldn’t naturally go to the stock market and use all its cleverness to make a lot more money in the market? And if you thought all it takes is better algorithms in order to make money in the market, you think, well, then it’ll just take over the world because it’ll own everything. Just thinking in your own head is not so great a way to just figure out how to win the markets. Most of the people who make money in the markets, they’re connected to the world and they’re getting information about the world and they’re talking to people and they’re using that information. It’s very hard to compete in that space. So how is this machine so much better at that?

Richard: Well, this scenario is a little bit easier for me to imagine because it’s connected to the internet. It can read every language. It can read the newspapers of every language and get all that information. It could maybe read into people’s emails. It can…

Robin: But all the other hedge funds have that ability too. So we’re postulating in a world where there’s lots of AIs. In order to postulate that this AI takes over, we have to have it be much better than the rest. That’s the trick of the…

Richard: Why can’t it just be one hedge fund that invents this AI?

Robin: Well, at the moment, there are many hedge funds that already have AIs. They’re already trying to use those. So you can’t be the first hedge fund with an AI anymore. You have to be the first one of a certain kind that was much better than the rest. And so again, we get back to the scenario, somehow you postulate somebody’s system with an AI has this innovation that makes it vastly better than all the other systems in the world, and then it can rapidly and vastly approve its capabilities in a very short time.

“Less Than 1%”

Richard: Yeah. We’ll move on from the doomerism scenario. Right now, before we do, let me just ask you, what do you think is the possibility that Eliezer Yudkowsky is completely right and our heads will explode sometime in the near future?

Robin: Well, less than 1%.

Richard: [laughs] Okay. Well, is it less than 0.1%? This matters for existential risk.

Robin: So the subtler question is, if there was a big problem, would this be the right time to work on it? And I’m even more confident that if there is a big problem, we’re still not at the point where this is the time to work on it. That is, we need to see if there’s going to be a problematic out of control system, we need to see a version of it that’s more concrete and closer to what the problematic system is, and then we could start to work on how to deal with that. But at the stage where we hardly have any idea what this problematic system would look like, there’s just not that much we can do.

Richard: Are you not convinced that ChatGPT is getting to something closer to intelligence, and with DALL-E and you can make these into a module or you…

Robin: Of course, we’re getting closer. But look, so as you probably know, roughly every 30 years since at least the 1930s, we’ve had these bursts of concern and attention to exciting, interesting demos of automation and then computers and then AI that make people think, oh my goodness, this could do a lot more than we realized, and could it almost be there? And that’s what they said when I was a student in the early 1980s when I left grad school to go off to Silicon Valley to be an AI researcher because I read all these news articles about how AI was about to take over everything.

I was duped and wrong at that time, but I think we’re just in another era like this, and this’ll happen again 30 years from now. ChatGPT is just not almost human. Sorry. It’s just not close. But in the past, people also felt, wow, they looked at the new systems. Every new system, every decade, you say, “Nobody’s ever done something like that before.” And that’s been the case every decade the whole period, and it’ll continue to be the case. The question is, is this system capable of doing most everything a human mind could do? And no, it’s just not close to that.

Richard: No. Okay. So you’re not going to give me a less than a... You’re saying more than 99. Okay, that’s good enough. You don’t have to give me one in a thousand or one in a million or one in a billion or whatever. Yeah, that might be asking too much, but just very very unlikely, and for practical purposes, probably not worth worrying about.

Robin: Well, if you’re going to worry about it, what you should do is save up resources so you’re ready to go when you actually get enough data to do something.

Richard: You should just build economic growth and we should just be as rich as possible.

Robin: We need to pivot. When the problem shows up in a concrete form, you’re ready to pivot. You’re watching for it. You’re monitoring. You’re looking for the chance that you have a system that might be this sort of a problem and you’re ready to jump on it.

What Principal-Agent Problem?

Richard: Yeah. You have another interesting argument about AI, and I think this applies to not just the possibility of foom, but the other things that people are worried about. “Oh, the AI is going to become a powerful government or it’s going to become an oligarch or something.” The idea that there’s the principal-agent problem. So the way I take your argument is that basically it’s sometimes better if you’re the principal to have an agent that is not aligned with you, but that is really competent and good at what it’s doing, rather than one that’s, say, more aligned and, say, not as good. I think what you’re thinking is in terms of economic growth. The AI, if it’s really really smart and really really good at doing things, we’re going to get so much wealthier that even if we get a smaller portion of a pie, we really shouldn’t worry about that. Is that the argument?

Robin: Well, so I’ve got two related arguments. One argument I have in a post a long time ago that says, “Prefer law to values.” I’d say, if you were thinking about moving to a foreign country to retire, imagine there’s two questions you could ask about this country. One question is, do the people there agree with me about values? And another is, do people there obey the law? And I think you’d want that second question to be answered as a higher priority. That is, in order to keep the peace with the other people in this new country you move to, it’s more important to know that property rights are respected and that the law will be an intermediary between you than to know that they actually share your values. That’s because the law is this thing that makes you care less about their values as long as they obey the law.

So that says that for AI as well, what we’ll want is for them to be embedded in a larger social legal system wherein they fit in that system and they keep the peace within that system. That is, they follow the rules of that system. And that’s the important thing you want to know about them, is they’ve been designed and habituated to sit in the social rules that we can interact with comfortably and peaceably. We don’t need to know what they want or what their values are. We need to know that they can relate to us through property rights and law. So that’s one sort of claim about what you should care more about, they’re being law-abiding and that you have some sort of legal system between you and them that can adjudicate disputes and encourage good behavior.

Then another issue is agency failures, as you talked about the principal-agent. So some people in the AI risk community have said that if you have an agent and it gets smarter, your problem of controlling it gets harder. And they have said that you should really worry about a very smart agent because it will just outsmart you and then you really won’t get much of what you want with a really smart agent. And I have a post where I say basically, we have a large economic literature on this principal-agent problem. We know a number of parameters that make the agency problem harder such that agents are less well aligned and they get more of the pie relative to you, but intelligence isn’t one of those parameters.

And I don’t believe that intelligence is such a parameter. I just don’t believe that on average, having a smarter agent is worse for you. I think if you were thinking about hiring a butler or a driver or an executive assistant, you’d probably want a smarter one. In general, that would just go better, even though in principle they could trick you better, but overall I think it’ll go well.

Richard: Yeah. Do you see an analogy here with nationalism? So you have these decolonization movements in the 20th century, and then you have even ethnic politics today, where it seems like many people want to be ruled by people whose values align with them or who look like them or who share their cultural background. But often those people are much worse at governing and these countries and these communities often end up being worse off. Do you see as this a…

Robin: You’re right.

Richard: ... similar mistake?

Robin: I think I do. For example, multinationals seem to be just better firms, just better run generally, and so when nations prefer their local firms over multinationals, I think they’re making a mistake. They would get more of most everything they want if they more encouraged multinationals to come participate in their economy. And I actually think instead of electing local people in political races, I think it would be great if management consulting firms ran for office on an international reputation, basically said, “Look, here’s the hundred other places we have run over the last few decades and a record of how we ran them, and we’re going to do this for you here if you elect us.” And I think that would probably be better than electing the local guy who says he loves you and he grew up in your town and he’s going to do well for you, but he doesn’t have a track record, and people like him haven’t done so well.

Richard: And so even in the economy where, even if it’s within the same country, we love small businesses and we hate big business. And the big business outcompetes again small business, but we don’t care. We somehow think the small business has better values.

Robin: And the local small business, especially. Our nearby small business, we supposedly try to favor.

Richard: Yeah. You see these signs at the store.

Robin: “I buy local,” yeah.

Richard: Yeah. Who cares? [laughs] It’s a very silly thing. Would this have an implication for the federalism debate? The American federal government... Generally, the federal government jobs I think pay better than state government jobs. I think most people think the FBI is probably more professional than most local police forces.

Robin: There’s certainly a related thing for non-profits versus for-profits. Some people would rather go to a non-profit hospital thinking that somehow they care more about them. And I don’t think it actually helps you.

Richard: Caring about you is great, but it’s not necessary. Society didn’t have an economic takeoff after the Industrial Revolution because people started caring about each other more. They probably cared about each other much more in the distant past, in probably small villages.

Robin: But in the forager world above a million years ago and up until even the farming revolution probably it just was really important to gauge who around you liked you and shared your values and that probably was a big indicator of who you could work with well and trust. It’s just, the main thing that’s happened in the last few centuries, we’ve vastly expanded the size of organizations and they’re quite alien to our evolved experience. And yes, we are in a world where we’re trying to trust our intuitions, which are based on very small-scale groups in order to deal with pretty alien, strange, new superintelligence.

The Robin Hanson Version of Doomerism

Richard: This has been an optimistic conversation. I want to believe you because the other side is selling doom and gloom.

Robin: Let me give you the doom side of it. I’ll give you my doom side, which is just to say if we continue to have a competitive world, then we will continue to have competition that changes the world and then the things that win the competition in the future, won’t be you, they’ll be maybe descendants of you, but they’ll be quite different from you and their values will probably be different from you. That is the world will select for competitive winners and whatever values it is that produce those competitive wins and that’s probably not your values. You should probably expect the future to be different if it competes. And I think many people are scared of that. They don’t want competition because they think competition will drive our descendants to be strange.

Richard: Robin, what if I’m a Nietzschean and my value is just winning and whatever’s winning is great?

Robin: Then you’re pretty lucky then. [laughter] How many people are like that?

Richard: I just love winning. Whatever wins, I’ll be satisfied with.

Robin: If you just want a universe where something wins, then you’re even happier, right? Doesn’t even matter if it’s you, right? Just as long as something wins, I like it.

Richard: I want it to be conscious. If they’re unconscious machines that would be…

Robin: Okay, well you have other constraints now.

Richard: That would be it. I’d prefer the species homosapien and although I’m not completely committed to it, but I want the consciousness, that scares me if they’re robots and maybe it’s a gradual thing and the robots are fighting and humans, but there’s no consciousness…

Robin: This is why I’ve actually tried to think a fair bit about what will win the competition in the long run. And I think I’ve come to some at least rough conclusions that I can guess about how we will change as a result of competition over the coming centuries. And I don’t know if you’ll like them or not, but at least it’s something we can think about. We could draw some actual concrete conclusions.

Richard: Say more. I’d be interested.

Robin: Okay. For example, we have a good theory about why humans discount the future, which is because our children share roughly half our genes and that’s a reason when we’re trading off resources for us now or our children, a generation from now, we have roughly a factor of two discount rate for generation. That’s a result of sexual selection. And with asexually reproducing creatures like investment funds, they should not have that discount rate. And eventually the universe should be dominated by creatures who do not discount the future. They might discount growth with the usual logarithmic discount of growth size, but they would not discount time. And eventually, the claim is, we will no longer neglect the future. We now neglect the future because of this innate discount rate of a factor of two per generation. But there will come a time when we do not neglect the future anymore.

Richard: When you say something like, today, you imagined this could be, probably ongoing, you mentioned investment firms, there are probably some that chase quick returns and there are probably some that are long term. Do you think in the long term in a hundred years we’ll have investment firms that have longer time horizons?

Robin: We know a lot actually about the selection effects among investment firms in a competitive investment world. People have done a lot of mathematical work on that actually. And we know that if we allowed it, investment firms would just grow because historically say the rate of return on investment has been higher than the rate of return of growth, the rate of growth of the economy, investment would in fact grow as a fraction of the economy. And the reason that it hasn’t so far is that we have prevented that through law. We have in fact in the past prevented investment firms from just reinvesting all their money and just devoting their cause to that. When you create a foundation, say when you die, there’s rules about how much of the foundation’s money has to be spent each year so that it cannot grow in this way.

But if we allowed investment firms to last arbitrarily long with arbitrarily giving a high percentage of their reinvestment, then they would come to dominate the economy and they would force interest rates down to equal growth rates and they would dominate investment choices in the economy. And people have envisioned that and that’s why they made these legal rules to prevent it because they said, we don’t want the dead hand of the past to determine our world. And if we allowed this then the dead hand of people who died and who gave their money to investment firms would be ruling our world. But I still think eventually whether through investment funds or just other some form of selection, we will have creatures who no longer discount the future. And that’s a thing we can predict about the future.

Richard: So I can’t leave behind an estate that spends money and, I say, “Don’t spend any money for a hundred years,” and then do X, Y, Z?

Robin: You cannot do that. You can leave it to your uncle and give them, you can leave it to your nephew and give them those instructions. But you know what? He might not follow them.

Richard: Right.

Robin: He wouldn’t be legally obligated to do so.

Richard: And what are the restrictions on investment firms that stop them from just investing all their money?

Robin: Because they have owners and the owners will ask for some of the money that is there.

Richard: But you said that there was some restriction that made them…

Robin: If you try to have a will that creates a foundation you see after you die and just make it and reinvest all the money, then it isn’t allowed to do that. If you try to reinvest all your personal money, you can do that. But then, when you die, that money will go to whoever inherits your money and they may not continue this policy.

Richard: I thought that you said there was an analog to this “dead hand of the past” rule in the way investment firms work. Did I misunderstand this?

Robin: There’s just that rule in wills.

Richard: Okay.

Robin: You can’t use your will to create a foundation that just reinvests its money. You can do that yourself as long as you’re alive, but as soon as you die, whoever you give that money to will get to choose their own policy and they may want to spend it all.

Richard: Okay, that’s one area of life we prevent from letting things change. What else is going to maximize over time? What do you think about even families and certain genetic populations and communities? There’s the Amish, who, this is not conscious, I don’t think they think that in a hundred years they’re going to…

Robin: I actually think relatively soon we’re just going to be replaced with artificial descendants who aren’t going to reproduce with the usual DNA method. That method of reproduction is not going to last that long, much longer. We’re going to make our descendants in factories out of explicit designs. But natural selection will still continue in that world. It’ll just continue with these new kinds of genes which aren’t DNA, but I think we can predict things about that world. In fact, I think we can predict that eventually we will have creatures whose value is in their minds, “I want to reproduce.” Today, we reproduce because we have preferences that are indirectly inducing reproduction. We want sex, we want status, etc. And by wanting these things, our behavior tends to produce children. And that’s what evolution’s been counting on for us to reproduce. But in fact, that’s not very reliable. As the environments change, these evolved habits don’t necessarily make us reproduce as much as we could. And that’s why we’re suffering this vast fertility decline.

Richard: But why can’t the human communities that already have this in their brain, why isn’t it that just they would take over the world? Because there are communities that just want to maximize…

Robin: Sure. Artificial is just better in so many other ways. This isn’t the reason why artificial wins, this is just something that happens with artificial after artificial wins.

Richard: Maybe those people are the least inclined to pursue artificial. There’s the humans who don’t want to reproduce are going extinct but they’re the technophiles and the humans who do want to explicitly reproduce are technophobes.

Robin: Over the next millennia, there’ll be a slow selection effect among humans for the ones who reproduce. But the switch to artificial may just happen well before then, in which case we’ll just have a world… My book, Age of Em is a scenario like that where basically the emulations are artificial creatures and they reproduce a different way and then they quickly dominate humans in that scenario.

On Consciousness

Richard: Do you think they’ll be conscious or not? They will be conscious?

Robin: Yes, I think so.

Richard: And why do you think that?

Robin: First of all, I’m a physics person who just thinks stuff in our physical universe is conscious when it can be because that’s the only reason it may make sense for our brains to be conscious. There’s nothing special about our brains that makes us conscious. They’re just ordinary physical devices in our universe. If our brains are conscious, I’d guess most everything else that could be is.

Richard: You subscribe to Panpsychism?

Robin: I said that could be, that’s the key constraint.

Richard: It could be…

Robin: Panpsychism would be everything. Nothing couldn’t be.

Richard: What percentage of things are conscious to you?

Robin: I would think if it can compute what its conscious feelings are, that’s a good indication. It really can’t have conscious feelings unless they can compute them. And computing conscious feelings is actually quite a restriction on a system. Most systems don’t do that. In order to feel something your body has, your mind has to compute what you feel, it has to calculate that. And that’s a bunch of work.

Richard: My computer, it can compute, it’s getting too hot or it’s getting too whatever, and it has the system updates and things. Do you think that our computers are conscious?

Robin: In some way, but not in the way you are because their consciousness doesn’t feed into other things in the way yours does.

Richard: Interesting.

Robin: I just don’t think…there’s just physics. There isn’t anything else. The answer to these questions just have to be in the physics. Some physical arrangements are conscious, clearly. And the question is, how does the universe tell which physical arrangements are conscious? The simplest answer would be any of them that could be conscious are, that would be a simple way the universe could figure that out. Anything else would have to be a lot more complicated. And the question is, where does the universe compute this, figuring out which things are conscious? I’d say the fundamental principle is computation happens in physics.

Richard: Yeah. Okay.

Robin: If there’s something that has to be computed in order for things to work in the universe, it happens in physics. Physics is the thing that computes it. And if you’re conscious that’s being computed somehow. And it’s either a very simple rule, it doesn’t require much computation or it’s a complicated one, and then it’s complicated, something that computes like your brain.

Richard: Whatever we create will be somewhat like a human and somewhat like a computer. And humans we know are conscious. And you think computers are probably in some way conscious, so a thing that’s a sort of human-computer hybrid would probably itself be conscious. Is that the idea?

Robin: What else could it be I guess is the question?

Richard: I don’t know. It could be that there’s just something. I guess what else could it be? It could be that just carbon-based life is different than silicon-based. I don’t know.

Robin: I know enough that carbon is just a certain number of protons in the nucleus. That’s all it is. How does the universe care about the number of protons in the nucleus? How does that work? It doesn’t make any sense as a theory.

Richard: The universe, not the universe cares, I don’t know if that’s the right way to think about it. But for example, the human body has certain properties that machines don’t have in the sense that, it’s wet, for example, and machines tend to be dry. This is one of the simplest things you can imagine. And consciousness can just be like, we don’t know. We don’t know.

Robin: This is where being a physicist comes in. As a physicist, I say, look, these are the concepts that make sense as the fundamental physics concept. So if there’s something true about the universe that needs to be expressed in terms of these fundamental physics concepts and you know what? Wet isn’t one of them. Wet is a very high level abstraction and in some sense you don’t know if any of the rest of other humans in the world are conscious. You don’t know if you were even ever conscious in the past. In principle, you could just be remembering when you thought you were conscious and you never were.

You’re postulating perhaps that all the brains in the world of humans are all conscious. But what’s the basis for that? You’re basically assuming some generality because they’re similar, but I’d say wet versus not wet, that’s actually similar in terms of basic physics. There’s nothing fundamental about wet versus not wet that makes any sense as a distinction. You might say only people in the Northern Hemisphere are conscious. That’s a line you can draw, but it seems pretty arbitrary to me. Wet versus not wet seems just as arbitrary in terms of basic physics.

Richard: So processing information to you seems more fundamental in the role of physics than wet versus dry?

Robin: The key thing is, if there’s going to be a distinction, it has to be computed somehow, that is there has to be a physical process that results in that distinction being figured out. And the universe, either there’s a physical law by which an evolution of things figure it out or it’s an arbitrary label. Where does this label come from? I have a strong prior toward integrating whatever this other thing is with all the other physics things we understand. There’s a physical universe that’s a certain set of properties, a certain set of things. We understand those things and you know what? I’m going to stick with those things and will be reluctant to add other things unless you show me some evidence that there’s other things.

Richard: I started reading The Age of Em, a long time ago. And it lost me when it got to the sci-fi aspects. But these ideas, your ideas of consciousness, are they found in there?

Robin: No. I tried to avoid that because that’s a rabbit hole that people just get sucked down to. As you may know, there’s some honeypot topics out there that just suck people in and there’s just not much value in them. And mostly people should avoid them. And that’s one of them. Consciousness is one of these topics. If you get sucked in, there’s just endless cycles you can go through talking about things and there’s just not much that ever comes out of it. There’s nothing you can do with any of this stuff. And I’m very attentive to, let’s think about the stuff we could do something with if you figured it out. And that’s where I go.

Richard: And I guess you must think AI alignment then is the biggest honeypot of them all?

Robin: It is a big honeypot. Yeah, a lot of people in our world are sucked into it. And it’s interesting to speculate why. I actually think one of, we’ve talked about abstractions, one of the key distinctions in the world between the people I’ve liked and the people I don’t like as much is I like to hang around people who just have a taste for abstraction. People with a taste for abstraction, they tend to think abstractly about decision theory and about quantum mechanics and about utilitarianism. And there’s just a whole range of abstractions that they gravitate toward because you can talk about things at an abstract level and you don’t have to get dragged down with the details. And they just like that. And this is one of those topics, people can talk about AI alignment in the abstract and they hardly need to know any details.

And it seems important and almost a comic book story and it gets sucked in. But in general, I like abstraction. I like to think about abstractions. I think it’s fun to think about quantum mechanics and utilitarianism and algorithm design and all these sorts of things. But one of the most important skills in the world is to judge when to reason abstractly and when to reason concretely. And often in a conversation you have to go back and forth several times. And I’m wary to get sucked into certain details and I say, “Where are we going with this? I don’t get it. I don’t see it, where all the value is?” And I have to be wary about certain kinds of abstractions, certain kind of abstractions I go, “Is this just a word, or is there really a thing behind this?” And we talked about that with intelligence, “Okay, what kind of an abstraction is intelligence? What kind of a thing is behind that? Is that just like betterness, another name for stuff we like or is there a thing in the world that corresponds to it?”

Richard: Yeah. And it sounds like your difference with a lot of these people, they like abstractions and I like abstractions too, you like to get somewhere too, you are selective.

Robin: For example, I have abstractions about economic growth but tied to our history of economic growth, that is, I’ve seen what we know about the history of economic growth and the history of innovation. And I’ve tried to tie my abstractions to those observations so that they’re grounded in that way. And then I feel much more confident in what I would say about innovation because it’s tied to that concrete data. And that’s what I want to do with abstractions. If they drift too far away from concrete data, then they can often just go off the rails in strange directions. And it’s quite an art I think to have concrete data to jump away from it to the right level of abstraction so you’re not lost, you’re not dragged in by arbitrary details, but you don’t get too far away from it because otherwise you could...

For example, people who talk about capitalism or other things like that, the whole world of Marxists, has this world of these floating abstractions and they just are often quite a distance from any concrete social phenomena you might be interested in.

They talk about exploitation, you go, “Exploitation, what is that? Let’s look at, show me where exploitation is.” But they don’t care that you can’t like cash it out in concrete situations. They just talk in these abstractions. And that’s just a big risk of people who like abstractions is they will talk about abstractions that have just become detached, that are not tied down well enough by concrete details.

A Theory of the Sacred

Richard: People, when they talk about politics, I see this a lot, I saw this tweet a week or two ago. It’s like, “Don’t be fooled. The left will never give up power voluntarily.” And there are so many abstractions, first of all.

Robin: Who’s the left? What’s power exactly?

Richard: What is the power? What is “give up voluntarily”? They lose an election and then Obama leaves office and then Trump takes office, is that voluntarily or they lost the election? And they’re trying to make it almost like, “We have to go to complete war against these people,” but it’s just meaningless and so much of political discussions are like this, the right is this, the left is that, have you ever read Curtis Yarvin’s stuff?

Robin: I have in the past. Not so much lately, but I’ve actually, in my last few months, the last six months I’ve been focused on the sacred. And we don’t have time now, but we could talk about it later. But I think I’ve recently had an insight that explains a fair bit of this habit that is a frustrating habit. There’s certain kinds of topics and people just get abstract and floaty and they don’t even want to be very, just even imagining what that would mean concretely is not a habit they have. And I think I’ve come up with a way of understanding why that happens on sacred related topics. That is a temptation for later. We don’t have time for that today, but if you want to come back and talk we will do that some other time.

Richard: Can you give us a nutshell version?

Robin: Basically, the sacred’s been in my way in a sense all my life, I finally was frustrated enough, I said, “Let’s study this thing.” And I collected 50 or so correlates of the sacred, things people say go along with things that are called sacred. And I looked for theories that people had to explain these things. I picked one I thought was pretty good and then it explained some of them but not others. And I came up with an alternative add-on theory to explain all of them together. So I think I have a nice unified theory of the sacred making sense of all these 50 correlates of the sacred, which then is a powerful toolkit by which you can think about the sacred, you can understand how the sacred affects other people and yourself because everybody has stuff that’s sacred. Even you or I, we won’t give it up, nobody’s going to give up all of the sacred. And so it’s important to figure out what it is, how it works, and how to minimize its harms and maximize its advantages.

Richard: Is there a level of synthesis of these things or is it just like a 50-item checklist?

Robin: I take the 50 items, I clump them into seven clusters, each of which has a general theme. And then I try to explain these clusters. And one simple theory I take from Durkheim explains three of the seven clusters and then I add one other theory and explains the other four. And so now I’ve got a unified theory of all the clusters from one simple ancient theory plus my one new add-on and I’ve got a unified account. It’s not just a list of them, it’s a unified theory explaining why all these 50 things are there.

Richard: I’m looking forward to this. As you’re going to write this up, what’s the timeline on this?

Robin: I have written it up, you can see my recent post on the sacred, I have a paper whose title, basically I think is, “We see the Sacred from afar, to see it the same.” That’s my key insight to explain these other four correlates.

Richard: Okay. I look forward to reading it.

Robin: This is a tease. We have to end soon here because we’ve been talking for an hour and 20 minutes. I’ve been working on it the last six months, so I’m pretty proud of coming up with a coherent account of what seems to be a pretty fundamental human behavior.

Richard: Okay. Conscious of your time, Robin, is there anything else you’re working on that you want to let people know about before we let you go?

Robin: That was it. That was my pitch.

Richard: Okay, great. It’s been great having you on and we’ll have you back to talk about that other stuff.

---