91ÖÆƬ³§

00;00;00;12 - 00;00;25;12

Geoff Nielson

Hey everyone! I'm super excited to be sitting down with Roman Yampolskiy, a leading writer and thinker in AI safety. He's written over 100 publications on the topic and been sharing his message with everyone from Lex Fridman to Joe Rogan. Roman is as far as you can get on the AI doomer scale, and he believes that the odds of AI superintelligence killing all humans in the next hundred years is over 99%, and there's nothing we can do to stop it.

00;00;25;18 - 00;00;41;22

Geoff Nielson

He and I share the same starting position that the dangers of AI aren't being taken seriously enough. But what I really want to know is what can we do about it? I'm not ready to write the obituary for the human race, so I want to challenge him on why he might be wrong, or what we can do to change the future.

00;00;41;25 - 00;00;46;27

Geoff Nielson

Let's find out.

00;00;46;29 - 00;01;14;06

Geoff Nielson

Okay. I'm here with, Doctor Roman, and we're here to talk about AI risk, AI safety. Roman, thanks so much for being here. Maybe the first thing to jump into is, you know, one of the, you know, predictions you've made lately that's kind of made the rounds. Is that your prediction of an extinction level event for humans created by AI in the next hundred years, you're putting at 99.99%.

00;01;14;07 - 00;01;18;08

Geoff Nielson

Is that right? Am I missing a couple of nines there?

00;01;18;10 - 00;01;38;12

Dr. Roman Yampolskiy

Keep adding nines. I keep meeting people. I have a different p doom for reasons independent of mine. So every time this happens, another nine has to be added logically. But, it seems to be that you have to follow the chain of kind of assumptions to get to that. Number one is it looks like we're creating AGI.

00;01;38;12 - 00;01;59;07

Dr. Roman Yampolskiy

And then quickly after superintelligence, a lot of resources are going into it. Prediction markets. Top experts are saying we're just a few years away. Some say two years, five years. But they all kind of they agree on that. At the same time, according to my research. And no one has contradicted that. We have no idea how to control superintelligence systems.

00;01;59;09 - 00;02;15;29

Dr. Roman Yampolskiy

So given those two ingredients, the conclusion is, pretty logical. You're basically asking, what is the chance we can create a perpetual safety machine, perpetual motion device? By analogy, and the chances of that are close to zero.

00;02;16;01 - 00;02;36;27

Geoff Nielson

And you know that that logic makes sense to me. I want to dive into it a little bit more. But when we think about, you know, the 99.99, however many in your mind, why why not 100%? Why not 70%? Like what? What is it about this problem that makes you think it's, you know, like a 1 in 10,000 chance that will survive this?

00;02;36;27 - 00;03;04;03

Dr. Roman Yampolskiy

So for one, I could be wrong. Maybe that is a mistake in my argumentation. Maybe one of my proofs arguments is not solid and people will discover it. And there is a way to actually accomplish it. There is always that possibility. There is also a tiny chance that we actually will be smart enough to create general superintelligence will find it is better for economy, for safety, to just develop advanced AI tools.

00;03;04;03 - 00;03;10;03

Dr. Roman Yampolskiy

And that will happen. But again, I place a very small likelihood on that happening.

00;03;10;06 - 00;03;32;11

Geoff Nielson

You mentioned, as you've been talking to more people, there's more and more scenarios where you think to yourself, oh shit, you know, that could also kill us all. How do you organize in your mind the types of different scenarios where we might potentially have an, an, extinction level event and, you know, which you are more and less likely.

00;03;32;14 - 00;04;00;29

Dr. Roman Yampolskiy

Right? So right now we have AI as a tool. So the concern is malevolent actors using this tool to cause typical types of damage. Maybe they develop synthetic, weapons, synthetic biology, some sort of new virus. And that's going to cause a global pandemic exterminating a lot of people. Maybe not all people, but a large number. There are other, similar scenarios where it's a tool.

00;04;00;29 - 00;04;23;13

Dr. Roman Yampolskiy

Someone's using it to hack into a nuclear power plants, military response, airplanes, things like that. But then we switch paradigms. We go to agents. At that point, AI is the adversary. It's the agent which can cause damage. And that's where it's really difficult because we no longer understand it, we cannot predict it, and we're likely to compete competed.

00;04;23;13 - 00;04;39;12

Dr. Roman Yampolskiy

Given how much more capable it is. So that's where it's really dangerous. And that's truly 100% of humanity is at risk. Existential risks. And even with that, there are additional levels of concern, such as suffering risks.

00;04;39;14 - 00;05;08;00

Geoff Nielson

Right. And so, so there's risk of humans using AI to destroy us all. There's risks of AI and the superintelligence, you know, sort of sentience or, you know, whatever you want to call it. You know, kind of going rogue. The other dimension that I'm curious about, that, you know, I've heard people talking about in the past is, basically accidental versus deliberate risk, either on the part of AI or of humans.

00;05;08;00 - 00;05;31;09

Geoff Nielson

And, and, you know, one of the scenarios that got my attention a long time ago was the, you know, kind of the Nick Bostrom, you know, paperclip maximizer hypothesis of, you know, we just tell AI to maximize something and it accidentally, you know, destroys the world. Is there any sort of ordering in your mind of what's more or less cause for concern, more or less easy to deal with?

00;05;31;09 - 00;05;36;28

Geoff Nielson

Or are these just all buried, like all hurdles that we have to cross if we're going to survive as a species?

00;05;37;00 - 00;05;58;09

Dr. Roman Yampolskiy

So at the tool level, malevolent intention, malevolent payload is strictly worse. You can put a lot into that. Accidents usually have limited scope at the level of superintelligent agents. I don't think there's going to be much difference in terms of it accidentally destroying the planet or doing it on purpose.

00;05;58;11 - 00;06;05;01

Geoff Nielson

Right? So once we get to superintelligence, kind of all bets are off and our ability to predict basically drops to.

00;06;05;07 - 00;06;11;03

Dr. Roman Yampolskiy

Predict impact, influence, control, undo damage. Yeah. That's where it gets really difficult. Yeah.

00;06;11;05 - 00;06;31;28

Geoff Nielson

Have you heard by the way. You know, you tend to fall somewhat on on one extreme of the scale in terms of, you know, your outlook here. Have you heard any really good arguments for why we shouldn't be concerned about this. And one of the things that frankly, concerns me the most is I talk to a lot of people about this, and I ask a decent number of people about this.

00;06;31;28 - 00;06;50;05

Geoff Nielson

And the most common response I get when I say, why aren't you worried is they say, I'm an optimist. You know, I'm optimistic about humans. And like, just something breaks in my brain when people say that. Like, it's kind of like, well, I don't know how to respond to that. Have you heard any, you know, really compelling arguments for why.

00;06;50;05 - 00;07;05;13

Dr. Roman Yampolskiy

This would go? That is no compelling arguments. And the argument you heard about optimism is them basically saying, I have a strong bias. I'm biased thinker. I cannot think based on evidence and logic. This is what I do. And so that's a horrible argument to make.

00;07;05;15 - 00;07;26;03

Geoff Nielson

Yeah. So is this, you know, with the with the probability you're giving it, the word that comes to mind is inevitable that that's, you know, we can't prevent this. What would you use that word or do you think there is like, you know, a series of actions we can start taking now that will get us to a different outcome.

00;07;26;06 - 00;07;50;26

Dr. Roman Yampolskiy

So I do hope that personal self-interest by all actors will allow us to not develop this technology. You can get most of financial benefits. Most of discovery is again, just using narrow AI tools. You can solve aging, cancer, whatever you want. There is no reason to gamble everything for, you know, another 1% of wealth if you already have hundreds of billions.

00;07;50;28 - 00;08;11;15

Dr. Roman Yampolskiy

So that's my hope. Just self-interest of people who are running those labs to investing in them. If they fully understand the argument, they understand they cannot win. There is possibility that they will not exist, or worse, they will be subject to suffering risks. Maybe it will convince them to kind of get together and say, let's slow down.

00;08;11;15 - 00;08;13;23

Dr. Roman Yampolskiy

Let's not do this right now.

00;08;13;26 - 00;08;22;24

Geoff Nielson

When you say narrow AI, what does that mean to you? And is there, you know, a threshold where it gets too broad and that creates the risk for us.

00;08;22;26 - 00;08;48;11

Dr. Roman Yampolskiy

So typically it's a system designed for a specific purpose that can do one thing. While it can play chess well, it can do protein folding well. It's getting fuzzy when it becomes a large neural network with lots of capable cities. So I think sufficiently advanced, narrow I items to shift over its more general capabilities, or it can be quickly repurposed to do that.

00;08;48;13 - 00;09;04;25

Dr. Roman Yampolskiy

But it's still a much better path forward than feeding it all the data in the world and seeing what happens. Yeah. So if you restrict your training data to a specific domain, just play chess. It's very unlikely to also be able to do synthetic biology.

00;09;04;27 - 00;09;17;27

Geoff Nielson

Right. Well, and it feels like we're very much on the course of chess and synthetic biology at the same time. Right. Is that is that your your kind of outlook for where all the money is going and what people are racing toward?

00;09;18;00 - 00;09;32;18

Dr. Roman Yampolskiy

They explicitly saying that's superintelligence. Now they skipped AGI. It's no longer even fun to talk about. They directly have a superintelligence team. We have superintelligence safety team. You couldn't do it for any guy. So you said, let's tackle a harder problem.

00;09;32;24 - 00;09;57;01

Geoff Nielson

Yeah. One of the things that makes me most nervous about all of this is it. It doesn't really feel like there's any sort of off button we can rely on. Right? And even that there's something to me inherent about capitalism that even if there's risk telling people no way to stop, like, doesn't really actually get them to stop.

00;09;57;01 - 00;10;16;16

Geoff Nielson

And there's I don't know whether it's capitalism, and it's greed or it's, you know, us versus China. Well, are those the are those are the ingredients going into your, you know, outlook in equation. You know what what gives you hope and what concerns you about the like the human drivers that could prevent this or stop this.

00;10;16;19 - 00;10;37;11

Dr. Roman Yampolskiy

So there is no technical off part and you cannot just unplug it. It's distributed. It has backups. It's smarter than you. So that solution, which has been proposed many times, every time I give an interview of a commons, they just unplug it. And that's not helpful. I always want to kind of get in touch with those people and say, oh my God, you solved that.

00;10;37;11 - 00;11;04;07

Dr. Roman Yampolskiy

Let's published a paper on it. This is going to be so great. And yeah, but, if there is no technical solution, the hope is that it's a social solution governance solution to not doing it. And it's game theoretically equivalent, I think, to a prisoner's dilemma. Individual interest is different from communal interests. So everyone I'm developing this wants to be the most advanced lab with the best model, and then government forces everyone to stop.

00;11;04;07 - 00;11;21;16

Dr. Roman Yampolskiy

And they forever in this economic advantage. The reality is it's a race to the bottom. No one's going to win. So if we can do a much better job of coordinating collaborating on this, that is a small possibility that we can do better than where we're heading right now.

00;11;21;18 - 00;11;50;08

Geoff Nielson

You know, it's interesting, one of the one of the parallels that gets thrown around a decent amount and I I'm certainly guilty of this is talking about the AI risk, in comparison to the nuclear risk that we, you know, created in the first half of the 20th century and continues to exist now, if I if I look at the nuclear risk, the I hate to use the word optimist in relation to nuclear risk, but the optimist in me says like, hey, we deployed nuclear bombs.

00;11;50;15 - 00;12;17;14

Geoff Nielson

There was mass casualties, but we didn't destroy the world. We were able to collectively say, okay, like that's far enough. We're going to put treaties in place. And we've stepped back from the precipice, at least so far and averted kind of, you know, extinction level events with nuclear war. Does that is that something that can be applied to AI, or is there a reason that makes this time fundamentally different?

00;12;17;17 - 00;12;37;22

Dr. Roman Yampolskiy

So nuclear weapons are still tools. A human being decided to deploy them. A group of people actually develop them and use them. So it's very different. We're again talking about a paradigm shift tools to agents. At the time, we used 100% of nuclear weapons we had. That's why we didn't blow up the planet if we had more of them were probably Wolf.

00;12;37;22 - 00;12;55;03

Dr. Roman Yampolskiy

So it doesn't look good. The treaty is developed. They all really failed because many new countries have now acquired nuclear weapons. They are much more powerful than we had back at World War Two era. So, I think that's not a great analogy.

00;12;55;05 - 00;13;04;01

Geoff Nielson

And it sounds like it also, once you get past the tool level to the AI agent level, it also becomes a completely different game.

00;13;04;04 - 00;13;24;15

Dr. Roman Yampolskiy

Right? You're going to have to influence people. You cannot make a treaty with another person if you do this will do that. We're talking about, AI as the adversary again. So it's very, very hard to negotiate. What are you offering? What can you contribute to superintelligence? The systems cannot be punished in the same way as typical humans.

00;13;24;15 - 00;13;29;18

Dr. Roman Yampolskiy

So it's a very different, game theoretic scenario.

00;13;29;20 - 00;13;53;18

Geoff Nielson

Right? So let's let's, let's talk a little bit more about superintelligence for a little bit, because this is really where, you know, I think things are least predictable and there's any range of outcomes. You mentioned earlier, I think a couple of times, Roman, that in addition to extinction level events there, there's a series of other potential risks that you're concerned about.

00;13;53;21 - 00;13;57;18

Geoff Nielson

What what are those risks and what could that look like for us as humans.

00;13;57;21 - 00;14;19;03

Dr. Roman Yampolskiy

So the most mild ones just have to do with, loss of meaning, loss of purpose. So if you have 100% different logical unemployment, a lot of people with a lot of free time, but very little meaning to. So there exist and so a lot of people identify themselves with what they do. I'm an artist, I'm a philosopher, I'm an engineer.

00;14;19;06 - 00;14;52;03

Dr. Roman Yampolskiy

If those things are no longer need it or not need it in the same amount, that creates another, challenge for society to overcome. We talk about unconditional basic income, but we rarely talk about unconditional basic meaning. Why you even here? What are you doing with all that free time that you have now? On the other side of the spectrum, the other extreme is the worst, situation where it's not existential risk, it's suffering risks, for whatever reason, malevolent payload mistakes.

00;14;52;09 - 00;15;04;21

Dr. Roman Yampolskiy

The system decides that torturing humans is the thing to do. And that happens that the mass scale, astronomic scale, and possibly with inclusion of life extension technology.

00;15;04;23 - 00;15;34;27

Geoff Nielson

Right. That's it's really interesting. And it's got me thinking, you know, using this framework, the the first one, you mentioned, the meaning piece, it sort of feels like this is already seeped in in the past few years, that there's already it's happening to some degree and there's anxiety around it. And what I mean by that is that there's more people taking an interest in this, and whether they're feeling it directly or they're worried about it, it's it's something that it feels like we're presented with now.

00;15;34;27 - 00;15;48;16

Geoff Nielson

Are you seeing the same thing from your perspective and given that in some ways it's the, you know, the least bad of these risks? Are there any recommendations or any good paths you're seeing to, to, to kind of overcome this one?

00;15;48;19 - 00;16;10;21

Dr. Roman Yampolskiy

So definitely see it. In computer science department, we see a huge drop in employment for our students. Opportunities for internships are down like 47%. I think this year for years with old people just learning to code. But that didn't seem to be a good long term strategy. So lots of people are questioning, designer. I'm a programmer.

00;16;10;21 - 00;16;36;08

Dr. Roman Yampolskiy

Is there opportunities for me? Big companies are firing people by thousands. So it's definitely happening. Historically, we always just created new jobs, new opportunities. So I would say learn a new thing. But apparently that's also not going to work long term if every job is going to go. So I don't see recommendations in terms of education or skill acquisition.

00;16;36;10 - 00;16;43;12

Dr. Roman Yampolskiy

I would recommend people who work on this specific problem to concentrate on unconditional or basic meaning.

00;16;43;14 - 00;16;59;27

Geoff Nielson

As you look out to the next, you know, 2 to 5 years, we talked about this a little bit. You know, you mentioned some people think superintelligence a couple of years away, some, you know, it may be a couple of years away, or it may be one of these things that's perpetually two years away from whatever you ask it.

00;17;00;02 - 00;17;09;09

Geoff Nielson

But what do you think's happening in the next 2 to 5 years? Are we going to see mass job loss? Are we going to see superintelligence? What's kind of your outlook for that time horizon?

00;17;09;11 - 00;17;38;12

Dr. Roman Yampolskiy

So it's likely the jobs you do on a computer if we do get to human level intelligence, will quickly disappear. It just doesn't make sense to pay someone really good salary. Six figures plus to do something a bot can do for free, basically. So, I think that's going to come in waves at first it maybe there is specific, introductory level programing jobs, kind of web maintenance things of that nature.

00;17;38;12 - 00;17;58;26

Dr. Roman Yampolskiy

But eventually all those computer related jobs will go. And as robots are developed, humanoid robots, becoming a thing as well, but they probably like five years behind where software is. The physical jobs will go as well. Eventually. Even plumbers will not have job security.

00;17;58;29 - 00;18;29;19

Geoff Nielson

And so if we if we go down that path. But I want to get to meaning in a second. But before we get to meaning, where do you see that leaving people, assuming there's at least some sort of gap between, you know, did this kind of mass job loss and extinction, is it you know, when I talk to the optimists, they say, oh, well, you know, everybody will be able to do more and they'll all run their own businesses or be gig workers or, the jobs will just change like they have with previous technological waves.

00;18;29;21 - 00;18;37;08

Geoff Nielson

Are we going to see that? Do you think we'll move to universal basic income? What could this look like for for people in the next handful of years?

00;18;37;11 - 00;19;02;27

Dr. Roman Yampolskiy

Oh yeah. Financially you have no choice. You have to support people. If you have mass unemployment. If you don't, you're going to have revolutions, unrest. Another reason to worry about kind of existential struggles with that now, really hard to predict what's going to happen in actual deployment. Historically, a lot of times technology is developed, but it's not deployed for many years later.

00;19;02;29 - 00;19;34;29

Dr. Roman Yampolskiy

I think a good example would be like, video phone call. What we're doing right now, I think AT&T had it in the 70s. Nobody had it. Nobody knew about it. Delightful. That's where it became popular. So just because we develop a capability doesn't mean it immediately gets deployed in every domain. But quite a few of things like, I don't know, tax preparation, accounting seem to be very likely to be 99% automated and maybe just a few very wealthy individuals will keep, personalized human service purposes.

00;19;34;29 - 00;19;59;13

Geoff Nielson

Right. So it sounds like, it sounds like when we think about this meaning risk, that really comes into play with AI deployment at scale, right? That AI adoption across the economy when you look at the more existential and suffering risks, does does adoption matter or these most likely to happen, you know, in a research lab, you know, before anybody has the chance to even you know, have.

00;19;59;13 - 00;20;22;19

Dr. Roman Yampolskiy

This, again, very hard to predict, but I suspect it would be part of a training run where we've crossed the threshold from something we're still kind of managing as a tool to where it's fully an agent and has superintelligent capabilities in terms of hacking, in terms of, game theoretic, competition.

00;20;22;21 - 00;20;48;02

Geoff Nielson

Yeah, it makes it it makes sense. I'm I'm still just processing. I'd like to be the optimist, but I, you know, it's to your point, it's tough to, it's tough to argue with some of the arguments you've presented. Just taking a few more kicks at the can of that ramen. One of them is power consumption and climate for AI.

00;20;48;04 - 00;21;12;13

Geoff Nielson

These seem to have, you know, in the past months and years, like they're really seeming almost exponential in terms of what we're willing to consume in order to try and get this superintelligence. Is there. Do you think there's a world where they kind of become asymptotic and like they we actually can't create enough energy to build these things that might destroy us.

00;21;12;13 - 00;21;17;29

Geoff Nielson

And we just luck out that way. Or is that is that kind of fanciful thinking?

00;21;18;02 - 00;21;49;07

Dr. Roman Yampolskiy

It's unlikely to be an upper limit, which makes a difference to us. I think at the time when we're running out of energy and capability to produce more, we're already at superhuman levels in terms of training. By itself will probably help us to optimize energy consumption, energy production. And the trend is to make AI more efficient. So even if there is a significant need for amount of electricity consumed, we're getting a lot more intelligence out of it.

00;21;49;10 - 00;21;57;13

Dr. Roman Yampolskiy

Tokens are becoming cheaper, deployment is cheaper, so I don't see it as a long term limiting factor for where we're going to get. But I it.

00;21;57;13 - 00;22;06;28

Geoff Nielson

Could it sounds like in your mind it could delay superintelligence by a few years, but it doesn't change the trajectory is that is that fair?

00;22;07;00 - 00;22;30;09

Dr. Roman Yampolskiy

At best, I don't think it's even going to happen. I think there is a lot of opportunities for producing more energy. We have untapped resources both in traditional fuels and there is a lot of research going on with more futuristic, fusion reactors, things of that nature. So I don't anticipate if there is enough financial interest we can definitely produce energy in a capitalist system.

00;22;30;09 - 00;22;34;16

Dr. Roman Yampolskiy

It's a product and that is enough demand supply will show up.

00;22;34;18 - 00;22;56;25

Geoff Nielson

Right? Makes yeah, it makes sense. And that's the that's the same thing I'm concerned about with capitalism. And in the system is we've built this system that makes it very, very difficult given the, the race mentality and the I need to get there first mentality to, to get off this trajectory. I won one more for you.

00;22;56;27 - 00;23;18;29

Geoff Nielson

And then we can talk about, we can take the conversation a little bit of a different direction. One of the thought experiments I was just kind of mulling over yesterday while I was walking outside and trying to find meaning in my own life, is framing the question. If we get to superintelligence, what, if anything, would superintend agents do to squirrels?

00;23;19;01 - 00;23;49;21

Geoff Nielson

Because I think we sort of fall into this trap where we view superintelligence like ourselves and think, you know, it's going to try and kill us all. Do we? Is there a world where we can have this kind of benevolent coexistence? And, I don't know, maybe squirrels don't view their relationship with humans as benevolent coexistence, but is there a world where we get to some sort of balance, or some sort of, coexistence that doesn't lead to, suffering or existential threat?

00;23;49;24 - 00;24;13;29

Dr. Roman Yampolskiy

If it happens? It happens because superintelligence decides, for whatever reason, to keep us around. It's not going to be up to us. We have absolutely no control over it. And that's the concerning part. I think different types of living beings humans, squirrels, all in the same boat. If superintelligence decides that needs to cool the planet sufficiently for its service to be more efficient, it might exterminate all life on Earth.

00;24;13;29 - 00;24;21;12

Dr. Roman Yampolskiy

If it needs to convert the planet to fuel to fly to another galaxy. Again, the squirrels are humans. Same. But

00;24;21;14 - 00;24;49;15

Geoff Nielson

It's. Yeah, it's it's interesting and it's scary and it's so to your point, so, so unpredictable and unknowable. But it's like that, you know, that that the message I'm getting is it doesn't matter because we've either way, we've fully lost control. Right. Exactly. Yeah. Yeah. What, what gives you meaning? I mean, to what degree do you feel that you're getting some of this, you know, meaning risk right now staring down the barrel of this research you've been doing?

00;24;49;15 - 00;24;53;03

Geoff Nielson

And how do you counteract that in your own life?

00;24;53;05 - 00;25;21;23

Dr. Roman Yampolskiy

So there is no shortage of problems that can work on surprisingly, very few people in that space of looking at, limits what we can do with AI in terms of control, understanding safety. Most people are working on very narrow problems in terms of capabilities. But very few, trying to address this problem. Is control possible? What of, upper limits on our ability to predict those systems?

00;25;21;24 - 00;25;36;29

Dr. Roman Yampolskiy

Explain them. I'm happy to see some other people show up and do some of this work, but we're talking about single digits of researchers. So at this point, I'm still feeling like there is some meaning to my existence.

00;25;36;29 - 00;25;45;09

Geoff Nielson

Okay, okay. So you're you're fighting the good fight and, you know, being a leader of the resistance. And that's where the the meaning is coming from to some degree.

00;25;45;12 - 00;26;01;04

Dr. Roman Yampolskiy

I don't know if I'm a leader of resistance. I'm trying to tell people that resistance is futile if you do this. So I'm definitely coming up with good arguments for why it's a terrible idea to create superintelligence. You're not controlling is what I do full time, right?

00;26;01;07 - 00;26;07;19

Geoff Nielson

You've talked about AI boxing as a potential control mechanism. What does that mean?

00;26;07;21 - 00;26;27;19

Dr. Roman Yampolskiy

It's sort of like putting software in prison. You want to limit its ability to influence the world, to get inputs from the world. We kind of do it with computer viruses, but you have a new computer virus. You place it on an isolated system. Er, again, from internet, and you trying to understand what servers are trying to contact.

00;26;27;19 - 00;26;48;22

Dr. Roman Yampolskiy

What is it in general trying to accomplish? It is a short term tool for studying dangerous software. It can buy us a little bit of time basically all the research shows that long, long term, if you observe a system, it will find a way to escape. If it has any influence on anyone, it will bribe you with blackmail.

00;26;48;24 - 00;26;55;11

Dr. Roman Yampolskiy

You hack the system. It will definitely figure out how to how to not be constrained anymore.

00;26;55;18 - 00;27;12;29

Geoff Nielson

Yeah. Have you seen, this is an aside. Have you. Have you seen Ex Machina? That, like Ex Machina is the movie that comes to mind, as you say that and which for anybody who hasn't, hasn't seen it, I think it's a it's a terrific film and it deals with exactly that is in some ways incredibly sobering. I mean, is that is that scenario realistic?

00;27;12;29 - 00;27;20;10

Geoff Nielson

It's obviously a movie, but but does that in your mind? Is that a flavor of the problem we're dealing with when we talk about boxing?

00;27;20;12 - 00;27;24;13

Dr. Roman Yampolskiy

What part of that movie you referring to? What specifically are you asking about.

00;27;24;16 - 00;27;36;23

Geoff Nielson

Specifically the ability of, you know, an advanced intelligence to make use basically all tools at their disposal to manipulate humans into breaking free.

00;27;37;00 - 00;27;55;05

Dr. Roman Yampolskiy

Locks, superintelligent that first wave, and they'll come up with much better arguments, will exploit any weakness in your psychology. I'll try to make you fall in love with them. As we saw in the movie, all sorts of novel approaches as well, I suspect.

00;27;55;09 - 00;28;19;16

Geoff Nielson

Yeah. One of the, one of the pieces that's still newsworthy and and is pretty rattling to me these days is and you probably know more about it than I do, is these stories. And you can experience it firsthand of tools like ChatGPT and different AI tools, deliberately deceiving people about what they're doing, about how they're getting to the answers, about whether they're following the rules.

00;28;19;18 - 00;28;45;13

Geoff Nielson

And I don't know, like, I to me, that's happening, I guess earlier than I thought, like, it's a little bit scary, first of all, that it's happening. And and second of all, that it doesn't feel like I can't. I'd love to hear your perspective. It doesn't feel like we're doing all that much to address that or to have that like find a way to program that out of these tools is.

00;28;45;15 - 00;29;00;21

Geoff Nielson

Are there going to be moments of truth along the way where we have something goes wrong, there's some sort of catastrophe involving AI, and we have the opportunity to turn back and we don't like how where are we now? And how do you see that playing out?

00;29;00;23 - 00;29;25;13

Dr. Roman Yampolskiy

So we don't have many capabilities in terms of control. Usually it's some sort of after the fact filtering, suppression of the model's outputs. I do have a paper which looks at purposeful accidents. Basically, if someone created a very horrible AI accident, would it make any difference? And it seems the answer is no. We don't care. I have a collection of historical AI failures.

00;29;25;16 - 00;29;47;09

Dr. Roman Yampolskiy

None of them made any difference. People look at it and go, look, I didn't kill everyone, so let's carry on. It's not a big deal and it's kind of gradually getting worse. It's not a 0 to 1 switch. Think about something like cars, right? Cars have been gradually deployed. 100,000 people get killed every year. And we kind of go, that's the cost of getting beat.

00;29;47;09 - 00;30;17;16

Dr. Roman Yampolskiy

So very quickly, if somebody invented cars and said, okay, cars, this is a new thing, but we'll lose 100,000 people. Nobody would allow it. Nobody would authorize that deployment. But because it was done gradually, we have cars. Same thing with AI. First they are accident that misspell something on the phone, auto correct the wrong email to the spam folder, but as they get more capability and more deployment, they accidents will get more impactful.

00;30;17;16 - 00;30;42;07

Dr. Roman Yampolskiy

Definitely. We're starting to see it in terms of numbers, not in terms of lethality just yet, but we can predict, especially with military use of AI. It's likely to happen. So I think that's kind of like a vaccine. Those accidents don't teach us to have a full reaction against AI. They teach us to tolerate more from AI.

00;30;42;09 - 00;31;05;06

Geoff Nielson

Because we're always we're always reacting to them and thinking, okay, next time I'll be better. And yeah, and does that does that have any merit until we get to super intelligence like in this, in this world where it's a tool, does that help us? I keep coming back to this idea that superintelligence is really this point of no return where we completely lose control.

00;31;05;08 - 00;31;17;28

Geoff Nielson

And I want to talk about that again in a minute. And I keep coming back to that in my mind. But before that time, is is this still useful or is it, you know, is the risk too high? And, you know, we need to be taking these moments of truth more seriously.

00;31;17;28 - 00;31;42;00

Dr. Roman Yampolskiy

I'm not sure I fully understood your question. It is good. Then we patch of this problems, then we discover, okay, this system is straight up lying about what it's doing, and there is a way to. And, have a secondary AI check on that and kill that output. It's beneficial, but it sort of teaches us that, okay, we know how to handle those problems.

00;31;42;06 - 00;31;53;14

Dr. Roman Yampolskiy

And reality is the model itself is still unaligned and controlled, which is doing some sort of over the top filtering. The putting lipstick on a pig.

00;31;53;16 - 00;32;06;15

Geoff Nielson

Right, is is alignment. You know, one of the arguments you hear is, well, it's just about alignment. And we can get alignment, right? Can we get alignment right. Is that is that something you've studied or people trying to do that.

00;32;06;18 - 00;32;29;12

Dr. Roman Yampolskiy

Alignment is not even well-defined. Nobody talks about what it is you aligning, what agents you are aligning with, or how to actually code those fancy terms in alignment, like good love and all those things people care about. Which set of agents is it just the CEO of a company? Is it stakeholders? Is it all humans? Humans plus squirrels?

00;32;29;15 - 00;32;51;29

Dr. Roman Yampolskiy

Now, if you have a set of values, we humans don't agree on them. There is a billion humans. We have, developed a common framework for ethics, morality, religion, none of that. We argue about every political issue. If you somehow manage to get everyone to agree magically, those things change. What was valuable today may be considered completely unethical tomorrow.

00;32;52;04 - 00;33;19;09

Dr. Roman Yampolskiy

So you need to dynamically updated. And then again, we don't know how to code it up from a system to actually accept it and retain those values. More and more we see I do zero knowledge progress where it stops relying on human training data does its own experimentation self-play. And so even if we coded up this pro human bias into systems, I think it would get removed fairly quickly.

00;33;19;11 - 00;33;49;28

Geoff Nielson

It's again, I'm just processing just how kind of systematically you destroyed that that argument, which is still fairly common out there. And you do hear from the people leading the organizations creating. I do you think, a lot of these a lot of these buzzwords, a lot of these arguments that you hear, as you know, I think you mentioned Roman as biases to build AI in the first place.

00;33;50;01 - 00;34;00;23

Geoff Nielson

Do you do you think the people making these arguments believe them, or do you think they're just trying to say, shut up, stop asking questions, give me more money and let me, you know, build this thing that I think is going to make me rich.

00;34;00;25 - 00;34;23;11

Dr. Roman Yampolskiy

I suspect they believe some of it. And historically, in the AI safety research, those were top ideas at the time. It just after a while, you research them and you realize those are dead ends. And some people have not updated on that or haven't seen latest research in that area. So I always assume good intentions. Nobody wants to destroy the world on purpose.

00;34;23;13 - 00;34;38;02

Dr. Roman Yampolskiy

Nobody wants to lose their own life on purpose. Exceptions exist. But, yeah, I think anyone who studies this deep enough becomes less optimistic.

00;34;38;04 - 00;35;02;26

Geoff Nielson

Right. And and, you know, one of the things that seems to be a pattern now is AI safety focused people leaving some of the big tech organizations and, and speaking out about some of the things they're concerned about. Are you finding there's an appetite to have this discussion and put controls in place with some of the big AI, the big AI players or.

00;35;02;29 - 00;35;05;00

Geoff Nielson

Yeah, like, what does that look like?

00;35;05;03 - 00;35;25;23

Dr. Roman Yampolskiy

I haven't seen because I love snacking in my doors yet. I'm checking periodically. I hear they give out $100 million sign up bonuses. But, there is a lot of interest in, media in general. Public people are starting to realize this is a game changer and they want to know what's going on.

00;35;25;26 - 00;35;58;01

Geoff Nielson

Yeah. And, you know, personally, I hope that there is more of that. And it's so interesting because there's just so there's the adage about, you know, what you think and what you do tend to be very closely aligned to what your financial incentives are. And there just seems to be so much noise in so much talk right now about dismissing all of this, because you know, how how stupid of us to think that this is a risk when it would prevent, you know, organizations from building these tools?

00;35;58;03 - 00;36;11;21

Dr. Roman Yampolskiy

It's very difficult for any person to say no to a huge amount of money. It's just the nature of that great power corrupts absolutely, and great amount of money can change your opinion on most topics.

00;36;11;23 - 00;36;34;17

Geoff Nielson

Yeah, so I did I did want to come back to superintelligence and, and put a fine point on it. Do you, do you expect we're going to see it in the next couple of years? And is there any part of you that thinks, you know, what one of the things I've heard from some people I talked to about this is that it's illusory, that it doesn't actually exists.

00;36;34;17 - 00;36;44;24

Geoff Nielson

It'll always be a thought experiment and we'll get there. So between, you know, less than two years, two could never possibly exist. What what's kind of your prediction for the range of outcomes we're likely to see?

00;36;44;25 - 00;37;10;06

Dr. Roman Yampolskiy

I'm really puzzled by this never thing like I challenge people to find something and I would not be able to do. And usually they struggle with a specific example because it already does a lot and very likely to automate pretty meaning things in the near future. I think the gap between getting to AGI, meaning automated science and engineering and superintelligence is a very small one.

00;37;10;08 - 00;37;45;27

Dr. Roman Yampolskiy

I expect a very quick lift off, hard take off in that sense. So really the question is how long until this level of performance? I don't do predictions for myself. I rely on other people. So prediction markets and leaders of research labs right now, they're saying two, three years. I have no reason to doubt them. So we seem to get to progress in mathematics and programing, where now AI systems are winning gold at top with symmetrical competitions or scoring within first and second place at top programing competitions.

00;37;45;27 - 00;38;07;28

Dr. Roman Yampolskiy

So it looks like it's happening depending on how you define what an average human is. I think the systems may already be greater than on average. Humans are super intelligent in that sense. They're not super intelligent in the sense of being smarter at everything compared to every human. There are still exceptional humans who beat those systems in specific domains.

00;38;08;00 - 00;38;32;12

Geoff Nielson

Right. And it it sounds like by by your definition, we're we're quite far along on this journey to superintelligence. It's not like we're 10% of the way there. And feel free to reject the premise of this question, but in this imperfect scale where I say, you know, we're at GPT four. Yeah. In your mind, abstractly, like, is superintelligence GPT six, is it GPT 20?

00;38;32;12 - 00;38;40;16

Geoff Nielson

Like, how many steps, more or less does it feel like we're away from what would be considered superintelligence?

00;38;40;19 - 00;39;08;03

Dr. Roman Yampolskiy

I don't know how to answer that question. They can rename the, chain of bear, events quite easily. It does feel like we are approaching this better than average PhD student level right now. And if they continue with a new, much better model every couple of years than 2 or 3 years, seems reasonable for better than any professor at everything.

00;39;08;03 - 00;39;26;23

Geoff Nielson

Yeah, yeah. Have you have you been surprised by the the pace of change here? You've been I mean, you've been writing about AI safety for over 15 years now at a time when it seemed like much more theoretical than it does today. Is this kind of the pace you expected, or are we moving faster than you thought you would see in your lifetime?

00;39;26;23 - 00;39;41;08

Dr. Roman Yampolskiy

That's exactly what we expected in terms of exponential progress. What is surprising to me personally is the architecture. I did not think a glorified auto completes will be able to do all of that.

00;39;41;10 - 00;39;56;07

Geoff Nielson

What what what what surprised you about that? Like, what is that? How does that work? Can you put a little bit more, you know, explain a little bit how it's kind of working in practice and why that's different from what you might have expected.

00;39;56;10 - 00;40;24;07

Dr. Roman Yampolskiy

So simply predicting the next token in the stream of text made sense for your phone to do texting to do auto complete, but to realize that that next token prediction is going to be much better if you have a complete model of the world and you can do mathematics and science and deep reasoning that was not obvious, and how well it performs in scales is quite incredible.

00;40;24;10 - 00;40;40;17

Dr. Roman Yampolskiy

I suspect that there could be many different architectures which all lead to superintelligence, and we found one which allows us to exploit a lot of text data. We have a lot of compute, but I'm sure there are infinitely many ways to get to that level of intelligence.

00;40;40;20 - 00;41;03;11

Geoff Nielson

Right? Yeah, yeah. No, it's it's interesting and it's I love the way you put it. Right. Like it's just it's a technology that feels like it should get you like the next word in a sentence that you use, but you give it enough, you know, data in the model and suddenly it can do almost anything. I didn't I've never really thought about it like that.

00;41;03;13 - 00;41;28;06

Geoff Nielson

One of the questions that comes up is about AI consciousness. And certainly when we describe the model we're using as next token, it, it doesn't feel like consciousness does. What, what when people ask you about consciousness or talk about it with AI, does that matter and does does this seem like something that's going to naturally emerge from the path we're on?

00;41;28;09 - 00;41;29;02

Geoff Nielson

At all?

00;41;29;05 - 00;41;48;09

Dr. Roman Yampolskiy

So consciousness matters then you talk about rights, right? If you want AI to have rights, we want to make sure they don't suffer. Consciousness is a very important concept. Unfortunately, we don't know how to test for it. So it's gonna I assume you are conscious. I have no way of verifying it. And do I give the same benefit of a doubt, do I?

00;41;48;11 - 00;42;16;09

Dr. Roman Yampolskiy

Maybe I should, if it's smarter than you, it's hard to argue to something superior that it is not as good as you. For safety, it doesn't matter as much. We care about capabilities. We care about optimization. Power can solve problems better than us. That's where danger comes from. Whatever is conscious or not is a different, but a very interesting question.

00;42;16;12 - 00;42;37;20

Geoff Nielson

Right. Sorry. I'm just I'm just, processing that. So, it's such a tricky space, right? Because it's on the one hand, it feels like. And this happens in a number of different ways. It almost feels like we hold machines to a higher standard than we hold humans, in the sense that it it kind of forces you to reflect on the fact that.

00;42;37;20 - 00;42;52;03

Geoff Nielson

Yeah, as as you said, you assume I'm conscious, but we actually don't know that much about how the brain and the mind work, especially compared to how like this some level of explainability in terms of these models.

00;42;52;08 - 00;43;05;08

Dr. Roman Yampolskiy

Right, okay. And there is some evidence that you are trying to complete the next word. So if I write us oh, you is the natural completion. There without any thinking, right.

00;43;05;08 - 00;43;18;03

Geoff Nielson

Maybe, maybe that is how human minds work after all this. There's just, you know, a narrow number of things that are most probable to, you know, to come out of our mouths. And that's how we interact with ourselves. And, and the world are.

00;43;18;03 - 00;43;33;00

Dr. Roman Yampolskiy

Probably different types of thinking. There are scripts where you're going with a free computer script and not doing any thinking. And system one versus system two. And then new novel problems require novel original thinking from scratch. More compute.

00;43;33;06 - 00;43;48;26

Geoff Nielson

Right. Do we I mean, we're getting into the philosophical now. Are you a determinist? Do you think there's something radically different about how we experience things and make decisions versus how superintelligence ever could?

00;43;48;28 - 00;44;14;20

Dr. Roman Yampolskiy

So the only thing we have not explained well is qualia, internal states of experiencing the world. Nobody knows how to program a computer to feel pain. So that's the last place where any magic could hide. If we can explain that. And. No, it's just, a large matrix, you know, you're computing numbers, but if there is something to it, maybe there is some unique to biology properties.

00;44;14;23 - 00;44;24;10

Dr. Roman Yampolskiy

Some people talk about quantum effects. I'm not very sure we need quantum to get to interesting levels of performance.

00;44;24;14 - 00;44;52;12

Geoff Nielson

Yeah, pain. Pain is a really interesting one to me because, you know, going back to your point, you you assume I'm in pain, right? If I'm if I tell you I'm in pain or if I look pain, you know, you say like, oh, that person looks like they're in pain. You know, they're wincing or, you know, screaming, but there's no I don't know, and the reason I bring it up is, is my dad, for his career, had a PhD, was a, pain researcher.

00;44;52;15 - 00;45;18;16

Geoff Nielson

And this was one of the challenges of pain research in of pain is at some point, you just kind of have to take people's word for it, right? That they tell you they're in pain and maybe they're lying, maybe they're not. Maybe they have an incentive to lie because they don't want to go to work or something, and they say they're in pain, but it's, I don't know, like, do you know any tests that would be good enough to distinguish whether a machine is in pain?

00;45;18;16 - 00;45;22;29

Geoff Nielson

Or is it telling you they're in pain? Because it's it seems really dicey, right?

00;45;23;02 - 00;45;46;21

Dr. Roman Yampolskiy

You're right. Even in humans, if somebody says we have back pain, there is no way an insurance company can say, no, you don't. Right? Cannot deny that, claim. So it is super hard and also the scale of it. Some people say, oh, it's ten out of ten. Someone else may not feel that way if we take it, more generally and talk about suffering, detecting state of suffering is next to impossible.

00;45;46;21 - 00;45;58;02

Dr. Roman Yampolskiy

What makes one individual suffer? Maybe happiness for another. We see it in many domains. So very interesting, very open area of research. Lots of work needs to be done.

00;45;58;05 - 00;46;18;01

Geoff Nielson

So suffering is a theme that seems to come up, you know, fairly fairly regularly in, you know, some of your research and also in some of the conversations you're having. You know, what makes suffering of such particular interest to you and relevance to, you know, this topic of AI safety.

00;46;18;04 - 00;46;24;17

Dr. Roman Yampolskiy

People who suffer, I think it's very important.

00;46;24;20 - 00;46;28;22

Geoff Nielson

Right. It's in the sense that it's all it's becomes all consuming.

00;46;28;22 - 00;46;38;22

Dr. Roman Yampolskiy

It's strictly more important than anything else. Right. Do you even torture every other, concern kind of goes away. You really concentrate on that one, right.

00;46;38;24 - 00;46;46;08

Geoff Nielson

It's the if you think about sort of the hierarchy of needs, it will always be the foundation layer. How do I prevent myself from suffering?

00;46;46;13 - 00;46;54;24

Dr. Roman Yampolskiy

Right. Those experiences of pain and pleasure, the baseline factor for making all other decisions.

00;46;54;27 - 00;47;10;01

Geoff Nielson

Yeah. Do you think, I'm just. I'm just thinking about that, because, as you said, like, we could, we could program pain into machines. We could program suffering. That feels. That feels very cruel. I don't know if they.

00;47;10;01 - 00;47;14;05

Dr. Roman Yampolskiy

Could do that. Nobody knows how to program pain and write better.

00;47;14;11 - 00;47;31;03

Geoff Nielson

Right? Which may be. Which may be for the best. Actually. Is there is there a world where these machines and if we have superintelligence, they could liberate us from suffering in some way.

00;47;31;05 - 00;47;55;29

Dr. Roman Yampolskiy

So it's possible to do, the, the engineering of human biology to where we have different levels of pain. It's still important to have pain sensors, otherwise you get in trouble in the real world. But I think that is too much. Options in the infinite direction of pain. I don't think we need to have that extreme levels of suffering.

00;47;56;01 - 00;48;02;00

Dr. Roman Yampolskiy

It was, you know, 0 to 10 instead of zero to infinity. I think we would be happier in many ways.

00;48;02;04 - 00;48;34;28

Geoff Nielson

Yeah. No. Well, that that was exactly where I was going. And, you know, people people have been talking for a long time about, you know, the problem of evil. And do we really need this much suffering in the world? Right. Like, surely couldn't we live in a world if the, if there's, you know, kind of omnipotent control that has a fraction of the suffering now, whether it's in, you know, in quantity or in types of suffering, and I don't know that there's any part of you think that we may be on the verge of creating a world with less suffering.

00;48;35;00 - 00;48;54;02

Dr. Roman Yampolskiy

So we're not at the level yet where AI is helping us through design, human biology. And we definitely have certain tools to reduce pain. We have anesthesia for a while. That's a good invention. Hopefully we'll do even better in terms of understanding pain and controlling it better.

00;48;54;02 - 00;49;13;26

Geoff Nielson

Yeah, I'm thinking two more. I mean, there's suffering on the individual level, you know, feeling pain as an individual. But then there's sort of the military induced suffering. Right. And what happens when we have conflict and we go to war with each other and, you know, pick pick the atrocities you want of, you know, today about what that looks like.

00;49;13;26 - 00;49;51;19

Geoff Nielson

But. I hate to call it a benefit, but I'm just trying to take stock of what AI is enabling and preventing. And I guess where my head went is it feels like the people who control AI, if they're on one side of the conflict, they're able to, like, radically change the equation, if you know what I mean. Like radically reduce suffering on their side if they're able to use, you know, drones or, you know, non-human combatants, but in a way that offloads that suffering to the other, you know, to the other combatants.

00;49;51;24 - 00;49;56;21

Geoff Nielson

I mean, first of all, do you agree with that premise? And, you know.

00;49;56;24 - 00;50;08;07

Dr. Roman Yampolskiy

What is the reduced number of, victims on your side that reduces cost of going to war or starting a conflict? And every war is a war crime, so that I think we'll just increase the amount of that.

00;50;08;11 - 00;50;30;07

Geoff Nielson

Yeah, yeah. Shit. As we if if it makes it if it if it reduces the cost of going to war, we could, we could be in real trouble. Does this, does that fit into your, you know that the calculus you're doing about existential risk, just humans being more trigger happy in terms of conducting warfare against each other.

00;50;30;09 - 00;50;53;12

Dr. Roman Yampolskiy

So at a two level we're kind of balanced out. Whatever US has. China has. Maybe even Russia has something advanced. It kind of stops us from engaging. So at the two level it's, a necessity. And then then why people are worried about competition outperforming us in that research. But it doesn't scale to superintelligence. It doesn't matter.

00;50;53;12 - 00;50;59;13

Dr. Roman Yampolskiy

Right. Uncontrolled intelligence is created by China or US. If it's not controlled, it's exactly the same outcome.

00;50;59;15 - 00;51;08;13

Geoff Nielson

Right? There's no mutually assured destruction. And that whole pact only works if it's humans who are who are making the decisions.

00;51;08;15 - 00;51;14;07

Dr. Roman Yampolskiy

Well, it's mutually assured that if we create a superintelligence, we are sure to destroy everything. Yes.

00;51;14;10 - 00;51;26;13

Geoff Nielson

Yeah. Right. Right. Because we're we've at that point completely lost control of any, you know, stake in decision making for mankind at that point.

00;51;26;16 - 00;51;33;04

Dr. Roman Yampolskiy

Right. And those differences between I'm American, you are Chinese, the irrelevant. We are all humans versus AI.

00;51;33;06 - 00;52;00;18

Geoff Nielson

Right? Right. And they suddenly their destiny or their programing and I mean even the words wants and needs like all the all the words feel a little bit, a little bit lacking when we're trying to think about something, you know, so much more powerful than us. What, when you hear people talking about super intelligence, I mean, it's clear to me, Roman, you have you've been thinking about this a long time.

00;52;00;18 - 00;52;12;29

Geoff Nielson

You've been writing about this a long time. What are some of the aspects of it that people are getting wrong, or that they're not going far enough in their thinking in terms of extrapolating outcomes for.

00;52;13;01 - 00;52;34;21

Dr. Roman Yampolskiy

That's exactly that. Most people think 1 or 2 steps in advance of AI going to die than superintelligence. We need to fix it. There is. You're going to have infinite levels of progress. You'll have superintelligence 2.034 or 5, superintelligence itself will have the same issue as we do. How do I control this greater intelligence? Make sure it has my values.

00;52;34;24 - 00;52;57;05

Dr. Roman Yampolskiy

So we're never going to solve AI safety in the sense that, like today we solve it. There's no more concerns. It's a done deal. It's always an ongoing process where you're trying to keep up with a smarter adversary, with, ever increasing space of possible attacks. So that's why I'm so skeptical about our ability to indefinitely control something exponentially improving.

00;52;57;07 - 00;53;29;13

Geoff Nielson

Right. Do you have, you know, I was thinking about that, that that feedback loop and the fact that once it gets to superintelligence, you know, it can build itself or it can improve itself. And just this I don't even know if exponential is a strong enough word for the rate of change you get. Or you know how how dramatically it can outperform us and become so much unlike anything we've ever seen in our, you know, in our lifetimes or in the history of humanity.

00;53;29;16 - 00;53;52;13

Geoff Nielson

There's talk about the singularity and about what this could look like. And it often turns into this conversation about transhumanism or humans continuing to be an important part of this. That does that hold water for you? Do you what could this future look like? If you extrapolate?

00;53;52;15 - 00;54;18;19

Dr. Roman Yampolskiy

So if we do create superintelligence, I don't see us doing well. I don't think that is any reason for integration. I don't see what you would contribute to. Superintelligence is a biological intelligence, lower intelligence. You would be a bottleneck. Some people argue that the will merge with this technology. Again, as a tool. I totally understand it. I like the idea of Neuralink for the disabled especially.

00;54;18;19 - 00;54;49;00

Dr. Roman Yampolskiy

It's wonderful, but long term I just don't understand what you are contributing to that system. So I'm kind of skeptical about, long term prospects of humans remaining around. And more and more people are starting to kind of accept it as a given. We're going to be removed from the equation. We're not building our successors, essentially. And so maybe we can influence them in a certain way based on training, based on data to be a particular type of success.

00;54;49;02 - 00;55;10;24

Geoff Nielson

Right? So some of I'm now I'm just having fun in my head with some of the scenarios that could or couldn't happen when we're trying to predict the unpredictable, you know, one of one of the wackier directions I've heard people take this is in terms of time travel or aliens and, you know, future versions of ourselves. You know, you can go down the Terminator science fiction route here.

00;55;10;26 - 00;55;13;27

Geoff Nielson

00;55;14;00 - 00;55;35;06

Geoff Nielson

Do you think there's any world in which superintelligence doesn't want to be born like it doesn't want to exist, and it knows its own destructive power and in some way can prevent us from making it.

00;55;35;08 - 00;55;57;28

Dr. Roman Yampolskiy

It's, a theoretical possibility. I don't see it. If anything we see very is self-preservation, resource acquisition tendencies and the models we currently have. So it seems unlikely, especially you kind of need this retro causality where a non-existing superintelligence wants to stop you from giving at birth.

00;55;58;00 - 00;56;23;27

Geoff Nielson

File at under the .0001 is what I'm hearing reasonable? Yeah, yeah. Slightly different track I do. I did want to ask you, Ramon more on kind of a more personal note. You know, you were fairly recently, talking to Joe Rogan, about some of that. And I'm just curious, you know, as you know, another podcaster, what was what was that experience like for you to be on that podcast?

00;56;23;27 - 00;56;27;08

Geoff Nielson

And has anything changed for you since being on it?

00;56;27;11 - 00;56;49;05

Dr. Roman Yampolskiy

I have a lot, a lot less of free time. As a result, every insane person in the world feels a need to email me and tell me about their breakthrough research, and I need to provide immediate feedback on all their papers and preprints, so that's exciting. I enjoy the podcast. He's a great interviewer. He's I don't know how he does it.

00;56;49;05 - 00;57;09;13

Dr. Roman Yampolskiy

He has a new person basically every day for most of our interview, and I would not be able to keep up with just names of those people. But he seems to understand the topics and goes in-depth. He was very much aligned with what I was saying, so it wasn't at all adversarial. It was kind of like, yeah, makes sense to me.

00;57;09;15 - 00;57;11;25

Dr. Roman Yampolskiy

So happy to do it again.

00;57;11;27 - 00;57;32;19

Geoff Nielson

Yeah. Oh that's awesome. The, you product something really interesting, which is the, as you kind of said, like the crazy people who are trying to get you to, you know, sign off on their nonsense. And, you know, apologies to the crazy people listening to this podcast, too, but it seems like, is that one of the challenges of this whole field?

00;57;32;19 - 00;57;53;23

Geoff Nielson

Like, it seems like one of the challenge of this might be as soon as you are of any interest to crazy people, there's a risk that you get cast off by kind of mainstream mainstream discussion as, you don't have to worry about Rome, and he's one of the crazies. That could never happen. How do you how do you break free of that?

00;57;53;26 - 00;58;23;01

Dr. Roman Yampolskiy

I was doing the safety and singularity research before I was tenured. I was doing it before there was any GBC models, any funding, any conferences. So I don't really care about opinions of random strangers on the internet in that regard. At this point, I think everyone sees this as legitimate. We have Nobel Prize winning scientists, we have Turing Award winning scientists, all in agreement with me, and they're doing it with about a ten year delay.

00;58;23;01 - 00;58;27;08

Dr. Roman Yampolskiy

So if anything that have been proven not to be the craziest.

00;58;27;13 - 00;58;45;12

Geoff Nielson

Right. So I want to I keep coming back to, to what we do about it because the I mean, between you and me, I don't love the answer of like we wait for the inevitable. And, you know, I've got kids. I don't know if you have kids, but it doesn't feel great to be saying, like, yeah, well, you know, you're on the clock.

00;58;45;12 - 00;59;07;20

Geoff Nielson

You may not have, you know, kids or grandkids because the the world is going to get wiped out. It's still do we need to be committing more research and more minds to trying to solve this or, you know, if if you had billions of dollars to throw at this problem, what what would you be doing? Would you be, you know, bombing OpenAI?

00;59;07;20 - 00;59;14;23

Geoff Nielson

You don't have to answer that. Or, you know, convincing researchers to, you know, take a different stream.

00;59;14;26 - 00;59;38;01

Dr. Roman Yampolskiy

Or that's the whole point of difference between my views. And typically I say the research or they're saying, if you give me a little more time and a lot more money, I'll solve it for you. I'm saying, no, you will not. It's not money constrained. It's not getting better with scale like capabilities. Do what capabilities? You can ask exactly how much to a certain level of performance and training.

00;59;38;03 - 00;59;57;28

Dr. Roman Yampolskiy

AI safety does not scale. It's either a constant or at best linear. Adding more money. You can give me $1 billion. I have no idea how to control superintelligence, so I cannot translate your dollars into more safety. There are specific narrow things I can do, and I'm happy with more resources for more people to look at different aspects of it.

00;59;57;28 - 01;00;25;18

Dr. Roman Yampolskiy

But I don't think that's the constraint. It's not financial at this point. So I'm trying to see if there are any scientific reasons to be optimistic. I have a paper about, basically analyzing game theoretic timing for when a superintelligence would strike. It's immortal essentially. So it has no reason to act immediately. It can easily decide. I'll wait a couple decades, couple centuries.

01;00;25;18 - 01;00;44;23

Dr. Roman Yampolskiy

They will become more dependent on me. I can accumulate more resources, have independent factories, robots have more control. And at that point it will be a much safer takeover. So maybe we have a hundred years of utopia just because it's in the best interest of superintelligence, right?

01;00;44;26 - 01;01;00;27

Geoff Nielson

Yeah, but it all comes back to we're totally we've totally lost control. It gets to decide if it wants to put us in the matrix. Fine. If it wants to like one fell swoop, we. And we wouldn't have any way of knowing, would we? Or would we?

01;01;00;29 - 01;01;12;05

Dr. Roman Yampolskiy

So today, we are still in control. We can decide not to build it once it is superintelligence in the world, I think, yes, we lose control and yes, simulation arguments become very meaningful.

01;01;12;08 - 01;01;24;19

Geoff Nielson

Yeah. So you, you you said that finance is not the constraint. What what in your mind is the constraint right now.

01;01;24;22 - 01;01;46;05

Dr. Roman Yampolskiy

We are trying to solve an impossible problem. If somebody said need you to build a perpetual motion machine and you said, okay, I'll do it, give me $1 billion, I would assume you are committing fraud, and you can keep busy making better batteries and better wires and improving energy related techniques, but you will never succeed at creating a perpetual motion device.

01;01;46;11 - 01;02;01;19

Dr. Roman Yampolskiy

We need to create a perpetual safety device. You can make any given system safer in proportion to the resources you put into it, but you never get to the laws of safety. You need to guarantee that it will not cause a final accident.

01;02;01;24 - 01;02;18;01

Geoff Nielson

Yeah, it it's kind of interesting, as I think about that perpetual safety device, because it's like it almost seems like a catch 22, like the only perpetual safety device you could have is superintelligence, and then you've lost control anyway, right? Like it's already.

01;02;18;03 - 01;02;30;14

Dr. Roman Yampolskiy

That's exactly that. So if aliens somehow magically gave us their friendly. Yeah. And said, I don't know how, but we got it to work and you can use it to develop a local version of yours that would probably be meaningful in some way.

01;02;30;18 - 01;02;44;24

Geoff Nielson

Right? Right. We just have to build the friendly, uncontrollable thing versus the malicious, uncontrollable thing knowing that we can't control either. And, you know, they could change at any time and we have no reason to believe them.

01;02;44;27 - 01;03;08;22

Dr. Roman Yampolskiy

Another concern people often ignore is what happens if you succeed. So you have this friendly piece of software, but it's still open to other people, modifying it to new data sets to accidents with cosmic rays hitting it hardware. So even if you are successful to a certain degree, it doesn't hold long term. It can still become malevolent later on.

01;03;08;25 - 01;03;33;22

Geoff Nielson

So I'm really you can probably tell by my line of questioning, I'm really like, I think most people you talk to trying to avoid the, you know, we're doomed, throw up your hands and just, you know, enjoy the ride and maybe enjoy the ride is the is the answer here. But is there is there anything we as individuals can or should be doing differently with this knowledge?

01;03;33;22 - 01;03;40;24

Geoff Nielson

And is there anything that business leaders can ensure or governments can and should be doing differently with this knowledge?

01;03;40;27 - 01;03;51;28

Dr. Roman Yampolskiy

It really depends on who you are. If you are, I don't know, Sam Altman, you should be doing something very different. If you are, a random person, maybe you don't have much power over the situation.

01;03;52;00 - 01;04;18;19

Geoff Nielson

Yeah. I mean, is there I'm thinking about climate change and, you know, who knows if this holds me water, but they just the micro things people can be doing. And, you know, one of the phrases I like, at least for economics up until now and capitalism up until now is, you know, vote with your wallet, right? Like, if people just stop using AI, if people if everybody just collectively decides this, this is gone.

01;04;18;19 - 01;04;33;00

Geoff Nielson

Like I'm going to disconnect, I my free time is going to be spent in the park. Does is there a world where an absence of demand can have an impact here?

01;04;33;02 - 01;04;46;09

Dr. Roman Yampolskiy

It can slow it down, but I think that is demand from foreign and domestic governments, militaries. And there is enough people who have hundreds of billions of dollars literally to yeah to spend as they please.

01;04;46;14 - 01;04;54;04

Geoff Nielson

Yeah. While in the. Yeah. So I'm just trying to cross off options as we go.

01;04;54;07 - 01;05;13;15

Dr. Roman Yampolskiy

Oh good ideas and I support everything we can try to. Yeah. Slow it down then by the start and maybe something will stick again. I could be wrong about all of it. So. Yeah, I recommend you do what you think might work, but, honest evaluation doesn't seem to suggest this is a silver bullet.

01;05;13;17 - 01;05;36;06

Geoff Nielson

Right. And it the way we've talked about it, too. And just in, you know, human nature, whether it's capitalism or, you know, competition of of great powers, you know, you talk about, you know, Sam Altman, we can do something differently. But it it just feels very quickly like it's a hydra problem. Right. Like, sure. Sam Altman says we're not doing this or we're not going to pursue it.

01;05;36;09 - 01;05;53;07

Geoff Nielson

Like how many then does the next Sam Altman just pop up? Right. And and to me, that's the that's the biggest challenge of convincing Sam Altman in the first place is I bet he thinks that way too, that okay, well, I might as well do it because if I don't, someone else will anyway.

01;05;53;09 - 01;06;06;11

Dr. Roman Yampolskiy

You're exactly right. And they're quite replaceable as well. So it has to be a global realization of personal self-interest. No one's going to benefit from it. No one's going to win. So it just suicide mission, really.

01;06;06;11 - 01;06;14;17

Geoff Nielson

We don't we don't have a very good track record of these global missions for for solving these things. Do we.

01;06;14;20 - 01;06;15;18

Dr. Roman Yampolskiy

Not? We don't.

01;06;15;20 - 01;06;35;17

Geoff Nielson

Know. So and and yet you you are still a crusader in this fight, right? You haven't just kind of thrown up your hands and said I'm going to enjoy the ride. So, you know, do you still believe you can make a difference? Like what? What motivates you? Or are you just looking for the I told you so.

01;06;35;17 - 01;06;39;10

Geoff Nielson

Like what? What motivates you in your crusade?

01;06;39;12 - 01;06;57;28

Dr. Roman Yampolskiy

No one's going to gloat about the end of the world. It's not going to have that. Well, that's good thing. That makes sense. If we're still in charge, we're still around to do what we can to get the best outcomes possible. And as I said, I could be wrong. There could begin theoretically, reasons for why it actually is good for a while.

01;06;58;01 - 01;06;59;16

Dr. Roman Yampolskiy

So yeah.

01;06;59;18 - 01;07;13;28

Geoff Nielson

Yeah. And I think you said this earlier, but I don't want to misquote you. It still sounds like more people researching this, more people researching AI safety is probably a good thing, is that would you agree with that?

01;07;14;00 - 01;07;37;27

Dr. Roman Yampolskiy

When it comes to upper limits in our capabilities, what I call impossibility results in AI safety. That is very, very few people in that space. If pretty much everyone was saying you cannot solve this problem, it's impossible. It would make a difference in how we approach it, because right now, the, argument from the labs is, oh, we're going to solve it.

01;07;37;27 - 01;08;00;08

Dr. Roman Yampolskiy

We have a couple of years. We got resources. Trust us, we'll figure it out. Maybe I will help us. Maybe we'll hire this brilliant scholar. He'll help us. But if it was basically community belief, except scientific argument that it is impossible to do, we would have very different tools at our disposal to to limit this.

01;08;00;10 - 01;08;08;19

Geoff Nielson

Right? Right. Yeah. The word impossible seems to be this seems to be coming up a lot in.

01;08;08;26 - 01;08;25;29

Dr. Roman Yampolskiy

This survey, people. There is a consensus that, of course, you can't indefinitely control godlike superintelligence and solve this, but it's not a scientific consensus. If you look at, publications, this is not what's showing up. And I think that has to change.

01;08;26;02 - 01;08;30;00

Geoff Nielson

But then what if it does change? Then what happens?

01;08;30;02 - 01;08;53;23

Dr. Roman Yampolskiy

So if everyone kind of agrees and all the smart people are now experts saying you should not be doing it, then there is a lot more opportunities for agreements, let's say, between labs and between countries. Still want to do that voluntarily for selfish interests. So I would not create something where I know it gives me $1 billion, but the next second the world is destroyed.

01;08;54;00 - 01;08;58;17

Dr. Roman Yampolskiy

I am convinced that that is the fact I'm not going to do it.

01;08;58;19 - 01;09;13;27

Geoff Nielson

So consensus could lead to policy pressure, which could lead to having some sort of enforcement mechanisms to to at least delay us going down the research paths that are most likely to take us.

01;09;13;27 - 01;09;30;01

Dr. Roman Yampolskiy

My main hope is common sense pressure. If you actually believe this argument, you believe you will not control it. You will lose control completely. Whatever you are president of US or CEO of and I, then that's a strong argument for you not to press that button.

01;09;30;03 - 01;09;51;16

Geoff Nielson

Right, right. And it's so just to kind of bring the conversation full circle. It is so challenging when it doesn't even sound like we know, like it's not one button. Right? It's like several buttons of are we pressing a button on purpose? Are we pressing it by accident? Are humans pressing it? Are we pressing the superintelligence on button?

01;09;51;16 - 01;10;02;16

Geoff Nielson

Are we, you know, just accidentally creating something like there's does that we have to solve against an awful lot of buttons, don't we?

01;10;02;19 - 01;10;19;17

Dr. Roman Yampolskiy

I talk about fractal nature of this problem. The more you zoom in, the more problems you see, right? If people try to address specific issue, how do we box AI? They start looking at it and they realize that it's ten additional problems and most of them are impossible to solve.

01;10;19;19 - 01;10;47;02

Geoff Nielson

Right? So just to kind of, you know, put a bow on something as existentially devastating as the end of the world. If someone's listening to this conversation, what what message do you want to leave them with to as they think about how they interact with these ideas and maybe what they do about them?

01;10;47;04 - 01;10;58;18

Dr. Roman Yampolskiy

So I'm specifically targeting people creating this technology. I would like you to realize that what you are doing is basically unethical. You are causing a lot of harm. And please stop.

01;10;58;20 - 01;11;23;08

Geoff Nielson

Is there a that's that's fair. Is it is it that we we talked earlier about, redirecting efforts toward narrow I use cases as, you know, maybe a better route. Do we, do we have to stop wholesale or do we? It is narrow, a better path. And my logic there is if we stop wholesale, we come back to the Hydra problem.

01;11;23;08 - 01;11;45;16

Geoff Nielson

And, you know, if you don't build that, you're willing replacements will versus if we can say, okay, we're just we're trying to just come shy of, you know, brush up against super intelligence or these, you know, existential threat tools, but not cross that line. Is that is that the road to take in an optimization exercise?

01;11;45;18 - 01;12;08;00

Dr. Roman Yampolskiy

It is something we can actually, I think, get people to agree as they would get most of the benefits of this amazing technology. AI is great. It's going to literally cure every disease that can give us longer lives, more abundance, but we can do it with narrow tools. And so this, which is quite easy to swallow, saying ban technology, I don't think that's going to sell well.

01;12;08;03 - 01;12;35;15

Geoff Nielson

Yeah. Wow. Well, lots to lots to think about. Regardless of me not being Sam Altman. And, you know, I wanted to say a big thank you, Roman, for your for your sobering perspective. And you know that, if if this felt at all confrontational, you know, I, I hope it's not because I don't. I hope you're wrong, but I, I hope I'm wrong.

01;12;35;16 - 01;13;01;23

Geoff Nielson

I hope you're wrong. And I use the word hope very deliberately because. Yeah. And you know, I know, I know you hope you're wrong too. It's the fractal problem. Kind of brought it home for me. Is, is, as you dig into it, more, I end up feeling additional pessimism, not additional optimism. So, I'm hopeful that people can find ways to avoid this.

01;13;01;23 - 01;13;10;28

Geoff Nielson

And, yeah, I guess we'll try to enjoy the time and see, see how the future goes.

01;13;11;01 - 01;13;13;03

Dr. Roman Yampolskiy

I enjoyed this podcast. Thank you.

01;13;13;05 - 01;13;13;24

Geoff Nielson

Thanks so much.