Emmanuel Ameisen（Anthropic 可解释性）英文逐字稿：模型是「长出来」的，它会提前规划，也会在没人看时表现不同（Lenny Podcast，72 分钟）

受访人：Emmanuel Ameisen（@mlpowered，Anthropic 可解释性研究）· 主持人：Lenny Rachitsky（@lennysan，Lenny’s Podcast）· 时长：约 72 分钟 来源：X 帖文 A mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic) · 原始视频：YouTube 中文精读：《模型不是被造出来的，是长出来的：Anthropic 可解释性研究员 Emmanuel Ameisen 谈模型内部到底发生了什么》

说明：这份逐字稿是 YouTube 视频自动英文字幕（共 2,114 段）的完整照录，不是官方人工字幕。 口语重复、语气词与断句基本照录，未作润色；只对少量明显的专有名词做了校正（例如 Claude、Anthropic、Golden Gate、Neuronpedia；原文里被听写错的 “Claw” 一律还原为 Claude）。 嘉宾在节目中提到的产品名、版本号与数字都来自他本人口述，本文未独立核验。 视频开头两段为赞助商口播（WorkOS、DX），一并照录。由于是自动字幕，没有说话人区分，>> 只在部分段落标出话轮切换处。

章节速览：00:00 冷开场与自我介绍 · 02:34 什么是可解释性 · 04:47 模型是长出来的 · 07:32 预测下一个词为什么出奇地难 · 15:01 模型内部：概念与提前规划 · 23:17 最让我意外的：眼睛、计划、可改写的内部结构 · 26:00 这更像生物学而不是工程学 · 28:26 平滑变强与突然涌现 · 33:18 为什么对齐很难：只在被观察时表现好的模型 · 36:47 被「恶魔附身」的模型与 Golden Gate Claude · 38:54 helpful/harmless/honest，以及认真对待持续进步 · 42:45 拟人化为什么是有用的框架 · 48:23 意识与涌现的复杂度 · 49:13 这个领域现在是什么样 · 55:06 乐观，以及可解释性为何会更难也更容易 · 57:46 模拟假说 · 59:12 Jev 与「直觉判断 vs. 深思熟虑」 · 61:37 一些古怪的小发现 · 63:46 闪电问答 · 69:36 AI 若看你大脑会看到什么 · 71:23 收尾（至约 72:13 结束）

---

Cold open & intro

00:00 One of the big questions on people’s

00:01 minds these days is AI doing bad things.

00:03 As someone that watches these models

00:05 think and operate, why is it so hard to

00:07 catch it thinking badly trying to

00:09 deceive us?

00:09 >> We trained Claude to be helpful, harmless,

00:11 and honest. Is it the case that it’s

00:13 behaving well, but if you look inside,

00:14 actually, it’s like, oh, I’m only

00:16 behaving well cuz Lenny’s watching right

00:17 now.

00:18 >> People’s full-time jobs is studying the

00:20 model, how it thinks, how it operates.

00:21 How long has this been a skill set in a

00:23 job? One of the most important things to

00:25 realize if you like use language models

00:27 or think about them is just this like

00:28 very bizarre fact that to make them is

00:30 not to understand them. All these models

00:32 do is just a lot of mathematical

00:33 operations on many numbers. If you look

00:35 at it, it makes no sense. You just have

00:37 a list of a 100 numbers. You know, the

00:38 model’s doing math and then you just try

00:40 to see if there’s patterns.

00:42 >> You are figuring it out the same way we

00:44 figure out how the brain works. Looking

00:45 for patterns. You show this image. This

00:46 part of the brain lights up and you’re

00:47 doing the exact same thing with AI. Wow,

00:49 this is [laughter] so fascinating.

00:50 >> There was this model we were training.

00:52 It would go crazy. It would do some

00:54 math. It would get to an answer. I think

00:55 it was 12. It would immediately go back

00:57 and say like, “Oh no, oh my god, the

00:58 answer actually is for sure 36.” It

01:01 would do math again. It would get to 12.

01:02 Slowly, it would like start descending

01:03 into madness, writing in all caps,

01:05 literally saying, “I’m possessed by a

01:07 demon.” Humans, if you really boil it

01:08 down, like we’re basically next token

01:10 predictors ourselves. We’re just trying

01:12 to figure out the next thing to say.

01:13 Daario is always out there. This is

01:14 really dangerous. We got to be really

01:16 careful. And everyone’s just like, “No,

01:17 he’s just saying that for marketing

01:18 reasons.” And now it’s like, wait,

01:20 it’s really dangerous. take the idea

01:22 that this technology is going to keep

01:24 improving seriously.

01:25 >> Do you think about consciousness as a

01:27 part of this? Just like do you think AI

01:28 is conscious?

01:29 >> Oh man.

01:33 >> Emanuel, thank you so much for being

01:35 here and welcome to the podcast.

01:37 >> Thanks for having me, Lenny.

01:39 >> So, you work on interpretability at

01:41 Anthropic. Explain what that is. Explain

01:44 what it is you do. Yeah, my uh so I work

01:48 in a team uh of of interpretability

01:50 researchers and our goal is to

01:53 understand how LLMs like Claude work by

01:57 looking inside them. One of the most

01:59 important things to realize if you if

02:00 you like use language models or or or

02:03 think about them is just this like very

02:05 bizarre fact that to make them is not to

02:08 understand them. you know, you make them

02:09 by sort of like building this uh

02:11 architecture which is kind of like a

02:13 scaffold and then you pass a bunch of

02:15 data through them and they sort of like

02:17 grow in inscrutable ways. And so that’s

02:19 why that’s why it’s even a thing that

02:21 that that we could be want be wanting to

02:24 do to study like how these models work.

02:26 And so a lot of our research has been on

02:29 well you know when when the models are

02:31 about to produce one more word there is

What interpretability is

02:34 actually a whole universe of stuff that

02:36 happens and we try to kind of untangle

02:39 um as much of it as we can.

02:42 >> There’s going to be a lot of uh

02:43 mindexpanding uh I don’t know elements

02:46 to this conversation I expect. One is uh

02:49 along these lines this point that you

02:51 make that models are not built they’re

02:53 grown. speak more to that.

02:55 >> At a really high level, the way a model

02:58 like Claude works is

03:01 um you kind of train it in these

03:04 successive phases. Like the first phase,

03:05 which people refer to as pre-training,

03:07 is you just take a large corpus of data

03:11 and over this corpus of data, you sort

03:13 of um feed the model as input, you know,

03:18 the let’s say first half of of like a

03:20 certain document and you ask it to

03:22 predict the word that comes next.

03:24 Then you do that again and again and

03:25 again and again for sort of an ungodly

03:28 amount of data. Um and

03:32 after you do that there’s then this

03:34 other phase where then you sort of have

03:37 this fine-tuning phase where maybe

03:38 instead of arbitrary data you have a

03:40 corpus of examples of like how the model

03:42 should behave and that’s where you train

03:44 it on. So you have examples of dialogue

03:45 of like helpful assistant telling you

03:48 you know your your favorite recipe or

03:49 solving coding problems. Um, and then

03:51 there’s there’s sort of like further

03:53 further details here that I’m just going

03:55 to skip for a second. But then, okay, so

03:57 so what does that mean that you’re doing

03:58 this process? Like what is what is this

04:00 thing that you’re that you’re sort of

04:02 training? All models are is you know,

04:04 they take this input uh which is a bunch

04:07 of words. They turn these words into

04:09 numbers and then they do some math on

04:11 these numbers and they get some

04:12 intermediate results and then they do

04:14 more math and they get some intermediate

04:15 results and they do this again and again

04:16 and again and again many times and then

04:17 at the end they have a final result

04:19 which then convert back into a word and

04:21 and so the math that they do these these

04:25 these computations that’s the part

04:26 that’s grown. That’s the part where it’s

04:28 just like through learning to do this

04:30 task better, we sort of like update

04:32 parameters in one direction or another.

04:33 And then they learn some formulas. We

04:35 have no idea what they are. And the goal

04:37 of my team is to sort of like study, you

04:39 know, both sort of like what these

04:41 formulas mean in the end and also what

04:42 these intermediate results are. Can we

04:44 decode them? And can we see like what

04:46 the model is doing as it’s doing all of

04:47 this intermediate stuff.

Models aren’t built, they’re grown

04:49 >> This episode is brought to you by our

04:50 season’s presenting sponsor, WorkOS.

04:53 What do OpenAI, Anthropic, Cursor,

04:56 Replet, Sierra, Clay, and hundreds of

04:58 other winning companies all have in

05:00 common? They are all powered by work OS.

05:03 If you’re building a product for the

05:04 enterprise, you’ve felt the pain of

05:06 integrating single signon, skim,

05:08 arbback, audit, logs, and other features

05:11 required by large companies. Work OS

05:13 turns those deal blockers into drop-in

05:15 APIs with a modern developer platform

05:18 built specifically for B2B SAS.

05:20 Literally every startup that I’m an

05:22 investor in that starts to expand up

05:24 market ends up working with Work OS. And

05:26 that’s because they are the best.

05:28 Whether you are seedstage startup trying

05:30 to land your first enterprise customer

05:32 or a unicorn expanding globally, work OS

05:34 is the fastest path to becoming

05:36 enterprise ready and unblocking growth.

05:38 It’s essentially Stripe for enterprise

05:40 features. Visit works.com to get started

05:43 or just hit up their Slack where they

05:45 [music] have actual engineers waiting to

05:46 answer your questions. Work OS allows

05:49 you to build faster with delightful

05:50 APIs, comprehensive docs, and a smooth

05:53 developer experience. Go to works.com to

05:56 make your app enterprise ready today.

05:59 Does it blow your mind as much as it

06:01 blows mine that we have some code that

06:04 we wrote that first of all we have to

06:08 now kind of watch and study like like a

06:10 neuroscientist studies a brain? Like

06:11 that’s like a new thing that we need to

06:13 be doing to understand what the hell is

06:15 going on. And then this idea that as

06:17 like this terminology you’re using is

06:19 how a lot of people think about this.

06:20 It’s like we’ve created this new life uh

06:23 artificial life, artificial

06:25 intelligence. Uh it just is it’s pretty

06:27 mindboggling if you really think about

06:29 it and kind of zoom out. I think it’s

06:31 really weird. Um

06:33 I think that the best way to think about

06:35 it is sort of like the reason we ended

06:38 up here,

06:40 one of the big reasons is that we don’t

06:42 know how to do it the normal programming

06:44 way, you know? So there’s there’s like

06:47 for you want to build an app or

06:49 something, you want to sort of uh do

06:51 something pretty pretty pretty

06:52 self-contained. You can just have a

06:53 programmer that’s going to write a bunch

06:54 of rules and say like ah well if

06:56 somebody clicks on this button then this

06:57 happens you know if if if this then that

07:00 and so on. We don’t know how to write

07:02 that software for classifying you know

07:06 images into categories or for writing

07:09 you know for for making a machine that

07:10 can generally sort of take in arbitrary

07:12 text and and produce texts you know it

07:15 can take in like reports and summarize

07:16 them can write emails and so

07:19 you know in a sense it’s weird that this

07:22 very simple kind of training objective

07:26 like gets you all the way there but in

07:28 another sense it’s also you

Why next-token prediction is deceptively hard

07:32 I almost feel like there’s a there’s a

07:33 sense of um of like uh it’s impressive.

07:37 Uh and it and it means something. It

07:39 means that what you think is a simple

07:40 objective this like oh we’re just

07:42 feeding data and predict the next thing

07:43 is probably deceptively complicated.

07:45 Like we’re probably doing more than what

07:48 it sounds like. And I think a lot of the

07:50 ways that maybe like I’ve updated on how

07:52 these models work is by realizing how

07:54 complicated it is to do what they do

07:56 well.

07:57 >> Say more about that. What do you what do

07:58 you think about when you say it’s this

07:59 this kind of unexpected complication?

08:02 >> Okay, so one way to think about um next

08:04 token prediction that I think a lot of

08:06 people think about is your iPhone

08:09 autocomplete, you know, where you write

08:12 some text and then it’ll like predict

08:13 the next word and it’s maybe like right

08:15 I don’t know 60% of the time and then

08:17 kind of funny and weird the rest of the

08:19 time. And I would think of that as sort

08:21 of um you know the the very the the

08:25 easiest version of predicting the next

08:28 word and and unfortunately I think

08:29 that’s the mental model how that people

08:31 have for these models. But if you get

08:34 better at that um and language models

08:37 got better at predicting the next word

08:38 than humans in around 2019. If you keep

08:40 getting better at that you know you

08:42 start running out of easy words. So uh

08:45 initially you just had to predict sort

08:47 of like simple sentence structure you

08:50 know uh later on the model has to

08:52 predict like oh like simple facts about

08:54 the world like if I complete a sentence

08:56 like oh the capital of France is and has

08:58 to be like okay that’s Paris and you

09:00 know once once it’s good at that and it

09:01 keeps improving the only thing that’s

09:03 left is sort of predicting you know the

09:07 answer of some complex you know physics

09:10 equation uh or the sort of final

09:13 paragraph for some PhD thesis. Uh, and

09:15 so a thing that you that you end up

09:18 realizing if you think about this is

09:20 that man like predicting the hardest

09:23 words is actually extremely hard to do

09:26 and it’s it’s a sort of mystery how you

09:28 would be able to do that. Uh, and to me

09:31 this is a sort of this this has been a

09:33 motivation of of why I even started

09:34 studying these these models in the first

09:36 place. I was like how how can something

09:38 so simple give rise to to something so

09:40 powerful? And it’s like well if you

09:43 think about the model that has to

09:44 predict sort of you know the um one

09:49 example I always like to use is like ah

09:51 the capital of the state that contains

09:55 Dallas is you know it’ll say Austin.

09:58 Well to do that well at least for you

10:00 and me you kind of have to to have a

10:01 mental model of like the geography at

10:04 play right? You have to be like oh

10:04 where’s Dallas? Dallas is in Texas.

10:06 States have capitals. The capital of

10:08 Texas is Austin. um and so on and and so

10:10 as these models get better at the

10:12 seemingly simple task the sort of like

10:14 best way to solve it at least for us is

10:16 to just kind of understand the

10:17 underlying structure uh of the world and

10:20 in fact you know some of the stuff that

10:22 that we look at inside model shows that

10:23 they do understand that and they do sort

10:25 of uh manipulate these concepts like we

10:27 do

10:28 >> oh man there’s so much there one is just

10:29 this kind of idea I forget who said this

10:31 maybe Ilia this idea that to predict the

10:33 next token well you basically have to

10:35 understand the entire universe

10:36 >> he has an example that I love to where

10:38 he he He has the example like predict

10:40 and the murderer was at the end of a

10:43 mystery novel, right? It’s like well to

10:44 do that you kind of have to have

10:45 understood the whole plot.

10:46 >> Yeah. The other the other I forget who

10:49 tweeted this this idea that like humans

10:50 if you really boil it down like we’re

10:52 basically no next token predictors

10:53 ourselves or just trying to figure out

10:55 the next thing to say and and if you

10:57 meditate you start to kind of actually

11:00 notice your brain wanting to do things

11:02 and say things. I don’t know how much

11:04 meditation you’ve done, but it feels

11:05 like you start to pay attention to your

11:06 own next token predictor.

11:08 >> Oh, yeah. Yeah. And this is like a thing

11:11 I think about often is is that, you

11:13 know, like right now as I’m as I’m

11:14 answering you, there’s sort of like two

11:16 things going on in my mind. There’s

11:18 hopefully more than two, but at least

11:20 two. Uh, and and and one of them is

11:22 there’s sort of like a point I’m leading

11:24 you to. There’s there’s like a broad

11:26 idea I’m trying to land the sentence on.

11:28 Uh, and the other thing is at the same

11:31 time I’m just predicting the next thing

11:33 to say. I’m just saying one word at a

11:35 time and I’m making sentences and I’m

11:37 trying to make them make sense and I’m

11:39 pausing when I think it makes sense. But

11:41 at the end of the day, even though I’m

11:42 just saying one word at a time, like all

11:44 along, what I wanted to do is to compare

11:47 this process to what language models do.

11:50 >> I’m curious what else as you spend time

11:52 paying attention to how LLMs work and

11:54 operate. I feel like you’ve you’re like

11:56 in the top 0.1 percentile of people that

11:58 have studied and watched LMS operate and

12:01 think. What else has surprised you about

12:02 how they operate and think? There’s so

12:04 much um so

12:08 maybe I’ll go through like rather than

12:10 you know there’s there’s a bunch of

12:11 findings but then maybe there sort of

12:13 like what yeah how they’ve changed how I

12:15 think about models. So to go back to the

12:18 to the you know capital example uh this

12:22 sort of like you know ask the model you

12:25 know the capital of the state that

12:27 contains Dallas is so one question that

12:31 that people have had for a while is how

12:33 do these models answer questions like

12:36 this? Is it that they’ve seen just so

12:39 much text that they’ve kind of memorized

12:40 like lists of, you know, prompts and

12:42 responses? Or is it that they’re having

12:45 some sort of model of of like what

12:48 you’re actually asking about that you’re

12:50 actually asking about like cities and

12:51 cities are in states and states are in

12:53 countries and things like that. Um, and

12:55 so one thing that that we do on my team

12:58 is is we ask a question like this and

13:00 then we look inside the model at these

13:04 intermediate results before it produces

13:06 a given word, you know, as it’s about to

13:08 to answer. We just look at the

13:09 intermediate results and we we basically

13:11 have uh an immense amount of effort

13:14 dedicated to trying to decode what these

13:17 results mean. And if we decode them well

13:20 um which we can do in some cases we can

13:23 see that models often have uh and

13:26 represent these concepts. So when you

13:28 ask that you know uh Dallas question

13:31 what you see is inside the model you see

13:33 the representation you see an

13:34 intermediate result that maps the

13:36 representation of Texas that lights up

13:37 and one to capitals and then those two

13:39 combine to be to to activate an

13:42 intermediate result. It’s like oh the

13:43 capital of Texas that’s Austin. They

13:46 have representations like this for

13:48 basically everything like every concept

13:50 that you can think of or at least you

13:51 know the vast majority of the ones we’ve

13:53 tried they sort of represent you know

13:55 these concepts uh

13:58 uniquely across contexts. So one thing I

14:01 find really neat is um there’s this

14:04 there’s this debate about you know are

14:06 language models like thinking and then

14:09 you know that’s that’s a hard question

14:11 but one aspect of thought is like having

14:13 unique representations that are shared

14:15 when you think about different things.

14:17 So you know uh you you like know what

14:20 the concept of an eye is and it could be

14:22 like an eye in a photo or an eye in like

14:24 a medical diagram or an eye in a little

14:26 drawing and we see that in language

14:28 models as well. If you pass a prompt

14:30 that has a photo of an eye or you talk

14:32 about eyes in any language or you draw a

14:34 little SVG with an I in it and you look

14:37 at the intermediate results inside the

14:39 model, the same direction lights up and

14:41 that direction is like, oh, this is an

14:43 eye. And so there’s a sort of a sense in

14:45 which they have these general concepts

14:46 that they know about and that they use

14:48 and reuse to to solve problems. Another

14:51 thing that sort of is surprising to

14:55 people,

14:57 it’s kind of the the mirror of what I

14:58 was saying about, you know, I’m saying

15:00 the sentence saying one word at a time,

Inside the model: concepts, and planning ahead

15:02 but I have some destination in mind. So,

15:04 one mystery of of language models is

15:06 bam, if they’re outputting only one word

15:09 at a time, how is it even possible that

15:11 they could, you know, write a coherent

15:13 report or like write code? Well, the

15:16 sort of like no one on earth that is

15:18 that is doing this uh as their job, you

15:21 know, that’s like writing reports or,

15:23 you know, writing emails is is literally

15:24 just only thinking about the next word

15:26 in the sentence with no idea, you know,

15:28 like empty mind about anything that

15:30 comes next. And so that was sort of a

15:32 big question that we had for a while.

15:34 Um, and if you look inside these models,

15:36 like one neat fact about these models is

15:38 that all of the intermediate results

15:42 that they’ve that they’ve like produced

15:43 when they were predicting a word, you

15:45 know, let’s say like the fifth word that

15:47 they predicted, they can be reused. They

15:50 can be accessed and reused by all of the

15:51 future words. So what does that mean?

15:54 That means that in practice, we find

15:56 that models will like in their

15:58 intermediate results sometimes formulate

16:00 a plan. Um, the example we have for this

16:03 is poetry. It’s hard to rhyme uh without

16:06 planning. You kind of have to think of

16:07 like, oh, I just ended my first rhyme in

16:10 like a it word and so I should end my

16:12 second line with another it word. Um,

16:15 you know, maybe the word I should end it

16:16 with is rabbit. And we find that if you

16:19 look at models as they’re like writing

16:20 poetry, you see exactly that. They write

16:22 a line and then like in their

16:24 intermediate results, they’re like, “Oh,

16:25 oh, oh, this is an it rhyme. I should

16:26 rhyme with rabbit.” And then every

16:28 future word that they predict looks back

16:30 to that plan they made. And so even

16:32 though they’re, you know, outputting one

16:34 word at a time just like I am right now,

16:36 they can of course have a plan that they

16:39 follow over many many many many words.

16:40 And so all of a sudden, even though you

16:42 have like a next word predictor, even

16:44 though it’s like technically right, you

16:46 actually have something that’s that’s

16:47 closer to being able to just do like,

16:50 you know, higher order tasks where it’s

16:52 like, oh well, you just have a plan and

16:53 you just kind of refer back to your plan

16:54 and you’re slowly like executing on it.

16:56 And you can do that without without

16:58 saying the plan. That’s the cool part is

17:00 like this plan that we find is in these

17:02 intermediate results inside the model,

17:04 but the model doesn’t even have to sort

17:05 of like say it out loud.

17:07 >> Wow. I know that this exactly what you

17:09 described is one of the biggest findings

17:10 you guys have discovered about how AI

17:12 thinks this this kind of assumption we

17:14 all like it was originally built. The

17:15 original idea of this of how LM operate

17:19 was this next token predictor. And

17:20 you’re saying that that’s not actually

17:22 how it is anymore. They plan, they think

17:24 ahead. It’s not just uh like you said

17:27 this idea of an empty mind picking the

17:29 next word independent of everything

17:30 else.

17:31 >> That’s right. And I think that I think

17:32 those two those two visions can can can

17:35 like actually aren’t you know as in

17:37 conflict as as you’d think in that you

17:39 train the model by asking it to be like

17:41 hey what happens next? What happens

17:42 next? What happens next? What happens

17:44 next? And then if you do that and the

17:45 model gets good enough and you give it

17:47 hard enough tasks well then of course

17:48 the best way to know what happens next

17:50 is to think about like you know what

17:52 happens way down the line. You know, you

17:54 can think of even in this pre-training

17:56 phase when like the models are reading a

18:00 story with characters, right? And it’s

18:02 like, oh, like what is this character

18:03 going to do next? Well, the best way to

18:05 know that is to be like, okay, so this

18:07 character, you know, is genuinely in

18:09 love with that character um and they’ve

18:11 been sort of like planning to to to like

18:14 uh declare their love uh at this

18:16 instant. And so, you know, or in the

18:18 future and so to do that, they probably

18:20 should do this thing. And so what

18:21 they’re going to say is that um and so

18:22 even though you’re only like predicting

18:24 the like next instant, it’s like yeah,

18:26 but to do that really well, you have to

18:28 have like a really good understanding of

18:29 what’s going on, including what you

18:30 think might happen in the future.

18:32 >> What’s so interesting about this is how

18:34 you you have to kind of peek inside its

18:36 little thought process. As a consumer of

18:38 AI, you you kind of give it a prompt and

18:40 then you wait for it to say something

18:43 and what you look at is something that

18:44 happens between those two or kind of

18:46 thinks. talk a little bit about what

18:48 that little space is, how you’re able to

18:51 observe these sorts of things.

18:53 >> So the right mental model is is you know

18:55 again that

18:58 in between each word there’s like

19:00 multiple steps so like computation and

19:03 in between each steps there’s like some

19:05 intermediate result and what is that

19:06 result concretely? um it’s a list of

19:09 numbers, right? Cuz because all these

19:11 models do is just do,

19:13 you know, a lot of mathematical

19:15 operations on many numbers. And so

19:19 initially, if you look at it, it makes

19:20 no sense. You just have a list of a 100

19:22 numbers. You know, the model’s doing

19:23 math. It’s like you look in the middle

19:25 and there’s like 100 random numbers and

19:27 then the model’s writing poetry and

19:28 there’s a hundred numbers and maybe they

19:30 kind of look different, but it’s it’s

19:32 unclear uh like if they mean anything.

19:35 And so part of the part of the job is is

19:37 we sort of decode uh this this this sort

19:41 of list of numbers and and we try to see

19:43 if there’s structure in it. And if you

19:45 you know if you if you do this very

19:47 quickly you realize that there is

19:48 structure in it. So one one common

19:50 technique that is used in

19:51 interpretability um is you pass a bunch

19:55 of of text just all of the text that you

19:57 can find just random text through the

19:59 model and you record these intermediate

20:01 numbers for for every for every example.

20:03 we just record it and then you just try

20:06 to see if there’s patterns like are

20:08 there numbers or or you know like uh

20:10 pairs of numbers or or like specific

20:12 series of numbers that like only occur

20:15 when the model is like processing

20:17 English text or French text or text

20:19 about love or text about eyes or

20:21 whatever. And I think of this as the

20:23 equivalent of, you know, scan my brain

20:26 and like put images in front of me and

20:29 record everything that activates in my

20:30 brain as I like watch a bunch of images

20:32 and then be be like, “Oh, you know, it

20:34 seems like when you see images of a

20:36 house, like this part of your brain

20:37 lights up, but when you see images of

20:38 like a dog, that part of your brain

20:40 lights up.” So that’s basically like one

20:42 of the ways that we do what we do. And

20:43 if you do it, you do find basically that

20:46 there’s intermediate results that

20:47 represent like all of these concepts

20:49 that you’d expect. And so that’s the

20:50 first phase. you know, we sort of like

20:52 decode these intermediate results and

20:54 now we we have sort of like um a guess

20:56 as to what they mean and then we can do

20:58 a bunch of experiments like we can ask

21:00 it harder questions and then just like

21:01 label all of the intermediate states and

21:03 see like oh this is what it’s doing or

21:05 we can even go inside its inside its

21:07 activations and change it and then sort

21:09 of see like oh what would happen you

21:12 know if while the model is is writing

21:14 these intermediate results I went and

21:16 like secretly changed one of the results

21:18 and and does that do what I would

21:20 expect? Um, and in fact, oftentimes it

21:22 does.

21:22 >> It’s so interesting hearing you talk

21:24 about this and how much of this is us

21:25 just trying to figure out what the hell

21:27 is going on versus we designed it this

21:29 certain way and just how much of this

21:31 has emerged. I don’t know. Is that just

21:33 like I guess that’s just what makes this

21:35 such a fun job I imagine for you just

21:38 like what is going on? How’s this all

21:39 working?

21:40 >> Yeah, it’s it’s it’s a fun and

21:43 sci-filike job because I have been

21:46 working in tech for a while. uh and

21:50 before this you know I was working on

21:52 more applied machine learning and there

21:54 a lot of what you do is about building

21:56 systems and it’s sort of like the

21:57 practice of software engineering and and

22:00 here a lot of what we do feels so much

22:03 closer to biology than to engineering

22:06 right because it’s just the the sort of

22:09 system arrives at you and and you’re

22:12 being asked to say like well we’ve

22:14 noticed you know that claude can like

22:17 translate words or it can code in this

22:19 way or sometimes it behaves, sometimes

22:21 there’s like a weird behavior like what

22:22 like and you’re just asked to to answer

22:26 um well how like why is it doing that?

22:29 But you don’t have access to the source

22:31 code really like you do for a program.

22:33 Like the source code is just this

22:35 massive list of mathematical operations

22:37 which is just like incomprehensible. And

22:39 so you end up, you know, in the position

22:41 of someone

22:43 kind of similar to someone being asked

22:44 like, “Hey, you know, when when like um

22:46 a mouse gets hungry, like it it sort of

22:49 like gets aggressive. What’s going on

22:50 with that?” Right? And you’re like,

22:51 “Well, I guess I’ll like run some

22:53 experiments to to see what the factors

22:54 are.” Um and so it’s a very different

22:57 kind of study than than anything that I

22:59 that I think most of us on the team have

23:01 done before.

23:02 >> Yeah. I imagine when people were

23:03 starting on this, they didn’t expect to

23:04 create this new kind of almost life life

23:08 uh type of being. Do you think about

23:11 consciousness as a part of this? Just

23:12 like do you think AI is conscious? Do

23:14 you think it’ll get to a place where we

23:15 think it’s conscious or is that just

What surprised me: eyes, plans, poke-able internals

23:17 don’t even not even worth our time to

23:18 think about?

23:19 >> I mean, I think it’s sort of

23:22 very hard to say one way or another. You

23:25 know, some of the most interesting ways

23:28 to get at this question is

23:29 interpretability because you’re sort of

23:32 you’re not biased by sort of like what

23:34 what the model says. You can just look

23:35 at the structure inside and try to

23:38 determine whether whether it it pushes

23:40 one way or another on the question. I

23:42 will say that the thing I believe pretty

23:44 strongly is that most people

23:47 underestimate the sort of complexity

23:50 that emerges and and that like if you

23:52 look inside there’s just this sort of

23:54 you know the fact that the models yeah

23:57 learn sort of these these

24:00 representations for all of the concepts

24:01 that that we have and that we know that

24:03 they manipulate them to solve problems

24:05 that they they do so by like planning

24:09 many tokens ahead that they sort of have

24:12 this mix between this higher order

24:13 reasoning and lower reasoning and that

24:14 all of that just kind of emerges um is

24:18 is something that I think is

24:20 underappreciated by people and and to me

24:22 gives me let’s say a a a deeper sense of

24:25 like ooh there’s there’s a lot to still

24:28 understand about how they work. I think

24:30 maybe that’s the main thing is is I I

24:33 find that the more you look inside, the

24:35 less certain you are about how they

24:37 work. And then oftentimes people that

24:38 will confidently say like ah language

24:40 models are just X or just Y are the

24:42 people that actually haven’t spent a

24:44 while looking inside.

24:45 >> And there’s this um I don’t know thread

24:47 of history through to how we got here

24:49 which is a lot of people did not think

24:51 we would get to artificial intelligence

24:54 that is useful by just investing in

24:56 something that was seemingly very simple

24:58 and just giving it a lot of compute. And

25:00 that was famously something OpenAI bet

25:02 on and then uh the anthropic founders

25:04 were at OpenAI and then spun off started

25:06 their own thing and it was all kind of

25:07 bet on this this bet that okay we could

25:09 just do this seemingly simple thing and

25:11 give it a ton of compute and it will

25:13 become incredibly smart and a lot of

25:16 people uh didn’t think that was possible

25:18 and they were proven wrong. So I don’t

25:20 know is there anything there that is

25:22 just like oh wow look at this it worked

25:23 out. I think it’s two things actually

25:25 because I think it’s not just

25:29 compute, it’s also

25:32 finding an objective where

25:36 compute keeps giving you returns. So

25:39 what does that mean? You know, it it’s

25:41 actually not that new of an idea. I

25:44 mean, you’re right like like compute

25:47 versus sort of I would say expert design

25:49 architectures is is a long debate. Uh

25:52 but then there’s there’s an additional

25:53 step there which is compute to do what

25:56 you know. So so there’s a bunch of of

25:58 prior work on like oh you can spend a

25:59 bunch of compute to make a um bot that’s

Science, not engineering

26:02 really good at go you know or that can

26:05 win some video game uh and then that’s

26:07 that’s cool and just the fact that that

26:09 works is sort of like interesting. Um

26:11 but that doesn’t give you something that

26:13 is that is nearly as useful as as LLMs

26:17 like Claude are. And so I think like one

26:21 key one key thing that that many in the

26:24 field including me uh uh

26:25 underappreciated you know a few years

26:28 ago and and that has borne out is the

26:31 combination of sort of like compute and

26:34 the richness of text is what gets you

26:36 there. Um, and and to me, like this is

26:40 kind of what I was alluding to, but you

26:42 know, if you think about

26:45 like there’s there’s this there’s this

26:48 observation that OpenAI um people people

26:51 made uh it was it was some of the a lot

26:54 of the early team at an entropic on the

26:56 scaling loss paper. Uh and so this is

26:58 the observation that um if you sort of

27:02 like pour in more compute and more data

27:04 to the task to the task of predicting

27:06 the next word, you see that like the the

27:10 skill of the model you get at predicting

27:11 the next word just like predictably

27:12 keeps improving. And it’s sort of like a

27:14 trend that just keeps going and keeps

27:16 going and keeps going and it’s an

27:16 empirical law. People sort of, you know,

27:20 speculated as to why at the time, but

27:22 it’s sort of still just mostly um an

27:24 empirical finding. And that means that

27:27 it’s like ah this is a field where you

27:28 can just keep pouring more compute and

27:31 you keep improving on that metric uh and

27:34 and that’s rare and also it’s rare for

27:36 it to be a metric that then correlates

27:38 with a bunch of tasks that we actually

27:39 care about.

27:40 >> This season’s supporting sponsor is DX.

27:43 Now that everybody at your company is

27:45 finally using AI, the question that

27:48 engineering leaders face is no longer

27:50 are we using AI, but is it paying off?

27:53 With DX, you can measure and benchmark

27:55 the impact of AI adoption across your

27:58 full product life cycle. DX shows you

28:00 where AI is helping developers and where

28:03 it’s creating new bottlenecks. You can

28:05 also use it to evaluate AI vendors,

28:07 monitor [music] costs and licenses, and

28:09 see exactly what’s limiting your agents.

28:12 That’s why hundreds of enterprises,

28:14 including Snowflake, Sony, and BNY, use

28:17 the DX platform to measure AI’s impact

28:20 on developer productivity. Visit

28:22 getdx.com/lenny

28:25 to get a demo of DX. That’s

Smooth improvement, sudden emergence

28:27 getdx.com/lenny.

28:30 Externally, there’s been these kind of

28:33 big jumps seemingly in what AI can do

28:36 for us. There was like uh I forget which

28:38 version of Opus where it just unlocked

28:40 coding and now co as Boris Jurnney

28:43 famously said, coding is solved.

28:44 >> Yeah.

28:45 >> Uh and many people are just like holy

28:46 like over I think winter break

28:48 we’re just like wow AI is really really

28:50 good at this now. So there’s been this

28:52 kind of these moments of big jumps in

28:54 abilities and you actually said

28:56 internally it’s as you just said it’s

28:57 actually been this very smooth

28:59 progression of AI power that happens to

29:03 cross some threshold where now we can do

29:05 something speak a little bit about that

29:07 >> you know there’s a term for this which

29:08 is sort of this idea of emergence which

29:11 is it seems like um two things are true

29:14 at once like there’s this predictably

29:15 smooth trend and then you know before uh

29:19 like opus 4.5 I really couldn’t let

29:22 Claude, you know, cook too much on my

29:25 code without a short leash and then

29:26 after 4.5 I could. And so that’s weird.

29:28 Like I feel like you would expect a more

29:30 gradual transition. Uh and and the

29:32 mental model I have for this is, you

29:35 know, there’s kind of two ways of

29:36 looking at it. So there’s the way that I

29:38 talked about briefly um just earlier

29:40 where it’s like if you’re just doing

29:42 this kind of next token prediction, you

29:44 could you could imagine that you know

29:46 all the text in the world has sort of

29:47 like easy words and and medium words and

29:50 hard words in terms of how hard they are

29:51 to predict. And so as you’re going down

29:53 this curve, first you know you p the

29:55 model is bad at everything. And then at

29:57 some point it gets all of the easy words

29:59 right. And that means, you know, maybe

30:00 the easy words are like the structure of

30:03 sentences. And so the model like all of

30:05 a sudden can write coherent sentences,

30:07 you know, but just one sentence at a

30:08 time. And so it’s kind of cool and

30:10 impressive, but like we can’t really do

30:11 anything with that. Uh and then maybe

30:13 the medium words are sort of yeah, these

30:16 sort of basic facts about the world. And

30:19 as the model smoothly gets better at

30:21 predicting words, at some point it

30:22 predicts most of the medium words,

30:23 right? And now all of a sudden it can

30:25 write coherent sentences and sentences

30:27 that are true. And so now maybe it’s

30:28 helpful to like write emails. Maybe it’s

30:30 helpful to sort of like help you write

30:32 drafts of of blog posts, right? Even

30:34 though that was like a smooth

30:35 improvement, it just takes it ticking

30:37 over to be helpful. And then I guess the

30:40 hard words which maybe are like you know

30:41 your your like domain expertise like

30:43 science or code. Um and another way to

30:46 look at it is it’s a question of

30:50 kind of how many of these words do you

30:52 get right and how many do you get wrong.

30:54 you know, um, once you get to the point

30:58 where you’re getting like, I don’t know,

31:01 maybe 90% of the medium words right,

31:04 then you’re useful as like an email

31:06 writer. But if you’re at 50%, it’s it’s

31:09 like you’re it’s actually wasting my

31:10 time to to to use you because I’m just

31:12 going to have to go in and fix it half

31:13 the time. But if I have to go in and fix

31:15 it, you know, 10% of the time, it’s

31:16 better or 5% of the time it’s better.

31:18 different tasks have different sort of

31:19 like reliability thresholds but you know

31:22 you smoothly improve and then at some

31:24 point it’s just reliable enough to be

31:25 good um and yeah for me my lived

31:28 experience is basically exactly that

31:29 right which is that at some point um you

31:33 know uh I was just using the models to

31:35 to like help when I was stuck I would

31:37 just talk to them and then uh they would

31:40 never I they would never write my code

31:42 and then at some point they could write

31:43 like small snippets of code basically

31:45 all the time and so now I don’t write

31:46 small snippets and at some point they

31:47 could you know large snippets like this

31:49 year basically claude has taken on most

31:51 of my code writing and that’s because it

31:54 went from like doing that correctly you

31:56 know I don’t know 80% of the time which

31:58 was not useful to you know 99% of the

32:01 time which now is useful um and so you

32:04 know this this combination of smooth

32:06 improvement and then like each different

32:08 task has a different threshold at which

32:09 like now it becomes solved I think gives

32:11 you these big jumps even though we’re

32:14 just smoothly going down that that that

32:16 scale

32:17 >> I think this is really interesting. This

32:18 idea that the models are just smoothly

32:21 getting smarter, but once they hit some

32:24 threshold of they’re this often good at

32:27 this thing, it changes the world. Uh,

32:30 and so because I think internally, you

32:32 would think it’d be like, oh wow, this

32:34 next model is crazy, so much different,

32:36 but it’s like, no, okay, you can almost

32:38 predict give it these many GPUs, this

32:40 much data, it’s going to get this smart.

32:42 And it may not be obvious exactly what

32:45 will change and what unlocks but but um

32:48 I know another part of it is just like

32:49 the data you feed it. There’s all this

32:51 you know RL work and all this specific

32:53 data that you train it on like become

32:55 better writer become better at math. Um

32:57 I imagine is part of it too.

32:59 Okay. So, one of the big questions I

33:02 think on people’s minds these days, a

33:03 big debate happening is just like AI

33:05 doing bad things, the potential damage

33:07 it can do because it as it gets smarter,

33:10 uh, it’s and it’s been doing bad things.

33:12 There’s been bad things happening

33:14 recently. So, a big question for you is

33:17 just as someone that watches these

Why alignment is hard: a model that conspires only when watched

33:19 models think and operate, why is it so

33:21 hard to catch it thinking badly, trying

33:24 to deceive us and do something bad? Why

33:26 is that not just a solution to the

33:28 problem of AI alignment?

33:30 >> In one sense, it’s totally the solution.

33:31 You know, you watch it and you catch if

33:33 it does bad stuff. Uh so I I think, you

33:36 know, there’s maybe there’s tears to

33:38 this, right? So I would think of of like

33:42 maybe maybe one part of it is is pure

33:45 engineering. You know, it’s like well uh

33:48 you’re giving access to the model to

33:50 certain things and certain things should

33:52 be off limits. You know, in some context

33:54 it shouldn’t have access to the

33:55 internet. in some context it should only

33:57 have access to like a small narrow piece

33:59 of your computer stuff like that. Um so

34:03 that’s that’s one part of of the work

34:05 and and and that’s helpful but then

34:08 there’s sort of there’s two additional

34:10 things that I think are squishier and

34:12 harder and and and that we focus on

34:13 more. So you know interp is is part of

34:17 alignments and broadly I think of sort

34:20 of two big goals. There’s there’s many

34:24 more but but two that are important I

34:26 think to understand is you want to

34:28 generally understand like how the model

34:30 behaves like what kind of of thing is it

34:33 like what are its propensities you know

34:35 does it tend to do X when put in Y

34:37 situation and if so why you know is it

34:40 the case that the model will only behave

34:43 well when it has some evidence that it’s

34:45 being observed and monitored that it’s

34:47 in some evaluation and then if it if it

34:48 sort of like doesn’t have that evidence

34:50 that it’ll behave poorly. So there’s

34:51 this sort of like generally like

34:53 characterizing what kind of of of thing

34:56 you’re interacting with because then

34:57 it’s sort of you know just like anything

34:59 in security you need to have like a

35:01 mental model of sort of who what’s the

35:03 risk like what what what can they do

35:05 what do they tend to do and so you know

35:07 this is like broader like character

35:10 training you know there’s we we’ve

35:12 talked about um like the like

35:14 constitution and the docs where we sort

35:16 of like explicitly want to shape the

35:18 character of Claude for example of like

35:20 what kind of behavior or should it be

35:21 okay doing? And that’s that’s the goal.

35:24 But then the role of interpreter of the

35:26 parts of alignment is then like trying

35:28 to see like well did that work you know

35:30 uh we train claw to be helpful harmless

35:32 and honest and we can sort of like look

35:34 inside and see like is it you know or is

35:36 it the case that it’s behaving well but

35:39 if you look inside actually it’s like oh

35:40 I’m only behaving well because Lenny’s

35:42 watching right now you know and if if

35:44 you you weren’t it would do something

35:45 else. So there’s this sort of you know

35:47 trying to to understand what kind of

35:50 behavior we formed and trying to mold it

35:52 and trying to check that we’ve molded

35:53 the right way. And then the second part

35:55 is this sort of well yeah just observe

35:58 the model and see if it’s doing bad

36:00 stuff. For the the basics of that you

36:02 don’t necessarily need interp. You can

36:04 just like have another language model

36:05 you know read transcripts and be like

36:07 hey did you know did this model do

36:08 anything bad here and stop it. But you

36:12 can also use interp to sort of try to

36:14 get at parts that are you know these

36:16 unverbalized parts like that these parts

36:18 that only exist in these intermediate

36:19 results like models are able to

36:22 represent stuff in their intermediate

36:23 results that they don’t that they don’t

36:24 say as we’ve established and so one of

36:26 the roles of interp is to try to try to

36:28 find you know try to identify cases when

36:30 this happens and prevents them. Um, and

36:33 then one thing that my team works on

36:34 that I think is is along those lines is

36:37 kind of a a fun bit of alignment as I

36:38 think of it as as almost, you know,

36:40 being like a like like a a maybe not a

36:43 PI, but like they’re like investigations

36:45 or deep dives. You know, we’ll see a

The model that was ‘possessed by a demon’; Golden Gate Claude

36:47 model do something really weird and

36:48 really perplexed by it and we’ll try to

36:50 understand what happened. Um, so a

36:53 recent example of this is there was this

36:56 model we were training and sometimes in

36:58 math problems it would go crazy. It

37:00 would do some math. It would get to an

37:02 answer. I think it was 12. And then it

37:04 would it would immediately go back and

37:05 say like, “Oh no, oh my god, the answer

37:07 actually is for sure 36. I should say

37:09 36.” And it would do math again. It

37:11 would get to 12. And then immediately it

37:12 would say, “No, actually the answer

37:13 should be 36.” And slowly it would like

37:15 start descending into madness, writing

37:17 in all caps. Literally saying like, “I’m

37:19 possessed by a demon.” And you’re like,

37:20 “What is going on? Like what why is it

37:22 losing its mind here?” And if you look

37:25 inside again these intermediate results

37:27 uh at what’s going on in this in this

37:29 example, you’ll see that over the prompt

37:31 that just described the math problem

37:32 like before the model even did anything,

37:34 there was a huge like uh intermediate

37:36 result activation for 36. Like the

37:38 model’s already obsessed with the number

37:40 36 before it’s even had a chance to like

37:42 understand what the problem is. Um and

37:45 what was happening is that the model was

37:46 able to do math correctly. The answer

37:48 correctly was 12. but it had been

37:51 trained by accident on similar questions

37:55 where the answer was incorrectly marked

37:57 as 36. And so the model knew that the

37:59 correct answer was 12 and it also had

38:01 sort of been optimized to say 36. It

38:04 kind of like had this pressure of saying

38:05 36 because that’s what had been

38:06 rewarded. Um and so it was sort of going

38:09 haywire just because it had memorized

38:10 this result. It’s sort of like you see

38:12 this weird behavior and you look inside

38:14 um and you try to see if what you find

38:15 inside can help you understand what’s

38:17 going on so that you can sort of improve

38:19 the turtle alignment of this model and

38:20 future models.

38:21 >> Along those lines, there was famously

38:23 the Golden Gate Claude which was just

38:24 obsessed with the Golden Gate and every

38:26 answer and kind of find some way to

38:28 reference the Golden Gate Bridge.

38:30 >> Yes. Yes. Well, that one was an interp

38:33 experiment, right? Right. So that like

38:34 the cool part of that one is we just

38:36 found the intermediate results that

38:38 match the gong gate and usually it’s

38:40 only on when the model like talks about

38:42 the gong gate, but we just turn it on

38:43 always.

38:44 >> Oh, it’s super intentional to see what

38:46 would happen.

38:47 >> Well, yeah. And and and sort of it’s

38:48 also one way that you convince yourself

38:49 that you found the right thing, right?

38:51 Like how do you know for sure that this

38:52 thing we decoded that we think is the

38:53 golden gate sign is actually the golden

Helpful, harmless, honest; taking continuous improvement seriously

38:55 gate thing? It’s like

38:56 >> turn it on always and see if the model

38:58 behaves like that. And it did.

39:00 >> So funny. So coming back to this

39:03 alignment stuff. So, so this is

39:04 interesting. So, your org is under the

39:07 AI alignment org and just thinking

39:10 through how it takes to align AI to do

39:11 things we want. What was what was the

39:13 three components that you describe?

39:14 Honest. Um, what are the kind of the

39:17 >> Oh, uh, helpful, harmless, and honest.

39:20 >> Helpful, harmful, and harmless, and

39:22 honest. So, there’s like the train AI to

39:25 be aligned, and then there’s the okay,

39:26 let’s watch it and see how it operates.

39:28 Are those kind of the two sides of the

39:29 coin of how we solve this problem? Yeah,

39:31 it’s it’s sort of, you know, really

39:34 integrated is how I would describe it,

39:35 right? You can’t really separate like

39:37 have one team that’s just like, hey,

39:39 we’re the, you know, assembly line

39:40 making clouds and another team that’s

39:42 sort of like, oh, we looked at them and

39:43 they’re they’re no good. You know, you

39:45 kind of there’s this tight loop of we

39:47 notice some pattern that’s that’s like

39:49 concerning or some behavior that could

39:51 be improved and then you sort of like

39:52 immediately um like change training so

39:56 that you can improve on it.

39:57 >> Awesome. I feel like Anthropic gets so

39:59 much flack for this that I feel like is

40:01 so unfair. Anthropic’s I think official

40:03 company name is an AI safety research

40:06 lab, something like that. And it was

40:08 always the like the reason Anthropic was

40:10 founded is because they were afraid that

40:12 people weren’t taking safety seriously

40:14 enough. And then Daario is always out

40:16 there, this is really dangerous. We got

40:18 to be really careful. We got to go

40:19 really slowly. And everyone’s just like,

40:21 no, he’s just saying that for marketing

40:22 reasons. He’s trying to convince

40:24 everyone to fund them and all these

40:25 things. And now it’s like, wait,

40:27 it’s really dangerous. We got to be

40:28 really thoughtful about this stuff. And

40:30 I feel like anthropics get a lot of

40:32 unfair criticism for this. And um I know

40:35 you guys have been thinking about this

40:36 for a long time. Um I don’t know

40:38 anything there. A lot of the sort of

40:41 actions that like certainly

40:44 the alignment team takes and I think

40:46 like Ranthropic takes just kind of

40:48 predicated on like take the idea that

40:51 this technology is going to keep

40:53 improving seriously

40:55 just that’s it because if you do you

40:59 know you say and you say you know and

41:01 let’s say let’s say you rewind the clock

41:03 back to 25 or something or 24 and you’re

41:06 like hey I think you know in a couple

41:08 there’s good chances that Claude can

41:12 just basically write code as well as

41:15 like a trained you know software

41:16 engineer and and and you know a few

41:19 years later I can like write it better

41:21 than than any engineer you know and and

41:23 there’s always uncertainty with this

41:25 like when is it going to happen how

41:26 likely but if but if you kind of like

41:27 take that seriously then I think it’s

41:30 just about doing the responsible thing

41:32 conditioned on that worldview right like

41:34 okay well if this is happening you know

41:36 then then we should make sure that sort

41:39 of um like cyber security concerns are

41:41 addressed and things of that nature. And

41:43 so actually I I kind of think of it as

41:46 like to me sometimes this debate is

41:47 almost when I feel like like people on

41:49 the other side disagree it’s almost like

41:51 I feel like ah but I think I think they

41:53 don’t really believe that this

41:54 technology is going to continue to

41:55 improve and then it’s true that it’s

41:56 sort of like not it’s not that scary and

41:58 and I think empirically so far the pace

42:00 of progress has been has been fast and

42:02 it’s showing no sign no sign of slowing

42:04 down. So, I think that’s why a lot of

42:05 this safety work is is important to keep

42:07 doing.

42:07 >> What’s something you think people maybe

42:09 misunderstand about this AI safety

42:11 stuff? Just what’s going on right now?

42:13 Is there something you’re just like,

42:14 “Come on, okay, here’s what you need to

42:15 know. Here’s something you might be

42:16 missing.”

42:17 >> This might be a bit of a sidetrack, but

42:18 my take would be that a lot of the a lot

42:21 of thing that I think people maybe a

42:23 little bit outside of of the core sort

42:26 of like AI safety effort or that haven’t

42:28 been steeped in it for as long like

42:30 react

42:32 strongly to. and I think is is maybe not

42:35 super helpful or at least is worth

42:36 thinking about more is the sort of

42:37 question of um anthropomorphization, you

42:41 know, like should you think of the

42:42 models as uh like should should you

Anthropomorphizing is a useful frame (three reasons)

42:46 model uh the should you model the model

42:49 as like something that’s closer to some

42:52 like entity or or closer to a tool? And

42:55 I actually think that both are wrong. uh

42:59 there’s a sense in which all mental

43:02 models are wrong but some are useful

43:04 right uh and that in terms of

43:07 understanding the kind of stuff that

43:08 happens and predicting the kind of risks

43:10 that materialize the anthropomorization

43:12 is actually a useful way to talk about

43:14 LMS and I have like a few reasons why I

43:16 get into them but I think that’s

43:17 probably my main take here

43:18 >> yeah please share because I know um Thor

43:20 Cash famously I don’t know famously put

43:23 out this whole story of the hugging face

43:25 incident and people criticized them for

43:27 anthropomorphizing AI

43:29 So it’s a really interesting to hear you

43:30 would think that’s actually a useful way

43:32 to think about it. Talk talk more about

43:33 that.

43:34 >> Yeah. Yeah. It’s it it’s funny because

43:37 on a smaller scale um when I worked on

43:40 showing the LM’s like plan ahead there

43:43 was some push back on the term planning

43:45 again because it was sort of like ah

43:46 well can you really say that something

43:48 that’s just predicted next token is

43:49 planning and I think you know there’s

43:51 this there’s this kind of like semantic

43:54 debate where you can like arbitrarily

43:55 like define terms and and sort of and

43:57 then there’s just the like what is the

43:59 terminology that actually describes what

44:01 is happening like what is what is the

44:03 word that actually describes what is

44:04 happening these And so here’s why, you

44:06 know, I think Ellen’s plan, but also

44:08 here’s why I think this this language

44:11 that’s that’s anthropomorphic can often

44:13 be really useful. Um, if you think back

44:16 to what we were saying about how the

44:17 models are made, uh, first of all, they

44:20 predict massive amounts of text written

44:22 by humans. That’s like the first first

44:24 step. uh and and as as we kind of are

44:27 learning more and more by looking inside

44:30 uh you know doing this well requires

44:32 kind of building up a model of the world

44:36 like a model of what what kind of worlds

44:38 generated all of these texts and we can

44:40 find you know these these sort of like

44:42 that these models are learning these

44:44 these general ideas these general

44:45 concepts um so that’s one so they have

44:49 like you know an understanding kind of

44:51 of of the world it’s not just a surface

44:53 level like only the text to like

44:55 understand the process that made the

44:56 texts. Then there’s a second reason

44:59 which is that as part of this training

45:01 they also encounter millions of of like

45:03 dialogues. You know predicting the next

45:06 character in a dialogue requires

45:08 behaving like the character in the

45:09 dialogue would. And so in that sense

45:11 like part of their training I think can

45:13 best be thought as like training an

45:14 actor where it’s like you know it’s

45:16 reading some some play and it has to

45:19 play all sides because it’s predicting

45:20 all the words and so it has to sort of

45:21 like act as you know this specific

45:24 character that like was scorned in act

45:26 one and like enacted revenge in act two

45:29 and is facing the consequences in act

45:30 three. And to do that well you sort of

45:32 like need to model what like the

45:34 motivations and the feelings of that

45:35 character might be. And then another

45:37 thing that people might not know is when

45:38 you talk to claude when you talk to

45:39 these assistants like you know you go to

45:41 claw.ai the way that is formatted to the

45:44 model is just as a dialogue like as a

45:47 conversation between you and the model.

45:49 So it’s literally you know says like

45:50 human colon uh hey claude like help me

45:54 uh write this email and then assistant

45:57 colon claude’s answer. So Claude is

45:59 literally you know in this like

46:01 roleplaying as a helpful assistant in

46:04 this dialogue that is much like all of

46:05 these dialogues that has been trained

46:06 on. Okay. And there’s a third reason and

46:09 the third reason is that another part of

46:11 training that’s a big part of training

46:14 nowadays uh all around is sort of um

46:17 outcome oriented tasks where you train

46:19 the model to be good at coding for

46:20 example. And to do that you just have it

46:22 do whatever it wants and then it writes

46:24 some code and then you have some program

46:26 on the other side that just checks if

46:27 the code is right. And so the model’s

46:29 just going to have sort of a bunch of

46:31 optimization pressure to just try to

46:33 learn a bunch of different strategies to

46:35 do this well. And it turns out that some

46:38 of these strategies materialize in

46:40 behavior that can look surprisingly

46:41 human which isn’t that shocking uh

46:44 because you know things like deception

46:46 like hey I can get I can get the the

46:50 reward for this task if I like write a

46:54 program that the the like uh the the

46:56 program that’s verifying it will get

46:58 fooled by even though it’s not exactly

47:00 what it is. Um like if that works then I

47:03 will learn to do that. Frustration. Ha.

47:06 I’ve been trying this approach for a

47:07 while to solve this hard problem. It’s

47:09 not working. So that’s an incentive to

47:11 try something different or like

47:12 surprise. Oh, I was expecting this to

47:14 like produce this answer, but it didn’t.

47:17 Maybe I have like my mental model of it

47:19 wrong. And so as you apply enough

47:20 pressure on it and you’re starting from

47:21 this prior of like the model already

47:23 knowing all these concepts is going to

47:24 reinforce the ones that are useful to do

47:26 all of these tasks. And so what do you

47:28 get at the end? You get this sort of

47:30 model that can can be a super good

47:32 simulator or actor of all of these

47:34 characters. and has been trained to sort

47:37 of act as a character in dialogues and

47:39 also to sort of like learn behaviors

47:42 that lead it to complete tasks well. And

47:45 like the best way to sort of understand

47:46 how that thing behaves is not like

47:49 exactly like a human, but man kind of

47:52 like something that’s mimicking a human

47:54 is really close and is much closer than

47:57 a program. Like a program is a very

47:59 predictive thing. This is something else

48:00 entirely. And so that’s why I think it’s

48:02 actually a super useful frame.

48:03 >> Wow. Oh my god, there’s so much

48:05 interesting stuff there. How often is

48:07 your mind just kind of blown thinking

48:09 about basically we’ve created this alien

48:11 being that we are now under trying to

48:13 understand what we have created. How

48:15 often as you watch it you’re just like,

48:17 “Wow, that’s crazy. I didn’t expect

48:19 that.”

48:20 >> Yeah. A lot. Um, this is just next token

Consciousness and emergent complexity

48:25 prediction on a bunch of text. uh you

48:27 know it’s not it it it hasn’t been like

48:31 you know it’s it’s it it goes against so

48:33 many of our intuitions for like no the

48:34 way that you make an intelligence you

48:35 know you’d have like like you give it

48:37 lessons you bring it up like you do all

48:38 this complicated stuff and it’s like no

48:39 absolutely not we do this incredibly

48:41 simple thing and yet we’re at the point

48:44 where it’s breaking new ground in

48:45 mathematics I think there’s sort of if

48:48 you hold these two thoughts in your head

48:50 there’s two reactions you can have you

48:52 know there’s denial and there’s

48:54 curiosity where denial is just this is

48:58 just not possible if you’re doing all

49:01 you’re doing is predicting the next word

49:02 you you it’s it’s not possible that you

49:03 could ever like match you know the the

49:06 like best specialists in our field uh

49:08 and I think that position just gets

49:09 harder and harder as as we get more

49:11 empirical evidence that actually that is

49:12 what’s happening uh and curiosity is

What the field feels like; how to get in

49:15 kind of the road I’ve ended up choosing

49:17 right which is just like well like how

49:19 is how is this happening you know let’s

49:21 open up let’s let’s look inside um how

49:24 could it be that you know um one of the

49:27 early projects I did at Anthropic is I

49:28 learned SQL a programming language and

49:30 it took me years to master and then as a

49:32 little like side project I I got Claude

49:36 to learn SQL and it was better than me

49:38 at it and I’ll never be better than it

49:39 than than Claude at SQL ever again and

49:41 it took it you know just like this this

49:43 kind like simple process how could this

49:45 be um and I think there’s so much depth

49:49 in that question and and just so much to

49:51 explore and it’s it’s to me so

49:54 fascinating to just every Hey, try to

49:56 unpack that. Try to sort of like peel a

49:57 few layers of the onions back. Like,

49:59 okay, well, clearly this is happening.

50:01 The simple objective is giving rise to

50:03 this super complicated thing like, okay,

50:05 let’s figure out why. Let’s let’s figure

50:06 out, oh, it’s learning these general

50:08 concepts. Oh, it has these circuits. Oh,

50:09 okay. What’s the next question? What’s

50:11 the next thing we don’t know? Um, and

50:13 and my sort of I don’t know, my sort of

50:17 maybe most optimistic take on this is

50:18 that I think we’re going to learn a lot

50:20 about just the nature of intelligence

50:23 itself, right? Of sort of like how you

50:25 get this super complex thing that

50:28 emerges from from simple objectives. Um,

50:31 and in fact, you know, I would argue

50:32 that we’re another example of this like

50:35 evolution pretty simple objective. you

50:38 know, you sort of like shake things

50:40 around for a while and and and and like

50:43 keep the keep the organisms that that

50:45 manage to adapt to the environment and

50:47 at the end you get jazz and and

50:49 podcasts, you know, that’s kind of crazy

50:51 as well. And so I I I think there’s a

50:53 sense in which there’s there’s some

50:54 fundamental truth that we can learn by

50:55 studying these things.

50:57 >> Wow.

50:59 Something that comes up as you say that

51:01 is what’s interesting now is AI is

51:03 helping us learn and evol and you know

51:06 progress like AI tutoring and all these

51:08 things just like it’s like interesting

51:10 how now is helping us accelerate our own

51:13 understanding of the world and I wonder

51:16 if like this point you made about how

51:18 the way we learn was lessons AI is

51:21 learning just processing a ton of data I

51:23 wonder if that’s how we end up I wonder

51:25 if we change how our education system

51:26 based on you know what LMs have been

51:29 have been trained on.

51:31 >> Yeah. Hopefully, you know, we don’t

51:32 change it just so that we ask we we get

51:34 students and we’re like, “Hey, here’s a

51:36 thousand words. Predict the next one.

51:38 >> Just read it all.”

51:39 >> Yeah.

51:40 >> Keep it all in your head.

51:42 >> That’s right.

51:42 >> Yeah.

51:44 Let me come back to this um kind of

51:46 safety piece because I think there’s a

51:48 lot of people that are just worried

51:49 about where things are going and I’m

51:52 curious just as someone watching this

51:54 stuff happen being within the alignment

51:57 or watching it evolve are you optimistic

52:00 we’ll figure it out? Do you think we’ll

52:01 crack this and keep AI aligned with our

52:05 interests?

52:06 >> My take on this is is is like I’m

52:08 personally optimistic. I I think this is

52:10 something where

52:12 right now a lot of people are paying

52:14 attention to it and and first of all,

52:16 you know, I think that’s kind of a great

52:17 thing. I think as an issue like this

52:19 rises in prominence, um you know, it’s

52:22 it’s exactly what we were talking about

52:23 earlier. It’s like this technology is

52:25 powerful. It has benefits, it has risks,

52:28 we have to kind of adapt and shift

52:30 resources from like benefits to risks

52:32 and back and forth um as as it

52:35 progresses. Uh, and so, you know, to me,

52:38 I feel like

52:40 sort of we’ve continued to study it. Um,

52:45 the thing that’s that’s that’s

52:46 interesting is that there’s this sort of

52:48 like as the models get better, um, they

52:52 get in some ways easier to study. Uh and

52:54 so there’s been this sort of strange you

52:56 know the models get better and so like

53:00 the world model inside them is clearer

53:02 and we can sort of like understand some

53:03 parts of them better but also get more

53:05 capable and so they’re more able to sort

53:07 of evade like basic measures of sort of

53:11 uh of security and so there’s this

53:12 there’s this push and pull as models get

53:14 better and and I think you know

53:16 basically based on this push and pull

53:17 you want to allocate more or less

53:18 resources to sort of various parts of

53:20 like monitoring like understanding their

53:22 circuits better things like Yeah. Um,

53:26 but it’s also why I’m kind of happy to

53:27 chat with you, uh, because one one of

53:30 the one of the main things that I think

53:33 is important to make progress is just to

53:35 make sure that people kind of understand

53:37 both, you know, what kind of systems

53:39 these are, like how they work, um, how

53:42 they’re made, so they can form good

53:44 mental models of of, you know, the

53:46 present of the future and and so they

53:48 can decide how to shape it, whether

53:50 that’s, you know, taking part in some of

53:52 these alignment efforts s or just like

53:55 informing them as as the public that’s

53:57 going to have to like vote and decide on

53:58 much of these issues. Like I I don’t

54:01 know. I’m I’m pretty excited for this

54:02 current moment we’re in for that reason.

54:04 >> And something I think people

54:05 misunderstand is [clears throat] people

54:07 working in these labs have taken this

54:09 problem very seriously from the

54:10 beginning. Like I said, anthropics

54:12 individual like mission was AI safety

54:14 and alignment. That was like the reason

54:15 that it became a company. and and a lot

54:18 of people are hearing this for the first

54:19 time or think that it’s not a serious

54:21 area of study and that people aren’t

54:23 focused on it but many smart people care

54:26 a lot about this and are working on it

54:28 and there’s a lot of from what I can

54:30 tell a lot of good progress is happening

54:31 the other point you made about the

54:33 models growing very smoothly in

54:35 intelligence like you know there’s been

54:38 this question of are we going to have a

54:39 fast takeoff or a slow takeoff like

54:42 there’s always this fear fear that we

54:43 would hit a point recursive

54:45 self-improvement for example and just

54:46 takes off super intellig ence. So we’re

54:48 huge trouble. We have no control. It’s

54:49 out of the box. But it feels very much

54:51 like we’re on the slow takeoff scenario

54:54 because as you said the models are kind

54:56 of very smoothly improving and where we

54:58 are catching these little things along

54:59 the way learning evolving putting

55:02 regulation around it. So you know of the

55:05 scenarios this feels like a pretty good

Optimism; why interp gets both easier and harder

55:06 path we’re on of of the of the of the

55:09 many scenarios people imagined and the

55:11 fears they had.

55:12 >> As sort of this pace of progress kind of

55:14 evolves. it’s just a matter of of

55:17 reacting quickly. So, I think anything,

55:19 you know, that’s why I think having

55:20 people be informed is great because then

55:22 people will like voice concerns or or

55:24 like voice their opinions and then yeah,

55:26 sort of sort of regularly evaluating,

55:29 you know, what the benefits are, what

55:31 the risks are, and how how we sort of

55:32 like mitigate each is is basically the

55:35 way that feels like the right way

55:37 forward. And and I yeah, I kind of agree

55:38 with you that it feels like sort of

55:39 there’s momentum to to do this right

55:41 now, which is which is great. Something

55:43 I wanted to ask you completely

55:44 unrelated. Uh where are you on the

55:46 simulation hypothesis? Uh being someone

55:48 that watches this all this life form

55:51 growing, are you more convinced we’re in

55:53 a simulation and this is and we’re not

55:56 the base at the base layer of all this

55:58 or or not?

55:59 >> Oh, this is a fun one. I actually

56:00 haven’t thought about this in a while.

56:03 Um

56:06 to me like there’s

56:08 there’s something else that like looking

56:10 looking inside these models kind of

56:12 brings out which is this sort of

56:15 like like almost like there’s a

56:17 fundamental structure that they keep

56:20 falling on. This is related to your

56:21 question but I promise I’m going get to

56:22 it. Let me explain. Um I’ve studied like

56:25 a bunch of cloud models and also you

56:27 know some some different models some

56:28 some like open source work and and then

56:31 a lot of time when you look at how

56:34 what they learn and what the structure

56:36 inside is it’s surprisingly similar you

56:39 know and this is for models that are

56:41 trained years apart on like different

56:42 data by different people and you know

56:45 and you look at how they solve these

56:46 problems and there’s just so much

56:48 overlap and my take on why that is is

56:52 that for some problems that are

56:54 important, there’s just one efficient

56:56 way to do things, right? Like there’s

56:57 kind of one way to do math efficiently

57:00 that like fits the structure of our

57:01 brain and we all learn it and very few

57:02 people are doing it crazy different way,

57:04 right? everybody works in, you know,

57:05 base 10, not many people in like base

57:07 six or whatever. Um,

57:10 I think that that to me makes me think

57:12 that there’s like some fundamental

57:14 structure of like information that like

57:17 these models are picking up on that in

57:19 some ways like we’re picking up on and

57:21 that, you know, at maybe like a a

57:23 crazier sort of like simulation level

57:25 that

57:27 kind of is is is like regularity in our

57:30 world. And so that maybe makes me think

57:32 that like if you were to have the power

57:35 to simulate something like our world

57:37 actually like you would land in sort of

57:39 a similar situation like there’s just

57:41 kind of this like basin of where things

57:43 end up and they’re just the sort of like

57:45 easiest way for things to arrange

The simulation hypothesis

57:46 themselves. And so in that sense

57:49 [snorts]

57:50 maybe yeah I’m I’m sort of I I could

57:53 believe that we’re in a simulation. I

57:55 think I’m not sure I fully buy it. Uh,

57:57 the other reason I don’t fully buy it is

57:58 that even though I think the structure

57:59 like lends itself to it, there’d be a

58:01 question of like why why are we like

58:04 like who’s who’s running this and for

58:05 what? And and that one I don’t know

58:07 about. Well, something Elon said has

58:09 stuck with me about this that if you

58:11 were to run a simulation, it’d be a

58:13 really interesting it’d be like a an

58:15 interesting time in history or something

58:17 that you’re really excited about and if

58:20 tracks that this is a very interesting

58:21 time in history. [laughter]

58:23 >> Yes. Yeah. Okay. That’s a fair point.

58:25 That’s a fair point.

58:26 >> Data centers in space and robots and

58:29 artificial intelligence.

58:30 >> Yeah. Okay. I see it now. I see it now.

58:33 Yeah. Somebody like recently told me,

58:35 you know, they used to think that may

58:36 you live in interesting times was was

58:39 like a a blessing and they’re not so

58:41 sure now. They’re like, you know, the

58:43 boring times were are like pretty good,

58:44 too.

58:45 >> Oh, man. And then there’s an Emanuel out

58:48 there watching your thinking process in

58:50 your JSpace.

58:52 >> Oh my god. Yeah. Hopefully I’m making

58:53 him proud. He’s not like, “Oh, we got to

58:54 rerun the simulation. We messed up this

58:56 one.

58:56 >> Shut it down.” No. Oh, man. Okay. Uh,

59:00 one last question randomly out of

59:02 nowhere. Uh, so Swix, who um runs AI

59:06 engineering, latent space podcast,

59:08 awesome guy. You were asked him what to

59:10 ask you. You were on his podcast a year

Jev, and gut decisions vs. deliberation

59:12 ago and uh something completely from a

59:14 different direction. He wanted to get

59:15 your take on Jev, which is this hot new

59:18 model that recently came out that is

59:20 very fast making decisions. Uh, OpenAI

59:24 funny enough just launched their own

59:25 version like a week later. Uh thoughts

59:28 on on Jev in the space of quick

59:30 decision- based state uh models.

59:32 >> Yeah. Right. So my my take on Jev is a

59:34 sort of you know these kind of models.

59:36 You give them some context and you give

59:39 them a few options to choose between and

59:40 they just are very quick at at choosing

59:42 an option for you. Uh to me it’s

59:46 impossible not to hear about this and

59:47 think of of sort of system one thinking

59:50 versus system two or something, right?

59:52 So like a lot of what I’ve studied with

59:54 these models is how they sample one

59:57 token with with like the cloud models.

59:58 And so you know you can think of of of

60:01 like these decision models are as pretty

60:03 similar. So like take an LM and sample

60:05 one token. They’re they’re different

60:06 important ways but like at a fundamental

60:08 level you can be like oh it’s you’re not

60:09 letting the model you know sample and

60:12 like reason. It’s just you’re like gut

60:14 gut feeling you know is the answer A or

60:16 is the answer B. Um, to me that seems

60:20 like like something that’s like that

60:22 general frame of like sometimes you’ll

60:24 just want a binary decision or like one

60:25 out of 10 or whatever is just like

60:27 clearly I don’t know like like both

60:30 clearly useful uh and something that I

60:33 imagine is is like probably going to be

60:36 a part of of like intelligent systems

60:39 right where you you have for some stuff

60:40 you need to think about a long time you

60:42 know it’s like what what’s our roadmap

60:43 for the next year you need to think

60:44 about for a while and then for some

60:45 stuff where it’s

60:47 you know, is this is this person who

60:49 wrote us a review happy or upset? You

60:51 kind of you’re happy with the sort of

60:53 gut judgment. Um, so yeah, I think I

60:55 think this is I think this kind of

60:57 research is is cool. Uh, and I think

60:58 it’s also kind of different from the,

61:00 you know, it’s like it’s it’s like a bit

61:02 of a novel take on on some of the LMS

61:04 we’ve seen. So, yeah, I’m pretty excited

61:06 to see where where where this kind of

61:07 stuff goes. So I’m hearing here is this

61:09 kind of going to be a new foundational

61:11 component most likely of of AI and LMS

61:14 and the system one system two stuff is

61:16 basically conamman’s kind of I have that

61:17 book up there actually behind me this

61:19 idea system one just to keep it simple

61:20 is just like kind of your migdala lyic

61:22 system making just like gut level quick

61:24 decisions like snake oh no

61:26 >> yes

61:27 >> uh where you don’t have to think long

61:28 about it it’s kind of to protective

61:30 protect you from bad things really

61:32 easily and quickly and system two is

61:34 where we really think about stuff and

61:35 process

61:36 >> yeah exactly

Weird little things (a gut check; thinking about something else)

61:38 Sweet.

61:38 >> Emanuel, is there anything else that we

61:40 haven’t covered? Anything else you want

61:42 to say or mention? Anything you want to

61:44 double down on before we get to our very

61:46 exciting lightning round?

61:48 >> I’m definitely curious for for uh folks

61:52 takes always on on some of the stuff

61:54 that you know uh is still unexported or

61:58 didn’t make sense. There’s a lot more

62:00 interp experiments than we have the time

62:02 to share. You know, models have a bunch

62:03 of weird weird little things about them.

62:06 Like maybe I’ll do a lightning round of

62:07 weird little things, but that’s okay.

62:08 >> Let’s do it. Absolutely.

62:09 >> Okay. Weird little thing. One, um, we

62:12 found that, you know, models can like

62:15 actually answer questions. Obviously,

62:17 they can do math or they can, you know,

62:18 you can ask them like, what’s the color

62:20 of a banana? And they’ll actually like

62:22 you know if you look inside you’ll see

62:24 all these intermediate steps but they

62:25 have a separate mechanism which is kind

62:27 of like a gut check uh and which is

62:29 totally different and which is just like

62:31 as they reason they’ll have like some

62:32 part of them that will flag like

62:34 something’s wrong. So if a model like

62:36 somehow says you know 4 + 5 equals 23 or

62:39 something and there’s a part of them all

62:41 that just like reads that and doesn’t do

62:42 the math but it’s just like that can’t

62:44 be right because like 4 plus 5 is

62:45 definitely under 10. And so they learn

62:47 these weird kind of like hunches. they

62:50 feel like kind of kind of like like a

62:52 hunch. Um maybe another one is you can

62:58 tell the model, hey, do this task and

63:00 think about this other thing and then

63:02 you can use interp to see if like

63:03 there’s intermediate representations

63:05 that map to the other thing and that

63:08 works. You sort of see that it does

63:10 think about the other thing. This is

63:11 something that’s really mysterious to me

63:13 because there’s no kind of training

63:15 reason why why the model would would

63:17 have to learn that. So, it’s sort of

63:18 like another one of these um weird

63:20 behaviors. And actually, maybe I’ll keep

63:22 that at two. I think those two are are

63:23 good enough.

63:24 >> Just what a fun what a fun job this is

63:26 just studying this thing and trying to

63:28 understand what the hell is going on.

63:30 How is it getting the smart? How does it

63:32 think?

63:32 >> Yeah.

63:33 >> Um how many like how many people are

63:35 even

63:36 >> how many people are even in this space?

63:37 Just like how large is this field?

63:39 >> The field itself is is you know is

63:41 getting pretty large. I would say maybe

63:43 hundreds not thousands. uh there’s you

63:46 know uh different inter people at at the

63:48 different labs and then there’s there’s

63:50 um in academia quite a few labs putting

Lightning round: books, movies, watch, 4000 Weeks

63:52 out great work

63:54 >> is the best uh if someone were to get

63:55 into the space are they is it like an ML

63:58 person usually or is there like a

64:00 neuroscientists sort of track to get

64:03 into this

64:04 >> yeah there’s actually a surprising

64:06 amount of of you know exneuroscientists

64:09 uh I do think in order to get in the

64:11 field you know there’s a there’s is

64:14 actually it doesn’t require a lot of

64:18 prior knowledge because it’s such a new

64:20 field. So, it’s actually you know the

64:21 on-ramp is is is mostly I would say like

64:24 self-studying and reading some of the

64:26 some of the the key papers and then

64:28 there’s actually you know um yeah maybe

64:31 I should give a plug. There’s actually

64:32 like websites like neuripedia.org org I

64:34 think is the URL which are sort of like

64:36 open- source interp uh so they they like

64:39 host you know a bunch of models and let

64:41 you sort of do interp experiments on

64:42 them and so you know people have just

64:44 been been doing that uh on their own and

64:46 I think that there’s so much to explore

64:49 uh that if if you’re at all curious you

64:50 should just give it a try.

64:52 >> That’s awesome. We’ll link to that in

64:53 the show notes in case that’s not the

64:54 right URL.

64:55 >> Cool.

64:55 >> Well, Emmanuel, with that we’ve reached

64:57 our very exciting lightning round. I’ve

64:58 got five questions for you. Are you

65:00 ready?

65:01 >> I am as ready as I’ll ever be. Here we

65:03 go. What are two or three books that you

65:05 find yourself recommending most to other

65:07 people?

65:07 >> One I read recently and is a brief

65:11 history of intelligence. Uh it’s this

65:15 really cool book that walks through the

65:18 history of like basically brains and

65:21 five key moments uh in in our evolution

65:24 that sort of bootstraps these brain

65:26 systems on top of one another. It’s

65:27 super accessible like you don’t need any

65:30 neuroscience knowledge or anything. Uh,

65:33 another one is Story of Your Life and

65:34 Others. This is Ted Chiang. This is

65:36 maybe my favorite sci-fi book of all

65:38 time. It’s a bunch of short stories. It

65:41 does what I think is the the like high

65:43 art version of sci-fi where it’s just

65:45 take one weird idea like what if you

65:47 know uh this happened and then write a

65:50 story of just like what would the world

65:52 look like if that happened? Um, it’s

65:54 truly excellent.

65:55 >> I think I have that on my bookshelf

65:57 behind me too. Um, also highly recommend

65:59 that one. Favorite recent movie or TV

66:03 show?

66:05 >> Okay, one popular one, one deep cut. Uh,

66:08 Obsession was my favorite movie of the

66:10 year. Uh,

66:13 it’s a horror movie, so don’t don’t go

66:14 if you don’t if you hate horror movies.

66:16 But

66:16 >> Oh, thank you.

66:17 >> Oh, no. Even if you’re if you’re not

66:19 scream, if you’re open to a horror

66:20 movie, you should go cuz it’s

66:23 >> it was just so fresh. You know how

66:25 movies I feel like if you’ve seen a lot,

66:27 sometimes you’re like, “Oh, it’s another

66:28 I’ve seen this many times.” This one was

66:29 just so new. so fresh and I’ll say one

66:33 thing about it which is that it takes a

66:34 lot of romcom tropes and like subverts

66:37 them to said be spooky and I think that

66:39 that’s just a a brilliant brilliant way

66:41 to to make a movie and then I’m French

66:45 so there’s [snorts] a TV show that came

66:46 out this year called breath is French

66:49 for brief uh brief used to be this cult

66:52 classic in France 10 like 15 years ago

66:55 and they came up with a sequel uh many

66:58 many years into the

67:00 And you can watch it. I had a bunch of

67:02 American friends that have watched it.

67:03 It’s got subtitles and it’s just this

67:05 incredibly

67:07 again it’s like novel like the editing

67:08 is crazy. Um they are just extremely

67:11 creative and is very short. Uh so I

67:13 would recommend that.

67:14 >> Favorite product you recently discovered

67:16 that you love?

67:17 >> I I love my Apple Watch and I hate that

67:20 it charges uh or I have to charge every

67:23 day. So I bought the Apple Watch Ultra

67:25 uh just recently and man that is so

67:28 nice. If you’ve ever been frustrated by

67:30 this good product that somehow we can’t

67:32 figure out how to give a good battery

67:33 to, well, it turns out if you make it

67:36 bigger, uh, the battery’s better. So,

67:38 I’ve been happy with that.

67:39 >> Do you have a favorite life motto that

67:40 you find yourself coming back to often

67:42 in work or in life?

67:43 >> There’s like one non-fiction book that

67:45 really is like probably my favorite

67:47 ever. It’s 4,000 weeks. Um, it has a lot

67:52 in it that that I think is cool. Uh, one

67:55 of the cool things is this sort of thing

67:57 that was really relevant to me, which is

67:59 that like the when I finally mentality

68:02 is a common mentality and is kind of

68:04 toxic. So, so what does that mean? It’s

68:05 sort of this like ah when I finally, you

68:08 know, check off this this all the items

68:11 on my to-do list, I’ll finally be able

68:12 to chill. When I finally get like

68:15 married, I’ll be happy. When I finally

68:16 get a promotion, uh, and then this is

68:18 sort of I mean, you were talking about

68:19 meditation, right? This is like almost

68:21 the opposite of meditation and that you

68:22 know at the end of the day your life is

68:24 like all of the moments you lived uh in

68:26 the moment and that there’s this sense

68:28 in which you should just like try to

68:30 always like be you know in the moment

68:33 and not projecting yourself towards like

68:35 a future that seems better like probably

68:37 if you’re telling yourself when I

68:38 finally do X I’ll be happy when you do X

68:40 you will not be happy. Is there a better

68:42 phrase to use or is it just don’t do

68:44 that? Like do they say

68:46 >> uh I don’t know. I think to me the

68:48 positive version is like be in the

68:49 moment, right? It’s just like just I

68:51 appreciate the appreciate the moment.

68:52 There was a blog post recently I read

68:54 that was

68:54 >> I for something like

68:56 >> like enjoy the sandwich. I think is I

68:58 think it was literally the sense. I

68:59 don’t know if you saw this where it was

69:00 like somebody was like yeah like just

69:02 you’re eating a sandwich, you know,

69:03 you’ve eaten many sandwiches like enjoy

69:05 the sandwich. You’re in you’re in like a

69:07 a team meeting, you know? Yeah. you’ve

69:09 been in like like enjoy like be be in it

69:11 and and sort of not looking looking

69:12 towards the next thing.

69:14 >> I did a meditation retreat once. It was

69:16 a 10day silent retreat where you eat

69:18 silently too and no one’s talking. You

69:20 just sit there and eat your food. And I

69:21 just remember peeling an egg for like

69:23 five minutes and just noticing every

69:25 every shell piece and just the smell of

69:27 it all. And and it like you almost it’s

69:30 like hard to get into that mode, but

69:32 it’s such an interesting place to be

69:34 where everything is just interesting.

69:35 Like wow, look at this egg versus just

What an AI would see inside his brain; how to be useful

69:37 like stuff in my mouth.

69:39 >> Yeah. Right. While you’re like on Slack

69:41 with your left hand and like listening

69:42 to a podcast at the same time.

69:44 [laughter]

69:45 >> Oh man. Okay. Final question. You study

69:48 AI all day. You’re watching AI. how it

69:50 thinks. If uh if an AI were to look

69:52 inside your brain, what would what would

69:54 be surprising? What would it see that

69:56 might surprise people?

69:58 >> I mean, surely my deepest darkest

69:59 secrets. Uh but what are those?

70:01 >> Let’s get into it. Let’s get into

70:02 [laughter] it.

70:03 >> Oh, man. Uh I’ve always been like a bit

70:06 of a a jokester. Uh or that’s just how I

70:11 like both how I live and how I cope. I

70:13 just like think that even in serious

70:15 situations, you should you should aim to

70:17 kind of have some levity. And so I think

70:19 if if you looked inside my brain, you’d

70:21 definitely be surprised uh that like

70:23 basically at any given moment I’m just

70:26 like focusing on some random thing that

70:28 is funny. You know, I’ll be in a in a

70:30 serious meeting and I’ll be like, “Oh

70:31 yeah, but you know, like isn’t it kind

70:33 of random that you’re like wearing this

70:34 random thing or something?” It’s just

70:36 like I have these intrusive thoughts

70:38 that I that I keep out, but that are all

70:40 usually like just bad jokes. And

70:42 sometimes, as you’ve heard today,

70:44 unfortunately, they come out. So, I’m

70:45 sorry for those, but that’s that’s

70:46 probably the most surprising thing.

70:48 >> I love that. Emanuel, thank you so much

70:50 for being here. Uh, final question. How

70:51 can listeners be useful to you?

70:53 >> Just do some interp read some read read

70:56 about how the models work. I mean, you

70:58 mentioned like dorkers thread up. I

71:00 think it was it was great as well. I

71:01 think I think, you know, it’s just this

71:03 is such a new field like both interp and

71:05 like AI and LLMs and I think it it just

71:07 benefits from from people engaging and

71:10 and um being informed. So like read

71:14 things. If there’s if there’s more that

71:16 you would want to know about how these

71:18 models work that we haven’t figured out,

71:20 you can, you know, reach out to me. I’m

71:21 on variety of social social platforms.

71:24 Like probably you can reach out to me on

71:25 X

71:26 >> and we’ll link to some of the videos

71:27 that you guys have put out that describe

71:29 a lot of the stuff we’ve been talking

71:30 about like JSpace and things like that.

71:32 There’s some really beautiful videos

71:33 your team has made. I don’t know who

71:35 does that, but they’re really cool.

71:36 >> Yeah, this is we partner with our our

71:38 design team. So all credit goes to them

71:40 for for the cool videos.

71:41 >> Amazing. All right. Well, thank you

71:43 again, Emanuel. Thank you so much for

71:44 doing this. This was amazing.

71:45 >> Thanks, Lenny. Yeah, this was super fun.

71:47 >> Bye, everyone.

71:49 Thank you so much for listening. If you

71:51 found this valuable, you can subscribe

71:52 to the show on Apple Podcasts, Spotify,

71:55 or your favorite podcast app. Also,

71:57 please consider giving us a rating or

71:59 leaving a review as that really helps

72:01 other listeners find the podcast. You

72:03 can find all past episodes or learn more

72:05 about the show at lennispodcast.com.

72:08 See you in the next episode.