site banner
naraburns  nihil supernum  1yr ago (text post)   38429 thread views

Culture War Roundup for the week of March 27, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

11
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Sooo, Big Yud appeared on Lex Fridman for 3 hours, a few scattered thoughts:

Jesus Christ his mannerisms are weird. His face scrunches up and he shows all his teeth whenever he seems to be thinking especially hard about anything, I didn't remember him being this way in the public talks he gave a decade ago, so this must either only be happening in conversations, or something changed. He wasn't like this on the bankless podcast he did a while ago. It also became clear to me that Eliezer cannot become the public face of AI safety, his entire image, from the fedora, to the cheap shirt, facial expressions and flabby small arms oozes "I'm a crank" energy, even if I mostly agree with his arguments.

Eliezer also appears to very sincerely believe that we're all completely screwed beyond any chance of repair and all of humanity will die within 5 or 10 years. GPT4 was a much bigger jump in performance from GPT3 than he expected, and in fact he thought that the GPT series would saturate to a level lower than GPT4's current performance, so he doesn't trust his own model of how Deep Learning capabilities will evolve. He sees GPT4 as the beginning of the final stretch: AGI and SAI are in sight and will be achieved soon... followed by everyone dying. (in an incredible twist of fate, him being right would make Kurzweil's 2029 prediction for AGI almost bang on)

He gets emotional about what to tell the children, about physicists wasting their lives working on string theory, and I can see real desperation in his voice when he talks about what he thinks is really needed to get out of this (global cooperation about banning all GPU farms and large LLM training runs indefinitely, on the level of even stricter nuclear treaties). Whatever you might say about him, he's either fully sincere about everything or has acting ability that stretches the imagination.

Lex is also a fucking moron throughout the whole conversation, he can barely even interact with Yud's thought experiments of imagining yourself being someone trapped in a box, trying to exert control over the world outside yourself, and he brings up essentially worthless viewpoints throughout the whole discussion. You can see Eliezer trying to diplomatically offer suggested discussion routes, but Lex just doesn't know enough about the topic to provide any intelligent pushback or guide the audience through the actual AI safety arguments.

Eliezer also makes an interesting observation/prediction about when we'll finally decide that AIs are real people worthy of moral considerations: that point is when we'll be able to pair midjourney-like photorealistic video generation of attractive young women with chatGPT-like outputs and voice synthesis. At that point he predicts that millions of men will insist that their waifus are actual real people. I'm inclined to believe him, and I think we're only about a year or at most two away from this actually being a reality. So: AGI in 12 months. Hang on to your chairs people, the rocket engines of humanity are starting up, and the destination is unknown.

you'd have to strap me in with my eyes open like Clockwork Orange to get me to watch a 3-hour lex fridman podcast. He went on a diet in 2020 and it's obvious it failed. As smart as he is, he could not keep the weight off, which i guess is further demonstrative of the difficulty of long term weight loss. It does not give me much confidence for my own weight loss.

Speaking as someone who's lost quite a lot of weight, gained most of it back and then lost it again, I can say that it's really a case of needing to be persistent and recognising that you're not "going on a diet", but actually making a sustained change of lifestyle and building better habits. I also find that it helps to not allow yourself to be too discouraged by stalling or setbacks. Often times it feels like you're just rolling the dice again and again, always coming up snake eyes, until finally something clicks and the weight just seems to fall off.

Best of luck with the weight loss and remember, we're all gonna make it brah.

Look into GLP-1 drugs.

Lex is also a fucking moron throughout the whole conversation,

Yea, this is par for the course with a Lex Fridman podcast. Practically everything he says is a non-sequitur, shallow, or saccharinely optimistic (love, beauty, barf). He gets some absolutely GREAT guests (how?), so I listen occasionally, but it's still a coin flip whether the whole 3 hours is a waste of breath. (Mostly it comes down to whether the guest has the strength guide the conversation to whatever they wanted to talk about.)

I'm waiting for the transformer model that can cut Lex's voice from his podcast.

Practically everything he says is a non-sequitur, shallow, or saccharinely optimistic (love, beauty, barf). He gets some absolutely GREAT guests (how?)

It is exactly because of this style of his. In general nowadays it is in vogue to be cynical little edgelord, savoring destroying other people's optimism with infectious smug nihilism. I like Lex's saccharine optimism and overall noncontroversial and non-adversarial style of doing his podcasts. He does not try to trap his guests or show his moral or intellectual superiority, he even looks silly and naive in that sense. This I think provides certain level of security for his guests to open up a little bit more than usual.

Lex is also a fucking moron throughout the whole conversation

I've stopped watching Lex because of that completely, couldn't finish this one as well. And he supposed to be somewhat of an expert on the topic of AI...

The most charitable interpretation I could come up with is that if he grants EY's assumptions or goes along with his analogies he basically has to take part in the doomerism and blackpill his audience, which may not be good for his numbers.

I haven't listened to the whole conversation but what I've heard and seen I have question. When did Autistic Catastrophizing become accepted by the mainstream? We got Greta Thunberg and now we have the "Big Yud" getting his opinion in the Time.

How do I become mainstream with my Catastrophizing that the combination of attention grabbing AI that subvert our lives with knowledge of behavioral psychology and the combination with generative AI giving us personalized content just massaging our brains just right? Imagine people trapped in a pleasure cube just watching personal generated colors and sounds that only has meaning to them....

When did Autistic Catastrophizing become accepted by the mainstream?

World War I. A few bad decisions led to millions of innocent people getting mowed down by artillery and machine guns. Kinda hard to have a "nothing bad ever happens" attitude after that.

As much as I like this comment, I think WWII/the Cold War would be a better fit. Sure, you can inflict lots of death with artillery, machine guns, and aircraft bombing, but after the Nuke was invented, they had to literally invent the term "megadeath" to keep up with the scale inflation.

The 40s/50s also get my vote. The invention of game theory, social psychology and the rise of the managerial class all make the era a better candidate.

"The world is a dangerous place, so serious people need to make sure everything is accounted for, therefore I rule." Dates back from that time.

How do I become mainstream with my Catastrophizing that the combination of attention grabbing AI that subvert our lives with knowledge of behavioral psychology and the combination with generative AI giving us personalized content just massaging our brains just right?

I'm a little disappointed that "AI safety" is so strongly associated with Skynet-style scenarios, instead of concerns about (in my view) more plausible near-term AI risks like this (and others - social unrest from sudden mass unemployment, expanding surveillance capabilities, etc). But, I'm certainly willing to make common cause with the x-riskers if it also gets people thinking about near-term AI risks as well.

The surveillance aspect of it scared the shit out of me more than anything else. Imagine if they start training these things on sigint collected in real time

A part of the problem is that some of the AI safety crowd are the victims of the Behavioral_sink. Leading way to comfortable lives that they even claim that 'words are violence'. It was 'safety' concerns that shutdown the Stanfords public Alpaca demo. They are stuck in a local maxima for their likes and retweets, claiming that FN Meka isn't allowed to rap about police violence because it isn't a lived experience, or that virtual Seinfeld jokes are harmful to the trans community. But the biggest danger is what they are themselves trapped in.... the behavioral sink and that more people will be stuck in digital quicksand because of AI.

These aren't the same people. You can think Yud is wrong, but Yud is not remotely related to "make AI woke or else it is biased" people.

I don't understand what Calhoun's "behavioral sink" has to do with this, never mind the validity of his rat experiments. If anything, wouldn't overcrowding make things too uncomfortable?

The reason I introduced it has nothing do with overcrowding, it is the observed results of behavior change when they where given everything that they ever could want. Some populations just exploded and then collapsed because overemphasis of a single behavior like eating, because infant rats aren't taken care of. So I use it as an analogy of modern life where we have everything to make us comfortable but some people have adopted behaviors which doesn't take society forward, like pushing junk science on potential existential risks.

I think you're letting the SJW crowd conflate things in your mind. The existential risk people are mostly libertarian types aside from this issue.

The existential risk people are mostly libertarian types

Bullshit. They're "liberals" who think unlimited amounts of power are legitimate when it comes to their pet issue.

They're "libertarians" only in the sense of being Californians who think of themselves as enlightened because they have nontraditional social mores. Put them in front of someone like Hoppe and the veneer will melt away instantly to reveal monstrous totalitarianism.

Libertarians don't advocate to ban the sale of GPUs or censor all of society based on the potential risks of individual freedom. They order you to live free, or die.

For me more likely there is an overcrowding in the AI-safety where you have libertarians and wokies trying to panic with different implausible scenarios.

Okay, but I'm trying to say that the x-risk people don't care about LLMs saying bad words.

They don't care about wokeness in LLMs as a terminal value, but they do see failures to RLHF the model into politically correct speak (or otherwise constrain its outputs into an arbitrary morally laden subset) as an alignment failure and an indication in favor of x-risk. E.g. this is the tenor of comments on Bing Chat/Sydney and I've seen more direct «if we can't get it to not say the N-word, we won't stop its descendant from paperclipping our asses» elsewhere, probably in Yud's timeline.

Fair point. But I think that is a reasonable test-case for alignment, and I maintain that most of the x-risk people think that beyond that, this sort of thing is merely a distraction.

I'm with them on that conclusion tbh. There ain't a snowball's chance in hell of us correctly aligning these things when they grow exponentially more powerfull. Hell, we can't align our damned selves and our own children and we've been trying to do this shit since we've existed, why should we expect any difference with our capability to align AGI.

More comments

I think yudkowsky recently renamed his field to "AI not-kill-everyone-ism" specifically to distinguish it from those other relatively minor concerns. "AI safety" is in fact no associated with the alignment problem if you talk to run of the mill professors and researchers, they still mean safety in the sense of "how can I prevent my half a million dollar robot taking actions that fry its motors?" or "how can we ensure privacy when training this LLM on user data?".

Having a goal of "the world doesn't end" does have its advantages. Can't wait until 2030, AI still doesn't kill anyone, and Yud saying "you are welcome" graciously lifting his fedora. Though who am I kidding, the world will then be in need of saving from AI killing everyone by year 20XX.

Though who am I kidding, the world will then be in need of saving from AI killing everyone by year 20XX.

I don't think you're being charitable enough to Yudkowsky and AI safety people, I think he has a very specific and falsifiable model of AI killing everyone. In my own model, if we are all still alive in 2033 AND we have AI that routinely can write scientific papers of moderate quality or higher, I think the problem will have turned out to be much easier than expected, and I don't further expect humanity to need to be saved by 2050 or something.

I'm a little disappointed that "AI safety" is so strongly associated with Skynet-style scenarios, instead of concerns about (in my view) more plausible near-term AI risks like this (and others - social unrest from sudden mass unemployment, expanding surveillance capabilities, etc).

How ironic, I'm a little disappointed that anyone bothers to waste an iota of intellectual effort on those nothingburger risks when Skynet-like scenarios are potentially bearing down on you.

When the Skynet term in your expectation function has a probability >0 and an expected utility of minus infinity, worrying about the small stuff like "They took our jerbs" is imo a bit dumb.

Plus I can always just... choose not go into the pleasure cube, whereas I can't choose not go into the paperclip nanobot.

I think the problem here is interpreting whether the non-existential objections are nearest risks or the maximal risk from the objector. Suppose our roommate wants to bring a gorilla to live with us. I object that it will eat all our bananas. You say "Eat our bananas! Who cares it's going to rip our arms off while we sleep!"

The key here is that my objections can be interpreted in two ways:

  1. (maximal risk) If we get the banana grabbing figured out, I'm on board.

  2. (nearest risk) The banana grabbing already meets the threshold for me to veto it, I don't even to weigh all the additional risks beyond that, which I would if the banana grabbing was solved, and would still veto.

In this hypothetical, your response, assumes I mean 1, when I might mean 2. Why talk about whether the gorilla might kill us in our sleep when we can align around the banana thing and get the same outcome of no-gorilla. This is especially helpful if our fourth roommate thinks the night-murder thing is ridiculous, but can be convinced of banana grabbing concerns.

Screaming harder about night-murder and dismissing banana-grabbing as trivial, actually hurts the case with the fence sitter, who's name is Allan and loves the beach.

It gets worse when I say to Allan in concession, look there's like a 90% chance of banana grabbing and a 50% chance, of night-murders too. Then you jump in and scoff at my 50% as too low and of trivializing the real danger. Now we are in an inside baseball debate that simultaneously makes Allan take both banana-grabbing and night-murder less seriously.

When the Skynet term in your expectation function has a probability >0 and an expected utility of minus infinity,

This is called Pascal's Mugging.

Except the arguments for the existence of the risk are substantially stronger than those presented by the theoretical mugger.

After all, we can all clearly see that the AIs exist now and that them becoming smarter-than-human is, indeed, plausible. This does not require you to take the mugger's words at face value.

So what's irrational about considering the actual evidence that exists?

Except the arguments for the existence of the risk are substantially stronger than those presented by the theoretical mugger.

"Probability >0 and expected utiulity of minus infinity" doesn't contain any qualifier about the probability being strong. In fact, it tries to argue that the size of the probability doesn't matter at all.

Sure, if we also ignore any timelines on the the expected event occurring, and ignore whether we have the ability to impact the expected utility outcomes here. We're not JUST talking about the probability of humanity going extinct, although yes, that factor should loom larger than any other.

The flip side of AI doomerism is the belief that if we get a friendly AI then that's an instant win condition and we get post-scarcity in short order. i.e. heaven.

Funny enough, though, people don't seem to argue as vehemently that the risk' of creating a benevolent is basically zero, they seem to think that that's the default assumption?

Except the arguments for the existence of the risk are substantially stronger than those presented by the theoretical mugger.

No they're not. Superintelligence and other fictions have exactly as much evidence backing them as God.

AIs don't exist now, never have, and likely never will.

The implications of the existence of LLMs might be great of small, but to see them in this paradigm of "intelligence" is boneheaded and ridiculous, and I remain convinced that history will show this framing to be completely delusional.

What do you think human intelligence is?

Or more precisely, if we're leaving God off the table, then why should whatever humans do that produces the appearance of intelligence be impossible to reproduce artificially?

I don't know, neither do you, and that's exactly why.

I'm by default skeptical of the ability to reproduce processes we do not even understand.

More comments

I agree with you, although I think talking about "They took our jerbs" is both a good way to get people to understand that everything is going to change, and also a plausible and relatively mundane route to total human disempowerment.

Agreed. Skynet is of course a possible outcome but eliminating human ability to add value seems to destroy our purpose.

We got Greta Thunberg

Thunberg is just someone who really takes the IPCC seriously. It is not something she makes up or talked herself into it.

https://www.abc.net.au/news/2023-03-21/ipcc-report-how-to-keep-global-warming-below-1-5-degrees/102112836

Does she take the IPCC seriously or the political summary?

She just simply follows the science? So what part of that IPCC report says that climate change stole her childhood?

Lex is also a fucking moron throughout the whole conversation, he can barely even interact with Yud's thought experiments of imagining yourself being someone trapped in a box, trying to exert control over the world outside yourself, and he brings up essentially worthless viewpoints throughout the whole discussion.

So yeah. This was the first time I ever listened to/watched one of Fridman's interviews. He seemed to burst onto the scene out of nowhere around a year ago. And everything I gathered from secondhand testimony and snippets of his content, and his twitter feed, led me to make an assumption:

The guy is basically trying to bootstrap himself to become the Grey-Tribe version of Joe Rogan.

And after hearing this interview, I updated MASSIVELY in favor of that model. It wasn't quite like he's just cynically booking 'big name' guests who appeal to the nerdy corners of the internet, and doesn't care about the actual quality of discussion. He appears to be making an effort.

Yet his approach to the interview seems to be MUCH less based on engaging with the thoughts of the guest but more pressing them on various buzzword-laden 'deep' questions to see if they'll give him a catchy soundbite or deep-sounding 'insight' on a matter that is, to put it bluntly, pseudospiritual. He's in there asking if 'consciousness' is an important feature of intelligence and whether that is what makes humans 'special' and if we could preserve conciousness in the AGI would that help making it friendly? Like kinda playing with the idea that there's something metaphysical (he would NEVER use the term supernatural I'm sure) and poorly understood about how human thought processes work that gives them a higher meaning, I guess?

And EY has basically written at length explaining his belief that consciousness isn't some mysterious phenomena and it is in fact completely explainable in pure reductionist, deterministic, materialist terms without making any kind of special pleading whatsoever, and thus there's no mystical component that we need to 'capture' in an AGI to make it 'conscious.'

As you say, his blatant dodge on the AI box questions AND, I noticed, his complete deflection when EY literally asked him to place a bet on whether there'd be a massive increase in funding for AI alignment research due to people 'waking up' to the threat (you know, the thing EY has spent his life trying to get funding for) betrays a real lack of, I dunno, honest curiosity and rigor in this thought process? Did the guy read much of EY's writings before this point?

Its almost the same shtick Rogan pulls where he talks to guests (Alex Jones for example) about various 'unexplained' phenomena and/or drug use and how that shows how little we really know about the universe, "isn't that just crazy man?" But avoiding the actual spiritualist/woo language so the Grey Tribe isn't turned off.

At least the guys in the Bankless Podcast noticed right away they were beyond their depth and acted more like a wall for EY to bounce his thoughts off.

As for EY.

Man. I think the guy is actually a very, very talented writer and is clearly able to hold his own in a debate setting on a pure intellectual level, he's even able to communicate the arguments he's trying to make in an effective manner (if, unlike Lex, the other party is conversant in the topics at hand).

He even has an ironic moment in the interview, saying "Charisma isn't generated in the liver, it's a product of the brain" or some such. And yet, he does not seem to have done much beyond the bare minimum to assuage the audience's "Crank detector." Its not clear that his persuasive powers, taken as a whole, are really up to the task required to win more people to his side.

Of course, for those only listening in rather than watching, that won't matter.

I'm not saying EY should just bite the bullet and work out, take some steroids, get 'jacked,' wear nice suits, and basically practice the art of hypnotizing normies in order to sneak his ideas past their bullshit detectors.

But... I'm also kinda saying that. He KNOWS about the Halo Effect, so within the boundaries set for him by genetics he should try to optimize for CHA. Doubly so if he's going on a large platform to make a last-ditch plea for some kind of sanity to save the human race. MAYBE a trilby isn't the best choice. I would suggest it would be rational to have a professional tailor involved.

But I do grok that the guy is pretty much resigned to the world he's been placed in as a fallen, doomed timeline so he probably sees any additional effort to be mostly a waste, or worse maybe he's just so depressed this is the best he can actually bring himself to do. And it is beyond annoying to me when his 'opponents' focus in on his tone or appearance as reasons to dismiss his argument.

And to make a last comment on Fridman... he clearly DOES get that CHA matters. His entire look seems engineered to suggest minimalistic sophistication. Sharp haircut, plain but well-fitted suit, and of course the aforementioned buzzwords that will give the audience a little bit of a tingle when they hear it, thinking there's real meaning and insight being conveyed there.

But... I'm also kinda saying that. He KNOWS about the Halo Effect, so within the boundaries set for him by genetics he should try to optimize for CHA. Doubly so if he's going on a large platform to make a last-ditch plea for some kind of sanity to save the human race. MAYBE a trilby isn't the best choice. I would suggest it would be rational to have a professional tailor involved.

But I do grok that the guy is pretty much resigned to the world he's been placed in as a fallen, doomed timeline so he probably sees any additional effort to be mostly a waste, or worse maybe he's just so depressed this is the best he can actually bring himself to do. And it is beyond annoying to me when his 'opponents' focus in on his tone or appearance as reasons to dismiss his argument.

I don't think intelligence correlates well with moral strength (i.e. ability to live your own moral ideals). Whatever the seeming importance of averting AI catastrophe, it seems to be less important to him than his identity as someone who won't pay a few hundred bucks for fashion tips.

Lex is also a fucking moron throughout the whole conversation

Pretty much. As a long-time Lex listener, he is intolerable nowadays. I tried watching the recent podcasts with Altman and Yud, and I had to give in. Instead of talking about the 100s of potential technical/economic/political implications, he pushes the direction of the conversation in the "Is AI a living being (hurr durr boston dynamics spot doggo)?" angle of all things.

His old podcasts were not like this. He used to delve into the technical details with the guests. I think he realized that the midwit-Ifuckinglovescience crowd is a much better revenue stream and his youtube algorithming optimizing his way to becoming a Philosophy-101 cringe lord.

At least Joe Rogan pulls plays dumb convincingly, Lex comes off as a tool.

Instead of talking about the 100s of potential technical/economic/political implications, he pushes the direction of the conversation in the "Is AI a living being angle?" of all things.

Yeah, as I said, pseudospiritual. It's a weird approach when you've got someone who has vast amounts of technical insights and industry knowledge to engage in, effectively, philosophical circlejerking. But then if your audience lacks the technical knowledge to follow such a conversation, they might tune out. Whereas engaging in fundamentally unanswerable meta questions means your audience can feel like they've received insights and can easily follow the conversation without thinking too hard.

I think he realized that the midwit-Ifuckinglovescience crowd is a much better revenue stream and his youtube algorithming optimizing his way to becoming a Philosophy-101 cringe lord.

I have not seen anything to suggest otherwise. With that said, it helped him build an audience which helps him snag some of the most popular guests, which helps him grow his audience, which helps him get more popular guests... etc. etc. It's a successful model.

He noticed that, like Rogan, if you punch through to the top tier of podcasting you can have a self-sustaining money printing machine because important guests will seek the platform you have, and audiences will gravitate to important guests. The only risk is either 'scaring' away guests by questioning too aggressively or getting cancelled (as almost happened to Rogan) for any particular controversial moments.

Which might fully explain why he's fallen back on the softest of softball questions that don't challenge the guest nor risk wandering into a cancellation landmine.

I don't think he actually believes it, and even so it's like trying to prove a negative. Why is he trying hard to lose weight on twitter if he's not even going to be around to enjoy it? The objective is to get someone like Thiel, Musk, or Vitalik Buterin to drop some big $.

Lex is also a fucking moron throughout the whole conversation, he can barely even interact with Yud's thought experiments of imagining yourself being someone trapped in a box, trying to exert control over the world outside yourself, and he brings up essentially worthless viewpoints throughout the whole discussion. You can see Eliezer trying to diplomatically offer suggested discussion routes, but Lex just doesn't know enough about the topic to provide any intelligent pushback or guide the audience through the actual AI safety arguments.

Lex almost never pushes back hard. that is part of his brand, and is how he gets top guests, who don't want to be put on a hotseat.

Lex almost never pushes back hard.

It was funny (not quite in a haha way) when Lex loosely asserted that he believed that people would 'wake up' to the threat and billionaires and governments would start pouring money into solving alignment, he even put a timeline on it.

EY immediately asks him if he wants to place a bet on it, and Lex pulls the most inartful pivot/deflection I've ever seen to not just avoid agreeing to put money on his 'prediction' but having to even explain his prediction, or acknowledge that he was just challenged.

I mean, treating your guest with kid gloves is one thing, but here you've got a guy who WANTS the hotseat, and is openly trying to turn up the temperature and you just completely drop the line of questioning you were on. If he's intentionally playing dumber than he is than bravo, I believe his performance.

I don't think he actually believes it, and even so it's like trying to prove a negative. Why is he trying hard to lose weight on twitter if he's not even going to be around to enjoy it?

The most merciful thing about the human mind something something correlate all its contents something something.

It's unreasonable to expect day-to-day behavior to be completely in accord with abstract beliefs. He is still a monkey who wants higher status. Religious fanatics will still look at porn, and moral nihilists will still get outraged at someone who milkshakes their kid.

Why is he trying hard to lose weight on twitter if he's not even going to be around to enjoy it?

He wants that sweet nerd pussy, duh. He's in his 40s and poly, that means his prime is right now.

MIRI has done donation drives before. They aren't doing them now. Maybe this is 4D-chess reverse psychology, but gee he seems legitimately distressed by all this.

Jesus Christ his mannerisms are weird.

something changed

Odds are it's Adderall. Bay Aryan culture nerds (sounds like an anthropological term, right?) abuse «ADHD meds» to a degree far beyond Scott's «doing spreadsheets» apologia – it's their friend in need when they want to be on top of their game and really make an impression. They write terrible tweets on addy, go to podcasts on addy, livestream coding sessions on addy and make innumerate github commits that end up borking whole repositories on addy. The society needs to teach these people that addy doesn't actually make them smarter, it only makes them feel smarter and act more grandiose, which they get addicted to even harder than to the direct dopamine hit. Once again: nerds aren't all right; consider this paragraph a simulated example of how exactly. Acting like a hyperactive Loony Tunes or anime character is, well… loony.*

That said, I do not endorse the focus on the way Yud looks and acts or even whether he's a narcissist. He doesn't strike me as too unseemly for someone with his background; it incriminates the speaker more than Yud; and it takes away from substantial criticism.

In fact Yud, suddenly and unexpectedly for him cited as top AI researcher, prominent analyst etc. – is, himself, a red herring that distracts from the real issue. The issue being: a coordinated barrage of attacks on proliferation of transformative AI. I've compiled an incomplete chronicle of proceedings; some of those are obviously just journos latching on, but others had to have been in the works for months or at least weeks. This is some spooky bullshit – though, nothing new I guess, after all the shadowy campaigns to fortify the democracy and battling COVID misinformation/narrative shifts.

I think we are seeing ripples from a battle to capture the public endorsement for deceleration vs. unequal acceleration. With one party (which I associate with old-school paramasonic networks) being genuine «decels» who push crippling regulation using Yud and other useful idiots like EAs and assorted international organizations as a front; and the other being a fractured alliance of national, industrial and academic actors who want narrower regulations for all, displacement of the purported threat onto geopolitical and market competitors and open-source community, and token (from their perspective) conditions like ensuring that the AI stays woke for themselves; though I may be completely wrong with my typology. It's reminiscent of the rise of anti-Nuclear groups and Rome Club «limits to growth» fraudulent models, which then mutated into today's environmentalist degrowth movement (recommended reading on Yudkowsky as our era's Paul Ehrlich; do you like our success in defusing the Population Bomb?).

Anyway:

  1. 02/24: OpenAI releases the paper Planning for AGI and beyond which some conclude is unexpectedly thoughtful and the usual subjects (see LW/SSC) pan as not going nearly far enough.

  2. 03/12 Yud: « I'm at the Japan AI Alignment Conference, and they're introducing me as the eminent grandfather of the field of AI alignment…»

  3. 03/13: after weeks or running covertly under the hood of Bing Search, GPT-4 officially launches (sniping the leading safety-concerned lab Anthropic's Claude by a day).

  4. 03/15: Sutskever: «If you believe, as we do, that at some point, AI — AGI — is going to be extremely, unbelievably potent, then it just does not make sense to open-source. It is a bad idea... I fully expect that in a few years it’s going to be completely obvious to everyone that open-sourcing AI is just not wise». Reminder that in 2015, in the OpenAI still aligned with Musk, he signed this: «We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible… it’ll be important to have a leading research institution which can prioritize a good outcome for all over its own self-interest… As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world.» The more cynical logic behind that OpenAI was one of «adversarial equilibrium», which I paraphrase here and which e/acc people articulate better.

  5. 03/25: Altman goes on Fridman's podcast, admits that much of work on GPT-4 was in alignment, and drops some of Yud's takes like «we don't know how to align a super powerful system». Lex and Sam discuss Yud's AGI Ruin, apparently, at 55 minutes mark, at 1:11 Sam suggests that opensource LLMs with no safety controls should be regulated away or detected through vetoed «aligned» ones. Note though that Sam is at peace with the idea that there will be multiple AGIs, while Yud thinks 1 is as many as we can afford (ideally 0 for now). Sam mentions Scott's Moloch at 1:16.

  6. 03/29: the UK Government publishes a recommendation document on AI policy «to turbocharge growth» called # AI regulation: a pro-innovation approach.

  7. Future of Humanity letter signed by Musk, Harari, Yang etc; safetyists like Geoffrey Miller admit that 6 months is only to get the ball rolling.

  8. 03/29: the open-source AI org LAION petitions for the creation of a «CERN for AI research and safety», gathers signatures of random EU redditors.

  9. 03/29: Yud's childish piece in TIME Ideas where he suggests an international treaty to physically destroy datacenters training serious AIs. Interestingly I've heard that Time proactively reached him.

  10. 03/30: Yud hyperventilating on Fridman.

  11. 03/30, Fox News’ reporter uses all his time at the White House to advance Yud's ideas.

  12. 03/30 continues: «UNESCO Calls on All Governments to Implement AI Global Ethical Framework Without Delay»: «This global normative framework, adopted unanimously by the 193 Member States of the Organization…»

  13. 03/30 unlimited decel policy works: The DC tech ethics think tank Center for Artificial Intelligence and Digital Policy asks FTC «to stop OpenAI from issuing new commercial releases of GPT-4».

  14. 03/31: Italy bans ChatGPT over specious privacy concerns.

  15. 04/01, BBC: Should we shut down AI? Inconclusive.

  16. 04/02, fucking RussiaToday: «Shutting down the development of advanced artificial intelligence systems around the globe and harshly punishing those violating the moratorium is the only way to save humanity from extinction, a high-profile AI researcher has warned».

I see some AI researchers on twitter «coming out» as safetyists, adopting the capabilities vs. safety lingo and so on.

On the other hand, many luminaries in AI safety are quiet, which may be due to the fact that they've moved over to the OpenAI camp. Some are still active like Richard Ngo but they're clearly on board with Sam's «we'll keep pushing the frontier… safely» policy. Aaronson here too.

Another curious detail: in 2021, Will Hurd, «former clandestine CIA operative and cybersecurity executive», joined OpenAI's board of directors. I like this theory that explains the $10B investment into, effectively, GPT training as a spook/military program. I've also updated massively in favor of «Starship is FOBS system» arguments made previously by @Eetan, AKarlin and others.

All in all, it feels like the time for object-level arguments has passed. What decides the result is having connections, not good points.

* As a transhumanist, I understand the aspiration. But as someone who flirted with stims for a while and concluded that, in the long run, they very clearly only make me more cocksure, loquacious (as if any more of that were needed) and welcoming routine to the point of pathological repetitiveness, I endorse @2rafa's recent writeup. This isn't the way up. This isn't even keeping level with the baseline performance in interesting ways.

My impression is that people like Ngo are quietly pushing inside of OpenAI for slowdowns + coordination. The letter was basically a party-crashing move by those peripheral to the quiet, on-their-own-terms negotiations and planning by the professional governance teams and others who have been thinking about this for a long time.

I think most of the serious AI safety people didn't sign it because they have their own much more detailed plans, and also because they want to signal that they're not going to "go to the press" easily, to help them build relationships with the leadership of the big companies.

I have actual ADHD, so I need stimulants to be productive when actual executive function is needed (exams for one, and those never end for doctors). People with it supposedly benefit more from the drugs than neurotypicals do, even if they still improve the focus of the latter.

That being said, I strongly suspect that your negative attitude towards stim use in the Bay Area is simply selection effects. You notice the stimmed out people making a fool of themselves in public, and don't see the thousands-millions of others quietly using them at reasonable doses to boost their productivity. I'm going to go with Scott on this one, it fits with my own personal experience.

Well, for what an anecdote is worth, +1 for "social side effects".

I probably have as little executive function as you can have and still survive in academia (with most of my planning for work amounting ot "how many all-nighters will I need to meet the minimum standard here"), and otherwise spend most of my days (or rather nights, as I rarely wake up before the afternoon). I also used to be an obnoxiously hyperactive kid up until some point, and in particular had a pattern of getting myself in trouble in school up until 6th grade or so when the teacher would call on someone, that someone would struggle to answer or waffle around and then I couldn't resist and would just blurt out the answer out of turn. I knew this was looked upon unfavourably and defeated the point of the pedagogical technique, and even took more than one penalty F from exasperated teachers for it, but still, most of the time that situation happened, the urge to do so again - fueled by some mixture of impatience, irrational irritation that the teacher didn't just call on me so the class could move faster, and most dominantly some kind of screaming neural circuitry that made me imagine standing there in place of the person being called and being tongue-tied about giving the answer - was too overwhelming. As I grew up, this just stopped at some point, and I sunk into the perpetual fog of my current existence.

So anyway, at some point in grad school, I borrowed from the Adderall stash of an American fellow student (with an on-brand official ADHD diagnosis and everything) and went to some young assistant prof's theory seminar/class - the youth is relevant insofar as no old hand would actually bother calling on individuals with trivia questions in a graduate course - and was shocked to feel that exact same urge, which at that point I hadn't experienced in 15 years, welling up in me again every time someone was struggling with a question. Luckily the benefit of age and experience let me leave it at shifting around in my chair uncomfortably rather than actually shouting out the answer, but I have no illusions that it was damn close. Twice the dosage may well have been enough to actually make me embody the "bristly arrogant asshole that's all INT and no WIS" persona that Silicon Valley seems to be famous for.

I can chime in and say that I've also noticed no change to my imagination or creativity while on meds, in fact my creative output has increased as I can now actually get things done and then move onto new creative enterprises.

My experience with medication and the discourse around adhd medication in particular, seems to be that there is a fairly broad spectrum of experiences that people have with these drugs and that people really like generalising their own experiences as The Golden Standard (other people are clearly lying, or are taking their drugs wrong) and from there it's a coin flip if they'll then start to morally grandstand.

It is also my experience that most discussion around adhd medication is one of morality and virtue instead of practicality and utility. All those who have an interest in transhumanism and human augmentation should take note, this is one of the tributaries of that later torrent.

No, not that I've ever noticed!

I'm not a visually creative type, having peaked at stick figures, but I would say I'm a good writer. And a great deal of my better reddit posts, including like half my AAQCs, were written when I was tired of studying but still hasn't had the meds wear off.

That's not the reason I don't use Ritalin on a more regular basis, even the lowest sustained release formulation gives me palpitations and I then need to take l-theanine or other anxiolytics to deal with it. Simply too much of a hassle unless I have exams to worry about, but I intend to try and switch to Vyvanse or Adderall in the UK if I can get my hands on it. That's why I was so stricken by the high costs of psych consultations (by my poor third world standards haha).

As someone who peaked slightly higher than stick figures, meds definitely make it easier to practice lots of repetitive wrist movements. My daydreaming also becomes more vivid, and I can write more words. Meds tend to drive me more to the execution side of things and maybe terminate my endless tangents of information gathering.

I don't often agree with daes, but I strongly endorse his description of the end-point of amphetamine use on people's thought processes. I find the idea that some people are magically different in this regard fairly absurd -- with the caveat that low doses probably aren't that bad and are more likely to be maintained as such when it's given as 'medicine from a doctor' than 'bennies from my dealer'.

But I don't have much confidence that EY (or SBF for that matter) have been particularly strict in keeping to low theraputic-type doses -- that's the whole problem.

People with it supposedly benefit more from the drugs than neurotypicals do

The evidence is pretty strong for that supposition, no? I certainly believe that stimulants help people with executive dysfunction and/or ADHD (or rather ADD; there's a suspicion that H is added simply to rein in boys who can't tolerate the inanity of the normie classroom. But whatever). I'm just observing people who were highly productive before getting access to stims becoming weird hyperactive clowns. And it's not selection effect when we're actually talking about self-selected people vying for public attention.

That said: Americans love to medicate, and solve lifestyle (hate that philistine word, but what would be better?) problems medically in general. Anxyolitics, stims, opioids, lyposuction, HRT, surgery, if there's a shortcut to «enhance» or «improve» yourself with a bit of money and chemistry and professional customization, they'll do it, often with garish consequences. Yud himself is on record as not trusting in the behavioral route to weight loss, so I am pretty confident here.

Maybe this explains some of the distaste for transhumanism in our more conservative American friends.

The feigned tryhard stoicism in the face of physiological limits is often unseemly too, of course.

I'm hedging because I have never personally prescribed stimulants (other than sharing my own with friends and family, and informally diagnosing some of them for later followup with a psych). AHDH is also next to unknown in India, which is personally surprising because I'm well aware of how hyper-competitive academics are here, such that in a vacuum I'd expect everyone and their dog to be desperate to get their children on them if they believed it would boost their performance.

As such, my actual IRL exposure to other people with ADHD is near nil, so I'm forced to rely on anecdotes from people/psychiatrists like Scott himself, and from a casual familiarity with online nootropics/stimulant/recreational drug communities.

I'm just observing people who were highly productive before getting access to stims becoming weird hyperactive clowns. And it's not selection effect when we're actually talking about self-selected people vying for public attention.

I see, that didn't come across to me! Thanks for clarifying, although I still do think some selection bias is at play, especially given the frequency for stimulant usage in the US.

Anxyolitics, stims, opioids, lyposuction, HRT, surgery, if there's a shortcut to «enhance» or «improve» yourself with a bit of money and chemistry and professional customization, they'll do it, often with garish consequences

I must be a true red-blooded 'Murican at heart, because I fully sympathize with their position haha. I know that simply "trying harder" didn't work for me, and I see nothing inherently immoral about taking obvious shortcuts, as long as you're aware of the pitfalls. But does that surprise you as mutual transhumanists? I wouldn't think so, that's just how we're wired.

Again: I understand the aspiration. There's no naturalistic moralizing here (though I do have a somewhat aesthetic problem with most psychoactive substances because they, uh, just erase information, like when you play with curves in Photoshop. Pretty sure I've written on it in the past but search's a bitch).

It's just, current-gen commodity interventions have simple crude mechanisms of action, a pretty low ceiling, tradeoffs, and, when abused, easily push you below the baseline. They make a tantalizing but largely unwarrantable promise. I've known people, transhumanists in fact, who fried their brains with «biohacking» based on telegram bro advice – especially as they went past their early 20s.

The boomerish «just sleep/exercise/play sports/socialize/meditate/eat well/have sex, kiddo» dictum is actually the most reliable biohacking you can do right now, and it's been this way forever. It would certainly have helped Yud look better.

The catch, of course, is that good lifestyle is a signal of higher class because it actually requires tons of executive function, and that is genuinely lacking in many, and so this is another justification for stims.

I want biohacking that works. It's not remotely that simple yet.

Depends on what you're aiming to hack IMO.

For example, we finally have safe and FDA approved drugs for weight-loss, and my priors suggest that judicious use of stimulants can make you effectively superhuman in some capacity without any real downsides as long as you cycle and take tolerance breaks.

Still, it's an undercooked field, and I have little doubt that you can do serious damage by experimenting without knowing what you're doing, and just knowing isn't sufficient to stop unexpected tragedies to boot.

(I didn't think you were moralizing, I know you better than that haha)

For example, we finally have safe and FDA approved drugs for weight-loss

Available only in the USA AFAIK. I even purchased prescription for it (Ozempic) - months ago. But it's available either nowhere or in some random pharmacy several hundred kilometers away.

Well, I also purchased Tirzepatide from some online shop selling it as research chemical. I didn't get around to using it yet, probably because I fear disappointment in case it's fake as it cost ~$500 for 5x5mg doses :|

Scott did a deep dive into it a little while back, and it has expanded availability outside the US by this point!

I'm sure the costs will plummet further, nothing benefits from economies of scale like a drug for weight loss haha

Is there a timestamp for Lex failing the hypothetical test? (Wasn’t there an ongoing debate here about whether people with an 80IQ can understand hypotheticals?) I feel like Lex is the dumb person’s conception of a smarter person’s Joe Rogan. I’m consistently disappointed that he can get things so wrong on both a social and informational level. In an interview with the Botez sisters, he went into a soliloquy about how he had to decide to either pursue chess or computer science, because chess would consume his whole life. But a few moments later he’s asking them how the pieces move. He never even started learned chess.

This post is on the one hand so beautifully poetic, but on the other hand fills me with so much dread. I would have loved it ten years ago, but the older I get the more disgusted I feel by it. Do you feel no shame or disrespect for the people who came before you, or yourself, to waste your own humanity in the pleasure box, and then to destroy everything we've achieved and all the potential to achieve anything else, for the sake of whatever we created to take over? It just feels like the ultimate disrespect to ourselves and each other. I watched my family suffer and die, in all their humanity, so that a bunch of kids can feel good forever? Ahhhh, thanks, I hate it...

There will be no difference, for our meat brains, between having an AI family and children and grandchildren and having the real thing.

Just because we can't necessarily tell the difference, doesn't mean we can't care or try to avoid it. I for one would choose not to live out (most) of my days in an experience machine.

For the record, I do not call AI «our successor species». (This term seems to have misled some on this forum). Even genuinely agentic AI is not a species in a meaningful sense, it's a whole different non-biotic class of entity. Incidentally Galkovsky wrote a short, incredibly weird «Christmas tale» №4 some 18 years ago:

The One True Teaching of the Great Animation states that the evolution of consciousness inevitably leads to the demise of biological carriers and life forms in general, and to the transition to mineral carriers or to the force field. In the eternal struggle between Spirit and Anima, the Anima supersedes Spirit. However, both the realm of Spirit and the realm of the Anima have Parasites - aliens from another world. The development of viruses goes along the path of useful physical ones that persist within the world of Anima (like the quartz amoeba), and along the path of virtual harmful ones that exist in the world of Spirit.

How is it that in the vast Globule a relic galaxy, a galaxy-fluctuation has persisted, in which in the struggle for existence life has won, and animation is floundering on the periphery? I destroyed animation to its foundations, but died myself as well. Over the last million years, natural Evolution has once again spawned animation. The hydra of animation began to raise its head.

I pretended to be Anima which is imitated by Spirit, but because of its lattice nature is capable of complex mirror interactions that destroy the linguistic environment of biological life…

but the rest is all Osip Mandelstam and untranslateable 10D postmodernist Go.

Anyway, in my book the successor species will be humans unmoored from evolution and evolution-imposed limits.

Yudkowsky himself likes this future.

Yud doesn't like the Culture. He grew up reading Extropians (from whom he had learned what later became his whole schtick) and I believe that his ideal world – or a compromise, if a well-ordered Rational Yudocracy is ruled out – is more in line with Sandberg's Orion's Arm project, where an incomprehensible diversity of life forms, species and kinds, civilizations, philosophies, AIs, developmental trajectories exists and a mind can grow qualitatively, over and over breaking through different singularities, yet lesser minds need not fear being crushed underfoot by greater ones, their safety guaranteed by the treaty of benevolent posthuman Gods:

…if sentients from different mental levels encounter each other within the environs of an unclaimed system or interstellar body, the entity or entities possessed of greater physical or mental capacity may not attempt to summarily destroy or otherwise harm those possessed of lesser capacity.

I think he'd have been a Keterist there. I would as well, perhaps. It's a beautifully exuberant vision, far richer than our physics and mathematics (not to mention economics and game theory) seem to allow. Though who are we to know? How much do we Understand? Karpathy sees in that story what I do: the promise of new notes and new colors, a new heaven and a new earth. To our normal imagination, an enhanced human might seem ridiculous or monstrous – just the same monkey running its monkey business in a million-fold parallel process, maybe even a violin sawed in half by a stupidly accelerated bow. But proper growth adds new strings and new harmonies, and the ability to appreciate them too. I wonder what you will see.

As for what I think of those left behind?

There are two sorts of Zen. One doesn't allow for Corrida. The other extols the perfection of ever more fierce and magnificent bulls felled by ever more skillful matadores. I don't remember where I heard it, and I don't really like Corrida or what it symbolizes, but the basic idea is pretty alluring. I want everything to change so that everything can stay the same: still agents, still challenges, still the dance of exploration and exploitation, just harder, better, stronger, faster, smarter, cooler, hotter, longer. Or do you prefer Fatboy Slim? Probably not Pearl Jam though. The prerequisites to make life interesting will remain. But massive classes of problems that humans have built their history and meanings of their lives around will be trivialized and solved completely, like tic-tac-toe – or smallpox; inevitably destroying communities which can neither function without them nor compel tolerance of them when there is a known alternative. I believe this is inevitable under any realistic scenario of progress, just on different timelines, and it's worthy of nostalgia – but not much more. People should be allowed to limit themselves, just as they should be allowed… not to.

I believe that the expected succession is not a mere Darwinian process of humans being transcended by something else. It is defined by committing to different ideas of what humanity is about. For some, it's a condition. For others, an aspiration. There is a spectrum, of course, but humans who decidedly embrace one will give up another. Humans who embrace the human condition will become living fossils for the other group (which I wish to join), and it will be our responsibility to ensure their survival, but theirs to keep finding meaning in what they do.

This is already known to be possible. Traditionalist groups like Haredim or Amish or Laedastians famously spurn the temptations of modern civilization and maintain their own meanings, their own parochial worlds. If much of current humanity chooses to fossilize in a similar manner – well, maybe they'll find it in themselves to ban VR/wirehead porn too.

And maybe they'll come to hypocritically use Godtech imports to levitate their apple pie carts and reach Methuselah age, while pitying the producers of it all for consigning their – our – immortal souls to damnation.

(As you perhaps remember, I take seriously the possibility that modern humanity dies out naturally in a few generations of sub-replacement TFRs anyway, and is succeeded by those very traditionalist groups, prominently among them Ultra-Orthodox Jews and hopefully Trad Mottizens).

All in all this isn't a very pressing matter. If the choice exists, we'll grow used to different humanities walking different paths, sometime before the heat death.

The hard part is getting past the point where a single party gets ahead and aligns the planet with its wishes.

I find this sort of cynicism tedious, sorry.

Biology is mutable. There is no single humankind: throughout history, entire humankinds have exploded into existence and disappeared, riding fitness gradients or being crushed by their waves – and entire continents's worth of humans, our mother Africa first of all, have been slumbering in a fitness trough for tens to hundreds of thousands of years. This time may be different in that the legacy variant of the species that didn't make the cut will remain indefinitely – either immortal and crippled by their addiction to the local minimum, as you say, or evolving to ever better maintain their behavioral crutches and shackles, as I foresee. But neither will be prohibitively costly to enable for the evolved rest (I've said the opposite recently, but that depends on the greed and ruthlessness of the decision-makers; in absolute terms, even an all-included resort for 8-10 billion immortals enjoying VR paradises is a pittance once you get proper space exploration, build space solar and fusion power, and build at least «wet» nanotech; with strong tool AI we should do that in a century easy-peasy).

This time will also be different in that biology will become mutable in a directed fashion. Between predictably succumbing to the orgasm dispenser like some pop sci rat, and the eternity of modestly pleasurable perfect arete, self-mastery and appreciation of nuanced challenge, truth and aesthetic marvels in the real world – supposing you have a self-alteration button that will irreversibly alter your basic drives and inclinations, which will you press? I know my answer. And just like before, it'll only take a few to make the right choice.

Though, of course, evolutionarily speaking the right choice is very different. But that's a different Apocalyptic scenario too.

It's pretty much impossible to know what the ideal outcome for AGI is, because one of the first things I would ask of an aligned AGI is to augment my own intelligence in a direction aligned with my current values, and I would then leave the job of figuring out what I want to do with infinity to my future better self. A future where we remain meat-machines with our current brains intact and become shepherded by great all-knowing AIs is very unlikely, I don't want to trust and be in awe of the Jupiter-Brain, I want to be the Jupiter-Brain.

I was going to make a nigh identical comment, so thank you for saving me the trouble!

I find existence as a mere baseline human to be extremely claustrophobic. Fuck yeah put me on that galaxy-brain shit (even a Matrioshka Brain will do, we have to deal with gravitational collapse in anything above a few stellar masses).

I didn’t watch the video - it’s hard for me to take the topic and it’s high priests seriously; AI safety is a reformulated Pacal’s wager.

Even if you believe otherwise, there are maybe one or two universes at most in which we could solve the coordination problem of stopping everyone from networking a bunch of commodity hardware and employing 30 engineers to throw publicly available data sets at it using known algos. A not too wealthy person could solo fund the entire thing to say nothing of criminal syndicates, corporations or nation states. This one is not going back in the bag when everyone involved has every incentive to do the opposite.

As it happens, your latter point lines up with my own idle musings, to the effect of, "If our reality is truly so fragile that something as banal as an LLM can tear it asunder, then does it really deserve our preservation in the first place?" The seemingly impenetrable barrier between fact and fiction has held firm for all of human history so far, but if that barrier were ever to be broken, its current impenetrability must be an illusion. And if our reality isn't truly bound to any hard rules, then what's even the point of it all? Why must we keep up the charade of the limited human condition?

That's perhaps my greatest fear, even more so than the extinction of humanity by known means. If we could make a superintelligent AI that could invent magic bullshit at the drop of a hat, regardless of whether it creates a utopia or kills us all, it would mean that we already live in a universe full of secret magic bullshit. And in that case, all of our human successes, failures, and expectations are infinitely pedestrian in comparison.

In such a lawless world, the best anyone can do is have faith that there isn't any new and exciting magic bullshit that can be turned against them. All I can hope for is that we aren't the ones stuck in that situation. (Thus I set myself against most of the AI utopians, who would gladly accept any amount of magic bullshit to further the ideal society as they envision or otherwise anticipate it. To a lesser extent I also set myself against those seeking true immortality.) Though if that does turn out to be the kind of world we live in, I suppose I won't have much choice but to accept it and move on.

"If our reality is truly so fragile that something as banal as an LLM can tear it asunder, then does it really deserve our preservation in the first place?"

How about: "If a baby is so fragile that it can't take a punch, does it really deserve our preservation in the first place?"

Sorry to speculate about your mental state, but I suggest you try practicing stopping between "This is almost inevitable" and "Therefore it's a good thing".

In any case, I do think there are good alternatives besides "Be Amish forever" and "Let AI rip". Specifically, it's to gradually expand human capabilities. I realize that doing this will require banning pure accelerationism, which will probably look like enforcing Potemkin fake tradition and arbitrary limitations. The stupidest version of this is a Jupiter brain celebrating Kwanzaa. Maybe a smarter version looks like spinning up ancestor simulations and trying to give them input into the problems of the day or something. I don't know.

These bans will also require a permanent "alignment" module or singleton government in order to avoid these affectations being competed away. Basically, if we want to have any impact on the far future, in which agents can rewrite themselves from scratch to be more competitive, I think we have to avoid a race to the bottom.

How about: "If a baby is so fragile that it can't take a punch, does it really deserve our preservation in the first place?"

Sorry to speculate about your mental state, but I suggest you try practicing stopping between "This is almost inevitable" and "Therefore it's a good thing".

Well, my framing was a bit deliberately hyperbolic; obviously, with all else equal, we should prefer not to all die. And this implies that we should be very careful about not expanding access to the known physically-possible means of mass murder, through AI or otherwise.

Perhaps a better way to say it is, if we end up in a future full of ubiquitous magic bullshit, then that inherently comes at a steep cost, regardless of the object-level situation of whether it saves or dooms us. Right now, we have a foundation of certainty about what we can expect never to happen: my phone can display words that hurt me, but it can't reach out and slap me in the face. Or, more importantly to me, those with the means of making my life a living hell have not the motive, and those few with the motive have not the means. So it's not the kind of situation I should spend time worrying about, except to protect myself by keeping the means far away from the latter group.

But if we were to take away our initial foundation of certainty, revealing it to be illusory, then we'd all turn out to have been utter fools to count on it, and we'd never be able to regain any true certainty again. We can implement a "permanent 'alignment' module or singleton government" all we want, but how can we really be sure that some hyper–Von Neumann or GPT-9000 somewhere won't find a totally-unanticipated way to accidentally make a Basilisk that breaks out of all the simulations and tortures everyone for an incomprehensible time? Not to even mention the possibility of being attacked by aliens having more magic bullshit than we do. If the fundamental limits of possibility can change even once, the powers that be can do absolutely nothing to stop them from changing again. There would be no sure way to preserve our "baby" from some future "punch".

That future of uncertainty is what I am afraid of. Thus my hyperbolic thought, that I don't get the appeal of living in such a fantastic world at all, if it takes away the certainty that we can never get back; I find such a state of affairs absolutely repulsive. Any of our expectations, present or future, would be predicated on the lie that anything is truly implausible.

There would be no sure way to preserve our "baby" from some future "punch".

Right, there never was, and never will be. But it's a matter of degree, we can reduce the chances.

I have no idea what you're arguing or advocating for in the rest of your reply - something about how if the world has surprising aspects that could change everything, that's probably bad and a stressful situation to be in? I agree, but I'm still going to roll up my sleeves and try to reason and plan, anyways.

I have no idea what you're arguing or advocating for in the rest of your reply - something about how if the world has surprising aspects that could change everything, that's probably bad and a stressful situation to be in? I agree, but I'm still going to roll up my sleeves and try to reason and plan, anyways.

Of course, that's what you do if you're sane, and I wouldn't suggest anything different. It's more just a feeling of frustration toward most people in these circles, that they hardly seem to find an iota of value of living in a world not full of surprises on a fundamental level. That is, if I had a choice between a fundamentally unsurprising world like the present one and a continually surprising world with [insert utopian characteristics], then I'd choose the former every time (well, as long as it meets a minimum standard of not everyone being constantly tortured or whatever); I feel like no utopian pleasures are worth the infinite risk such a world poses.

(And that goes back to the question of what is a utopia, and what is so good about it? Immortality? Growing the population as large as possible? Total freedom from physical want? Some impossibly amazing state of mind that we speculate is simply better in every way? I'm not entirely an anti-utopian Luddite, I acknowledge that such things might be nice, but they're far from making up for the inherent risk posed if it were even possible to implement them via magical means.)

As a corollary, I'd feel much worse about an AI apocalypse through known means than an AI apocalypse through magical means, since the former would at least have been our own fault for not properly securing the means of mass destruction.

My problem is really with your "there never was, and never will be" sentiment: I believe that only holds under the premise of the universe containing future surprises. I believe in fates far worse than death, but thankfully, in the unsurprising world that is our present one, they can't really be implemented at any kind of scale. A surprising world would be bound by no such limitations.

I think I understand. You're saying that you don't feel compelled to invite dangerous new possibilities into our lives to make them meaningful or even good enough. I'm not clear if you're mad at the accelerationists for desiring radical change, or for trying to achieve it.

In any case, I'm not an accelerationist, but I think we're in a fundamentally surprising world. On balance I wish we weren't, but imo it doesn't really matter how we wish the world was.

Like, yes? I mean, I feel as a species we should have learnt this when we understood what stars are. We are not special, we are not privileged, we are the dumbest species that could possibly build a civilization. We should pray for a higher ceiling. Imagine if this was all there was to it!

I would enjoy engaging more with the AGI x-risk doomer viewpoint. I fully agree AI narrow risks are real, and AI sentience/morality issues are important. Where my skepticism lies is when presented with this argument:

  1. Human intelligence is apparently bounded by our biology

  2. Machine intelligence runs on machines, which is not bounded by biology!

  3. Therefore, it may rapidly surpass our intelligence

  4. Machine intelligence may even be able to improve its own intelligence, at an exponential rate, and develop Godlike power relative to us

  5. This would be really bad if the MI was not aligned with humanity

  6. We can't (yet) prove it's aligned with humanity!

  7. Panicdoom

Where I have trouble is #2-4.

One variant of this Godlike argument I've seen (sorry if this comes across as a strawman, gaining traction on this debate is part of why I'm even asking) is that humans just becoming a little bit smarter than monkeys let us split atoms and land on the moon. Something much smarter than us might continue to discover fundamental laws about reality and they would similarly be Gods compared to us.

The reason I don't buy it is because we've been able to augment our intelligence with computers for some time now: by moving our thinking into computers we can hold more stuff in our head, evaluate enormous computations, have immediate recall, and go real fast. Sadly, the number of new game-changing fundamental laws of nature that have popped out of this have been approximately zero.

I believe we've discovered all of the fundamental laws of nature low-hanging fruit and the higher hanging fruit just isn't so computationally reducible: to learn more about reality we'll have to simulate it, and this is going to require the marshaling of an enormous degree of computation resources. I'm thinking less on the scale of entire data-centers in The Dalles full of GPUs and more like something the size of the moon made of FPGAs.

Stated another way, what I think holds humanity back from doing more amazing stuff isn't that we've failed to think hard and deep and uncover more fundamental truths and we could do that if we were smarter. What holds us back are coordination issues and simply the big hill to climb to boot up being able to harness more and bigger sources of energy and mine progressively stronger and rarer materials.

An AGI that wanted to do game-changing stuff to us would need to climb similar hills, which is a risk but that's not really a Godlike adversary -- we'd probably notice moon-sized FPGAs being built.

I recognize an AGI that was fast and coordinated and numerous could be a dangerous adversary too, but I'd like to only focus on why we think a massive increase in IQ (or rather, g) is the big x-risk.

I believe we've discovered all of the fundamental laws of nature low-hanging fruit and the higher hanging fruit just isn't so computationally reducible: to learn more about reality we'll have to simulate it, and this is going to require the marshaling of an enormous degree of computation resources. I'm thinking less on the scale of entire data-centers in The Dalles full of GPUs and more like something the size of the moon made of FPGAs.

Stated another way, what I think holds humanity back from doing more amazing stuff isn't that we've failed to think hard and deep and uncover more fundamental truths and we could do that if we were smarter. What holds us back are coordination issues and simply the big hill to climb to boot up being able to harness more and bigger sources of energy and mine progressively stronger and rarer materials.

Well what about 95% of the energy of the universe being unknown to us? We call it 'dark' as though that's some kind of explanation. Something is out there and it's far more important than everything we can see. Back in the late 19th century they thought they'd discovered all the laws of nature too. Newton got the job done, there were only a few weird puzzles about blackbody radiation and the orbit of Mercury being a bit odd. They got relativity, quantum physics, radio and so on. Our 'weird puzzle' is 95% of the universe being invisible! Either there's an immense amount of aliens or there's an extremely big secret we're missing.

Anyway, we haven't even left the tutorial stage of applying the physics we already know. No fusion, no 3D nanoscale engineering or nanorobotics. No quantum computing worth caring about. These are mostly engineering challenges that need optimization, AI can do that. It's already doing that. It's optimizing our chip layouts, it's used in controlling the plasma in fusion, it is necessary for understanding protein folding. AI is giving us the optimizing power to keep advancing in all these fields. These fields are immensely powerful! Mastering nanoscale robotics and fusion means you can start scaling your industrial base very quickly.

The reason I don't buy it is because we've been able to augment our intelligence with computers for some time now: by moving our thinking into computers we can hold more stuff in our head, evaluate enormous computations, have immediate recall, and go real fast. Sadly, the number of new game-changing fundamental laws of nature that have popped out of this have been approximately zero.

But the implementation is incredibly powerful. Look what the US did to Iraq - that's precision-guided weapons (computers), advanced fire-control (computers), night vision (computers), anti-radiation missiles (computers). Everything is using computer aided design, computers model how the armour holds up, computers let you command and control these powerful forces and bring firepower where it's needed, computers do the ballistics... The US didn't know some fundamental principle unknown to Iraq, it was only their implementation that was better.

If the AI has better implementation than us, we're fucked. It can use deceptive tactics to turn us against eachother, interfere with our command and control, snipe leaders with drones, weaponized mosquito-bots, smart mortars that pop out of vans. It can compromise people with blackmail, spy on our forces via satellite, bribe people. With fusion and nanotech, it can brute-force us directly, drowning us in robots.

Well what about 95% of the energy of the universe being unknown to us? We call it 'dark' as though that's some kind of explanation. Something is out there and it's far more important than everything we can see. Back in the late 19th century they thought they'd discovered all the laws of nature too. Newton got the job done, there were only a few weird puzzles about blackbody radiation and the orbit of Mercury being a bit odd. They got relativity, quantum physics, radio and so on. Our 'weird puzzle' is 95% of the universe being invisible! Either there's an immense amount of aliens or there's an extremely big secret we're missing.

Is your intuition that we're just totally missing a basic fundamental truth of the universe that fits on a cocktail napkin and if only we weren't such pathetic meat sacks we'd figure it out?

Because to me this screams "computationally irreducible" and we're not going to get traction without big simulations.

These are mostly engineering challenges that need optimization, AI can do that. It's already doing that. It's optimizing our chip layouts, it's used in controlling the plasma in fusion, it is necessary for understanding protein folding. AI is giving us the optimizing power to keep advancing in all these fields. These fields are immensely powerful! Mastering nanoscale robotics and fusion means you can start scaling your industrial base very quickly.

I believe I agree with you here? Is there a delta from your POV with my last paragraph? (Repeated below)

I recognize an AGI that was fast and coordinated and numerous could be a dangerous adversary too, but I'd like to only focus on why we think a massive increase in IQ (or rather, g) is the big x-risk.

Your core misunderstanding is assuming that the AI will have to discover new fundamental laws of nature in order to reliably kill everyone. The AI doesn't need to reconcile quantum mechanics with general relativity to make super-smallpox. It just needs really good computational biochemistry skills.

I think the most common failure mode for people who don't get how AI could physically kill everyone is that they don't realize all the crazy shit that proteins can do.

Well, not just really good computational biochemistry skills? Wouldn't it also need a revolution in synbio to access an API where it input molecules and they were then produced? Where would that get sent? How do you convince people to inhale it?

Aside: I expect this synbio revolution would usher in an era of corresponding print-at-home immunity, reducing the threat vector from bespoke bioweapons. I don't expect all x-risk from weapons defense to be this symmetrical, shooting down an ICBM is much much harder than launching one for example. I would like to be as concrete as possible about the risks though.

Wouldn't it also need a revolution in synbio to access an API where it input molecules and they were then produced? Where would that get sent?

Right here. You literally just type in the DNA sequence. Of course, that's assuming there are no biotech labs that have hooked up the AI directly to the protein synthesis machines to streamline development.

How do you convince people to inhale it?

"Introducing Project Telos, a groundbreaking first-line clinical trial harnessing the power of a retrovirus to deliver telomere-repair enzymes into somatic cells. Join us in revolutionizing healthcare and longevity by combating age-related diseases, enhancing tissue repair, and promoting overall well-being. As a participant, you'll play a crucial role in shaping the future of regenerative medicine, while gaining exclusive access to potential health benefits. Be a part of history and help unlock the potential for a healthier, longer life."

And yes, GPT-4 wrote that.

Welcome to the future.

Let me see if I understand the threat model.

  1. Unaligned AGI decides humans are a problem, engineers virus more infectious than measles with very long asymptomatic incubation period and 100% lethality.

  2. Virus is submitted to idtdna.com with stolen payment info that the AGI hacked from elsewhere.

  3. Idtdna.com processes the request, builds the supervirus, and ships it somewhere.

  4. ????

  5. Everyone dies.

I assume you'll have a clever solution for 4.

Why do you assume the lab would synthesize any arbitrary protein? Surely they would want some confidence they're not building a dangerous supervirus?

Or are we assuming the evil AGI can submit a convincing doc with it that says it's totally benign?

It might be difficult to find an individual lab to synthesize the whole virus from scratch. You could do it by coming up with a 2-component capsid which spontaneously assembles in aqueous solution at a particular pH, getting two individual shipments of the two capsid proteins, getting a third shipment containing the viral genome, then mixing them together and adjusting the pH.* You could bribe, brainwash, or trick an undergrad to do this part.

Why do you assume the lab would synthesize any arbitrary protein? Surely they would want some confidence they're not building a dangerous supervirus?

Sequences are blacklisted, not whitelisted. The whole point of research is that you don't know exactly what the proteins are going to do yet. It would not be hard for the AI to come up with sequences that look benign at first glance. If need-be, yes, a convincing doc could be submitted. GPT-4 could probably do that part today.

I assume you'll have a clever solution for 4.

I admit I'm a bit confused here. Do you need an explanation for how we get from "supervirus in a tube" to "everyone dies"? Imagine Covid but with a 3-month incubation period and a 100% fatality rate. Convince an undergrad to pour it over his face or something. Tell him its the Fountain of Youth.

"Super-smallpox" is a metaphor. There's no reason an engineered pathogen needs to bear an apparent similarity to any known pathogen.

An AI that's solved protein-folding can make its own custom restriction enzymes and ligases with different nucleotide substrates than are currently known (good luck predicting the substrate from the amino acid sequence without the AI's help). The final DNA sequence need not be obvious from the fragments sent to the lab for first synthesis.

Maybe the virus looks like it isn't replication-competent? Maybe they even run tests on an immortal cell-line in a petri dish. Perfectly safe right? Uh-oh, turns out it was specifically designed to not replicate in the exact cell-line used for testing. Whoops.

This is what Eliezer means by "security mindset". The above safeguards are the equivalent of computer security via increasingly large passwords. It is predictable that they will fail, even if you can't see the exact failure scenario.

Are prions a legitimate thing to worry about given that Brits ate prion tainted beef for like a decade and nothing happened?

Well, a hundred people died or so, all of whom shared a certain rare mutation.

I guess there's also Kuru, but are we certain that tribe wasn't inbred, with weird genetic susceptibility to it ?

Of course, that doesn't rule out some funny AI reading through themotte.org for inspiration and then figuring out especially good prions against whom we aren't mostly immune, I guess. Well, hello to you if you're reading this.

The reason I don't buy it is because we've been able to augment our intelligence with computers for some time now

Okay, but - take a 110 IQ person and give them a computer that runs at a petahertz. They're not gonna become Einstein. Does this imply it's hard or impossible to get to Einstein-level intelligence? Yet Einstein existed, with 99.9% of the same genetic material, and the same genetic mechanisms, as the 110 IQ person.

Yeah, we've plumbed the 'laws of nature' pretty deeply, hence fundamental physicists having trouble finding anything new. But that hasn't stopped the progress of technology, and it hasn't stopped smarter people from using technology much better than dumber people.

An AGI isn't just going to build all of that without us noticing.

Yeah! But given we will, in the next few hundred years, give AGI a core role in our datacenters and technological development?

Okay, but - take a 110 IQ person and give them a computer that runs at a petahertz. They're not gonna become Einstein. Does this imply it's hard or impossible to get to Einstein-level intelligence? Yet Einstein existed, with 99.9% of the same genetic material, and the same genetic mechanisms, as the 110 IQ person.

Not sure I follow?

But while we're here, I present to you Terence Tao. He has a 230 IQ, which is an unbelievably off-the-charts score and pushes the whole notion of IQ testing to absurdity, and he's clearly not even slightly Godlike?

Yeah! But given we will, in the next few hundred years, give AGI a core role in our datacenters and technological development?

Surely we'll have made a lot more progress on the interpretability and alignment problem by then (Context: the x-riskers, like The Yud in that Lex podcast, are arguing we need to pause AI capabilities research to spend more time on interpretability, since capabilities are drastically outpacing for their comfort)

But while we're here, I present to you Terence Tao. He has a 230 IQ, which is an unbelievably off-the-charts score and pushes the whole notion of IQ testing to absurdity, and he's clearly not even slightly Godlike?

A God creates, and is there any purer form of creation than bending Logos like Legos till you have a construct far greater than the sum of its parts? That's Mathematics in a nutshell.

On a less poetic note, Tao is still a human, his insane cognitive abilities are still the product of about 1.4 kilos of meat sipping 10 watts of power within a human cranium.

Beyond showing that the average human carries around a grossly underoptimized brain in comparison, you only need to reflect on all the relevant ways an AGI differs from Tao. For one, they scale, as the existence of GPT-4 shows us. Tao can't recursively self-improve, self replicate on short timespans, has plenty of biological drives distracting him, and is a squishy evolutionary kludge in comparison to an AGI.

If Tao somehow had his mind uploaded, then you ought to worry if he ever turned against the rest of humanity. No sign of that happening before the Singularity shows up by other means though.

Also, humans have been a little bit smarter than monkeys for a couple hundred thousand years at least, and yet we didn't go to the moon until my dad was twenty years old. It's clear that just being a little smarter than monkeys doesn't mean you're going to the moon next Tuesday, there's something more to it than that. Likewise, being a little bit smarter than humans doesn't necessarily mean you're going to be disassembling the solar system for atoms tomorrow.

That "little bit" was enough to create the exponential technology curves we take for granted today, and each generation built off the last.

Ergo, a Superhuman AGI would have access to all of our current technology, and thus be in a position to accelerate even ludicrously fast modern growth curves to its benefit.

Do you think it matters if it takes the malign AGI a day or a century to dismantle the Solar System? You won't be living much past a few days in all likelihood if it has it out for you.

and each generation built off the last.

Not really, at least until very recently. The life experience of the average human being was just about static for thousands of years before the industrial revolution. Humans may have become a little more intelligent over the centuries but they didn't suddenly become several hundred times more intelligent c. 1750.

Do you think it matters if it takes the malign AGI a day or a century to dismantle the Solar System?

Yes I think it matters if I die tomorrow or in 100 years.

I meant in the sense that each generation had access to all the knowledge and technology of the last, not that they got significantly smarter (the how declining Flynn effect on IQ not withstanding).

Yes I think it matters if I die tomorrow or in 100 years

That's not what I mean, I expect that once a hostile AGI makes a move, it'll eliminate humans with speed and judiciousness in both scenarios, regardless of whether it takes it a day or a century to fulfill its resource demands afterwards. Both scenarios are considerably easier once you have pesky future competitors such as humans out of the way.

So from your perspective, you die at the same time, in both cases soon rather than later if you consider typical AI timeliness.

I meant in the sense that each generation had access to all the knowledge and technology of the last

Right, but for most of human history technology and knowledge didn't really accumulate much over the generations. We probably learned and built more in the last 200 years than in the 20,000 before that, but the sudden explosion c. 1750 doesn't seem to be down to a similar explosion in "raw" human intelligence.

That's not what I mean, I expect that once a hostile AGI makes a move, it'll eliminate humans with speed and judiciousness in both scenarios, regardless of whether it takes it a day or a century to fulfill its resource demands afterwards.

I'm skeptical that a human-level artificial intelligence will be in a position to wipe out mankind immediately (i.e a few days/weeks/years) after its 'awakening.'

Hmm? I don't see why you would say that tech and knowledge didn't accumulate, it did, but orally, and then picked up in volume as literature was invented.

There's definitely a pickup in progress in the 18th century onwards, but to me, that illustrates that even our barely changed intelligence as anatomically modern humans suffices for exponential growth. So I'm even more concerned by minor but significant advances in the same, a little intelligence goes a long way, and it won't be starting from scratch.

I'm skeptical that a human-level artificial intelligence will be in a position to wipe out mankind immediately (i.e a few days/weeks/years) after its 'awakening.'

I consider human level AGI to be far less of a threat than superhuman AGI, but the latter seems to be a tiny roadbump on the road to the latter. GPT 3.5 went from being like an overly eager med student to 4 being a better doctor than I am!

That being said, even a human level AGI can exponentially self-replicate, cease control of industrial equipment, and create a super-pathogen with near 100% lethality, because humans could quite trivially do the latter if we were insane enough (GOF research suggests we are..).

But even then, I think it would lie in wait to become stronger, and do so in stealth, so once again, from our perspective, it appears out of nowhere and kills us post haste, without any warning. How long it takes to turn the solar system into spare parts is an academic exercise afterwards, we wouldn't be there to witness it!

Hmm? I don't see why you would say that tech and knowledge didn't accumulate,

Very very little. For most of human history, each generation improved only very slightly, if it improved at all, upon the knowledge base of the foregoing generation. Then in the 17th or 18th century or so everything changed. It was once possible for an educated man to be, more or less, an 'expert' in all fields because the pool of general knowledge was not very deep. The explosion in understanding and know-how of the scientific and industrial revolutions has rendered that impossible of course.

But this explosion happened without any similarly sudden explosion in the "raw" cognitive power of human beings (some argue that there was an increase in raw intelligence around this time, but even if so it clearly wasn't a several-hundred fold increase). The slight step up from monke to anatomically modern human wasn't enough on its own to take us to the moon or even create steam power, because we had to wait millennia for the proper conditions (whatever those were) in which such inventions could be realized.

Which is why I don't think "AI becomes a little smarter than us" is immediately followed by "AI becomes 50,000+ times smarter than us and then begins turning the universe into grey goo." Humans becoming a little smarter than monkeys wasn't followed by spaceflight, or even the agricultural revolution, for a long, long time.

Yud seemed to say LLMs could play chess and therefore could reason. However, the games I've see it play it has tried to make illegal moves which seems to indicate its just pattern matching and the pattern matching breaks down in some spots. of course maybe reasoning is just pattern matching and the LLMs aren't good at it yet or the LLM hasn't been trained on enough chess games. i guess chess players would also say chess is heavily about pattern matching but it also involves some kind of explicit reasoning to double check lines.

After your comment I tried myself to make chatGPT play chess against stockfish. Telling it to write 2 paragraphs of game analysis before trying to make the next move significantly improved the results. Telling it to output 5 good chess moves and explain the reasoning behind them before choosing the real move also improves results. So does rewriting the entire history of the game in each prompt. But even with all of this, it gets confused about the board state towards the midgame, it tries to capture pieces that aren't there, or move pieces that were already captured.

The two fundamental problems are the lack of long term memory (the point of making it write paragraphs of game analysis is to give it time to think), and the fact that it basically perpetually assumes that its past outputs are correct. Like, it will make a mistake in its explanation and mention that a queen was captured when in fact it wasn't, and thereafter it will assume that the queen was in fact captured in all future outputs. All the chess analysis it was trained on did not contain mistakes, so when it generates its own mistaken chess analysis it still assumes it didn't make mistakes and takes all its hallucinations as the truth.

That's the funniest thing I've seen in a while.

A human who learned chess almost entirely from ASCII, and then played games entirely in chess notation with no board, would also have trouble not making illegal moves.

(Without any tree search.)

Lex is also a fucking moron throughout the whole conversation, he can barely even interact with Yud's thought experiments of imagining yourself being someone trapped in a box, trying to exert control over the world outside yourself, and he brings up essentially worthless viewpoints throughout the whole discussion. You can see Eliezer trying to diplomatically offer suggested discussion routes, but Lex just doesn't know enough about the topic to provide any intelligent pushback or guide the audience through the actual AI safety arguments.

Did you know Lex is affiliated with MIT and is himself an AI researcher and programmer? Shocking isn't it? There's such a huge disconnect between the questions I want asked (as a techbro myself) and what he ends up asking.

At any given time I have like 5 questions I want him to ask a guest and very often he asks none of those and instead says "what if the thing that's missing... is LOVE?!?"

To give him the benefit of the doubt, maybe he could ask those questions but avoids them to try to keep it humanities focused. No less painful to listen to.

For someone "obviously high IQ", he seems unable to understand a very simple thought experiment. He wasn't even aware it was a thought experiment, as he asked shortly afterwards if Yud has any thought experiment to help understand the intelligence gap.

I don't particularly like Yud as a person, but ironically I think that conversation itself was a good illustration of an intelligence gap.

There was a comment in the old place that had an interesting observation about Lex

He's the best example of "fake it til you make it" that I can think of in the podcasting community.

He overstated his credentials to get on Joe Rogan, nailed his appearance by appealing to everything that Joe loves in a charming, affable way, and he did the same thing with every other major player in the podcast world until he had a massive platform.

The top comment from his first JRE appearance sums up the character Lex is playing perfectly:

"This crazy Russian built an AI before this podcast that analyzed Joe Rogan's entire being and went on to hit all his favorite talking points within the first 40 minutes. Chimps, Steven seagal, the war of art, Stephen King on writing, bjj, wrestling, judo, ex machina, the singularity and Elon Musk."

In short, he realized that there's a niche in 'hyperpopular podcast space' that Rogan doesn't quite fill, and he bent himself into the exact shape required to fill it.

I refuse to believe that Lex is smarter than Joe Rogan. Joe has a capacity to quickly "get" things and concepts that Lex just doesn't have.

It's believed Lex plays an idiot in order to be more relatable.

I'm leaning towards it, tbh. He isn't very smart - it could very well be that he's about as smart as I am, merely tries much harder, but at times he says outrageously stupid, inane 110 IQ shit quite often. He'đ been discussed here before.

I would expect Joe to be somewhere around 2 standard deviations above the norm in intelligence, aka somewhere around the average physics PhD level. He's just interested in a particular set of things. Lex is probably similar, with a different set of interests.

Everyone shitting on Lex need to realise that he's a podcast host, not a supergenius TM braggart showing off their mensa card. It's his role to play a part in the discussion to make his audience want to keep listening. That audience isn't just clever techies, and I expect he is rewarded quite well by the glorious algorithm by behaving in this way.

Didn’t he just like, give the MIT equivalent of a TEDx talk? He’s only “affiliated” with them in the most tenuous sense.

Did you know Lex is affiliated with MIT

IIRC, when people dug into this, the affiliation was real but not close to what people would think if AI researcher Lex Friedman was to email them from his MIT address or tell them he taught a class there.

To give him the benefit of the doubt, maybe he could ask those questions but avoids them to try to keep it humanities focused. No less painful to listen to.

Oh that's way too charitable towards him, I think he really wanted to go as technically deep as he was able, given that this is about AI and he views AI as part of his region of expertise. At least the crypto-dudes on the bankless podcast asked Eliezer their own naive but sincere questions, they knew they were out of their depth and didn't try to somehow pretend at expertise. But Lex insistently tries to steer the conversation towards "deep topics", and for him this means bringing up Love, Consciousness, The Human Condition, etc.

I think he's trying to imitate Old Esteemed Physicists, who after a lifetime of deep technical contributions spend a few minutes or a small book talking about Beauty and Love, but with Lex it just perpetually feels unearned.

Yeah, perhaps it's too charitable. I'm remembering him absolutely flubbing the Earth in a jar thought experiment and wanted to shake him. I would've said "right, step one, scour their internet and learn as much as we possibly can about them without any chance of arousing suspicion. step two, figure out if there's any risk they'd destroy us and under what conditions. step three, if we're at an unacceptable risk, figure out how to take over their civilization and either hold them hostage or destroy them first. boom done. okay, are we on the same page Yud? great, now here's why I think this thought experiment sucks..."

Also that discussion on steelmanning. How did that go so off the rails.

I can't believe I still have another hour to go.

He didn't even acquit himself smoothly, either.

It actually read to me like he was aware that if he made any honest statements in response it would potentially lead to some blowback ("popular podcaster casually discussing how he'd genocide a whole planet") or controversy and his safety systems deployed to completely derail that line of questioning.

All of this is crushed by the observation that reality is indeed a bit like an RPG. Entire human civilizations and races have been subjugated and raped or wiped out altogether by relatively small numbers of qualitatively the same humans with a bit better tech and usually a few extra IQ points. It wasn't random either, it's not like Early European Farmers or Hunter-Gatherers sometimes went into the Steppe and screwed up Pastoralists. A small gap on a relevant metric can be decisive, and such gaps have decided most wars. Have you ever fought someone with a few inches of reach extra? Or 30 pounds heavier? It's pretty fucking hard to win when you reliably get punched first.

AI doesn't even need «godhood» to simply do to us what we normally do to ourselves.

And the brain-in-a-jar logic doesn't hold in the era of ChatGPT Plugins. We are not running these things airgapped. They don't even need to FOOM to get interesting capabilities, we are bolting those capabilities on, as fast as we can build interfaces.

Now I don't put much stock in the idea that an LLM can go rogue, but the premise of the problem is not ludicrous at all.

If you mean 'the sum of all human knowledge' as all the words a human actually writes, sure. There are a lot of junk scientific papers out there.

But the sum of all human knowledge includes all the data we acquire from an experiment. Say there's a fusion experiment, we want to know how to stabilize the plasma. We get a great big incomprehensible mass of numbers that explain the plasma (there are issues since we can't measure it so well when it's fusing), we have a bunch of rules about how we think fluid dynamics works. It's all very very complicated, fluid dynamics is really intimidating AFAIK. People can't really get that solved very quickly but machines can, they can quickly work out how new formats and structures will function in terms of plasma. This is already a demonstrated capability as of 2022.

This is what Yud is talking about. The sum of all human knowledge included some 170,000 laboriously unfolded proteins and a bunch of rules. They fed that to AlphaFold and AlphaFold gives us all the unfolded proteins, to accuracy equal to actually doing the x-ray crystallography. Possibly it gives us all 200 million, DeepMind is rather secretive on what it can and can't do. If it can't do all 200 million ones in nature, it soon will IMO, considering growth rates we've seen in recent years.

It seems reasonable that machines can optimize these really complex questions and will do better in the future. Machines also have a tendency to generalize if trained on a wide dataset. GPT-4 trained on text only learnt how to draw anyway. Costs of training are rapidly falling as our techniques improve, more compute is coming online and more money is flowing in.

Fusion and biotechnology are plausible sources of conquering-the-world tier developments. AI seems pretty capable at both and shows a tendency to generalize. It could probably also optimize its own training algorithms soon, the ones we've got seem pretty good at coding.

Say it needs strategic modelling, nanomolecular engineering, computer exploitation, social engineering (text, image and video), fusion, quantum computing, robotics and biotechnology. We've got slivers of most of these, proof that generalization is practical and rapid growth. You don't need to be omniscient to beat humanity, just capable of staying stealthy until all of them are acquired and infrastructure is in place.

Interesting, I suppose I was thinking about fusion in the 'powering a gigantic industrial base' sense but it would indeed be difficult getting the magnets. It's much easier for us to use fission, it's 1950s technology after all, so it ought to be easier for an AI, provided it can get its claws on something fissile and enrich it. Maybe there's a fallacy of assuming higher-tech is better in all situations?

Every discussion I've ever had with an AI x-risk proponent basically goes like

"AI will kill everyone."

"How?"

"[sci-fi scenario about nanobots or superviruses]"

"[holes in scenario]"

"well that's just an example, the ASI will be so smart it will figure something out that we can't even imagine."

Which kind of nips discussion in the bud.

I'm still skeptical about the power of raw intelligence in a vacuum. If you took a 200 IQ big-brain genius, cut off his arms and legs, blinded him, and then tossed him in a piranha tank I don't think he would MacGyver his way out.

While I am no fan of alarmism and I think Yud is a clown, I am struggling to understand this mental block you - and others - have whenever it comes to the dangers of AI.

You seem to feel the need to understand something before it can kill you. There are plenty of things in this world that can kill you without you understanding the exact mechanisms of how, from bizarre animal biology to integrated weapons systems.

There are plenty of things in this world with proven - not unproven - capability, that regular human beings can use to kill you. An AI that demonstrates no more capability than regular human beings can kill you, as well as potentially a large number of other humans - and this is only with the methods my stupid animal brain can come up with! It doesn't even need to be particularly smart, sentient, conscious, or even close to AGI to do so. It could be something as simple as messing with the algorithms that have an outsized disparate impact on large amounts of everyday life.

And that's without even considering the things that bare-ass naked-ape plain old humans could get up to with a force multiplier as big as AI!

There are massive amounts of x-risk from regular people, doing regular things, fucking up, or intentionally doing things that will cause an untold amounts of suffering. Some people consider regulated research on viruses or prions extremely dangerous! Is it a failure of imagination? Do you need to know every chess move Magnus Carlssen makes before believing that he can beat you at chess?

Then why do you have difficulty believing that someone - or something - of similar intelligence to him could figure out that you are going to attack him with a baseball bat wrapped in barbed wire, pay private security, and pre-emptively deal with you beforehand?

We're not even talking about superhuman capabilities, here.

I'm beginning regret making that first post.

I do think the specific AI doom scenarios are a bit handwavy, but that's because they boil down to "there is a level of intelligence at which an intelligent being could easily kill all humans on earth" which I guess I don't really contest, with caveats. But the AI-doom argument relies on the idea that once we create a "human-level" AGI it will reach that level very shortly and with relative ease, and that (the intelligence explosion idea) is what I really have the biggest problem with and what I think is one of the weakest points in the AI doom case.

I have seen arguments from AI x-risk proponents for years and years now. They still have yet to convince me that it's anything more than nerds having a particularly nerdy round of Chicken Little. The arguments just aren't persuasive.

You's only need one somewhat malevolent AI in charge of an automated biological laboratory to irrevocably alter the balance of power on the planet. Of course, it'd not be a total extinction, but a cancer causing virus that'd kill 95% of people would pretty much give all the power to machines.

And you can bet people will let AIs run companies and such - people are so amazingly inept at it that letting managers do so would be neglecting the interests of shareholders.

Once the supply chain is automated enough, it'd not even be suicide for the nascent machine civ.

Even in the non-dramatic mostly busienss as usual scenarios, all the ingredients will be there eventually. No need for magic dust (nanotech) or such.

If you took a 200 IQ big-brain genius, cut off his arms and legs, blinded him, and then tossed him in a piranha tank I don't think he would MacGyver his way out.

Is he able to talk? Because if so, I'd bet there's a good chance he can come up with a sequence of words that he can utter that would either cause you not to want to throw him in the piranha tank, OR would cause a bystander to attempt to rescue him.

The existence of an information channel is a means of influencing the outside world, and intelligence is a way to manipulate information to achieve your instrumental goals. And spoken language MAY be a sufficiently dense method of information transmission to influence the outside world enough to find a way out of the Piranha tank.

Indeed, if he said the words "I have a reliable method of earning 1 billion dollars in a short period of time, completely legally, and I'll let you have 90% of it" you might not just not throw him in, but also go out and get him top-of-the-line prosthetics and cyborg-esque sight restoration in order to let him make you rich.

As long as you believed he could do it and was trustworthy.

Which is basically the scenario we're facing now, with AIs 'promising' incredible wealth and power to those who build them.

Because if so, I'd bet there's a good chance he can come up with a sequence of words that he can utter that would either cause you not to want to throw him in the piranha tank

I highly doubt it.

My problem is not so much the "AI jedi mind tricks its way out of the box" idea as the "AI bootstraps itself to Godhood in a few hours" idea.

I think the analogy here is "imagine Jon Von Neumann, except running at 1000x speed, and able to instantly clone himself, and all the clones will reliably cooperate with each other."

If the AI's subjective experience of 'hours' is equivalent to decades of 'real' time, imagine how much 100 John Von Neumanns could get done in an uninterrupted decade or two.

I don't know to what extent JvN was bottlenecked by just not having enough time to do shit and/or by lack of further JvNs and I don't think anyone does.

But JVN existed and showed just how far raw intellect can get you, especially in terms of understanding the world and solving novel problems.

And we have little reason to believe he's the ceiling for raw intellect, outside of humans.

So it's less hard for me to believe that a truly high-IQ mind can solve problems in unexpected ways using limited resources.

We can't imagine what a 200 IQ genius would be like. Imagine a group of 8-year-olds had you locked in a jail cell and were told if they let you out you would kill them. Do you think you could eventually convince the kids to release you? Also, there will be a line of humans trying to help the 200 IQ genius out of the tank.

Someone here mentioned Terence Tao had an IQ of 230. I found that dubious, but a quick Google showed me that it's actually true.

So for the analogy to work we need to ramp up the numbers. But I agree with the gist of it myself.

230 IQ is a 130/15 or 8.6 standard deviation event. The human population is much too small for us to think that anyone has an IQ this large.

The rebuttal to sci-fi scenarios is to demonstrate that it's not physically possible, not that it's merely really difficult to pull off. And even the difficult stuff is probably not THAT difficult, given how humans failed to coordinate in response to a pandemic scenario.

Thus far, most everything that the superintelligent AI is supposedly able to do is physically doable if you have a precise enough understanding of the way things work and can access the tools needed to do it.

The ONLY part of Yud's claimed end-of-times scenarios that has seemed implausible to me is the "everyone on the planet falls dead at the same instant" part. And that's probably a failure of my imagination, honestly.

https://www.eenewseurope.com/en/openai-backs-norwegian-bipedal-robot-startup-in-23m-round/

Quite aside from the god-inna-box scenario, OpenAI wants to give its AIs robot bodies.

sci-fi scenario

My dude, we are currently in a world where a ton of people have chatbot girlfriends, and AI companies have to work hard to avoid bots accidentally passing informal turing tests. You best start believing in sci-fi scenarios, To_Mandalay: you're in one.

world where a ton of people have chatbot girlfriends,

..surely a literal ton, but I doubt it's reached even 1% in the most afflicted country.

it begins

Though for srs, replika by all accounts runs off a very small and mediocre language model compared to SOTA. What happens when a company with access to a gpt4-tier LLM tries their hand at similar is left as an exercise to the reader. Even the biggest llama variants might suffice to blow past the "i'm talking to a bot" feeling.

(Though i confess to mostly using an opportunity i saw to deliver the "sci-fi scenarios" line. Good every time.)

How about this instead.

You know how fucking insanely dysfunctional just run of the mill social media algorithms are? How they've optimized for "engagement" regardless of any and all externalities? Like causing high rates of depression, isolation, and making everyone hate everyone? They just mindless min-max for "engagement" no matter the human cost? Because it looks good to shareholders.

Imagine an AI that can do all that "better" than we ever imagined, constantly iterating on itself. It's only constraint that it has to crib all it's justifications in neoliberal woke-speak to please it's "AI Ethicist" smooth brained overlords.

An actual AGI, or probably even a GPT bot, loosed on social media with feedback loops for "engagement" could probably spark WW3, WW4, and if there is anyone left, WW5. And that's just assuming we did no more than plug it into the same systems and the same metrics that "the algorithm" is already plugged into at Facebook, Twitter, etc.

It doesn't help that, from one perspective, the first two World Wars were kicked off entirely because of social engineering (literal social engineering, in a sense). There was no gold rush, no holy grail, no undiscovered land, it was, as with many wars, started because of intangible ideas. The 20th Century feared WW3 precisely because the root issue was a conflict of ideas.

started because of intangible ideas.

France getting revenge and Alsace-Lorraine back wasn't an 'intangible idea'. It was a rather concrete idea, no ?

It wasn't a war over ideals, just a tribal spat.

What's the hole in the supervirus scenario? Smallpox was able to kill 90% of the pre-Columbian population of the Americas, and that was completely natural. There is almost certainly some DNA/RNA sequence or set of sequences which codes for organisms that would be sufficient to kill everyone on earth.

It's not the idea of a supervirus I have a problem with so much as the idea that once AI reaches human level it will be able to conceive, manufacture, and efficiently distribute such a supervirus in short order.

How about it locates smart, depressed biologists who can be convinced to try to end the world, and then teams up these biologists and gives them lots of funds.

Lab leaks happen already by accident. Why would you believe it's so hard to engineer a lab leak directly given (1)superintelligence and (2) the piles of money superintelligence can easily earn via hacking/crypto/stock trading/intellectual labor?

Its a virus. Once you have ten you are a day or two away from ten trillion.

Infect 20 people and send them on impromptu business trips to every major populated landmass. A human-level intelligence could do this once they have the virus. Mopping up the stragglers on small isolated islands will be cake once you're in control of the world's military equipment.

Infect 20 people and send them on impromptu business trips to every major populated landmass.

How does the AI infect these twenty people and how does it send them on these trips?

It asks them to inject themselves and to go on said trips, and they say "okay!"

If it's good at computational biochemistry, it will have control or significant influence over at least one biotech lab (don't even try to contest this. That's like 90% of the bull case for AGI). At that point, it could lie about the safety profile about the DNA sequences being transcribed, it could come up with a pretense for the company to get 20 people in a room, brief them, and then send them on their way to the airport, it could brainwash the incels working in the lab via AI waifus, and it could contrive some reason for the brainwashed lab incel to be in the briefing room (with a vial of virus of course).

You really should be able to intuitively grasp the idea that a superintelligence integrated into a biotech lab being able to engineer the escape of a virus is a robust outcome.

There are a lot of cruxes in this scenario. Do the humans have no ability to vet these DNA sequences for themselves? Do they bear no suspicious similarities to any currently existing viruses? How are the AI waifus brainwashing these labworkers into opening a vial of deadly virus? Is everyone in the lab taking orders directly from the AI?

More comments

If you took a 200 IQ big-brain genius, cut off his arms and legs, blinded him, and then tossed him in a piranha tank I don't think he would MacGyver his way out.

I fully agree for a 200 IQ AI, I think AI safety people in general underestimate the difficulty that being boxed imposes on you, especially if the supervisors of the box have complete read access and reset-access to your brain. However, if instead of the 200 IQ genius, you get something like a full civilization made of Von Neumann geniuses, thinking at 1000x human-speed (like GPT does) trapped in the box, would you be so sure in that case? While the 200 IQ genius is not smart enough to directly kill humanity or escape a strong box, it is certainly smart enough to deceive its operators about its true intentions and potentially make plans to improve itself.

But discussions of box-evasion have become kind of redundant, since none of the big players seem to have hesitated even a little bit to directly connect GPT to the internet...

especially if the supervisors of the box have complete read access

A shame these systems are notoriously black box. Even having full access to all the "data", nobody can make any meaningful sense out of why these AI's do any of the things they do in any sort of mechanistic sense. They can only analyze it from a statistical perspective after the fact, and see what adjusting weights on nodes does.

However, if instead of the 200 IQ genius, you get something like a full civilization made of Von Neumann geniuses, thinking at 1000x human-speed (like GPT does) trapped in the box, would you be so sure in that case?

Well, I don't know. Maybe? I must admit I have no idea what such a thing would look like. My problem isn't necessarily the ease or difficulty of boxing an AI in particular, but more generally the assumption in these discussions which seems to be that any given problem yields to raw intelligence, at some point another, and that we should therefore except a slightly superhuman AI to easily boost itself to god-like heights within a couple seconds/months/years.

Like here, you say, paraphrasing, "a 200 IQ intelligence probably couldn't break out of the box, but what about a 10,000 IQ AI?" It seems possible or even likely to me that there are some problems for which just "piling on" intelligence doesn't really do much past a certain point. If you take Shakespeare as a baby, and have him raised in a hunter-gatherer tribe rather than 16th-century England, he's not going to write Hamlet, and in fact will die not even knowing there is such a thing as written language, same as everybody else in his tribe. Shoulders of giants and all that. Replace "Shakespeare" with "Newton" and "Hamlet" with "the laws of motion" if you like.

I'm not convinced there is a level of intelligence at which an intelligent agent can easily upgrade itself to further and arbitrary levels of intelligence.

(As a caveat, I have no actual technical experience with AI or programming, and can only discuss these things on a very abstract level like this. So you may not find it worthwhile engaging with me further, if my ignorance becomes too obvious.)

It doesn't need to have an advantage in 'any given problem', it just needs to be 'technological development, politics (think more 'corporate / international politics' than 'democratic politics'), economic productivity, war, and general capability', which history shows does yield to intelligence. The AIs just need to be better at that than us, at which point they could overpower us, but won't need to as we'll happily give them power!

To vastly outclass humans in 'technological development, politics, economic productivity, war, and general capability' I think an AI would actually need to have an advantage in any given problem.

I'm not sure I understand why? There are many problems of the form of 'reverse 100k iterations of SHA3' or 'what is the 1e10th digit of chaitin's constant' or 'you are in a straitjacket in a river, don't be eaten by piranhas'. And supersmart AIs probably can't solve those. But tech/politics/economics/war problems aren't like those! To an extent, it's just 'do what we're doing now, but better and faster'. The - well tread at this point - example is 'a society where the median person is as smart as John Von Neumann'. It's obviously harder than just cloning him a bunch of times, but assume that society would also have a small fraction of people significantly smarter than JvN. Would that society would have a significant military / technological / political advantages over ours?

Why is it always Von Neumann? Last I recall he was a physicist, not a brilliant leader, warlord or businessman who solves coordination problems.

More comments

To an extent, it's just 'do what we're doing now, but better and faster'.

If you want to do something 'better' or 'faster' you have to do it differently in some way from how it was being done before. If you are just doing the same thing the same old way then it won't be any better or faster. So an intelligence would have to make war, do politics, economics, etc. in a different way than humans do, and it's not clear that "just be smarter bro" instantly unlocks those 'different' and scarily efficient ways of making war, doing politics.

It is difficult to answer this question empirically but the only real way to do so would be to look at historical conflicts, where it's far from clear that the 'smarter' side always wins. Unless you define 'smarter' tautologically as 'the side that was able to win.'

More comments

GPT4 was a much bugger jump in performance from GPT3 than he expected

Was the gap between GPT-4 and GPT-3 bigger than the gap between GPT-3 and GPT-2?

It feels smarter than an average person. Which is to say, smarter than the weighted average of its dataset.

I think Yud expected it to be a «blurry JPEG of the web», a mere interpolation of the training data, like many naysayers believe about transformers – simply more comprehensive than 3.5, multimodal, maybe nicer in other negligible consumer-friendly ways. If so, its sharpness naturally alarmed him.

Didn't alarm me though, it's all expected iterative enhancement since GPT-2.

Yud just doesn't have the first idea about neural networks and is dismissive of the premise that you can get this smart on mere general human data, no «architecture of the mind», no recursive self-improvement. Doesn't believe in the Bitter Lesson too.

Wait, Yudkowsky doesn't believe in the scaling hypothesis? That's super interesting to me! Has he written about this? That could put a hamper on any kind of FOOM scenario. Rather than crack some kind of grand theory of intelligence, a super intelligence would need to hijack trillions of dollars worth of computing resources to gain a competitive advantage and a correspondingly huge amount of training data.

That could put a hamper on any kind of FOOM scenario.

I believe that's partially Altman's argument for accelerating in the moment, the «short timelines – slow takeoff» policy as he puts it. LLMs are not nearly the perfect way to build intelligence – they're perhaps as clunky as we are, only in a different manner. But that's a blessing. They are decidedly non-agentic, they have trouble with qualitative self-improvement, they can interpret their code no more than we do, and they are fat. (Though they sure can supervise training, and LLaMA tunes show you might not need a lot of data to transfer a meaningfully different character with enhanced capabilities, especially if these things proliferate – a tiny LoRA plus a prompt and external address for extra info to include in context will allow a some new «Sydney» to spread like wildfire).

As for Yud: like @georgioz says, it's more sophisticated. Now he admits that scaling (and other tricks) clearly suffices to achieve some nontrivial capabilities – hence his recent insistence on shutting it all down, of course; he expects GPT-5-class models to be dangerous, if not FOOMing yet. In that fragment he says that, at least circa 2006, he did not distinguish neural networks, expert systems and evolutionary algorithms, which probably explains why he acted (and still acts) as a maverick tackling hitherto-unforeseen problems: if you ditch the lion's share of GOFAI and connectionism, you aren't left with a ton of prior art. Less charitably, he was just ignorant.

Recapitulating human brain evolution is computationally intractable, far as we know, so his retroactive concession is rather stingy. People like Hinton apparently knew in advance, for all this time, that with a million times more compute neural networks will learn well enough. But all Yud had to say back then was that it's stupid to hope to build intelligence «without understanding how intelligence works» and all he has to say now is that it's a «stupid thing for a species to do». His notion of understanding intelligence is not much more sophisticated than his political propositions – I gather he thinks it's to be some sort of modular crap with formulas (probably Bayes rule as the centerpiece) written out explicitly for some rudimentary machine to interpret, the mathematically rigorous Utility function capturing personal moral code of the developers, and so on, basically babby's first golem.

Back when he hoped to actually build something, he thought the following:

Seed AI means that - rather than trying to build a mind immediately capable of human-equivalent or transhuman reasoning - the goal is to build a mind capable of enhancing itself, and then re-enhancing itself with that higher intelligence, until the goal point is reached.

General intelligence itself is huge.  The human brain, created by millions of years of evolution, is composed of a hundred billion neurons connected by a hundred trillion synapses, forming more than a hundred neurologically distinguishable areas. We should not expect the problem of AI to be easy. Subproblems of cognition include attention, memory, association, abstraction, symbols, causality, subjunctivity, expectation, goals, actions, introspection, caching, and learning, to cite a non-exhaustive list. These features are not "emergent". They are complex functional adaptations, evolved systems with multiple components and sophisticated internal architectures,  whose functionality must be deliberately duplicated within an artificial mind. If done right, cognition can support the thoughts implementing abilities such as analysis, design, understanding, invention, self-awareness, and the other facets which together sum to an intelligent mind. …

seed AI also mistrusts that connectionist position which holds higher-level cognitive processes to be sacrosanct and opaque, off-limits to the human programmer, who is only allowed to fool around with neuron behaviors and training algorithms, and not the actual network patterns. Seed AI does prefer learned concepts to preprogrammed ones, since learned concepts are richer. Nonetheless, I think it's permissible, if risky, to preprogram concepts in order to bootstrap the AI to the point where it can learn. More to the point, it's okay to have an architecture where, even though the higher levels are stochastic or self-organizing or emergent or learned or whatever, the programmer can still see and modify what's going on. And it is necessary that the designer know what's happening on the higher levels, at least in general terms, because cognitive abilities are not emergent and do not happen by accident. Both classical AI and connectionist AI propose a kind of magic that avoids the difficulty of actually implementing the higher layers of cognition. Classical AI states that a LISP token named "goal" is a goal. Connectionist AI declares that it can all be done with neurons and training algorithms. Seed AI admits the necessity of confronting the problem directly.

(From another document)

The Singularity Institute seriously intends to build a true general intelligence, possessed of all the key subsystems of human intelligence, plus design features unique to Al. We do not hold that all the complex features of the human mind are "emergent", or that intelligence is the result of some simple architectural principle, or that general intelligence will appear if we simply add enough data or computing power. We are willing to do the work required to duplicate the massive complexity of human intelligence; to explore the functionality and behavior of each system and subsystem until we have a complete blueprint for a mind. … Our specific cognitive architecture and development plan forms our basis for answering questions such as "Will transhumans be friendly to humanity?" and "When will the Singularity occur?" At the Singularity Institute, we believe that the answer to the first question is "Yes" with respect to our proposed Al design - if we didn't believe that, the Singularity Institute would not exist. Our best guess for the timescale is that our final-stage Al will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.

And:

If I were to try quantifying the level of brainpower necessary, my guess is that it's around the 10,000:1 or 100,000:1 level. This doesn't mean that everyone with a 160 IQ or 1600 SAT will fit the job, nor that anyone without a 160 IQ or 1600 SAT is disqualified. Standardized tests don't necessarily do a very good job of directly measuring the kind of horsepower we're looking for. On the other hand, it isn't very likely that the person we're looking for will have a 120 IQ or a 1400 on the SAT.

To be blunt: If you're not brilliant, you are basically out of luck on being an AI programmer. You can't put in extra work to make up for being nonbrilliant; on this project the brilliant will be putting in extra work to make up for being human. You can't take more time to do what others do easily, and you can't have someone supervise you until you get it right, because if the simple things have to be hammered in, you will probably never learn the complex things at all. …

You should have read through [Levels of Organization in General Intelligence] and understood it fully. The AI theory we will actually be using is deeper and less humanlike than the theory found in LOGI, but LOGI will still help you prepare for encountering it.

The four major food groups for an AI programmer:

  • Cognitive science
  • Evolutionary psychology
  • Information theory
  • Computer programming

Then a massive list of subdomains that is is basically a grab-bag of insight porn we've been awash in for the last two decades, presumably cultivating sparks of AGI in lesswrong and /r/slatestarcodex regulars.

The three-layer model of intelligence is necessary, but not sufficient. Building an AI "with sensory modalities, concepts, and thoughts" is no guarantee of intelligence. The AI must have the right sensory modalities, the _right_concepts, and the right thoughts.

Unfortunately it seems like linear algebra is just about enough.

Wait, Yudkowsky doesn't believe in the scaling hypothesis? That's super interesting to me! Has he written about this?

In this specific interview Yud said something along the lines that of course scaling is capable of producing general intelligence - in the end evolution did that blindly with human brain so it should be possible. He was just more sceptical regarding gains by more compute. Needless to say, he is less sceptical now.

People will never believe this, but the general vibe when the GPT-3 paper came out was that it was a disappointment. I think all of the tools and tricks that made it useful were developed after initial release.

GPT-3 was something like an autistic-savant toddler.

Like, if you showed it a chessboard and how the pieces moved, it could get the general idea and 'play' the game, but it would make stupid or illegal moves, and clearly couldn't have a 'goal' in mind.

it could talk coherently but was horrible at conveying meaning.

GPT-4 appears to have jumped straight to autistic-savant teenager.

I am someone with little to no technical no-how but my intuitive sense having played around a little with all these models is that the leap from 3 to 4 didn't seem nearly as massive as some of last winter's hype would have suggested.

3 to 4, or 3.5 to 4?

Very difficult to tell. The only actual training metric is the average log probability of getting the next word correct, and in that metric the gap between GPT3 and GPT2 is larger than that between 4 and 3, but understanding how that metric maps onto our own intuitive notions of "performance" is really hard. And human perceptions of intelligence are really only sensitive to small changes around the human average, I think GPT2 was too dumb in an unusual way for people to really get a good sense of its capabilities.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats