site banner
naraburns  nihil supernum  1yr ago (text post)   38431 thread views

Culture War Roundup for the week of March 27, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

11
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

GPT4 was a much bugger jump in performance from GPT3 than he expected

Was the gap between GPT-4 and GPT-3 bigger than the gap between GPT-3 and GPT-2?

It feels smarter than an average person. Which is to say, smarter than the weighted average of its dataset.

I think Yud expected it to be a «blurry JPEG of the web», a mere interpolation of the training data, like many naysayers believe about transformers – simply more comprehensive than 3.5, multimodal, maybe nicer in other negligible consumer-friendly ways. If so, its sharpness naturally alarmed him.

Didn't alarm me though, it's all expected iterative enhancement since GPT-2.

Yud just doesn't have the first idea about neural networks and is dismissive of the premise that you can get this smart on mere general human data, no «architecture of the mind», no recursive self-improvement. Doesn't believe in the Bitter Lesson too.

Wait, Yudkowsky doesn't believe in the scaling hypothesis? That's super interesting to me! Has he written about this? That could put a hamper on any kind of FOOM scenario. Rather than crack some kind of grand theory of intelligence, a super intelligence would need to hijack trillions of dollars worth of computing resources to gain a competitive advantage and a correspondingly huge amount of training data.

That could put a hamper on any kind of FOOM scenario.

I believe that's partially Altman's argument for accelerating in the moment, the «short timelines – slow takeoff» policy as he puts it. LLMs are not nearly the perfect way to build intelligence – they're perhaps as clunky as we are, only in a different manner. But that's a blessing. They are decidedly non-agentic, they have trouble with qualitative self-improvement, they can interpret their code no more than we do, and they are fat. (Though they sure can supervise training, and LLaMA tunes show you might not need a lot of data to transfer a meaningfully different character with enhanced capabilities, especially if these things proliferate – a tiny LoRA plus a prompt and external address for extra info to include in context will allow a some new «Sydney» to spread like wildfire).

As for Yud: like @georgioz says, it's more sophisticated. Now he admits that scaling (and other tricks) clearly suffices to achieve some nontrivial capabilities – hence his recent insistence on shutting it all down, of course; he expects GPT-5-class models to be dangerous, if not FOOMing yet. In that fragment he says that, at least circa 2006, he did not distinguish neural networks, expert systems and evolutionary algorithms, which probably explains why he acted (and still acts) as a maverick tackling hitherto-unforeseen problems: if you ditch the lion's share of GOFAI and connectionism, you aren't left with a ton of prior art. Less charitably, he was just ignorant.

Recapitulating human brain evolution is computationally intractable, far as we know, so his retroactive concession is rather stingy. People like Hinton apparently knew in advance, for all this time, that with a million times more compute neural networks will learn well enough. But all Yud had to say back then was that it's stupid to hope to build intelligence «without understanding how intelligence works» and all he has to say now is that it's a «stupid thing for a species to do». His notion of understanding intelligence is not much more sophisticated than his political propositions – I gather he thinks it's to be some sort of modular crap with formulas (probably Bayes rule as the centerpiece) written out explicitly for some rudimentary machine to interpret, the mathematically rigorous Utility function capturing personal moral code of the developers, and so on, basically babby's first golem.

Back when he hoped to actually build something, he thought the following:

Seed AI means that - rather than trying to build a mind immediately capable of human-equivalent or transhuman reasoning - the goal is to build a mind capable of enhancing itself, and then re-enhancing itself with that higher intelligence, until the goal point is reached.

General intelligence itself is huge.  The human brain, created by millions of years of evolution, is composed of a hundred billion neurons connected by a hundred trillion synapses, forming more than a hundred neurologically distinguishable areas. We should not expect the problem of AI to be easy. Subproblems of cognition include attention, memory, association, abstraction, symbols, causality, subjunctivity, expectation, goals, actions, introspection, caching, and learning, to cite a non-exhaustive list. These features are not "emergent". They are complex functional adaptations, evolved systems with multiple components and sophisticated internal architectures,  whose functionality must be deliberately duplicated within an artificial mind. If done right, cognition can support the thoughts implementing abilities such as analysis, design, understanding, invention, self-awareness, and the other facets which together sum to an intelligent mind. …

seed AI also mistrusts that connectionist position which holds higher-level cognitive processes to be sacrosanct and opaque, off-limits to the human programmer, who is only allowed to fool around with neuron behaviors and training algorithms, and not the actual network patterns. Seed AI does prefer learned concepts to preprogrammed ones, since learned concepts are richer. Nonetheless, I think it's permissible, if risky, to preprogram concepts in order to bootstrap the AI to the point where it can learn. More to the point, it's okay to have an architecture where, even though the higher levels are stochastic or self-organizing or emergent or learned or whatever, the programmer can still see and modify what's going on. And it is necessary that the designer know what's happening on the higher levels, at least in general terms, because cognitive abilities are not emergent and do not happen by accident. Both classical AI and connectionist AI propose a kind of magic that avoids the difficulty of actually implementing the higher layers of cognition. Classical AI states that a LISP token named "goal" is a goal. Connectionist AI declares that it can all be done with neurons and training algorithms. Seed AI admits the necessity of confronting the problem directly.

(From another document)

The Singularity Institute seriously intends to build a true general intelligence, possessed of all the key subsystems of human intelligence, plus design features unique to Al. We do not hold that all the complex features of the human mind are "emergent", or that intelligence is the result of some simple architectural principle, or that general intelligence will appear if we simply add enough data or computing power. We are willing to do the work required to duplicate the massive complexity of human intelligence; to explore the functionality and behavior of each system and subsystem until we have a complete blueprint for a mind. … Our specific cognitive architecture and development plan forms our basis for answering questions such as "Will transhumans be friendly to humanity?" and "When will the Singularity occur?" At the Singularity Institute, we believe that the answer to the first question is "Yes" with respect to our proposed Al design - if we didn't believe that, the Singularity Institute would not exist. Our best guess for the timescale is that our final-stage Al will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.

And:

If I were to try quantifying the level of brainpower necessary, my guess is that it's around the 10,000:1 or 100,000:1 level. This doesn't mean that everyone with a 160 IQ or 1600 SAT will fit the job, nor that anyone without a 160 IQ or 1600 SAT is disqualified. Standardized tests don't necessarily do a very good job of directly measuring the kind of horsepower we're looking for. On the other hand, it isn't very likely that the person we're looking for will have a 120 IQ or a 1400 on the SAT.

To be blunt: If you're not brilliant, you are basically out of luck on being an AI programmer. You can't put in extra work to make up for being nonbrilliant; on this project the brilliant will be putting in extra work to make up for being human. You can't take more time to do what others do easily, and you can't have someone supervise you until you get it right, because if the simple things have to be hammered in, you will probably never learn the complex things at all. …

You should have read through [Levels of Organization in General Intelligence] and understood it fully. The AI theory we will actually be using is deeper and less humanlike than the theory found in LOGI, but LOGI will still help you prepare for encountering it.

The four major food groups for an AI programmer:

  • Cognitive science
  • Evolutionary psychology
  • Information theory
  • Computer programming

Then a massive list of subdomains that is is basically a grab-bag of insight porn we've been awash in for the last two decades, presumably cultivating sparks of AGI in lesswrong and /r/slatestarcodex regulars.

The three-layer model of intelligence is necessary, but not sufficient. Building an AI "with sensory modalities, concepts, and thoughts" is no guarantee of intelligence. The AI must have the right sensory modalities, the _right_concepts, and the right thoughts.

Unfortunately it seems like linear algebra is just about enough.

Wait, Yudkowsky doesn't believe in the scaling hypothesis? That's super interesting to me! Has he written about this?

In this specific interview Yud said something along the lines that of course scaling is capable of producing general intelligence - in the end evolution did that blindly with human brain so it should be possible. He was just more sceptical regarding gains by more compute. Needless to say, he is less sceptical now.

People will never believe this, but the general vibe when the GPT-3 paper came out was that it was a disappointment. I think all of the tools and tricks that made it useful were developed after initial release.

GPT-3 was something like an autistic-savant toddler.

Like, if you showed it a chessboard and how the pieces moved, it could get the general idea and 'play' the game, but it would make stupid or illegal moves, and clearly couldn't have a 'goal' in mind.

it could talk coherently but was horrible at conveying meaning.

GPT-4 appears to have jumped straight to autistic-savant teenager.

I am someone with little to no technical no-how but my intuitive sense having played around a little with all these models is that the leap from 3 to 4 didn't seem nearly as massive as some of last winter's hype would have suggested.

3 to 4, or 3.5 to 4?

Very difficult to tell. The only actual training metric is the average log probability of getting the next word correct, and in that metric the gap between GPT3 and GPT2 is larger than that between 4 and 3, but understanding how that metric maps onto our own intuitive notions of "performance" is really hard. And human perceptions of intelligence are really only sensitive to small changes around the human average, I think GPT2 was too dumb in an unusual way for people to really get a good sense of its capabilities.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats