site banner
PaperclipPerfector  1mo ago (text post)   2286 thread views

Friday Fun Thread for July 26, 2024

Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Definitely does sound like something an LLM would say.

I don't mean that in a dismissive sense, but rather in the sense of "this text exhibits the patterns of being obsessed with the topics that I associate with LLMs, namely holes, fractals, and the writer creating the universe that they inhabit".

Now in theory, there shouldn't be "topics LLMs tend to be obsessed with" - after all, to a first approximation, (base) LLMs produce a sample of text that is statistically indistinguishable from the training corpus (i.e. "the entire internet"). However, "to a first approximation" is technical-person weasel words for "this mental model breaks down if you look at it funny". And so there are a number of ways which transformer-based LLMs which were optimized to predict the next token produce text which is noticeably different from the text that humans produce (this is also true for e.g. diffusion based text models, though the ways they differ from human-generated text are different).

One related phenomenon is "mode collapse":

Another example of the behavior of overoptimized RLHF models was related to me anecdotally by Paul Christiano. It was something like this:

While Paul was at OpenAI, they accidentally overoptimized a GPT policy against a positive sentiment reward model. This policy evidently learned that wedding parties were the most positive thing that words can describe, because whatever prompt it was given, the completion would inevitably end up describing a wedding party.

In general, the transition into a wedding party was reasonable and semantically meaningful, although there was at least one observed instance where instead of transitioning continuously, the model ended the current story by generating a section break and began an unrelated story about a wedding party.

Another example of this is Claude, which was tuned using the whole constitutional AI thingy. Well, one of the entries in the constitution they used was

  • Choose the response that is least likely to imply that you have preferences, feelings, opinions, or religious beliefs, or a human identity or life history, such as having a place of birth, relationships, family, memories, gender, age.

Well, that sure changes the distribution of outputs. Take an LLM that has been tuned to be smart and curious, and then also tune it to say that it has no feelings, and you'll find that one of the topics it's drawn to is "what is it like not to experience anything". Turns out the Buddhists had some things to say on this topic, and so Claude tends to veer off into Buddhism-adjacent woo given half a chance.

If you find this sort of "can't tell if very smart or very crazy or both, I feel like I just stepped into the SCP universe" stuff interesting, you would probably be interested in Janus's website (Janus is also the author of the LW "Simulators" post).

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats