site banner
magic9mushroom  If you're going to downvote me, and nobody's already voiced your objection, please reply and tell me  3mo ago (thezvi.wordpress.com)   1801 thread views

Danger, AI Scientist, Danger

thezvi.wordpress.com

Zvi Mowshowitz reporting on an LLM exhibiting unprompted instrumental convergence. Figured this might be an update to some Mottizens.

9
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Diffusion models work for text too.

Last I checked, diffusion models work at all for text but they don't work very well. More specifically, text diffusion models remind me quite strongly of the classic-style Markov chain text generators that used to be popular for generating amusing locally-coherent-globally-word-salad text. Here's the best concrete example of this I can point to (italicized text is the prompt, normal text is continuation, examples sourced from this JDP tweet, whole tweet is excellent but somewhat OT here):

Diffusion model:

This whole dream seems to be part of someone else's experiment.

In the end, I think the only way to get the full benefit of the dream is to get up and walk around.

A lot of the other strands of the theory of dreams, especially the one called yncrest are known to the person who writes this blog. Consequently, I've tested his reactions via a series of brief, televised interviews with the British Prime Minister, Margaret Thatcher.

“I don't care what anybody says. I'm not going to waste my time with that. The study was conducted in the high-flow, low-flow and quiescent periods of a butterfly species and thus provides evidence for the first three of the four hypotheses listed in the original paper. The only one not covered is the "need to change the rate of capture", where we now know that this is largely a policy instrument - and particularly relevant to Sudanese geese.

GPT-2:

This whole dream seems to be part of someone else's experiment. This is usually hard to see, since you're conscious that mental shortcuts are normal. But you're definitely moments away from being normal again, as it's obvious that you are being coached to think like that by another person and doing our own thing. Why are we having such thoughts when we're paying no attention because that's already taking place?

I'm totally appalled by what these key players were doing, yet I explain it in a very large way. After 2 minutes, I make my way through high teaching circles, recognizing that there is more to learning than just to learn.

Few other teachers would air this incessantly their students' struggles recount the stories of their own memories and teachers'. Whatever it takes is the big fat slut shaming.

All these legendary trainees made the same point:

Cognitive functional aspects: Bayesian modeling, learn science way. And the most important part is: Like the coming of a hurricane, the book is mucking between science and morals.

Twitter Mentions of the first book: Kent

Now obviously in the limit as computational power and training data volume go to infinity, diffusion models and transformer models will generate the same text, since in the limit they're pulling from the same distribution with minimal error. But in the very finite regime we find ourselves in, diffusion models "spend" their accuracy on making the text locally coherent (so if you take a random 10 token sequence, it looks very typical of 10 token sequences within the training set), while transformer LLMs "spend" their accuracy on global coherence (so if you take two 10 token sequences a few hundred tokens apart in the same generated output, you would say that those two sequences look like they came from the same document in the training set).

The blending of concepts that we see in MidJourney is probably less to do with the diffusion per se as with CLIP

Agreed. Obvious once you point it out but I hadn't thought about it that way before, so thanks.

'Self play' is relevant for text generation. There is a substantial cottage industry in using LLMs to evaluate the output of LLMs and learn from the feedback.

Notably, Anthropic's Constitutional AI (i.e. the process by which Anthropic turned a base LLM into the "helpful, honest, harmless" Claude) process used RLAIF, which is self play by another name. And that's one big cottage.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats