This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
I'm actually quite skeptical that there is anything that can be meaningfully described as a thought process or reasoning going on when an LLM responds to a problem like this. It may well be that if an LLM produces a step-by-step summary of how to go about answering a question, it then produces a better answer to that question, but I don't understand how you can draw any conclusions about the LLM's 'reasoning', to the extent that such a thing even exists, from that summary.
Or, well, I presume that the point of the CoT summary is to give a indicative look at the process by which the LLM developed a different piece of content. Let's set aside words like 'thought' or 'reasoning' entirely and just talk about systems and processes. My confusion is that I don't see any mechanism by which the CoT summary would correspond to the subsequent process.
It seems to me that what the paper does is ask the LLM to produce a step-by-step set of instructions, and then ask the LLM to iterate on those instructions. LLMs can do that, and obviously if you change the set of instructions, the iteration on the instructions is different. That's perfectly intuitive. But how does any of that correspond to, well, the idea of thoughts in the LLM's mind? Or the process by which it produces text? How is that different to the rather banal observation that if you change the input, you change the output?
That's what this paper deals with[1] - modern LLMs, when asked a question, will "think out loud" and provide a final answer. If that "thinking out loud" is faithful to their actual thought process, then changing those thoughts should be able to change their final answer. So what the researchers did is they asked an LLM a question like
The LLM then "thinks out loud" to generate an answer
The researchers then modify the reasoning and feed the input with altered reasoning back into the LLM to complete to see if the final answer changes, so e.g.
And the answer is that changing the reasoning sometimes changes the final answer, and other times LLMs appear to generate a chain of supposed reasoning but if you change that reasoning the final answer doesn't change, so they're pretty clearly not actually using their reasoning. Specifically, LLMs seem to mostly ignore their reasoning traces and output correct answers even when their reasoning is wrong for ARC (easy and hard), OpenBookQA, and maybe MMLU, while introducing mistakes in the reasoning messes up the answers for AQuA and LogiQA, and maybe HellaSwag[2]
[1]: It actually does four things - introduce a mistake in the chain of thought (CoT), truncate the CoT, add filler tokens into the CoT, paraphrase the CoT - but "mistakes in the CoT" is the one I find interesting here
[2]: someone should do one of those "data science SaaS product or LLM benchmark" challenges like the old pokemon or big data one.
More options
Context Copy link
More options
Context Copy link