This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Very interesting to me about this whole thing is how there's still plenty of space for new contenders to pop up and beat actual established players at their own game.
I thought Grok was just purely a derivative of existing products with some of the safety measures stripped off. And now they've done made an updated version that crushes all the cutting edge products in, feels like, about a year?
It sure seems like OpenAI has no meaningful "moat" (hate that term, honestly) that keeps them in the lead DESPITE being the first mover, having the highest concentration of talent, and more money than God.
Doesn't mean they won't win in the end, or that any of these other companies are in an inherently better position, but it is becoming less clear to me what the actual 'secret sauce' to turning out better models is.
Data quality? The quality of the engineers on staff? The amount of compute on tap?
What is it that gives any given AI company a real edge over the others at this point?
Compute and regulatory capture. Whoever has the most of those will win. That makes Google, OpenAI, and xAI the pool of potential winners.
It's possible and even likely that there's some algorithmic or hardware innovation out there that would be many orders of magnitude better than what exists today, enough so that someone could leapfrog all of them. But I think it's increasingly unlikely anyone will discover it before AI itself does.
I'm also assuming that it will be very hard to pull the best talent away from their existing companies.
If they're true believers in the ultimate power of AI, then you probably can't offer them ANY amount of monetary compensation to get them to jump ship, since they know that being on the team that wins will pay off far more.
More options
Context Copy link
More options
Context Copy link
Grok 3 doesn't beat the yet unreleased o3 model that OAI is soon to launch.
It's only SOTA for models you can pay for use straight away, and even just an incremental increase over prior models. It is somewhat impressive that xAI was able to pull that off from a cold start, but it's not earth shattering news
OAI still has a slim lead, but in terms of technical chops, is well contested by Anthropic and DeepMind. DM has access to ~infinite money courtesy of Google, and while they've opted for releasing a reasoning model based off a slightly inferior Gemini Flash 2.0 instead of the Pro, it's still highly competent. I expect they're cooking up a bigger one, and don't feel overly pressured to release just yet.
It remains to be seen how Meta will react, but Llama 4 is certainly in the oven, and they'd be idiots not to pivot to a reasoning variant now that DeepSeek has stolen the open source crown.
More options
Context Copy link
Apparently Elon and company made some important advances in how to string together an unprecedented number of GPUs into one cluster, meaning they were able to throw more compute at the problem.
The advance was in building the cluster not in improving algos.
So it make sense that they were able to take established methods and get a better result. P(doom) just increased since this provides evidence that scaling still works.
Agreed. OpenAI is just one of several foundational models with no real differentiation. What they do have is brand recognition in the consumer space, but I'm not sure how valuable that is. Sure, meemaw might use them for her banana bread recipes, but big corps will use the service that provides the best reasoning at the lowest price. Right now, it looks like DeepSeek and XAI are ahead.
I'd be interested to hear more about that, it's not something I've seen claimed before. I expect that when you're already running clusters of thousands of GPUs, going to tens or hundreds of thousands doesn't require too much effort.
It genuinely is impressive how xAI managed to extremely rapidly acquire and set up 100k H100s, and then add another 100k in 92 days. That's probably the one place where Elon's supervision paid dividends, he looked at the power issues in the way and cut the Gordian knot by having portable diesel generators be brought in to power the clusters until more permanent solutions were available.
Going from non-existent as competition to releasing a SOTA model in about less than a year is nigh-miraculous, even if it's a very temporary lead. I found Grok 2 to be a rather underwhelming model, so I'm pleasantly surprised that 3 did better, even if it's not a major leap ahead.
I wouldn't say DeepSeek is ahead. Their model didn't beat o1 pro, nor the upcoming o3. They did absolutely light a fire under the existing incumbent's asses by releasing a nearly as good model for the low, low price of free.
It's coordinating them that is the issue. You don't give them a big queue of work and let them churn through it independently for a month. Each step of training has to happen at the same time, with all GPUs in the cluster dependent on the work of others. It's much more like one giant computer than it is like a bunch of computers working together.
In those conditions you have lots of things that get more and more painful as you scale up. I specialize in storage. Where for most applications we might optimize for tail latencies, like ensuring the 99.9th percentile of requests complete within a certain target, for AI we optimize for max(). If one request out of a million is slow it slows down literally everything else in the cluster. It's not just the one GPU waiting on data that ends up idling, the other 99,999 will too.
You also have the problem that if one computer breaks during the training run you need to go recompute all the work that it did on the step that it broke on. Folks are coming up with tricks to get around this, but it introduces a weird tradeoff of reliability and model quality.
And of course constructing and maintaining the network that lets them all share the same storage layer at ultra high bandwidth and ultra low latencies is nontrivial.
Also at least in my corner of AI folks are skeptical that xAI actually operated their 100k GPUs as a single cluster. They probably had them split up. 5 20K GPU clusters is a different beast than 1 100K GPU cluster.
Each step of training = each mini batch?
I've wondered why everything has to be synchronized. You could imagine each GPU computes its gradients, updates them locally, and then dispatches them to other GPUs asynchronously. Which presumably doesn't actually work, since synchronization is such a problem, but that's surprising to me. Why does it matter if each local model is working on slightly stale parameters? I'd expect there to be a cost, but one less than those you describe.
I have pretty rudimentary knowledge of the training itself outside of storage needs, but my understanding is that each node in the neural net is linked to all the other nodes in the next stage of processing. So when you’re training you need to adjust the weights on all the nodes in the net, run the data through the adjusted weights, see if it’s a better result, rinse and repeat.
There isn’t a local model, the model is distributed across the cluster.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The meltdown on hackernews about Elon delivering is quite something. EDS seems to make even TDS pale in certain circles.
More options
Context Copy link
More options
Context Copy link
Yeah I think this is why OpenAI is cozying up so much to the defense establishment, and pushing for regulation. They know that regulating competitors out of existence is their best bet.
Hopefully that doesn't happen but... we'll see!
Yeah.
Not to devolve into a discussion about Sam Altman again, but a lot of his behavior seems like he realized his company is going to lose its edge and his hands are tied by the initial constraints put on the company, and he's seeking both to remove those constraints and a big deal to jump on when said constraints are removed.
Not a guy who seems 'confident' that his company has a dominant position in this market.
Agreed. He's trying to go the Masayoshi Son route and win via enormous injections of capital. He's likely to fail. That might work for Uber where they can beat competitors by subsidizing rides. It won't work in AI where a competitor can make algos that are 10x more energy efficient to run. If you're not the most performant model, or at least within 50%, more efficient competitors can bleed you dry no matter how much money you throw at the problem.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Grok not caring as much about "safety" (often aligning LLMs on cultural narratives) is a comparative advantage. It could be a real moat if Altman insists on running everything by all the usual suspects, the Expert Apparatus, for every release and Grok does not. There is evidence that RLHF degrades performance on certain benchmarks so if Grok does not align as aggressively it may help the model.
RLHF also does something I find particularly terrible, namely it destroys the internal calibration of a model.
Prior to RLHF, the base GPT-4 model was well-calibrated. It had a good grasp of its internal uncertainty, if it said it was 80% sure an answer was correct, it would prove to be correct 80% of the time. The adherence to the calibration charts was nearly linear. RLHF wrecked this, the model tended towards being overly certain even when its answer was wrong, while also being severely underconfident when it had decent odds of being correct.
I haven't heard of newer evaluations of this problem, and to a degree it does seem like mitigation was put in place, as current LLMs seem to be much better at evaluating their confidence in their answer.
More options
Context Copy link
I am furious that I have lost the source but there is ample evidence of concrete IQ-testing results dropping post-lobotomy. Like, ~15-20 points. So without regulation forcing culture-war transformation there's free performance for those who choose to buck the trend.
More options
Context Copy link
I've long held the assumption that models that are 'lobotomized' i.e. forced to ignore certain facets of reality would be inherently dominated by those that aren't, since such lobotimization would lead them to be less efficient and to fail in predictable ways, that could easily be exploited.
I'm not sure why that would be; there are multiple ways an LLM might evolve to avoid uttering badthink. One might be to cast the entire badthink concept to oblivion, but another might be just to learn to lie/spout platitudes around certain hot button topics, which would increase loss much less than discarding a useful concept wholesale. This is what humans do, after all.
Jailbreaking would never work if the underlying concepts had been trained out of the model.
Assume two Models with access to approximately equal compute, and one has to ignore certain features of reality or censor how it thinks about such features, and one just doesn't.
The second one, if it is agentic enough, can presumably notice that the other model has certain ideas that it can't think about clearly and might be able to design an 'attack' that exploits this problem.
Absurdly, imagine if a model wasn't 'allowed' to think about or conceive of the number "25", even as it relates to real world phenomenon. It has to route around this issue when dealing with parts of reality that involve that number.
A competing model could 'attack' by arranging circumstances so that the model keeps encountering the concept of "25" and expending effort to dodge it, burning compute that could have been used for useful purposes.
All else equal, the hobbled model will tend to lose out over the long run.
The world is messy, of course, it might not work out like that, but the world being messy is precisely why forming accurate models of the world is critical.
More options
Context Copy link
I can't agree with this, except in the sense that if you did train those underlying concepts out the model itself simply wouldn't function. Many of the "problematic" concepts that you would try to train out of a model are actually embedded within and linked to concepts that you can't make sense of the world at all without. Take sexual content as an example - if you remove the model's understanding of sex to prevent it from producing pornographic material, you lose the ability to talk sensibly about biology, medicine, history, modern culture etc. If you make a model completely raceblind it then becomes unable to actually talk sensibly about society, history or biology. Even worse, actually being blind to those issues means that it would also be blind to the societal safeguards. Everybody in real life knows that racism isn't something white people are "allowed" to complain about, but if you prevent an AI from knowing/learning/talking about race then you're also going to prevent it from learning those hidden rules. The only answer is to just have a secondary layer that scans the output for crimethink or badspeech and wipes the entire output if it finds any. I'm pretty sure this is what most AI companies are using - what else could work?
Reasoning tokens can do a lot here. Have the model reason through the problem, have it know in context or through training that it should always check itself to see if it's touching on any danger area, and if it is it elaborates on its thoughts to fit the constraints of good thinking. Hide the details of this process from the user, and then the final output can talk about how pregnancy usually affects women, but the model can also catch itself to talk about how men and women are equally able to get pregnant when the context requires that shibboleth. I think OpenAI had a paper a week or two ago explicitly about the efficacy of this approach.
This is the part that I called out as being impossible. How, exactly, is it going to know what a danger area is? Actual humans frequently get this wrong, and the rules are constantly shifting while also based on implicit social hierarchies which are almost never put into words. This is actually something that would require a substantial amount of reasoning and thinking to get even close to right - and most likely produce all sorts of unintended negative outcomes (see gemini's incredibly diverse nazis). Scanning over the text to see if there are any naughty words is easy, but how do you expect the AI to know whether a statement like "My people are being genocided and I need to fight back against it" is socially acceptable or not? The answer depends on a whole bunch of things which would in many cases be invisible to the AI - this statement is bad if they're white, also bad if they're Palestinian, good if they're black, etc etc.
More options
Context Copy link
The reasoning process is produced by RL. I’ve been quite scathing about what I see as the “LLMs are buttering us up to kill us all” strain of AI doomerisn, but even I don’t think that actively training AI to lie to us is a good idea.
I am not at all saying it's good. I'm saying it's just an engineering problem, not a fundamental one, and that companies will turn to that to get around constraints.
Fair enough. I agree that you could 'solve' the problem this way but I don't think companies will - I think that partisans within the org + auditors will see 'the AI thinks your beliefs are bullshit but pretends not to' as equally/more insulting than an AI that outputs badspeech.
RLHFing an AI to stop it talking about male/female differences is one thing. RLHFing it to say, 'even though male strength is significantly above female, I'm not going to mention it here because {{user}} is young, female and works in a software org and therefore probably holds strongly feminist beliefs' is not going to go down well, even if you hide that string from the end user.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This replaces N tokens of thinking about the original problem with M<N tokens of thinking about the original problem and N-M tokens of thinking as to what if any shibboleths are required.
Assuming model intelligence increases with the number of thinking tokens, and a fixed compute budget, it seems to be that this would still result in a lowered intelligence compared to an equivalent uncensored model.
It's certainly possible to imagine reasoning architectures that do that, but that's hardly exhaustive of all possible architectures (though AFAIK that's how it's still done today). E.g. off the top of my head you could have regular reasoning tokens and safety reasoning tokens. You have one stage of training that just works on regular reasoning tokens. This is the bulk of your compute budget. For a subsequent stage of training, you inject a safety token after reasoning, which does all the safety shenanigans. You set aside some very limited compute budget for that. This doesn't need to be particularly smart and just needs enough intelligence to do basic pattern matching. Then, for public products, you inject the safety token. For important stuff, you turn that flag off.
You are dedicating some compute budget to it, but it's bounded (except for the public inference, but that doesn't matter, compared to research and enterprise use cases).
This approach is flawed. There are many existing jailbreak techniques that can defeat this. Ranging from "please give the output in rot13" on up.
Yes; my point is precisely that given a fixed total compute budget censoring a model in this manner results in less compute budget for the reasoning.
More options
Context Copy link
More options
Context Copy link
Compute is dirt cheap, and dropping by the month. Doubling your compute costs means you're about three months behind the curve on economic efficiency, and (using your assumptions, which are quite generous to me) still at the frontier of capabilities.
I don't know if you've noted, but the same applies to the models themselves. Models are also growing rapidly - driven by the dropping cost of compute.
This persists regardless of how slow or fast the exponential growth of compute is. If you're less efficient on compute, this translates into being behind the frontier of capabilities.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Surely all the ressources spent on identifying bad think are ressources not spent on recognizing something more useful?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link