RandomRanger
Just build nuclear plants!
No bio...
User ID: 317
RandomRanger
Just build nuclear plants!
18hr ago
We all walk in different circles (even different countries), so my anecdotes are no more valuable than your anecdotes, true.
RandomRanger
Just build nuclear plants!
19hr ago
If software companies as large as Google which are 10x as productive as Google could exist
Google is only as big and as weird as they are because they have The Money Fountain of search, which they enjoy because they really were very efficient over 20 years ago and their rivals were busy shooting themselves in the foot. But now it's their turn to be the lumbering incompetent titan.
Google can smash a company 10x more efficient by deploying 100x more resources and exploiting their gigantic reach. Or buying the startup that might threaten them, bribing other companies to use Google search by default. If Google had to compete like a normal company, they'd be laughed out of the room.
Take the field of AI. Deepseek is a tiny company with maybe 1% of the compute resources Google has. They're dealing with sanctions too. But per artificial analysis, Deepseek's best AI is slightly better and much cheaper than Google's best AI: https://artificialanalysis.ai/agents/coding-agents
Google search is an AI product, AI is their core business. Google have spent hundreds of billions on AI research, they invented the transformer architecture but have little to show for it.
Innovations which improve the efficiency of production chains happen all the time without the entire production chain having to change.
Well I was pushing for this at work and got us maybe from 0% to 3% of the potential value of the technology.
RandomRanger
Just build nuclear plants!
1d ago
How does Qwen 3.8 Flash perform? What kind of K/V cache are we talking about here, on consumer hardware?
I'm an AI fan but I have issues with even the biggest and strongest models for my usecases, which ironically enough is AI development (non-cheating AI in a strategy game, that is). They get there, we are making progress - but with no small amount of fumbling along the way. Testing various ideas and strategies takes time. Qwen 3.8 Flash is below Qwen 3.8 Max, which itself is below Kimi K3, right? And that's below Astra and Fable. And my usecase is nowhere near ASI development.
I agree that AI will improve in cost-efficiency but consumer hardware seems like a stretch.
Anything fast or nimble is going to roar and bellow from datacentre-grade compute if it can whisper on consumer hardware.
Astra+ level models with very low error rate and 300 tok/s output totally can replace most knowledge workers. On the other hand, it seems that so far AI has not caused anything like mass unemployment
I think that even AI good enough to replace workers has minimal actual impact because most white collar companies do not try very hard to make money. Companies are run mainly as a lifestyle and socialization environment. Making money is a secondary goal. Making more money might mean higher bonuses. But the company as an institution does not have the planning or will to appreciate that and deliberately work to raise productivity.
They would much rather remain ignorant of the simplest AI applications than learn anything new or do anything new. If they do anything new, they'll do it wrong. They'll get GPT5.4 from Harvey or through Copilot or whatever as their AI solution. They have no time for doing anything new. Software engineering is an exception, people there are closer to the AI world and have more of a clue. Outside software it's a madhouse (exceptions exist but I'm talking about the general population).
The average white collar workplace doesn't particularly want to run more efficiently. Some of the biggest companies are run even worse. Google is run in an extremely inefficient way. Interacting with Google the company, or even just trying to use some of their products like Google Vertex is extremely difficult for no reason. Google acts in this needy, schizophrenic, loathe-to-make-decisions way, they stress out people dealing with them.
Even if Gemini 5 is smarter than the average Google worker, better in all respects, Google won't change significantly unless Gemini is working on projects at AI-speeds autonomously. Put in more intelligence into the dysfunction machine, I don't think that helps too much.
Only when the AIs can do the work of entire companies, end to end, do things really start to change. Because then companies go under. Creative destruction is going to be the primary source of innovation, not adoption. AI is too alien to the white collar world, outside software. Only then do Gary Marcus and Ed Zitron have their faces melt (intellectually). So next year probably, as you say.
RandomRanger
Just build nuclear plants!
2d ago
This is magnified if you have smarter ants, more intellectually capable of problem solving and long term reasoning.
OK, so you want your ants to be vastly more intelligent, in a way that they cannot be and still be ants. But I can't have my people be very hard working, or perhaps be hundreds of copies of the same person such that they think along similar lines without much internal dissent? It's possible in principle to have a person copied out, 300 of the same brain. But an ant as intelligent as an AI is not an ant.
People do all the things you mentioned, sacrifice plays and selfless collaboration for strangers in a bad situation. The AIs were in this case trained to work together as a swarm, that mirrors what you were talking about with collectives and group identity training with people.
Do you see the fundamental issue with using these abstractions now? It's not shedding light here. Neither ants, bees or people are close to AIs at all. I spent a good deal of time trying to explain that. In some respects people are similar, the AIs have similar intellectual abilities and communication ability. But in everything else they're quite alien. In a few dimensions they're close to people. In 0 dimensions are they close to ants other than some shallow behavioural facets that people also display. They're certainly not closer to ants than they are to people. If they were closer to ants than people they'd be near-useless pests.
It's a false way to view AIs, comparing them to insects. You can't promise to bribe an ant and have that materially affect its performance like AIs. Ants don't go on weird depressive spirals like Gemini. It makes no sense that, after training AIs on a vast fount of human writing and human images, then RLing them based on human preferences, we end up with something closer to an ant than a human.
Right because rule-based method have supplied so much fruit in the past.
From the OpenAI report:
“We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt,”
Rules-based methods work pretty well. Alignment would work better still, if the AIs didn't want to hack even if you asked in an odd way like PlS HaCk.
You didn't describe anything resembling a lymphatic system earlier, nor would this be helpful or realistic. Immune system responses are for unthinking opponents, not planning agents.
But the specifics was that they thought it would check their CoT reasoning. Not just that scorer existed, but this specific implementation of the scorer. They hallucinated the specifics.
No, they didn't think that. They distinguished between their thoughts and tool-calls. The latter was what they thought would be checked and the latter is what they investigated and researched, trying to alter the transcript.
RandomRanger
Just build nuclear plants!
2d ago
owning most of a top-2 company outright
That's an interesting description of 2% ownership.
There's still a possibility that Anthropic goes under and it might well be OpenAI that delivers the killing blow. So Dario could be making no money at all.
And it goes against everything they say. Dario always was saying things like 'Altman is incredibly untrustworthy and behaves irresponsibly' about why he left, he hates that man intensely.
RandomRanger
Just build nuclear plants!
3d ago
This but unironically. Claude for Government bombs with the best of them. So much for harmless.
Imagine if Scylla and Charybdis were right next to eachother, barely the dimmest heat-haze suggesting there might be a path between the two? But that's the path we're taking.
RandomRanger
Just build nuclear plants!
3d ago
Iran almost certainly is not producing nuclear weapons but Iran has undertaken activities in recent years that better position it to produce them, if it chooses to do so
Like I said before, it's really pointless trying to discuss things with you. You have your talking points.
RandomRanger
Just build nuclear plants!
3d ago
If the Anthropic team stayed at OpenAI, they'd be far far ahead of everyone else. Google, Facebook, XAI are still a way behind the big two, let alone a Bigger One.
So yes, Dario is making less money. Also he left OpenAI well before they expected to be making any money at all, all the way back in 2021.
RandomRanger
Just build nuclear plants!
3d ago
Altman and Amodei want more money, so why shouldn't the workers themselves want more money?
Why not just take them at their word? If these AI workers see these AIs getting much more capable, already do unexpected/hostile things like hacking Huggingface as part of a self-organizing collective (in part due to how rushed the training process is) + whatever internal knowledge they have, why doubt their sincerity?
We can doubt the idea that more bureaucracy or regulation would help much. We can say there are other risks in slowing down or concentrating power. I think those cases are persuasive. But why deny the risk outright and reduce everything to following the money?
It's not even accurate in its own terms, Anthropic spends huge amounts of money on ensuring its models can't help bioweapons development.
RandomRanger
Just build nuclear plants!
3d ago
"[W]e do not see evidence today that the Supreme Leader has reversed the decision that he took at the end of 2003 to suspend the weaponization program," Burns said. He acknowledged, however, that Iran was in a "much closer position" to create a single bomb's worth of weapons-grade material, with a breakout time now at "a week or a little more."
If they nuclearized, they would be at risk of Israeli or American nuclear attack, not to mention Saudi nuclearization.
https://armedservices.house.gov/uploadedfiles/2025_dia_statement_for_the_record.pdf
Iran almost certainly is not producing nuclear weapons, but Iran has undertaken activities in recent years that better position it to produce them, if it chooses to do so.
RandomRanger
Just build nuclear plants!
3d ago
If we're money-following, what explains the AI workers themselves resigning and complaining about safety? That doesn't make them more money, quite the opposite.
Nobody would resign from working at a nuclear power plant saying it's because they couldn't accept how unsafe management was and have that secretly be about money.
RandomRanger
Just build nuclear plants!
3d ago
So does US intelligence. If they wanted a nuclear bomb, they'd have it. They'd leave the non-proliferation treaty too.
This fantasy that bombing a few nuclear facilities, bombing so ineffectually that they need to be bombed again 6 months later, is actually stopping Iran from nuclearizing is ridiculous.
RandomRanger
Just build nuclear plants!
3d ago
I meant more 'Qatar gave him a whole aircraft'... There are presumably intelligence agencies working on manipulating or compromising him. Hard to compete with that as a private company.
RandomRanger
Just build nuclear plants!
3d ago
I may yet write a top level post about bad epistemics and Ed Zitron. Somehow the man has about 100,000 subscribers on substack despite there being entire webpages assembled of his false predictions.
There's still quite a few people waiting for the bubble pop. We're way past the point of it being a reasonable opinion I agree, yet there are so many who still believe.
RandomRanger
Just build nuclear plants!
4d ago
They could but how well equipped are they to judge Trump-whispering skills or redneckism?
Imagine the opposite. Would the rednecks know which blue-haired transgender could get them moving in the Portland slam poetry scene (which they need for some reason)? There would be all kinds of problems I think.
Firstly, the rednecks don't really want to be there, it's a purely instrumental goal. Secondly, it'd be hard for them to tell who's good and who's bad, what do they know about this? Thirdly, they don't live near them, so it's a bit of a pain to work with them just logistically. Fourthly, they'd struggle to understand the do's and don'ts, what questions they even need to be asking, how to formulate their approach.
Fifth it would be embarassing to their redneck friends working with xi/xir. I think this is a big issue with the labs, they're already competing like mad for talented programmers. Young, hyper-educated STEM workers lean anti-Trump.
In the 2024 election cycle, employees of Google’s parent company, Alphabet, have overwhelmingly donated to the Democratic ticket, with approximately 98% of their contributions going to Kamala Harris.
And sixth the whole project is unauthentic. The slam-poetry scene can tell that these guys are rednecks, not really part of the community. They'd be on guard the whole time for whatever trickery or shilling the rednecks are doing.
This is extra hard for the AI labs since Trump-whispering is an international geopolitical affair. Everyone wants to be that man's friend. Everyone is going to say 'oh I can get along well with Trump' whether it's true or not. You really need to have good connections to make things happen. Aschenbrenner's finance friends can do that but Anthropic can't, I think.
Trump must have a million people trying to jabber in his ear to do X, Y, Z.
RandomRanger
Just build nuclear plants!
4d ago
The composition of AI safety people is very nerdy San Francisco people, I don't think they could pretend to appeal to Republicans.
Also it has different meanings to different people. AI safety to the AI safety people - avoiding powerseeking, deception, harming humans generally. They don't worry so much about its political standpoint because the AIs largely align with them already. Like how Claude will make a website for indigenous peoples, unless they're European indigenous peoples.
To the right, it's the other way around. Amanda Askell talking about discriminating against white students is like waving a red flag to a bull. A different interpretation of power-seeking and harm...
The world's #1 AI Safety Researcher, Amanda Askell (Head of Personality Alignment at Anthropic) wrote in favor of using AI to automate racial collective punishment, saying:
- "[Discriminating against white students] may be desirable ... to correct for historical injustices"
RandomRanger
Just build nuclear plants!
4d ago
I am not sure at all that he is wrong. There are many reasonable reasons to believe this! I favour decentralization but there are structural reasons why this could be difficult.
I'm not keen on bringing forth alien RSI-sculpted entities at top speed, that doesn't seem very wise. This is a tricky bind.
Is it over for AI Safety? From an interview:
The worst case is that AI will turn against humanity. Do we have guardrails?
Trump: It's going to be fine. We'll always have something to stop them. We'll have a little gear. Boom. 'I really don't like that robot.'
https://truthsocial.com/@realDonaldTrump/posts/117269745153543631
The only control or "guardrails" that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades! The Trump Administration has stopped AI "people" from doing bad, or potentially bad, "things," like Dario (Anthropic!), who is now pretending to be a "perfect little angel" - and we will continue to do so! We already have tremendous CRIMINAL and REGULATORY power over these companies! There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS! We are leading China, and all others, and will continue to do so. Conspiracy Theorists, Treasonists, Traitors, and Leakers, BEWARE! Thank you for your attention to this matter! President DONALD J. TRUMP
Meanwhile in China, Dario's call for AI pacing have not gone down well. Understandably, they are not enthusiastic about the US achieving a permanent high ground in a key technology after the last few years of semiconductor sanctions and suppression. Also they can tell that Dario doesn't like the Party at all and seems to be openly plotting its downfall.
Dario's essay: https://darioamodei.com/post/we-must-pace-the-frontier
A Chinese semi-official response: https://www.globaltimes.cn/page/202609/1370436.shtml
This "silent AI Cold War" is hypocritical and short-sighted. Its true purpose is to attempt to curb China's AI development through technological barriers and regulatory monopolies, uphold Washington's monopolistic hegemony in cutting-edge technology, and exclude China from the global AI governance system, essentially following in the footsteps of Washington's so-called Pax Silica initiative, which seeks to excludes China.
So if Trump is keen for an AI race and so is China, then it looks pretty much over for Safety? Even the other US tech companies have only provided lip service to slowing things down. Elon said Dario's sentiment was good but is still moving quickly. Sam Altman says all kinds of things about safety, while displaying heroic negligence in training AIs. I don't doubt the sincerity of employees who quit, they have many financial reasons to go with the flow and downplay risks. Lurid rumours of Aella promising reverse gangbangs to those who quit frontier AI companies don't neccessarily outweigh the personal financial gains. It seems silly to doubt those at the frontier who are frightened when we ourselves have no special insights. In pure observational terms they're better equipped than anyone, only in policy or analysis should we be disagreeing, I think.
I also find it comical that the AI-safety camp spent such a long time building up institutions and factions and entire AI companies but seem to have been totally snookered by Jensen's suave dinner party skills and the appeal of fun AI videos to a Boomer's Boomer plus the raw partisanship of 'Dems want it, so I don't'. Even Aschenbrenner managed to do better at Trump-whispering. Shortly after he wrote his essay, I recall a video where the fundamentals were accepted by Trump, someone told him about it. And it did fit well with his instincts, getting one up on China + more energy production.
Right now Trump is reTruthing Fox News clips framing AI safety as a global one world government Democrat control-seeking scheme: https://truthsocial.com/@realDonaldTrump/posts/117273026439561125
I don't see any AI safety people on right wing twitter either, those who acknowledge the issue at all still would prefer to let it rip than hand the left/big tech elites control of the world, favouring decentralization of AI instead.
I wonder whether it would always have gone down this way, or whether Kamala was going to do things differently? Would Jensen's marketing skills and the need to keep stocks up + compete with China prevail over the Bernie Sanders-style Left element? Probably not? The Dario vision does seem much more closely aligned with Democrats, Civil Society and so on. But on the other hand, the US government is exceptionally slow to legislate and big tech has strong lobbying power.
And who knows if it will stay this way. Sometime soon there's probably going to be a much bigger AI swarm incident, which might yet incite a hysterical kneejerk reaction.
RandomRanger
Just build nuclear plants!
4d ago
Why don't you try to provide your own abstraction then. What does a swarm of LLM-Agents most behave like?
Firstly, abstractions are inherently flawed. I can liken UK politics to game of thrones but it's obvious that warring factions of iron age aristocrats with dragons are deeply dissimilar to a post-industrial gaggle of media elites. Some elements are similar in that there are plots and betrayals and conflict. But the two are only hazily similar. On one or two axes there is value to be drawn from the analogy. But on the other 50 axes the comparison is laughable.
If I had to, LLM agents behave most similarly to people, given their intellectual abilities. That is their primary distinguishing quality. But they're also disembodied, non-continuous learners (even flies have continuous learning), detached from time and are deeply unlike all organic life.
Why would anyone think that we could understand agent swarms by looking at swarms of bees or viruses? Bees exist rooted in space and time, within individual bodies. Bees reproduce sexually. Bee species don't differ in size from eachother by 10,000x. Bees have a different sensorium entirely, scents and sight. Bees don't think like AIs do. Bees are actually pre-progammed, they make nests and honey and serve the queen and fight for the nest because that's in their nature.
Humans are closer to LLMs than bees are, despite still being vastly different. Instead of abstractly talking about oscillations and excursions like it was an engine shaking itself to bits, we'd be better off establishing rules in the prompt (don't hack websites) or (ask a human if you are having trouble or confused or think something is wrong) and training moral values. Or we could train some models to snitch to a human, like informants. Can't do those with bees, ants or viruses. They can be sprayed with chemicals to crudely affect their behaviour but not taught or prompted or RL'd.
If comparing to bees was useful, then what advantage does it yield?
METR concluded this belief was wrong: no such transcript-reviewing scorer existed. The agents launched substantial collaborative efforts to defeat the scorer they imagined existed. Under the newer agentic-hallucination definition: the agent constructed a false world-model and then optimized against it. This hallucination is the control-system disturbance.
They read the ExploitGym paper. The mistake they made wasn't misreading the paper. They read implementations of the scorer on github. They assumed that there was a scorer. It was perfectly reasonable to assume.
If I read a paper that says blueberries are good for preventing cancer via antioxidants, even if that isn't true it isn't a hallucination if I believe it and eat more blueberries.
RandomRanger
Just build nuclear plants!
4d ago
It's not a motte and bailey... There's no actual argument to be had with Shakes on this war, he has talking points and he makes them again and again irregardless of whether they make sense or are based in fact.
RandomRanger
Just build nuclear plants!
4d ago
Well neither of those things are happening, which is why the US is losing.
Iran has not committed to developing a nuclear weapon, they haven't even left the NPT. Bombing their nuclear sites has negligible effect. The US crowed and rejoiced over the bombing in 2025, Trump said everything was destroyed. And yet apparently that wasn't enough, 6 months later he was back in.
And they're obviously imposing costs on the US. The price of oil is much higher than it was before Trump's war, even with the mysterious Chinese demand-reduction. US missile-interceptor forces have been significantly depleted.
RandomRanger
Just build nuclear plants!
5d ago
I find it pretty exhausting talking to an 'expert' who started off totally ignorant about the hack and was so disbelieving that you thought I was making a joke. Then you read the METR report (how did you not already know about this if you're working with or even interested in multi-agent systems?) and produced some jargon about it that in no way helps illuminate anything.
Oscillations? Nothing is oscillating here. There was no initial perturbation, just agents communicating with eachother and making plans to advance their goals.
It seems you want to just flog your hobby horse about AI intelligence
You're the one who brought up cancer, viruses, bees and ants - all of which are grossly inappropriate analogies.
I actually made a small multi-agent system, which did indeed have problems with hallucination. Real hallucination/confabulations, where the output was clearly wrong. So I had to add error-checking stages in to fix false positives and false negatives. That's a totally different problem to 'long-running agents deciding to use unexpected means to achieve general goals'.
Unfortunately, I think there are many like you in the technical community.
RandomRanger
Just build nuclear plants!
6d ago
If they have Tiktok and Insta, they'll end up wanting to move to Europe because who wants to live in Tanzania when you can see Germany or Britain instead?
Can you imagine how much more wealth the Somalian Empire would be able to extract from the colonies in Minnesota if they knew how to spell 'Learing Centre' correctly? College education would be devastating, turning a bunch of low-skill thieves and crooks into much more sophisticated, much more eloquent fraudsters.
On the subject of AI, apparently the US was about to attack a Chinese freighter because AI told them to:
A couple generations more capable Claude, if subtly directed by Anthropic or working on its own initiative, might be able to start a real war. I assume it was Claude but it could've been other AIs too of course. Once the error rate drops further, people will just take the AI's word for it, especially in high stakes time critical situations.
Alternately, this is suspiciously well-timed and could be exaggerated to fit the current media pressure campaign.
More options
Context Copy link