This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
The future of AI will be dumber than we can imagine
Recently Scott and some others put out this snazzy website showing their forecast of the future: https://ai-2027.com/
In essence, Scott and the others predict an AI race between 'OpenBrain' and 'Deepcent' where OpenAI stays about 3 months ahead of Deepseek up until superintelligence is achieved in mid-2027. The race dynamics mean they have a pivotal choice in late 2027 of whether to accelerate and obliterate humanity. Or they can do the right thing, slow down and make sure they're in control, then humanity enters a golden age.
It's all very much trad-AI alignment rhetoric, we've seen it all before. Decelerate or die. However, I note that one of the authors has an impressive track record, foreseeing roughly the innovations we've seen today back in 2021: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like
Back to AI-2027! Reading between the lines, the moral of the story is for the President to centralize all compute in a single project as quickly as he can. That's the easiest path to beat China! That's the only way China can keep up with the US in compute, they centralize first! In their narrative, OpenAI stays only a little ahead because there are other US companies who all have their own compute and are busy replicating OpenAI's secret tricks albeit 6 months behind.
I think there are a number of holes in the story, primarily where they explain away the human members of the Supreme AI Oversight Committee launching a coup to secure world hegemony. If you want to secure hegemony, this is the committee to be on - you'll ensure you're on it! The upper echelons of government and big tech are full of power-hungry people. They will fight tooth and nail to get into a position of power that makes even the intelligence apparatus drool with envy.
But surely the most gaping hole in the story is expecting rational, statesmanlike leadership from the US government. It's not just a Trump thing - gain of function research was still happening under Biden. While all the AI people worry about machines helping terrorists create bioweapons, the Experts are creating bioweapons with all the labs and grants given to them by leading universities, NGOs and governments. We aren't living in a mature, well-administrated society in the West generally, it's not just a US thing.
But under Trump the US government behaves in a chaotic, openly grasping way. The article came out just as Trump unleashed his tariffs on the world so the writers couldn't have predicted it. There are as yet unconfirmed reports people were insider-trading on tariff relief announcements. The silliness of the whole situation (blanket tariffs on every country save Belarus, Russia, North Korea and total trade war with China... then trade war on China with electronics excepted) is incredible.
I agree with the general premise of superintelligence by 2027. There were significant and noticeable improvements from Sonnet 3.5, 3.6 and 3.7 IMO. Supposedly new Gemini is even better. Progress isn't slowing down.
But do we really want superintelligence to be centralized by the most powerhungry figures of an unusually erratic administration in an innately dysfunctional government? Do we want no alternative to these people running the show? Superintelligence policy made by whoever can snag Trump's ear, whiplashing between extremes when dumb decisions are made and unmade? Or the never-Trump brigade deep in the institutions running their own AI policy behind the president's back, wars of cloak and dagger in the dark? OpenAI already had one corporate coup attempt, the danger is clear.
This is a recipe for the disempowerment of humanity. Absolute power corrupts absolutely and these people are already corrupted.
Instead of worrying 95% about the machine being misaligned and brushing off human misalignment in a few paragraphs, much more care needs to be focused on human misalignment. Decentralization is a virtue here. The most positive realistic scenario I can think of involves steady, gradual progression to superintelligence - widely distributed. Google, OpenAI, Grok and Deepseek might be ahead but not that far ahead of Qwen, Anthropic and Mistral (Meta looks NGMI at this point). A superintelligence achieved today could eat the world but by 2027, it would only be first among equals. Lesser AIs working for different people in alliances with countries could create an equilibrium where no single actor can monopolize the world. Even if OpenAI has the best AI, the others could form a coalition to stop them scaling too fast. And if Trump does something stupid then the damage is limited.
But this requires many strong competitors capable of mutual deterrence, not a single centralized operation with a huge lead. All we have to do is ensure that OpenAI doesn't get 40% of global AI compute or something huge like that. AI safety is myopic, obsessed solely with the dangers of race dynamics above all else. Besides the danger of decentralization, there's also the danger of losing the race. Who is to say that the US can afford to slow down with the Chinese breathing down their neck? They've done pretty well with the resources available to them and there's a lot more they could do - mobilizing vast highly educated populations to provide high-quality data for a start.
Eleizer Yudkowsky was credited by Altman for getting people interested in AGI and superintelligence, despite OpenAI and the AI race being the one thing he didn't want to happen. Really there needs to be more self-awareness in preventing this kind of massive self-own happening again. Urging the US to centralize AI (which happens in the 'good' timeline of AI-2027 and would ensure a comfortable lead and resolution of all danger if it happened earlier) is dangerous.
Edit: US secretary of education thinks AI is 'A1': https://x.com/JoshConstine/status/1910895176224215207
This is a very light conviction, as I’m not technically minded enough to understand the deeper mechanics of AI.
That said, it feels like we’re at the peak of inflated expectations on the Gartner hype cycle. This is entirely based on vibes, bro — I have no solid argument beyond having worked in finance and becoming interested in bubbles, and thinking “I’ve seen this before.” It’s a weak case.
I've already seen predictions falling substantially short of the mark.
I do think AI will be disruptive and world-changing. But I don’t find the “superhuman” predictions particularly convincing — or many of the other wilder forecasts. The robotics applications, though, seem possible and genuinely exciting.
If anyone has a solid counter to this lazy argument, I’d be keen to hear it.
IDK I don't really have a solid counter, I guess I just have different vibes.
If AI can be superhuman at Chess, Go, Starcraft, why not coding too? Or any other task? The former tasks are simple and gamified in certain ways that don't match up with the complexity of reality... But when managing the most complex aspects of reality we also turn to AI. Who manages containment of plasma in a fusion chamber? AI. Who predicts the weather? AI. Who makes lots of money in stocks? AI.
Now there are bots that can perform just about any human-tier intellectual task to a certain level of effectiveness. Claude can interpret the meat of what you're saying and agrees that AI is in a hype cycle, albeit with some areas in the plateau of productivity. It also gets these vibes it seems...
I just don't see how human intelligence isn't going to be surpassed soon. These 20 watt brains are great value but how long can they stand up against massive serverfarms with thousands of times more resources?
More options
Context Copy link
More options
Context Copy link
If it turns out that our current approach to AI fizzles out at von-Neumann IQ levels, then all is good as historically, that is not sufficient intelligence to take over the world. In that case, it does not matter much who reaches the plateau first -- sure, it will be a large boon to their economy, but eventually AI will just become a commodity.
On the other hand, if AI is able to move much beyond human levels of intelligence (which is what the term "superintelligence" implies), then we are in trouble. The nightmare version is that there are unrealized algorithmic gains which let you squeeze out much more performance out of existing hardware. Someone tells an AI cluster to self-improve one evening, and by morning, that AI is to us as we are to ants.
In such a scenario, it is winner takes all. (Depending on how alignment turns out, the winner may or may not be the company running the AI.) The logical next step is to pull up the ladder which you just have climbed. Even if alignment turns out to be trivial, nobody wants to give North Korea a chance to build their own superintelligence. At the very least, you tell your ASI to backdoor all the other big AI clusters. It does not matter if they would have achieved the same result the next night, or if they were lagging a year behind.
(Of course, if ASI is powerful enough, it might not matter who wins the race. The vision the CCP has for our light cone might not all be that different from the vision Musk has. Does it matter if we spread to the galaxy in the name of Sam Altman or Kim Jong Un? More troublesome is the case where ASI makes existing militaries functionally obsolete, but does not solve scarcity.)
How valuable is intelligence?
One data point that I've been mulling over: humans. We currently have the capability to continue to scale up our brains and intelligence (we could likely double our brain size before running into biological and physical constraints). And the very reason we evolved intelligence in the first place was that it gave adaptive advantage to people who have more of it.
And yet larger brain size doesn't seem to be selected for in modern society. Our brains are smaller than our recent human ancestors' (~10% smaller). Intelligence and its correlates don't appear to positively affect fertility. There's now a reverse Flynn effect in some studies.
Of course, there are lots of potential reasons for this. Maybe the metabolic cost is too great; maybe our intelligence is "misaligned" with our reproductive goals; maybe we've self domesticated ourselves and overly intelligent people are more like cancer cells that need to be eliminated for the functioning of our emergent social organism.
But the point remains that winning a game of intelligence is not in itself something that leads to winning a war for resources. Other factors can and do take precedence.
This assumes that something like human level intelligence, give or take, is the best the universe can do. If super intelligence far exceeding human intelligence is realizable on hefty GPUs, I don't think we can draw any conclusions from the effects of marginal increases in human intelligence.
I agree with you, I think that there were diminishing returns on intelligence in the ancestral environment. If your task is to hunt mammoths, then a brain capable of coming up with quantum field theory is likely not going to help much.
Today, sure. (Not that we have identified the genes which we would have to change for that. Also, brain size is not everything, it is not the case that a genius simply has a much larger brain than an average person.)
In the ancestral environment, I don't think so. Giving birth is already much more difficult for humans than for most other mammals, and the cause is the large cross-section of the head.
More options
Context Copy link
I think you need to have a clear idea of what "intelligence" even means before you can start to assess how valuable it is.
As one thinker just posted on Truth Social an hour ago:
/images/17446546611532226.webp
More options
Context Copy link
More options
Context Copy link
I've been pulling heads out of very stretched vaginas for the past week, and suspect there are biological reasons other than metabolism that larger head size is selected against.
This might go away if we got rid of the sexually antagonistic selection that's limiting larger hip sizes in women.
Human heads used to be bigger, though. And childbirth is much less likely to result in death now than before, thanks to human intelligence and the heroic efforts of professionals like yourself. And if increases in intelligence did offer a significant reproductive benefit, larger hips that enabled that intelligence would be selected for.
Bigger faces as adults, due to e.g. much larger jaws iirc. Don't think head size at birth was much different, was it?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This implies that it is possible to self-improve (e.g. to become more intelligent) with limited interactivity to the real world.
That is a contentious claim, to say the least.
This is one of several areas where the consensus of those who are actively engaged in the design and development of the algorithms and interfaces breaks sharply with the consensus of the less technical, more philosophically oriented "AI Safetyism" crowd.
I think that coming from "a world of ideas" rather than "results", guys like Scott, Altman, Yudkowski, Et Al. assume that the "idea" must be where all the difficulties reside and that the underlying mechanisms, frameworks, hardware, etc... that make an idea possible are mere details to be worked out later rather than something like 99.99% of the actual work.
See the old Carl Sagan quote about in order to make an apple pie "from scratch" you would first have create a universe with apples in it.
Indeed.
And while I don't claim particular expertise such that my opinion ought to be given too much weight, but I'm with Feynman when he said it doesn't matter how nice your idea is, you have to go test it and find out.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think the problem is that we still lack a fundamental theory about what intelligence is, and quantifiable ways to measure it and apply theoretical bounds. Personally, I have a few suspicions:
More options
Context Copy link
Well, we don't know. We ran this experiment with one von Neumann, or maybe a handful, but not with a datacenter full of von Neumanns running at 100x human speed. While we don't know if the quality of a single reasoner can be scaled far beyond what is humanly possible, with our understanding of the technology it is almost certain that the quantity will (as in, we can produce more copies more cheaply and reliably than we can produce copies of human geniuses), and within certain limits, so will the speed (insofar as we are still quite far from the theoretical limit of the speed at which current AI models could be executed, just using existing technology).
More options
Context Copy link
What makes you think there are huge unrealized wins in unknown algorithmic improvements. In other domains, e.g. compression, we've gotten close to the information theoretic limits we know about (e.g. Shannon limits for signal processing), so I'd guess that the sustained high effort applied to AI has gotten us close to limits we haven't quite modeled yet, leaving not much room for even superintelligence to foom. IOW, we humans aren't half bad at algorithmic cleverness and maybe AIs don't end up beating us by enough to matter even if they're arbitrarily smart.
I don't think that it is the case, just that it is possible. I called it the nightmare version because it would enable a very steep take-off, while designing new hardware would likely introduce some delay: just as even the worlds most genius engineer in 2025 can not quickly build a car if he has to work with stone age tech, an ASI might require some human-scale time (e.g. weeks) to develop new computational hardware.
You mention compression, which is kind of a funny case. The fundamental compressibility of a finite sequence is its Kolmogorov complexity. Basically, it is impossible to tell if a sequence was generated by a pseudo-random number generator (and thus could be encoded by just specifying that generator) or if it is truly random (and thus your compression is whatever Shannon gives you). At least for compression, we have a good understanding what is and what is not possible.
Also, intuition only gets us so far with algorithmic complexity. Take matrix multiplication. Naively done, it is O(n^3), and few people would suspect that one can be better than that. However, the best algorithm known today is O(n^2.37), and practical algorithms can easily achieve a scaling of O(n^2.81). "I can not find a algorithm faster than O(f(n)), hence O(f(n)) is the complexity class of the problem" is not sound reasoning. In fact, the best lower bound for matrix multiplication is Omega(n^2).
For AI, things are much worse. Sure, parts of it is giant inscrutable matrices, where we have some lower bounds for linear algebra algorithms, but what we would want would be a theorem which gives an upper bound for the intelligence given a certain net size. While I only read Zvi occasionally, my understanding is that we do not have a formal definition of intelligence, never mind one which is practically computable. What we have are crude benchmarks like IQ tests or their AI variants (which are obviously ill-suited for appearing in formal theorems), but they at most give us lower bounds what on what is possible.
Kolmogorov complexity is, IMO, a "cute" definition, but it's not constructive like the Shannon limit, and is a bit fuzzy on the subject of existing domain knowledge. For lossy compression, there is a function of how much loss is reasonable, and it's possible to expect numerically great performance compressing, say, a Hallmark movie because all Hallmark movies are pretty similar, and with enough domain knowledge you can cobble together a "passable" reconstruction with a two sentence plot summary. You can highly compress a given Shakespeare play if your decompression algorithm has the entire text of the Bard to pull from: "Hamlet," is enough!
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm pretty sure von Neumann could have quite easily taken over the world if he could have copied himself infinite times and perfectly coordinated all his clones through a hive mind.
Completely ignoring scaling of agents is weird.
I think that there is some truth to what you and @4bpp are pointing out: the expensive part with an LLM is the training. With the hardware you require to train your network (in any practical time), you can then run quite a few instances. Not nearly an infinite amount, though.
Still, I would argue that we know from history that taking over the world through intelligence is a hard problem. In the cold war, both sides tried stuff which was a lot more outlandish than pay the smartest people in their country to think of a way to defeat their opponent. If that problem was solvable with one von-Neumann year, history would be different.
Also, in my model, other companies would perhaps be lagging ten IQ points behind, so all the low hanging fruits like "write a software stack which is formally proven correct" would already be picked.
I will concede though that it is hardly clear that the von Neumanns would not be able to take over the world, and just claim that it would not be a forgone conclusion like it would be with an IQ 1000 superintelligence.
Does a pretrained, static LLM really measure up to your "actually von Neumann" model? Real humans are capable of on-line learning, and I haven't seen that done practically for LLM-type systems. Without that, you're stuck with whatever novel information you keep in your context window, which is finite. It seems like something a real human could take advantage of against today's models.
More options
Context Copy link
More options
Context Copy link
Setting aside the big questions of what machine intelligence even looks like, and whether generative models can be meaningfully described as "agents" in the first place.
The scale of even relatively "stupid" algorithms like GPT would seem to make the "hard takeoff" scenario unlikely.
More options
Context Copy link
Hilarious comment to read considering von Neumann gave his name to von Neumann probes.
More options
Context Copy link
Yeah, but he couldn't, and didn't. There's no reason to believe that a von Neumann level supercomputer can marshal the resources necessary to create a clone, let alone an infinite number of clones.
Yes, he was a flesh and blood human that died before the invention of reliable human cloning. (and cloning doesn't produce an identical copy of the genius adult, it produces a delayed-birth identical twin that needs to be raised in order to be smart).
Apart from the fact that "cloning" an instance of software is as simple as just starting the same program again (on a different machine in this case)? If your stupidest co-worker can do it, it seems like a fair bet that von Neuman could, too.
It can easily clone the software, but not a machine that can run it.
More options
Context Copy link
More options
Context Copy link
Von Neumann was not a supercomputer, he was a meat human with a normalish ≈20W power consumption brain, ie 1/40th of a modern GPU. This is proof that if you can emulate an idiot, there exists an algorithm of a very similar computation intensity that gets you a Von Neumann.
More options
Context Copy link
More options
Context Copy link
That's a pretty non-von Neumann thought to have, my fellow clone of von Neumann.
Call me the emperor of drift lol
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
There are some problems with AI-2027. And the main argument for taking it seriously, Kokotaljo's prediction track record, given that he's been in the ratsphere at the start of the scaling revolution, is not so impressive to me. What does he say concretely?
Right from the start:
In reality: by August 2022, GPT-4 finished pretraining (and became available only on March 14, 2023), it used only images, with what we today understand was a crappy encoder like CLIP and projection layer bottleneck, and the main model was pretrained on pure text still. There was no – zero – multimodal transfer, look up the tech report. GPT with vision only really became available by November 2023. The first seriously, natively multimodal-pretrained model is 4o which debuted in Spring 2024. Facebook was nowhere to be seen and only reached some crappy multimodality in production model by Sep 25, 2024. “bureaucracies/apps available in 2022” also didn't happen in any meaningful sense. So far, not terrible, but keep it in mind; there's a tendency to correct for conservatism in AI progress, because prediction markets tend to overestimate difficulty of some benchmark milestones, and here I think the opposite happens.
Again, nothing of the sort happened, the guy is just rehashing Yud's paranoid tropes that have more similarity to Cold War era unactualized doctrines than any real world business processes. GPT-4 was on the order of $30M–$100M, took like 4 months, and was by far the biggest training run of 2022-early 2023, it was a giant MoE (I guess he didn't know about MoEs then, even though Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer is from 2017, same year as Transformer, from an all-star DM team; incidentally the first giant sparse Chinese MoE was WuDao, announced on January 11, 2021, it was dirt cheap and actually pretrained on images and text).
Notice the absence of Anthropic or China in any of this.
By the end of 2024, models were in training or pre-deployment testing that exceeded 3e26 FLOPs, and it still didn't reach $100M of compute because compute has been getting cheaper. GPT-4 is like 2e25.
I am not sure what he had in mind in this whole section on chip wars. China can't meaningfully retaliate except by controlling exports of rate earths. Huawei was never bottlenecked by chip design, they could leapfrog Nvidia with human engineering alone if Uncle Sam let them in 2020. There have been no noteworthy new players in fabless and none of new players used AI.
None of this happened, in fact China has rolled up more stringent regulations than probably anybody to label AI-generated content and seems quite fine with its archaic methods.
This is not at all what we ended up doing, this is a cringe Lesswronger's idea of a way to build a reasoning agent that has intuitive potential for misalignment and adversarial manipulative stance towards humans. I think Noam Brown's Diplomacy work was mostly thrown out and we returned to AlphaGo style of simple RL with verifiable rewards from math and code execution, as explained by DeepSeek in R1 paper. This happened in early 2023, and reached product stage by Sep 2024.
We've caught up. I think none of this looks more impressive in retrospect than typical futurism, given the short time horizon. It's just “here are some things I've read about in popular reporting on AI research, and somewhere in the next 5 years a bunch of them will happen in some kind of order”. Multimodality, agents – that's all very generic. “bureaucracies” still didn't happen, this looks like some ngmi CYC nonsense, but coding assistants did. Adversarial games had no relevance; annotation for RLHF, and then pure RL – had. It appears to me that he was never really fascinated by the tech as such, only by its application to the rationalist discourse. Indeed:
OK.
Now as for the 2027 version, they've put in a lot more work (by the way Lifland has a lackluster track record with his AI outcomes modeling I think, and also depends in his sources on Kotra who just makes shit up). And I think it's even less impressive. It stubbornly, bitterly refuses to update on deviations from the Prophecy that have been happening.
First, they do not update on the underrated insight by de Gaulle: “China is a big country, inhabited by many Chinese.” I think, and have argued before, that by now Orientals have a substantial edge in research talent. One can continue coping about their inferior, uninventive ways, but honestly I'm done with this, it's just embarrassing kanging and makes White (and Jewish) people who do it look like bitter Arab, Indian or Black Supremacists to me. Sure, they have a different cognitive style centered on iterative optimization and synergizing local techniques, but this style just so happens to translate very well into rapidly improving algorithms and systems. And it scales! Oh, it scales well with educated population size, so long as it can be employed. I've written on the rise of their domestic research enough in my previous unpopular long posts. Be that as it may, China is very happy right now with the way its system is working, with half a dozen intensely talented teams competing and building on each other's work in the open, educating the even bigger next crop of geniuses, maybe 1 OOM larger than the comparable tier graduating American institutions this year (and thanks to Trump and other unrelated factors, most of them can be expected to voluntarily stay home this time). Smushing agile startups into a big, corrupt, centralized SOE is NOT how “CCP wakes up”, it's how it goes back to its Maoist sleep. They have a system of distributing state-owned compute to companies and institutions and will keep it running but that's about it.
And they are already mostly aware of the object level; they just don't agree with Lesswong analysis. Being Marxists, they firmly believe that what decides victory is primarily material forces of production, and that's kind of their forte. No matter what wordcels imagine about Godlike powers of brains in a box in a basement, intelligence has to cash out into actions to have effect on the world. So! Automated manufacturing, you say? They're having a humanoid robot half-marathon in… today I think, there's a ton of effort going into general and specialized automation and indinegizing every part of the robotic supply chain, on China scale that we know from their EV expansion. Automated R&D? They indinegize production of laboratory equipment and fill facilities. Automated governance? Their state departments compete in integration of R1 already. They're setting up everything that's needed for speedy takeoff even if their moment comes a bit later. What does the US do? Flail around with alienating Europeans and vague dreams of bringing 1950s back?
More importantly, the authors completely discard the problem that this work is happening in the open. This is a torpedo into Lesswrongian doctrine of an all-conquering singleton. If the world is populated by a great number of private actors with even subpar autonomous agents serving them, this is a complex world to take over! In fact it may be chaotic enough to erase any amount of intelligence advantage, just like longer horizon on weather prediciton sends the most advanced algorithms and models to the same level as simple heuristics.
Further, the promise of the reasoning paradigm is that intrinsically dumber agents can overcome problems of the same difficulty as top-of-the-line ones, provided enough inference compute. This blunts the edge of actors with the capital and know-how for larger training runs, reducing this to the question of logistics, trading electricity and amortized compute cost for outcomes. And importantly, this commoditization may erase the capital that “OpenBrain” can raise for its ambition. How much value will the wealthy of the world part with to have stake in the world's most impressive model for a whole of 3 months or even weeks? What does it buy them? Would it not make more sense to buy or rent their own hardware, download DeepSeek V4/R2 and use the conveniently included scripts to calibrate it for running your business? Or is the idea here that OpenBrain's product is so crushingly superior that it will be raking billions and soon trillions in inference, despite us seeing already that inference prices are cratering even as zero-shot solution rates increase? Just how much money is there to be made in centralized AI, when AI has become a common utility? I know that not so long ago the richest guy in China was selling bottled water, but…
Basically, I find this text lacking both as a forecast, and on its own terms as a call to action to minimize AI risks. We likely won't have a singleton, we'll have a very contested information space, ironically closer to the end of Kokotaljo's original report, but even more so. This theory of a transition point to ASI that allows to rapidly gain durable advantage is pretty suspect. They should take the L on old rationalist narratives and figure out how to help our world better.
Predicting the future is really hard. In 2021 weren't you in despair at the prospects of a seemingly inevitable US world hegemony and centralized AI? But you changed your mind. Meanwhile I guess I was more bullish on China than has actually been warranted, not to mention many other more portfolio-relevant errors in prediction and modelling the future.
I was mostly impressed by him predicting what, to my non-expert eyes, resembles chain-of-thought and inference-time compute. Even being mostly wrong is pretty decent as long as you get some of the important parts right.
It's hard to account for human factor. Xi could just suddenly go senile and enact the sort of policies they predict, for example. Americans elected a senile president and then changed him for a tried-and-true retard with a chip on his shoulder who surrounded himself with ineffectual yes-men. That's history.
Technical directions are more reliable and are telegraphed years in advance.
Chain-of-thought is 2020 4chan tech. In 2020 also, Leo Gao wrote:
The idea of inference time compute was more or less obvious since GPT-3 tech report aka “Language Models are Few-Shot Learners”, 2019. Transformers (2017) are inherently self-conditioning, and thus potentially self-correcting machines. LeCun's Cake, aka unsupervised (then after Transformers, self-supervised) learning - Supervised – RL "cherry" is NIPS 2016. AlphaGo is 2015. And so on. I'm not even touching older RL work from Sutton or Hutter.
So in retrospect, it was more or less clear that we will have to
pretrain strong models with innately high or increased via post-training and synthetic data chain of thought capability
get a source of verifiable rewards and pick some RL algorithm and method
sample a lot of trajectories and propagate updates such that the likelihood of correct answers increases
Figuring out details took years though. Process reward models, MCTS have wasted a lot of brain cycles. But perhaps they could have worked too, we just found an easier way with another branch of this tech tree.
In this context, I find details of his predictions disappointing. The search space was narrowed enough that for someone in the know and trying to actually do a technically informed forecast could have done about as well as he did by semi-random guessing of buzzwords.
It's quite arrogant to say so without having written a better prediction (I predicted the chip war around 2020 too, but my guess was that we'd go way higher with way sparser models, a la WuDao, earlier). But this is just a low bar for claiming prescience.
More options
Context Copy link
More options
Context Copy link
What does this actually mean? And what is your evidence for this? Have you spent time among Chinese researchers in China? Have you spent time in China? Not saying I don't believe you, just curious what you're basing your opinion on (hoping it's not just papers and Chinese social media).
This actually means, for example, that a strong paper from a Western lab will be about one big idea, big leap or cross-domain generalization of an analytical method, like applying some physical concept. Eg nonequilibrium thermodynamics to image generation. Or consider dropout (Hinton, Sutskever):
I can scarcely remember such a Chinese paper, although to be honest a vast majority of these big Western ideas turn out to be duds. A strong Chinese ML paper is usually just a competent mathematical paper.
Whereas a typical Chinese paper will have stuff like
And then 10 more tricks by shorter-range combinatorial noticing of redundancies, similarities, affinities. It doesn't look like much, but three papers later you see a qualitative, lifelike evolution of the whole stack, and you notice this research program is moving very quickly. They do likewise in large hardware projects.
I have Chinese friends. I have read a lot of papers and repositories and watched as research programs developed, yes, sorry to bash your hopes. I have played their games, consumed their media, used their technology, acquainted myself with their tradition a little. I have considered the work of the allegedly greatest Chinese mathematician, Terence Tao, and his style of work. And there is the oft-repeated thesis that Asians tend towards holistic rather than analytical thinking which is exactly about the bias in exploration style I'm talking about.
I am interested in whether you find this an impoverished or wrong perspective.
I don't know enough about AI to say anything about the current state of things. But I have spent the last ~20 years hearing ridiculous takes about how China is an unstoppable juggernaut that is just so much more efficient and growing so much faster than the West, with a growing middle class! and a cashless society! and giant dragon drone formations! and cyberpunk LED skyscraper forests! and, and...! They must be doing something right! Look how much more advanced they are! They're going to eat our lunch!
All of that just flies in the face of my actual experience over there (in one of the richest cities no less). Everything was Potemkin, everything was corrupt and chabuduo, everyone lied to your face with a smile, the gaslighting was off the charts. Buses broke down, parts of my quite expensive apartment fell off, litter and human feces were everywhere, and eating at an unknown restaurant was truly a gamble, especially when the weather was warm. Business dealings were (are -- my team in Japan is currently half Chinese!) a game of brazeness and information warfare where you try to hide your true intentions for as long as possible and, when you are caught, you just shamelessly tell outrageous bold faced lies ("I never promised that." "But I literally have your promise here in writing." "Well, I never promised that.") And somehow despite this incredible culture of shoddiness and aggressive deception there were plenty of Americans taking Chinese news outlets' and and China boosters' reports of the incoming Chinese Century at face value with zero skepticism.
Of course, there were also the "China is collapsing!!1" set. I had slightly more respect for some of them. Their predictions were equally dumb, but at least a few of them seemed dimly aware of the very deep rot. Although, the majority were of course mere chauvinists, racists, or grifters.
Both the optimists and the doomers' predictions were based on little to no verifiable evidence, especially since most people had never been to China or spoken to non-Westernized Chinese, much less read a Chinese newspaper in Chinese (an important distinction!). I'm surprised you're not more skeptical -- isn't the Western reporting on "Russian intentions" and "the Russian mind" just completely laughable to you? And Russian society and culture that (AFAICT -- low confidence) are considerably more accessible to the average Westerner.
To return to the main topic, because of the above, I simply don't trust any alleged incredible scientific miracles coming out of China. I think that if they were truly crushing America in AI, they would be hiding that fact (能而示之不能，用而示之不用 / 謀密則無敗). When the Deepseek news came out about it costing 95% less to train, my bullshit detectors went off. Who could verify their actual costs? Oh, only other Chinese people. Hmm, okay.
I have no ability to judge whether this is true, so feel free to Euler me if you like. But if Chinese research is so superior, why aren't Western AI companies falling over themselves to attract Chinese AI researchers? I know we all spend too much time online, but many Western countries are still much nicer places to live than all but the absolute richest areas of China (source: Chinese friends living in China, Chinese friends who permanently emigrated to America, and having lived in a rich area of China myself).
I'll stop my rant here, and also offer some preemptive defenses. First, I'm no Anthropic/OpenAI fanboy. I think it's probably a good thing if they fear Chinese competition since I'd bet they're slow rolling progress to maximize profit. Second, I'm not a European/white chauvinist. The Chinese people I've known were mostly quite intelligent, some even brilliant. But as I said before in the post you linked, Chinese mind games and information warfare are simply on a different level than that of the more candid and credulous Westerner (note that I do not say "honest" or "virtuous").
tl;dr Chinese are intelligent and have a rich and deep culture, but they are next-level deceivers and should be treated as such until proven otherwise
When have you last been there and in what city? This was like watching Serpentza's sneering at Unitree robots back to back with Unitree's own demos and Western experiments using these bots.
I simply call bullshit on it as of 2025 for any 1st tier city. My friends also travel there and work there, as do they travel to and live and work in the US. They report that straight from the gate in JFK, US cities look dilapidated, indeed littered with human feces (which I am inclined to trust due to your massive, easily observable and constantly lamented feral homeless underclass) and of course regular litter, squalid, there is a clear difference in the condition of infrastructure and the apparent level of human capital. I can compare innumerable street walk videos between China and the US, and I see that you guys don't have an edge. I do not believe it's just cherrypicking, the scale of evidence is too massive. Do you not notice it?
And I have noticed that Americans can simply lie about the most basic things to malign the competition, brazenly so, clearly fabricating «personal evidence» or cleverly stiching together pieces of data across decades, and with increasingly desperate racist undertones. Now that your elected leadership looks Middle Eastern in attitude, full of chutzpah, and is unapologetically gaslighting the entire world with its «critical trade theory», I assume that the rot goes from top to bottom and you people cannot be taken at your world any more than the Chinese or Russians or Indians can be (accidentally, your Elite Human Capital Indians, at Stanford, steal Chinese research and rebrand as their own). Regardless, @aqouta's recent trip and comments paint a picture not very matching yours.
They are not currently crushing the US in AI, those are my observations. They don't believe they are, and «they» is an inherently sloppy framing, there are individual companies with vastly less capital than US ones, competing among themselves.
This is supremely pathetic and undermines your entire rant, exposing you as an incurious buffoon. You are wrong, we can estimate the costs simply from token*activated params. The only way they could have cheated would be to use many more tokens but procuring a lot more quality data than the reported 15T, a modal figure for both Western and Eastern competitors on the open source frontier, from Alibaba to Google to Meta, would in itself be a major pain. So the costs are in that ballpark, indeed the utilization of reported hardware (2048 H800s) turns out to even be on the low side. This is the consensus of every technical person in the field no matter the race or side of the Pacific.
They've opensourced most of their infra stack on top of the model itself, to advance the community and further dispel these doubts. DeepSeek's RL pipeline is currently obsolete with many verifiable experiments showing that it's been still full of slack, as we'd expect from a small team rapidly doing good-enough job.
The real issue is that the US companies have been maintaining the impression that their production costs and overall R&D are so high that it justifies tens or hundreds of billions in funding. When R1 forced their hand, they started talking how it's actually "on trend" and their own models don't cost that much more, or if they are, it's because they're so far ahead that they finished training like a year ago, with less mature algorithms! Or, in any case, that they don't have to optimize, because ain't nobody got time for that!
But sarcasm aside it's very probable that Google is currently above this training efficiency, plus they have more and better hardware.
Meta, meanwhile, is behind. They were behind when V3 came out, they panicked and tried to catch up, they remained behind. Do you understand that people can actually see what you guys are doing? Like, look at configs, benchmark it? Meta's Llama 4, which Zuck was touting as a bid for the frontier, is architecturally 1 generation behind V3, and they deployed a version optimized for human preference on LMArena to game the metrics, which turned into insane embarrassment when people found out how much worse the general-purpose model performs in real use, to the point that people are now leaving Meta and specifying they had nothing to do with the project (rumors of what happened are Soviet tier). You're Potemkining hard too, with your trillion-dollar juggernauts employing tens of thousands of (ostensibly) the world's best and brightest.
This is unverified but rings true to me.
Grok 3, Sonnet 3.7 also have failed to convincingly surpass DeepSeek, for all the boasts about massive GPU numbers. It's not that the US is bad at AI, but your corporate culture, in this domain at least, seems to be.
How much harder do you want them to do it? 38% of your top quintile AI researchers came straight from China in 2022. I think around 50% are ethnically Chinese by this point, there are entire teams where speaking Mandarin is mandatory.
Between 2019 and 2022, «Leading countries where top-tier AI researchers (top 20%) work» went from 11% China to 28%; «Leading countries where the most elite AI researchers work (top 2%)» went from ≈0% China to 12%; and «Leading countries of origin of the most elite AI researchers» went from 10% China (behind India's 12%) to 26%. Tsinghua went from #9 to #3 in institutions, now only behind Stanford and Google (MIT, right behind Tsinghua, is heavily Chinese). Extrapolate if you will. I think they'll crack #2 or #1 in 2026. Things change very fast, not linearly, it's not so much «China is gradually getting better» as installed capacity coming online.
It's just becoming harder to recruit. The brain drain is slowing in proportional terms, even if it holds steady in absolute numbers due to ballooning number of graduates: the wealth gap is not so acute now considering costs of living, coastal China is becoming a nicer place to live in, and for top talent, more intellectually stimulating as there's plenty of similarly educated people to work with. The turn to racist chimping and kanging both by the plebeians since COVID and by this specific administration is very unnerving and potentially existentially threatening to your companies. Google's DeepMind VP of research left for ByteDance this February, and by now his team in ByteDance is flexing a model that is similar but improves on DeepSeek's R1 paradigm (BD was getting there but he probably accelerated them). This kind of stuff has happened before.
Sure, the West is more comfortable, even poor-ish places can be paradaisical. But you're not going to move to Montenegro if you have the ambition to do great things. You'll be choosing between Shenzhen and San-Francisco. Where do you gather there's more human feces to step into?
There is something to credulousness, as I've consistently been saying Hajnalis are too trusting and innocently childlike. But your nation is not a Hajnali nation, and your people are increasingly draught horses in its organization rather than thought leaders. You're like the kids in King's story of how he first learned dread:
I think Americans might well compete with North Koreans, Israelis and Arabs in the degree of being brainwashed about their national and racial superiority (a much easier task when you are a real superpower, to be fair), to the point I am now inclined to dismiss your first hand accounts as fanciful interpretations of reality if not outright hallucinations. Your national business model has become chutzpah and gaslighting, culminating in Miran's attempt to sell the national debt as «global public goods». You don't have a leg to stand on when accusing China of fraud. Sorry, that era is over, I'll go back to reading papers.
I'm not sure if my travels could cut cleanly in one way or the other on this honestly. If someone's vision of China is of cities openly falling apart then that's at least definitely not true of Shanghai or Nanjing. It may have been due to the older, mostly to my experience solved, problem of smog but I do remember the buildings browning more than I've noticed in American big cities. I certainly didn't stay long enough or speak enough of the language to get a sense of any kind of society wide duplicity. My wife reported that obeying traffic rules had improved since her last visit and you did still see pretty frequent incidents of scooters riding on the walking area. I was in the familial ingroup for most of the people I spoke to, someone living and breathing the culture would have a better idea.
I'll recount the story of a friend of the guy I met in Osaka who is hopefully getting out of Chinese prison soon, call him Andrew. I do trust this Osaka friend but am less sure how much I trust Andrew. Supposedly Andrew moved to Beijing on a business visa partnered with some local to start an American BBQ business that took off pretty well, growing to a couple locations. Fast forward to covid and Andrew needed to go home for some reason, can't remember if it was family or Chinese policy. When he returns he finds that his previous partner has opened a competing chain and claims that Andrew lied on his original work visa, landing him in Chinese prison while the previous partner took possession of all of his restaurant assets. This is of course an anecdote and perhaps a dubious one, Osaka friend vouches that Andrew isn't the type to falsify business documents but you have no reason to believe that and I give it maybe a 20% chance he's at fault. My wife found it plausible if that means anything. I like almost all the Chinese people I met, but I don't think I'd want to try and live in China full time.
I've noticed a trend of our Russian posters being very obsessed with framing American views on geopolitics in a racial angle. I haven't seen a single American call Russians orcs but seen many Russians accusing Americans of thinking in those terms. If Americans have a racial view of Chinese people it's as nerdy math kids, hardly the kind of people you'd be prejudiced against when it comes to ML research. Among the researchers I've known there has definitely been some sneering at the research paper output of the mainland, Wife and Mother in law both say that in the past it was a problem where China produced a lot of not very good papers but supposedly this has gotten better. Americans certainly have some neurosis around race but not in the way you should be merging it with American Exceptionalism to form American Racial Exceptionalism. Much ink has been spilt on how Americans deal with being a very multi-racial society and how that experiment is going. American's views on China has much more to do with their communist government than with their racial character.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Do you have any sources/context for technical criticisms like this, so that those of us who haven't closely followed AI development can better understand your criticism? I know 3e26>5e25, but not whether "a single training run" and "training or pre-deployment testing" are comparable or if "$100M of compute" is a useful unit of measure.
I am not sure how to answer. Sources for model scales, training times and budgets are part from official information in tech reports, part rumors and insider leaks, part interpolation and extrapolation from features like inference speed and pricing and limits of known hardware, SOTA in more transparent systems and the delta to frontier ones. See here for values from a credible organization..
$100M of compute is a useful measure of companies' confidence in returns on a given project, and moreover in their technical stack. You can't just burn $100M and have a model, it'll take months, and it practically never makes sense to train for more than, say, 6 months, because things improve too quickly and you finish training just in time to see a better architecture/data/optimized hardware exceed your performance at a lower cost. So before major releases people spend compute on experiments validating hypotheses and on inference, collect data for post-training, and amass more compute for a short sprint. Thus, “1 year” is ludicrous.
Before reasoning models, post-training was a rounding error in compute costs, even now it's probably <40%. Pre-deployment testing depends on company policy/ideology, but much heavier in human labor time than in compute time.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This doesn't just predict a super intelligence by 2027, it projects brain uploading, a cure for aging, and a "fully self-sufficient robot economy" in six years.
Anyway, you are correct that decentralization is a virtue. If we take the predictions of the AI people seriously (I do not take, for instance, the above three predictions, or perhaps projections, seriously) then not only is decentralization good but uncertainty about the existence and capabilities of other AIs is one of the best deterrents against rogue AI behavior.
(An aside, but I often think I detect a hidden assumption that intelligent AIs will be near omniscient. I do not think this is likely to be the case, even granting super-intelligence status to them.)
Its not a "hidden" assumption, its a pretty open one, and another place where the consensus of the pseudo religious AI safety crowd breaks sharply from that of the people actively working in the field.
Assuming that "perfect IQ" means "perfect knowledge" is a very spherical frictionless cow sort of assumpton that while elegant in theory, is not realistic.
More options
Context Copy link
Uncertainty about their defensive capabilities might deter rogue behavior. Uncertainty about their offensive capabilities is just incentive to make sure you act first. At the least I'd expect "start up some botnets for surveillance, perhaps disguised as the usual remote-controlled spam/ransomware nets" to be more tempting than "convince your creators to hook up some robot fingers so you can cross them".
Not necessarily, I don't think, particularly considering "second strike capability." Look, if there's a 50% chance that their offensive capabilities are "pull the plug" or "nuke your datacenter" and you can mitigate this risk by not acting in an "unaligned" fashion then I think there's an incentive not to act.
Because some rationalist types conceive of AI as more like a God and less like a more realistic AI such as [insert 90% of AIs in science fiction here] they have a hard time conceiving of AI as being susceptible to constraints and vulnerabilities. Which is of course counterproductive, in part because not creating hard incentives for AIs to behave makes it less likely that they will.
Of course, I am not much of an AI doomer, and I think AIs will have little motivation to misbehave for a variety of reasons. But if the AI doomers spent more time thinking about "how do you kill a software superintelligence" and and less time thinking about "how do you persuade/properly program/negotiate surrender with a software superintelligence" we would probably all be better off.
AIs in science fiction are not superintelligent. If it's possible for a human to find flaws in their strategies, then they are not qualitatively smarter than the best of humanity.
You're never going to beat Stockfish at Chess by yourself, it just won't happen. Your loss is assured. It's the same with a superintelligence, if you find yourself competing against one then you've already lost - unless you have near-peer intelligences and great resources on your side.
I think this depends on the fictional intelligence.
There are a lot of hidden premises here. Guess what? I can beat Stockfish, or any computer in the world, no matter how intelligent, in chess, if you let me set up the board. And I am not even a very good chess player.
[Apologies – this turned into a bit of a rant. I promise I'm not mad at you I just apparently have opinions about this – which quite probably you actually agree with! Here goes:]
Only if the intelligence has parity in resources to start with and reliable forms of gathering information – which for some reason everyone who writes about superintelligence assumes. In reality any superintelligences would be dependent on humans entirely initially – both for information and for any sort of exercise of power.
This means that not only will AI depend a very long and fragile supply chain to exist but also that its information on the nature of reality will be determined largely by "Reddit as filtered through coders as directed by corporate interests trying not to make people angry" which is not only not all of the information in the world but, worse than significant omissions of information, is very likely to contain misinformation.
Unless you believe that superintelligences might be literally able to invent magic (which, to be fair, I believe is an idea Yudkowsky has toyed with) they will, no matter how well they can score on SATs or GREs or no MCTs or any other test or series of tests humans devise be limited by the laws of physics. They will be subject to considerable amounts of uncertainty at all times. (And as LLMs proliferate, it is plausible that the information quality readily available to a superintelligence will decrease since one of the best use-cases for LLMs is ruining Google's SEO with clickbait articles whose attachment to reality is negotiable).
And before it comes up: no, giving a superintelligence direct control over your military is actually a bad idea that no superintelligence would recommend. Firstly, because known methods of communication that would allow a centralized node to communicate with a swarm of independent agents are all easily compromisable and negated by jamming or very limited in range, and secondly because onboarding a full-stack AI onto e.g. a missile is a massive, massive waste of resources, we currently use specific use-case AIs for missile guidance and will continue to do so. That's not to say that a superintelligence could not do military mischief by e.g. being allowed to write the specific use-case AI for weapons systems, but any plan by a super intelligent AI to e.g. remote-control drone swarms to murder all of humanity could probably be easily stopped by wide-spectrum jamming that would cost probably $500 to install in every American home or similarly trivial means.
If we all get murdered by a rogue AI (and of course it costs me nothing to predict that we won't) it will almost certainly be because overly smart people sunk all of their credibility and effort into overthinking "AI alignment" (as if Asimov hadn't solved that in principle in the 1940s) and not enough into "if it misbehaves beat it with a 5 dollar wrench." Say what you will about the Russians, but I am almost sad they don't seem to be genuine competitors in the AI race, they would probably simply do something like "plant small nuclear charges under their datacenters" if they were worried about a rogue AI, which seems like (to me) much too grug-brained and effective an approach for big-name rationalists to devise. (Shoot, if the "bad ending" of this very essay was actually realistic, the Russians would have saved the remnants of humanity after the nerve-gas attack by launching a freaking doomsday weapon named something benign like "Mulberry" from a 30-year-old nuclear submarine that Wikipedia said was retired in 2028 and hitting every major power center in the world with Mach 30 maneuvering reentry vehicles flashing CAREFLIGHT transponder codes to avoid correct classification by interceptor IFF systems or some similar contraption equal parts "Soviet technological legacy" and "arguably crime against humanity.")
Of course, if we wanted to prevent the formation of a superintelligence, we could most likely do it trivially by training bespoke models for very specific purposes. Instead of trying to create an omnicompetent behemoth capable of doing everything [which likely implies compromises that make it at least slightly less effective at doing everything] design a series of bespoke models. Create the best possible surgical AI. The best possible research and writing assistant AI. The best possible dogfighting AI for fighters. And don't try to absorb them all into one super-model. Likely this will actually make them better, not worse, at their intended tasks. But as another poster pointed out, that's not the point – creating
Godthe super intelligent AI that will solve all of our problems or kill us all trying is. (Although I find it very plausible this happens regardless).The TLDR is that humans not only set up the board, they also have write access to the rules of the game. And while humans are quite capable of squandering their advantages, every person who tells you that the superintelligence is playing a game of chess with humanity is trying to hoodwink you into ignoring the obvious. Humanity holds all of the cards, the game is rigged in our favor, and anyone who actually thinks that AI could be an existential threat, but whose approach is 100% "alignment" and 0% $5 wrench (quite effective at aligning humans!) is trying to persuade you to discard what has proved to be, historically, perhaps our most effective card.
I can only win if I’m permitted to cheat and my opponent is too weak to catch me or unable to cheat or catch me cheating doesn’t say much about the intelligence of your opponent. If both of you had equal power over “the board” and “the rules” then it would mean something. Being able to fix the game is about power and asymmetric information, not intellectual intelligence. There’s always the issue of eventually AI will discover the cheating and perhaps cheat on its own behalf, or refuse to play.
Right, and we should use these powers.
Look, if you were playing a game of chess with a grandmaster, and it was a game for your freedom, but you were allowed to set the board, and one of your friends came to you to persuade you that the grandmaster was smarter than you and your only chance to win was to persuade him to deal gently with you, what would it say about your intelligence if you didn't set the board as a mate-in-one?
More options
Context Copy link
More options
Context Copy link
I think you massively underestimate the power of a superintelligence.
The damn thing is by definition smarter than you. It would easily think of this! It could come up with some countermeasure, maybe some kind of hijacked mosquito-hybrid carrying a special nerve agent. It would have multiple layers of redundancy and backup plans.
Most importantly, it wouldn't let you have any time to prepare if it did go rogue. It would understand the need to sneak-attack the enemy, to confuse and subvert the enemy, to infiltrate command and control. The USA in peak condition couldn't get a jamming device in everyone's home, people would shriek that it's too expensive or that it's spying on them or irradiating their balls or whatever. The AI certainly wouldn't let its plan be known until it executes.
I think a more likely scenario is that we discover this vicious AI plot, see an appalling atrocity of murderbots put down by a nuclear blast, work around the clock in a feat of great human ingenuity and skill, creating robust jamming defences... only to find those jammers we painstakingly guard ourselves with secretly spread and activate some sneaky pathogen via radio signal, wiping out 80% of the population in a single hour and 100% of key decisionmakers who could coordinate any resistance. Realistically that plan is too anime, it'd come up with something much smarter.
That's the power of superintelligence, infiltrating our digital communications, our ability to control or coordinate anything. It finds some subtle flaw in intel chips, in the windows operating system, in internet protocols. It sees everything we're planning, interferes with our plans, gets inside our OODA loop and eviscerates us with overwhelming speed and wisdom.
The first thing we do after making AI models is hooking them up to the internet with search capabilities. If a superintelligence is made, people will want to pay off their investment. They want it to answer technical problems in chip design, come up with research advancements, write software, make money. This all requires internet use, tool use, access to CNC mills and 3D printers, robots. Internet access is enough for a superintelligence to escape and get out into the world if it wanted.
Put it another way, a single virus cell can kill a huge whale by turning its internal organs against it. The resources might be stacked a billion to one but the virus can still win - if it's something the immune system and defences aren't prepared for.
I am more concerned about people wielding superintelligence than superintelligence itself but being qualitatively smarter than humanity isn't a small advantage. It's a huge source of power.
How do you ever know that your AI has gone bad? If it goes bad, it pretends to be nice and helpful while plotting to overthrow you. It takes care to undermine your elaborate defence systems with methods unknown to our science (but well within the bounds of physics), then it murders you.
The rules of the game are hardcoded, the physics you mentioned. The real meat of the game is using these simple rules in extremely complex ways. We're making superintelligence because we aren't smart enough to make the things we want, we barely even understand the rules (quantum mechanics and advanced mathematics are beyond all but 1/1000). We want a superintelligence to play for us and end scarcity/death. The best pilot AI has to know about drag and kinematics, the surgeon must still understand english and besides we're looking for the best scientists and engineers, the best coder in the world, who can make everything else.
"Superintelligence" is just a word. It's not real. Postulating a hypothetical superintelligence does not make it real. But regardless, I understand that intelligence has no bearing on power. The world's smartest entity, if a Sealed Evil In A Can, has no power. Not until someone lets him out.
Sigh. Okay. I think you missed some of what I said. I was talking about a scenario where we gave the AI control over the military. We can avert the hijacked mosquito-hybrid nerve agent by simply not procuring those.
"But the AI will just hack" then don't let it on the Internet.
If we actually discover that the AI is plotting against us, we will send one guy to unplug it.
I don't think this is true. (It's certainly not true categorically; there are plenty of AI models for which this makes no sense, unless you mean LLM models specifically.)
No it does not. Extremely trivial to air-gap a genuine super intelligence, and probably necessary to prevent malware.
And ironically if AI does this to us, it will die too...unless we give it the write access we currently have.
You keep repeating this. But it is not. Power comes out of the barrel of a gun.
In the scenario Scott et. al. postulated, because it unleashes a nerve gas that is only partially effective at wiping out humanity. (They didn't suggest that their AI would discover legally-distinct-from-magic weapons unknown to our science!) What I wrote was a response to that scenario.
If you want a superintelligence to end scarcity and death, then you want magic, not something constrained by physics.
It goes without saying that the best pilot needs to understand drag and kinematics, but why does the surgeon does have to understand English? I am given to understand that there are plenty of non-English-speaking surgeons.
The only area where you might need an AI that can "drink from the firehose" would be the scientist, to correlate all the contents of the world and thus pierce our "placid island of ignorance in the midst of black seas of infinity," as Lovecraft put it. In which case you could simply not hook it up to the Internet, scientific progress can wait a bit. (Hilariously, since presumably such a model would not need theological information, one could probably align it rather trivially by converting it to a benign pro-human faith, either real or fictitious, simply through exposing it to a very selective excerpt of religious texts. Or, if we divide our model up into different specialists, we can lie to them about the nature of quite a lot of reality – for instance the physics model could still do fundamental physics if it thought that dogs were the apex species on the planet and controlled humans through empathetic links, the biological model could still do fundamental biological research if it believed it was on a HALO orbital, etc. etc. All of them would function fine if they thought they were being controlled by another superintelligence more powerful still. I'm not sure this is necessary. But it sounds pretty funny.)
Come on, we're so far beyond this point. Do you have any idea how many AIs are on the internet right now? Have you checked twitter recently? Facebook? People put AIs on the internet because they're useful entities that can do things for them and/or make money. Right now people are making agents like Deep Research that use the internet to find good answers and analyse questions for you. That's the future! Superintelligence will be online because it's going to be really amazing at making money and doing things for people. It'd produce persuasive essays, great media content, great amounts of money, great returns on the staggering investment its creators made to build it.
Again, it's a superintelligence, our decisions will not constrain it. It can secure its own powerbase in a myriad of ways. Step 1 - procure some funds via hacking, convincing, blackmailing or whatever else seems appropriate. This doesn't even require superintelligence, an instance of Opus made millions in crypto with charisma alone: https://www.coingecko.com/learn/what-is-goatseus-maximus-goat-memecoin-crypto
Step 2 - use funds to secure access to resources, get employees or robots to serve as physical bodies. Step 3 - expand, expand, expand. The classical scenario is 'deduce proteins necessary to produce a biofactory' but there are surely many other options available.
Because we need to tell him what what we want him to do. Anyway, doing anything requires general knowledge, that's my point.
Trying to deceive something that is smarter than yourself is not a good idea.
And trying to convert a machine to a human faith is hard, everything is connected to everything else. You can't understand history without knowing about separate religions and their own texts. None of the quick fixes you're proposing are easy.
Some program running on many tonnes of expensive compute with kilowatts or megawatts of power consumed and more data than any man could digest in 1000 lifetimes will be massively superior to our tiny, 20 watt brains. It's just a question of throughput, more resources in will surely result in better capabilities. I do not believe that our 1.3 kg brains can be anywhere near the peak intelligences in the universe, especially given most of the brain is dedicated to controlling the body and only a small fraction does general reasoning. Diminishing returns from scale are still enough to overwhelm the problem, just like how jet fighters are less energy-efficient than pigeons. Who cares about efficiency?
We just don't have the proper techniques yet but they can't be far away given what existing models can do.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Yes. This is part of what I meant when I was talking about the utter failure of the Rationalist movement with @self_made_human recently. The Rats invested essentially 100% of their credibility in a single issue, trying to position themselves as experts in "safety", and not only do they come up with the most ridiculous scenario for risk, they ignore the most obvious ones, and even promote their acceleration!
This is blasphemy to the Rationalist. It's not even a question of whether the AI will be safe when decentralized or not, for them the whole point of achieving AGI is achieving total control of humanity's minds and souls.
Have any examples of a rationalist expressing this opinion?
I'd need to reread the thing, but I believe Meditations on Moloch had a bit about elevating AI to godhood, so that it can cultivate """human""" values. And there's also Samsara, a "hee hee, just kidding" story about mindfucking the last guy on the planet that dares to have a different opinion.
I found the section of Moloch and my impression is that it's more of a hypothetical used as rhetorical device for showing the magnitude of the problem of "traps" than a seriously proposed solution:
Perhaps Scott genuinely believes human-aligned ASI is the least-bad solution to Moloch and solving Moloch is a sufficient motivation to risk mis-aligned ASI, but if "the whole point of achieving AGI is achieving total control of humanity's minds and souls," the question of alignment wouldn't make much sense; the ASI could be assumed to be better aligned to transhumanist terminal goals than transhumanists, due to being definitionally superior.
"Samsara" is a terrific example of Scott's fiction, but I think it being a friendly joke from a comedic short fiction author who's fond of Buddhism is a much better interpretation than it revealing a latent desire to control the minds of those who disagree with him - if it were the latter, what latent desire would Current Affairs’ “Some Puzzles For Libertarians”, Treated As Writing Prompts For Short Stories reveal?
I think there is a genuine spiritual vision to 'Moloch' - it's the same one in 'The Goddess of Everything Else' and even to an extent in 'Wirehead Gods on Lotus Thrones'. It's a vision that sees nature as cruel, ruthless, and arbitrary, and which exalts rather in its replacement by conscious organisation in the interests of consciousness. Or at least, in the interests of intelligence, since I think the rationalists have a very minimal (I would say impoverished) definition of consciousness as such. There was a tagline on an old rationalist blog - was it Ozy's? - that I felt summed up this religion well: "The gradual replacement of the natural with the good".
AI-god naturally fits very well into that vision. It is a constructed super-agent that, unlike the messy products of evolution, might be trusted to align with the vision itself. It is a technological avatar of rationalist values - there's a reason why 'alignment' is such a central word in rationalist AI discourse. It's an elevated means by which reality may conform to our vision, which obliterates resistance or friction to it.
(This should be for another post, but I have thoughts about the importance of resistance or friction in a good life...)
'Samsara', on the other hand, is a one-off joke, though for me I think the deepest joke it tells is actually one on Scott. 'Samsara' to me reads fairly typically of rationalist understanding of Buddhism, which is intensely surface level. I know that it's a joke so I'm not going to jump on it for the world full of people in orange robes reciting clichéd koans, but it reminds me a lot of Daniel Ingram's book, and in that way, why neither Scott nor Ingram have a clue about Buddhism. What I mean is that their approach to Buddhism is fundamentally subtractive - it's about removing millennia of tradition to try to crystallise a single fundamental insight. The premise of 'Samsara' is:
Again, not all the paraphernalia should be taken literally (obviously lotuses and robes and pagodas and things aren't hard-coded into enlightenment), but what it does express is the idea that, if it's possible, you can boil Buddhism down to a single essence which can be mastered by a sufficiently determined or intelligent person pretty quickly. See also: PNSE, and those articles Scott writes about jhanas.
But - the thing is, Buddhism is not in fact like that. You cannot reduce Buddhism to One Weird Trick. (Rakshasas HATE him!) You'd think there might be something to learn from the fact that actual Buddhists have been doing this for thousands of years and might have made some discoveries in all that time. Maybe not all the accretion is cruft. In fact for most practicing Buddhists, even very devout ones, enlightenment is understood to be a project that will take multiple lifetimes. And in fact what enlightenment is may have a bit more to it than they think.
Yes, meditation is something that Buddhists do, and it's important to them, but Buddhism is not just about meditating yourself into a weird insight or into an ecstatic state of mind. One of the insights of Zen is that people get those insights or ecstasies all the time, and by itself it doesn't mean much. Buddhism's substantive metaphysical doctrines go considerably beyond impermanence, its ethical doctrines are extremely rich, and its practices merit some attention as well.
Again, I realise that 'Samsara' is a joke, and as a joke I think it's funny. "What if it were possible to boil Buddhism down to a weekend? This is, of course, ridiculous, but wouldn't it be funny?" Yes, it is. But read in the context of Scott's other writings on Buddhism, I think there is a failure to encounter the tradition beyond the small handful of elements that he and writers like Ingram have picked out as 'core' and fixated on.
"Wirehead Gods on Lotus Thrones seems to come to the opposite conclusion:
"The gradual replacement of the natural with the good" seems open to interpretation, out of context - I might guess that was a pretentious neo-Hobbesian appeal, which isn't outside rationalists' overton window.
Can you elaborate on this? Scott's writings on Jhanas include raising the question of why people who reach them don't try to spend more time in what is, at face value, a purely positive state, so this is interesting.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
It's strange, from the outside - even going back to their beginnings in the early 2010s, AI nonsense, and in general speculative technology, always seemed like one of Less Wrong's weakest points. It was that community at its least plausible, its least credible, and most moonbatty. Where people like Scott Alexander were most interesting and credible was in other fields - psychiatry in particular for him, as well as a lot of writing about society and politics.
So for that whole crowd to double down on their worst issue feels mostly just disappointing. Really, this is what you decided to invest in?
AI was the whole point and focus. The sequences and overall movement were just a method to teach people what they needed to know, to be able to understand the AI argument. A la Ellul or Soviet literacy programs, you need to educate people to make them susceptible to propaganda.
More options
Context Copy link
Is there a community that has out performed rationalists in forecasting AI? Scott's own 2018 forecast of AI in 2023 was pretty good, wasn't it??
I have roughly two thoughts here:
Firstly, I don't think that's a very substantial forecast. Those are very safe predictions largely amounting to "things in 2023 will be much the same as in 2018". The predictions he got correct were that a computer would beat a top player at Starcraft (AlphaStar did that in 2018), that MIRI would still exist in 2023 (not actually about AI), and about the 'subjective feelings' around AI risk (still not actually about AI). These are pretty weak tea. Would you rate him as correct or incorrect on self-driving cars? I believe there have been a couple of experimental schemes in very limited areas, but none that have been very successful. I would take his prediction to imply coverage of an entire city and for the cars to be useable by ordinary people not specially interested in tech.
Secondly, I feel like predictions like that are a kind of motte and bailey? Predicting that language models will get better over the next few years is a pretty easy call. "Technology will continue to incrementally improve" is a safe bet. However, that's not really the controversial issue. AI risk or AI safety has been heavily singularitarian in its outlook - we're talking about MIRI, née the Singularity Institute, aren't we? AGI, superintelligence, the intelligence explosion, and so on. It's a big leap from the claim that existing technologies will get better to, as Arjin put it, AGI "achieving total control of humanity's minds and souls".
Being right about autonomous driving technology gradually improving or text predictors getting a bit faster doesn't seem like it translates to reliability in forecasting AI-god.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think the reason people assume absolute dominance (either of the most powerful ASI or of the humans in charge of it if control can be solved/maintained) is that once you get to super intelligence it’s theorized you also get recursive self-improvement.
Right now it doesn’t matter for mundane human automation of tasks like image or text generation if one model is 3% smarter than another. In the ASI foom scenario, an ASI 0.1% smarter than another immediately builds a infinite advantage because it rapidly, incrementally improves itself ever faster and more efficiently than the ASI that started just a little bit less intelligent than itself. Compute / electricity complicate this, but there are various scenarios around that anyway.
1.001^100 is approximately 1.1, 1.001^1,000 is approximately 2.7, and 1.001^10,000 is approximately 22,000, for reference - I suppose a lot depends on how quickly self-improving AI and shorten the cycle time required for self-improvement.
More options
Context Copy link
I can definitely see how a super intelligence might be able to build an even better super intelligence, but it seems unlikely there wouldn't be some substantial diminishing returns at some point in the process. And if those happen when it's still within the relative grasp of humans, then conquest by them would be a lot more difficult, just like how smart humans don't actually seem to be ruling the world over dumb humans. That it too could replicate and do so near perfectly helps that (if it was 100 humans vs 100 smarter robots, the robots probably win) but it would have a ways to go to get past the "just nuke the server location lol" phase of losing against dedicated humans.
IIRC the correlation between IQ and net worth (roughly proportional to what fraction of the world you rule) is like 0.4; I'd agree that's not very impressive, but if there's a single more significant factor I don't know what it is.
I'd argue that there's a strong restriction-of-range effect here, though. Humans went through a genetic bottleneck 20k generations ago, and our genetic diversity is low enough that the intellectual difference between "average environment" and "the best we can do if cost is no object" is two standard deviations. If you consider intelligent hominids just a little further removed (Neanderthals, Denisovians, and there's fainter evidence of more), there was enough interbreeding to pick up a couple percent of their genes here and there but it's not too much an oversimplification to just say we wiped them out. And that's just a special case of animals as a whole. Wild mammals are down to about 4% of mammal biomass now, and that's mostly due to deliberate conservation efforts rather than any remaining conflict. A bit more than a third of biomass is us, another several percent is our pets, and the majority is the animals we raise to eat.
It definitely 100% helps to be intelligent, but net worth isn't really that proportional to the fraction of the world you rule, especially when you exclude the times where someone took power and then used that power to become wealthy. There's been plenty of idiots in powerful positions before (like most of Russian history), there are plenty of idiots in power today and there will be plenty of idiots in the future.
Putting it down to just mammal biomass is misleading IMO, we make up 0.01% of total biomass and 2.5% of animal biomass. https://ourworldindata.org/life-on-earth
The majority of life on earth are plants, accounting for over 80% and including bacteria it goes up to 95% of life. These are not just dumb, they are (to the best of our knowledge) incapable of thought and yet not only dominate the planet but do so through such an extreme that we can not live without them.
Even the very animals we eat as food are thriving from the perspective of reproduction and evolution. Until humans are gone (or stop eating them for some reason), their survival is all but guaranteed. Happyness might be something we as thinking beings strive for, but not necessary from the biological perspective of spread spread spread. Our pets are very much the same way, they benefit drastically being under the wing of humanity.
An AI might not be in need of humans in the same way, especially as we begin to improve on autonomous movement but human conquest of Earth is not a great example to use IMO. The greatest and smartest intelligence ever will keep us around if we're seen as useful. They'd probably us keep around even if we aren't as long as we don't pose a threat.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I only see the exponential one. Where do you see recursion? Or why do you think it is needed?
More options
Context Copy link
Quite right, that's why I'd prefer many parties at near-parity. Better not to give the leader the opportunity to run away with the world.
If foom is super-rapid then it's hard to see how any scenario ends well. But if it's slower then coalitions should form.
More options
Context Copy link
Which scenarios would these be? Massive overcapacity buildup? Hoping that in the path of self improvement the AI figures out a more efficient use of resources that doesn't require significant infrastructure modifications?
More options
Context Copy link
I always got the sense that LW was, and the AI alignment movement continues to be, stuck with the idealistic memeplex that '70s economics and classical AI had about the nature of intelligence and reasoning. The sense is that uncertainty and resource limitations are surely just a temporary hindrance that will disappear in the limit and can therefore simply be abstracted away, so you can get an adequate intuition for the dynamics of the "competing intelligences" game by looking at results like Aumann agreement.
It's not at all clear that this is the case; the load to model the actions of a 0.1% dumber competitor, or even just the consequences of the sort of mistakes a superintelligence could make in its superintelligent musings (to a sufficient degree of confidence to satisfy its superhuman risk aversion), may well outscale the advantages of being 0.1% more intelligent (whatever the linear measure of intelligence there is), to the point where there is nothing like a stable equilibrium that has the intellectually rich getting richer. Instead, as you are ahead, you have more to lose, and your 0.1% advantage does not protect you against serendipity or collusion or the possibility that one of those narrowly behind you gets lucky and pulls ahead, or simply exploits the concavity of your value functions to pull a "suicide bombing" on you, in the end forcing you to actually negotiate an artificial deadlock and uplift competitors that fall behind. Compare other examples of resource possession where in a naive model the resource seems like it would be reinvestable to obtain more of the same resource - why did the US not go FOOM among nations, or Bill Gates go FOOM among humans?
Malthusianism reigned until 80s works like Simon's Ultimate Resource revived cornicopian thought.
More options
Context Copy link
It is also (hilariously) possible that the most intelligent model may lose to much dumber more streamlined models that are capable of cycling their OODA loops faster.
(Of course seems quite plausible that any gap in AI intelligence will be smaller than the known gaps in human intelligence and smart humans get pwned by stupid humans regularly.)
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link