site banner
PaperclipPerfector  24d ago (text post)   32055 thread views

Culture War Roundup for the week of March 10, 2025

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

SMBC gets this close.

I've been thinking about the Grossman-Stiglitz Paradox recently. From the Wiki, it

argues perfectly informationally efficient markets are an impossibility since, if prices perfectly reflected available information, there is no profit to gathering information, in which case there would be little reason to trade and markets would eventually collapse.

That is, if everyone is already essentially omniscient, then there's no real payoff to investing in information. I was even already thinking about AI and warfare. The classical theory is that, in order to have war, one must have both a substantive disagreement and a bargaining friction. SMBC invokes two such bargaining frictions, both in terms of limited information - uncertainty involved in a power rising and the intentional concealment of strength.

Of course, SMBC does not seem to properly embrace the widely-held prediction that AI is going to become essentially omniscient. This is somewhat of a side prediction of the main prediction that it will be a nearly perfectly efficient executor. The typical analogy given for how perfectly efficient it will be as an executor, especially in comparison to humans, is to think about chess engines playing against Magnus Carlsen. The former is just so unthinkably better than the latter that it is effectively hopeless; the AI is effectively a perfect executor compared to us.

As such, there can be no such thing as a "rising power" that the AI does not understand. There can be no such thing as a human country concealing its strength from the AI. Even if we tried to implement a system that created fog of war chess, the perfect AI will simply hack the program and steal the information, if it is so valuable. Certainly, there is nothing we can do to prevent it from getting the valuable information it desires.

So maybe, some people might think, it will be omniscient AIs vs omniscient AIs. But, uh, we can just look at the Top Chess Engine Competition. They intentionally choose only starting positions that are biased enough toward one side or the other in order to get some decisive results, rather than having essentially all draws. Humans aren't going to be able to do that. The omniscient AIs will be able to plan everything out so far, so perfectly, that they will simply know what the result will be. Not necessarily all draws, but they'll know the expected outcome of war. And they'll know the costs. And they'll have no bargaining frictions in terms of uncertainties. After watching enough William Spaniel, this implies bargains and settlements everywhere.

Isn't the inevitable conclusion that we've got ourselves a good ol' fashioned paradox? Omniscient AI sure seems like it will, indeed, end war.

AI doesn't change the problem of computational tractability. There are a lot of problems now where we know how to find the exact solution (to any arbitrary, finite precision) but where the solution is many orders of magnitude beyond what computers would be able to achieve in any reasonable timeframe, even assuming Moore's law. Like the exact energy spectrum of any medium sized atom in a reasonable basis set (yes, there are various approximations that can be computed which work well enough) because the problem scales factorially with the number of electrons and basis functions. There’s so much handwaving in "omniscient" where people are glossing over any serious thought about what it would actually take to achieve omniscience, and at least some of Yudkowski's arguments about the way a superintelligence could infer physics from between 1 and 3 frames of video are provably wrong if you know anything about math and physics (I wrote something about this on an SSC thread perhaps 8ish years ago).

EY's three frames scenario was excellently debunked in this Less Wrong post:

https://www.lesswrong.com/posts/ALsuxpdqeTXwgEJeZ/could-a-superintelligence-deduce-general-relativity-from-a

Someone also pithily asked what physical theories the AI would arrive at if the three frames involved a helium balloon.

Like the exact energy spectrum of any medium sized atom in a reasonable basis set

This problem is actually definitely computable. Any medium-sized atom does this "calculation" all the time just fine. We may need to employ quantum computers, but problems that the nature solves are not difficult.

There are a lot of problems that are not solvable, though, but this is not a good example.

As someone who has worked with quantum algorithms for quantum chemistry, that's. . . really silly, and kind of not turning out to be a practical or useful way of approaching problems, despite what Richard Feynman might have thought before quantum computers were really a thing. Addirionally, any claims that quantum computers have a low enough noise floor or long enough coherence times to do any useful calculations are currently overhyped, misleading BS, and it's not clear that there's a clear path out of that.

I think the response would be that you don't need arbitrary precision. You just need enough to get within a pretty wide range of bargaining solutions. That may be doable at a higher level of abstraction, and a perfect executing AI can find that proper level of abstraction.

Of course, this process might not even look like finding the right level of abstraction to our eyes. In chess, grandmasters sometimes look at computer moves, and they struggle to contextualize it within a level of abstraction that makes sense to them. Sometimes, they're able to, and they have an, "OHHHHHHHH, now I see what it's saying," even though it's not "saying".

My response to that would be tongo back to something like chemistry, which is computationally more complex than chess, less complicated than modeling a lot of other real world things, but also obeys known equations.

There are some surprisingly simple systems for which all our normal computational chemistry approximations fail and they require much more sophisticated solutions. And you can't always handwave it away to "AI will find a simpler approximation that works". How do you know? Is there a good enough approximation that "works" for factoring any large number? Why should computational scaling laws cease to apply in theory? Would, for example AI be able to solve any arbitrary NP hard problem even if we could prove P != NP?

How do you know?

I don't know! I'm just temporarily importing my understanding of the tenets held by the singulatarian doomerists. They seem convinced that there's nothing we can do, not militarily, not intelligence community, not nothing, to even hold a candle in comparison to how good it's going to be at executing. Presumably, a part of its ability to be so good is going to be understanding the world around it with significantly smaller error bars than we currently have. I don't think they even need it to be completely zero error bars; just that it's wayyyy better than ours. What I think is related is that we don't need to have perfectly zero error bars in order to avert war; we just need small enough error bars to overcome the bargaining frictions. Given the high costs of war, that seems pretty feasible.

the widely-held prediction that AI is going to become essentially omniscient.

Held by whom?

As someone directly involved in the design and development of ML algorithms, Yudkowsky's blind faith in the inevitability of omniscient/super-intelligent AI has always felt like the rationalist equivalent of "and then a miracle occurs". Sure, if A through E then possibly F, but that's all in theory, and even if we get to E, F is by no means a given.

Nah, the assumption here is "and then no miracle occurs".

If we're really improbably lucky, then we do get a miracle: the level of intelligence required for an ape to create civilization (i.e. the point we're basically still at, because the millennia of memetic evolution afterward has grossly outraced the eon of genetic evolution beforehand) turns out to be essentially the same as the maximum level of intelligence achievable by any technology. AI could pass the C3PO "somewhat annoying but helpful" level, but it couldn't possibly pass the Data "better at math but wouldn't clearly be better in command" level. All those log(N) curves turn out to actually be logistic(N) in the limit, and human thinking remains relevant indefinitely after all.

Even if we develop proper reasoning engines within the next 5-10 years, there is still a big leap to be made between basic reason and a truly general intelligence, much less general intelligence to super intelligence, and an even bigger jump from "super intelligence" to "omniscience".

And that's without considering Yudowsky and Altman's wider body of quasi-religious pronouncements.

There's a difference between this and "it becomes omniscient somehow" and other rationalist religious exclamations.

Could you cite "it becomes omniscient somehow" from a rationalist?

Does the OP count?

The one opposing "everyone in the Big Yud singularity doomerist community"? The opposition itself isn't a deal-breaker (though it's clearly at least a non-central example), but the word choices to maximize emotional reaction at the expense of clarity are.

I was hoping someone would at least point out an interesting source being paraphrased. You see ML papers that talk about the infinite-width limit of neural networks, and sometimes that's just for a proof by contradiction (as OP appears to be attempting, to be fair), and sometimes it leads to math that applies asymptotically in finite-width networks ... but you can see how after a couple rounds of playing Telephone it might be read as "stupid ML cult thinks they're gonna have infinitely powerful computers!"

I quoted Scott below, but yes, everyone in the Big Yud singularity doomerist community. My post is taking one of their tenets seriously and seeing the implications. My sense is that they won't be particularly happy with such implications. Of course, part of the bit is exposing that many many people don't believe their tenets, surfacing that disagreement, with a clear application of how it contrasts with their other claims.

And they'll have no bargaining frictions in terms of uncertainties. After watching enough William Spaniel, this implies bargains and settlements everywhere.

Fortunately SMBC made a comic about this too.

Not AI, of course, but if you are able to reliably predict the outcome of a conflict, you can just skip the conflict itself and go straight to the settlement.

Nice find!

Even an omniscient AI would still fight a war. War is about using force to achieve a political goal. If you have force and a goal, you can have a war. Rationality has nothing to do with it. Even if a party knows that it will lose a war, they will often continue fighting out of internal political considerations and spiteful hatred.

Hatred is rational. You would rather face a conciliatory pushover than a hateful, spiteful opponent.

Anyway, some AIs will be smarter than others and so they'll be stronger.

War is about using force to achieve a political goal.

That would be the substantive disagreement part. Classical theory says that that's not enough for war. You also need a bargaining friction, otherwise, you'll get a negotiated settlement.

Well yeah, omniscient AI will end war by taking over the world, leaving no possible adversaries.

Again I have to quote Boaz Barak (currently OpenAI): AI will change the world, but won’t take it over by playing “3-dimensional chess”.

Consider the task of predicting the consequences of a particular action in the future. In any sufficiently complex real-life scenario, the further away we attempt to predict, the more there is inherent uncertainty. For example, we can use advanced methods to predict the weather over a short time frame, but the further away the prediction, the more the system “regresses to the mean”, and the less advantage that highly complex models have over simpler ones (see Figure 4). As in meteorology, this story seems to play out similarly in macroeconomic forecasting. In general, we expect prediction success to behave like Figure 1 below—the error increases with the horizon until it plateaus to a baseline level of some simple heuristic(s). Hence while initially highly sophisticated models can beat simpler ones by a wide margin, this advantage eventually diminishes with the time horizon.

Tetlock’s first commandment to potential superforecasters is to triage: “Don’t waste time either on “clocklike” questions (where simple rules of thumb can get you close to the right answer) or on impenetrable “cloud-like” questions (where even fancy statistical models can’t beat the dart-throwing chimp). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.” Another way to say it is that outside of the Goldilocks zone, more effort or cognitive power does not give much returns.

Rather, based on what we know, it is likely that AI systems will have a “sweet spot” of a not-too-long horizon in which they can provide significant benefits. For strategic and long-term decisions that are far beyond this sweet spot, the superior information processing skills of AIs will give diminishing returns. (Although AIs will likely supply valuable input and analysis to the decision makers.). An AI engineer may well dominate a human engineer (or at least one that is not aided by AI tools), but an AI CEO’s advantage will be much more muted, if any, over its human counterpart. Like our world, such a world will still involve much conflict and competition, with all sides aided by advanced technology, but without one system that dominates all others.

In essence, irreducible error and chaotic events blunt the edge of any superintelligent predictor in a sufficiently high-dimensional environment.

What remains to be answered for me:

  1. Can AI planners interfere in the events with enough frequency and precision to proactively suppress chaos and reduce the world to a game of chess they can model to the draw?
  2. Is a decently superhuman prediction and execution not enough to eliminate warm, simply because humans are already close to this level and only initiate wars they won't win (instead of pragmatically retreating to some defensible compromise) in feats of retardation (see: Russia)?

This is definitely where I start to quibble with the concept of "superintelligence" as synonymous with "omniscient."

Irreducible error because your sensors aren't precise enough to resolve every single detail you need to make 'perfect' decisions and chaotic events that can't be predicted without spending WAY too much effort.

ALL THAT SAID, I do think that an AI that is able to formulate a long term goal will be RIDICULOUSLY effective at achieving it, even amidst chaos.

One thing I can imagine is if the superintelligence wants a particular person dead it could do something 'basic' like a genetically targeted bioweapon, or something more creative like getting the person to consume two separate substances each of which is individually innocuous or even beneficial, but have a fatal interaction effect if they are both introduced to the human body in a short period of time.

So in the morning, the AI ensures that the target consumes a dose of substance A, then later in the day gets them to consume substance B, and they die in a way that looks very accidental, or maybe even natural, and thus it would be hard to detect how it was achieved.

Maybe the AI is even able to design novel substances that will achieve this goal so there'd be no real way for the individual to defend against this approach.

Now, the next step that is harder for me to buy is that they could use this sort of precisely targeted, nigh-undetectable interventions to guide all events towards their preferred state, avoiding wars but never overtly showing their hand, even if people suspect some given event was due to its meddling.

Not sure what you suggest here is really new to AI, humans are pretty good at killing human beings (a state agency such as the KGB or CIA can kill anyone who wishes to remain relevant with around 100% certainty if they really want to, although making it ~undetectable is slightly less efficient and slower, and more likely to fail) and they are kinda iffy at using those sorts of interventions to guide events towards their preferred state.

I mean, I ignored that the AI would have a plethora of ways to kill a person directly.

Fly a drone in through a window and spray any given toxin in their face, then fly it out.

Hijack their car's software, disable the brakes at an opportune time.

I'm sort of gesturing at the fact that a superintelligent AI can probably carry out Rube-Goldberg-esque plans with enough precision to hit multiple targets at once, with the aim of achieving multiple goals at once, all without immediately tipping any observers off as to their ultimate plans.

So assuming their ultimate plan isn't to just kill humanity as a whole, there is an 'interesting' world that emerges that ultimately bends towards the AI's preferences but doesn't necessarily require omniscience and 'solving' the game. The AI still has to adjust the plan in progress, might miss some of its targets, and unforeseen events can still surprise it, but nonetheless, the state of the world ticks inexorably towards the outcome it wants.

And its moves can occur on such a high dimension that no single human, even given access to all the necessary information, could see what its doing or even hope to outsmart it.

Yes, it's an interesting theory. I guess my point is that due to information friction I think humans can carry out plans - perhaps ones that might not be as good as those of a theoretical superintelligence, but still plans that confound observers. I mean shoot there's still (good faith?) arguments about whether COVID-19 was a lab leak or not despite all the evidence there.

Now, and I apologize for the tangent, but if the scenario you describe came about (or even became plausible) it would be unfalsifiable, leading to a world where Superintelligence replaces the Illuminati as the hidden hand behind world events.

Yes, exactly.

And then we're living in a world where even our own motivations for taking a given action could be the result of an upstream manipulation.

I think the best illustration of this principle lies in the downfall of Kodak. Their bankruptcy is often cited as a cautionary tale of what happens when you obstinately stick to old technology in the midst of a changing landscape. But that it were true! Yes, Kodak was synonymous with film in the early 2000s, but, while digital cameras existed, they were expensive and people were still buying a ton of film. So they weren't going to just stop producing it (and they still haven't). But the idea that they didn't see the writing on the wall and failed to embrace digital photography is a myth. They wholeheartedly threw most of their effort into what they perceived the transition to digital would look like. They manufactured inexpensive digital cameras and supplies for making prints at home, and they put kiosks in stores and malls for people without the equipment to make prints. What they failed to anticipate was a world where the market for cheap cameras would move to smartphones, and where social media would replace the need to get prints of everything.

And the reason they didn't anticipate it was because they couldn't anticipate it. No one could. Digital cameras started gaining market share before the rise of social media and phones with acceptable cameras. If you told someone in 2003 what the low end of the photographic world would look like 5 years later, they'd tell you you were nuts.

Point and shoot digital cameras killed mass market film well before the iphone age. If Kodak made inexpensive digital cameras, then where are all of them today?

Maybe they got their arse handed to them by the Japanese, but that would be a failure to compete, not a failure to anticipate.

They did pretty well through the point and shoot era, and their cameras were everywhere if you cared to look; in 2005 they led the market in camera sales. They just weren't involved in the pro market the way their competitors were, so when that market died they had nothing to fall back on.

then where are all of them today?

In the recycling bin, or at the back of a drawer unused. Displaced in everyday use by phone cameras, just as physical prints have largley been replaced by Facebook and instagram.

The only people who use a seperate (non-phone) camera these day are professional photographers and high-end hobbiests who are looking for quality over price. This (not the ultimate shift to digital) is the shift that kodak failed to anticipate.

I've never held my hobby of photography highly enough to splurge for a DSLR.

My brother did his, and now it collects dust with the bulk of his photography done with his iPhone 15 Pro Max.

My family splurged for a DSLR a decade or more ago, but now it basically only gets pulled out when we need the 50-300mm lens for distant shots, or maybe once or twice a year when a few shots are so important that they're worth the extra hassle. We used to pull it out for low-light photography too, but at some point phone image sensors got so sensitive that it makes up for not having half a pound of glass in front of them.

Oh - I do still use the DSLR body with a telescope adapter. I tried an eyepiece-to-phone adapter for that, but the quality wasn't nearly as high. Maybe I just need to find a better one.

At this point, I'm not sure what utility a DSLR offers over a newer mirrorless camera. If you already own one, great, but they're a dying breed.

Frankly speaking, the computational photography that phone cameras pull of is nigh magical (though some of it is plain hallucinations of non-existent details), and I wish dedicated camera manufacturers took more inspiration from them rather than vice versa.

IMHO the mechanical mirrors are pointless; large lenses are really the only things phones lack. My DSLR is just old enough that mirrorless options were still kind of new. We also got a Nikon 1 around the same time, for portability, but unlike the DSLR that one's been completely obsoleted by our phones.

I'm not a fan of the current state of computational "photography", though. Detecting motion between multiple frames and trying to stack and deconvolve to get a sharp still image, that's fantastic, but when we reached the point where there's a "upsample moon photos using a neural net trained on moon photos" step, we'd lost the plot. If I wanted data from existing photos rather than my own photos then I'd be using the web browser, not the camera.

My Kodak DC220 sits unused on my bookcase, only barely hidden by my untidiness :-)

chess

If you pit two top engines against each other, you won't have any idea who will win. You know it'll be a coin toss but you won't know who will win.

There can be no such thing as a human country concealing its strength from the AI.

Time to read the three body problem again. It's fiction but it conceptualizes the idea of wallfacers who will deceive the enemy AI which can be everywhere in the world all at once.

Even if an AI can simulate the world with such accuracy that it becomes essentially a game, the opponent's moves are still unknown. Playing a game well is one thing, but solving a game (determining if a player can force a win) is entirely harder. Checkers, tic-tac-toe, and connect four are solved, while chess is not.

With current technology, nobody knows the outcome of a very lopsided chess game. The underdog AI still has a chance, and that's why people are still interested in watching.

If you pit two top engines against each other, you won't have any idea who will win. You know it'll be a coin toss but you won't know who will win.

Emphasis added. I don't need to know in order for the AI to tell me that the best outcome is a negotiated settlement within certain parameters.

the opponent's moves are still unknown.

Agreed, but sort of irrelevant. The chess engine is still executing perfectly, even though it doesn't actually know what moves the opponent will ultimately make.

Playing a game well is one thing, but solving a game (determining if a player can force a win) is entirely harder. Checkers, tic-tac-toe, and connect four are solved, while chess is not.

I think the answer here is again that it is ultimately irrelevant. We didn't need to solve chess or diplomacy to have an engine become a nearly perfect executor or to narrow the range of outcomes significantly (>90% draws unless you extremely bias the starting positions, for example).

You are being nonsensical in your handwaving of complexity. Chess has 32 total pieces each with an extremely contrained potential action across only 64 positions. You can't just handwave knowability there into the real world. There's no reason to believe enough computational power exists to be able to have 'omniscient level' understanding of the world. You are just speaking pure, unfounded fiction.

For the record, you don't have a problem with me. You have a problem with the people who hold the position that we are approaching an AI singularity and that doom is inevitable because the AI will have all these incredible characteristics. I don't actually hold that position; I'm just investigating it.

In any event, I again don't think it needs to be actually omniscient. It just needs to be able to reduce error bounds enough to eliminate the bargaining friction. Since war is very costly, it certainly doesn't need to be perfect; it just needs to get the error bars down enough. Think of it as a continuum. As the ability to gather information, model, and predict accurately goes up, the likelihood of war goes down, since the bargaining frictions due to uncertainty are reduced. Yes yes, it may be only when we take the limit that the likelihood of war goes down to precisely zero. I'm not even quite sure of that, because since war is so costly, we can probably still tolerate a fair amount of uncertainty and still remain in a region where settlements can be negotiated.

The AI singularity/doom people think that, for all intents and purposes, we're headed for that limit. They may be wrong. But if one believes their premise, then I think the conclusion would be that war goes to zero.

wallfacers

This is the absolute pinnacle concept of that series. I’m not exactly an AI skeptic, I truly think it will revolutionize the entire world in my lifetime.

But rationalists constantly underestimate the power and grace of intuition in service of subversion. Humans absolutely excel at it, and I can’t envisage a world where they are overtaken by machines in this particular task. It’s too messy, too inexact, too chaotic.

Under constant total surveillance and crushing power imbalances, prisoners develop their own occult economy, rituals, alliances, symbology, etc etc etc. the prison which is not in fact run by the prisoners is the unstable exception only bought by extreme and unwavering competence & creativity, not the rule.

People regularly deceive themselves in a richly woven pattern that only they themselves can unlock.

Deceiving a rationalistic / probabilistic super intelligence?

Child’s play. GG EZ.

I mean, even in the book almost every Wallfacer fails. Not just that, but most of the Wallfacers' plans are unraveled by their opposite "wallbreaker."

The one plan that worked out was due to the guy acting incredibly erratically for like a couple decades (because he had no intention of actually doing anything) then getting blackmailed into actually trying to succeed at his task, then managing to obtain an insight that would allow him to win but was also achievable without making any moves that would make his plan obvious. AND THEN, he was only able to beat the trisolarans because he was suicidally committed to said plan when the moment came.

Oh, and he almost got killed by the Trisolarans several times but happened to have a supremely competent and aware bodyguard around at the right time.

the widely-held prediction that AI is going to become essentially omniscient.

Can you name three people who would agree that they make this "widely" held prediction? I know a lot who predict "much better than humans at making predictions", and quite a few who predict something like "that which cannot be predicted will be controlled", and a handful who predict galaxy-brained strange-loop reflective cooperation, but all of these fall quite a bit short of "essentially omniscient".

The typical analogy given for how perfectly efficient it will be as an executor, especially in comparison to humans, is to think about chess engines playing against Magnus Carlsen. The former is just so unthinkably better than the latter that it is effectively hopeless; the AI is effectively a perfect executor compared to us.

The important question is whether they are effectively perfect executors compared to each other. Humans are effectively perfect executors in social conflict when compared to chimpanzees, but we still fight against each other.

Can you name three people who would agree that they make this "widely" held prediction?

Probably not. I don't keep track of names of people. Obviously, there's Big Yud. I quoted Scott below. I'd have to wade further into those doomerist circles to get a third name, and meh.

The important question is whether they are effectively perfect executors compared to each other.

This is where I'm appealing to things like the >90% draw rate in computer chess (when the starting positions are not specifically biased). We also see something similar in the main anti-inductive system that I'm making comparison to - financial markets. At one point, I had heard that an offhand estimate of how long a good trading idea lasts before it's discovered and proliferated is like 18 months. The models just keep getting better.

I don't think Yudkowsky would agree that he expects AI to be effectively omniscient in an absolute sense - relative to humans, sure, but that's a very different question. I do understand how it's possible to read him as saying that - a lot of the things he's written make more sense if his mental model is "an omniscient, reflexively consistent agent". However, I think that's because that represents the mental model he uses to think about AGI, rather than because that's something he expects to literally happen. In an interview a couple years ago he said

Planning is one way to succeed at search. I think for purposes of understanding alignment difficulty, you want to be thinking on the level of abstraction where you see that in some sense it is the search itself that is dangerous when it’s a strong enough search, rather than the danger seeming to come from details of the planning process.

One of my early experiences in successfully generalizing my notion of intelligence, what I’d later verbalize as “computationally efficient finding of actions that produce outcomes high in a preference ordering”, was in writing an (unpublished) story about time-travel in which the universe was globally consistent.

The requirement of global consistency, the way in which all events between Paradox start and Paradox finish had to map the Paradox’s initial conditions onto the endpoint that would go back and produce those exact initial conditions, ended up imposing strong complicated constraints on reality that the Paradox in effect had to navigate using its initial conditions. The time-traveler needed to end up going through certain particular experiences that would produce the state of mind in which he’d take the actions that would end up prodding his future self elsewhere into having those experiences.

The Paradox ended up killing the people who built the time machine, for example, because they would not otherwise have allowed that person to go back in time, or kept the temporal loop open that long for any other reason if they were still alive.

Just having two examples of strongly consequentialist general optimization in front of me – human intelligence, and evolutionary biology – hadn’t been enough for me to properly generalize over a notion of optimization. Having three examples of homework problems I’d worked – human intelligence, evolutionary biology, and the fictional Paradox – caused it to finally click for me.

which, when I read, it, was an "aha" moment where I understood why the stuff he wrote was Like That™ despite his insistence that people were misinterpreting his old writing.

This is where I'm appealing to things like the >90% draw rate in computer chess (when the starting positions are not specifically biased).

I think that's a fact particular to chess - I don't expect the same result in computer Go / othello / some other game that is less structurally prone to having draws.

We also see something similar in the main anti-inductive system that I'm making comparison to - financial markets. At one point, I had heard that an offhand estimate of how long a good trading idea lasts before it's discovered and proliferated is like 18 months. The models just keep getting better.

The models do keep getting better, but I don't see how improvement in those models means that there is a reachable point where winning strategies switch from being based on deception and trickery to being based on cooperation stemming from mutual knowledge of each others' strategies (here though I do expect Yud would take your side).

I mean, I kinda get your point that it's the way that he thinks about it, but he also says that it gives us straightforward bounds:

A paperclip-maximizing superintelligence is nowhere near as powerful as a paperclip-maximizing time machine. The time machine can do the equivalent of buying winning lottery tickets from lottery machines that have been thermodynamically randomized; a superintelligence can’t, at least not directly without rigging the lottery or whatever.

But a paperclip-maximizing strong general superintelligence is epistemically and instrumentally efficient, relative to you, or to me. Any time we see it can get at least X paperclips by doing Y, we should expect that it gets X or more paperclips by doing Y or something that leads to even more paperclips than that, because it’s not going to miss the strategy we see.

So in that sense, searching our own brains for how a time machine would get paperclips, asking ourselves how many paperclips are in principle possible and how they could be obtained, is a way of getting our own brains to consider lower bounds on the problem without the implicit stupidity assertions that our brains unwittingly use to constrain story characters. Part of the point of telling people to think about time machines instead of superintelligences was to get past the ways they imagine superintelligences being stupid. Of course that didn’t work either, but it was worth a try.

So, I guess, like, think about the best possible plans you could come up with to put some error bars on the expected value of war. Perhaps notice that political scientists don't just ask the question, "Why is there war at all?" (...coming up with the answer involving bargaining frictions...) but also the question of why war is actually still somewhat rare, especially if we think about all of the substantive disagreements there are out there. They point out that the vast majority of wars that are started actually end surprisingly quickly, often as some information is learned in the process, a settlement is quickly reached. Superintelligences are going to be wayyyyyyyyy better at driving down those error bars and finding acceptable settlements.

This is where I'm appealing to things like the >90% draw rate in computer chess (when the starting positions are not specifically biased).

I think that's a fact particular to chess - I don't expect the same result in computer Go / othello / some other game that is less structurally prone to having draws.

I guess it's not the draws, themselves, that are "the thing". Let me try to put it another way. One of the top GMs in the world made a comment not too long ago about their experience working with very powerful computers. He said something along the lines of, "With the computer, it's always either zeros or winning." That is, he basically viewed it as that once you have enough computanium, for many many many positions, either the computer sees a way to essentially just straight equalize or it can see out to a win. Now, obviously, this is not strictly true, and it's obviously not true in all positions, as you get closer to the start of the game. But they can see the expected outcome sooo vastly better than we can. In the same way that people want to blow up that ability to things like "can engage in warfare sooo vastly better than we can", it should also blow up their ability to see expected outcomes and come to negotiated settlements sooo vastly better than we can.

I don't see how improvement in those models means that there is a reachable point where winning strategies switch from being based on deception and trickery to being based on cooperation stemming from mutual knowledge of each others' strategies

The attempted resolution in the financial markets paradox is that people just stop investing in more information. Could they double down on deception and trickery? Perhaps. But that seems like an unlikely result, game-theoretically. "Babbling equilibrium" or "cheap talk" are sometimes invoked, depending on the specific formalization. There are others that aren't in that wiki article. I could walk through a bunch of different models for how humans try to deal with deception and trickery in different domains. Presumably a superintelligence will know all of them and more... and execute even better in implementing them. It took me a long time to realize this, but when you think of deception and trickery as part of the strategy set, then the correct game-theoretic notion of equilibrium is not necessarily "cooperation stemming from mutual knowledge of each others' strategy", but "the appropriate equilibrium stemming from mutual knowledge of each others' strategy, which may contain deception and trickery, and you are each reasoning about the other's ability to engage in deception and trickery, the value the other may obtain from such, etc." Of course I know that my opponent may try deception and trickery, so I need to reason about it. A superintelligence will reason about it even better. Probably the easiest thing to think about here is again the game Diplomacy.

Where the mere game of Diplomacy differs from actual war in the real world is that we have good reason to believe that the costs of engaging in war are much much much higher, so we have a very big bargaining range, and we need quite significant bargaining frictions to get in the way. I still don't see how a superintelligence doesn't reduce the bargaining friction.

Superintelligences are going to be wayyyyyyyyy better at driving down those error bars and finding acceptable settlements. [...] I still don't see how a superintelligence doesn't reduce the bargaining friction.

I hope you're right about that. I worry that a lot of the dynamics around retaliation and precommitment are anti-inductive, and as such the difficulty of determining where the bright lines actually are scales with the sophistication of the actors. This would happen because a hostile actor will go right up to the line of "most aggressive behavior that will not result in retaliation" but not cross said line, so it becomes advantageous to be a little unclear about where that line is, and that lack of clarity will be calibrated to your adversaries not to some absolute baseline. And this is the sense in which I don't see a reachable point where honesty and bargaining come to strictly dominate.

As a note I do expect that bargaining frictions will be reduced, but the existential question is whether they will be reduced by a factor large enough to compensate for the increased destructiveness of a conflict that escalates out of control. Signs look hopeful so far but our sample size is still small. Certainly not a large enough sample size that I would conclude

The omniscient AIs will be able to plan everything out so far, so perfectly, that they will simply know what the result will be. Not necessarily all draws, but they'll know the expected outcome of war. And they'll know the costs. And they'll have no bargaining frictions in terms of uncertainties.

Only a couple minor responses, as I think we're mostly understanding each other.

this is the sense in which I don't see a reachable point where honesty and bargaining come to strictly dominate.

My only quibble is that I don't think we really need the "honesty and" part. The question really is whether, even with dishonesty, bargaining can be achieved.

As a note I do expect that bargaining frictions will be reduced, but the existential question is whether they will be reduced by a factor large enough to compensate for the increased destructiveness of a conflict that escalates out of control.

The weirdly good thing about the increase in destructiveness ("good" only in the narrow sense of bargaining and likelihood of war, not necessarily in general) is that this increases costs to both sides in the event of war. As such, it increases the range of possible bargaining solutions that keep the peace. Both factors (this and the reduced bargaining frictions) should decrease the likelihood of war.

Thanks for the read, i think Omniscient AI is a long way of. Almost all current ai models "simply" condence known knowledge. Most new discoveries made is finding patterns that we didnt seen before, but where already present. The current AI models have no capacity to think and rationalize. They are just very complex and high dimensional information vectors (that is what the N-amount of parameters mostly are).

Simply said: Just because a LLM knows the relation between certain human words does not mean it it sentient. The models can only repeat what the humans trained them on.

It won’t end war because the omniscient Chinese AI will be able to omnisciently bullshit the omniscient American AI to the point that there is uncertainty again.

If there is value in weeding out the bullshit, the omniscient AI will weed out the bullshit. AI already plays diplomacy, trying to weed out bullshit. Just increase the scale. The best bullshitting Diplomacy players will be mere Magnus Carlsens against it. The Chinese AI and the American AI will both compute all the way out to the draw, just like the TCEC.

Arguably the Diplomacy players will do better than the AI. "A curious game, the only winning move is not to play." But humans have been able to create uncertainty about their willingness to play an unwinnable game nevertheless for decades.

Isn't the inevitable conclusion that we've got ourselves a good ol' fashioned paradox? Omniscient AI sure seems like it will, indeed, end war.

Omniscience, though, is a really high bar. The SMBC was positing only a rational AI, not an omniscient one.

I don't expect AI to become omniscient. The inherent problem in actually modeling the real world well enough to predict it perfectly, faster than events play out, particularly when said real world contains other equally-powerful modelers, seems likely to be unsolvable. Further, even if you could do that, you'd need perfect information on the initial conditions, which there's no way to get.

I think you're doing the thing where you haven't internalized "the thing". From Scott:

Consider weight-lifting. Your success in weight-lifting seems like a pretty straightforward combination of your biology and your training. Weight-lifting retains its excitement because we don’t fully understand either. There’s still a chance that any random guy could turn out to have a hidden weight-lifting talent. Or that you could discover the perfect regimen that lets you make gains beyond what the rest of the world thinks possible.

Suppose we truly understood both of these factors. You could send your genes to 23andMe and receive a perfectly-accurate estimate of your weightlifting potential. And scientists had long since discovered the perfect training regimen (including the perfect training regimen for people with your exact genes/lifestyle/limitations). Then you could plug your genotype and training regimen into a computer and get the exact amount you’d be able to lift after one year, two years, etc. The computer is never wrong. Would weightlifting really be a sport anymore? A few people whose genes put them in the 99.999th percentile for potential would compete to see who could follow the training regimen most perfectly. One of them would miss a session for their mother’s funeral and drop out of the running; the other guy would win gold at whatever passed for this society’s Olympics. Doesn’t sound too exciting.

A team sport like baseball or soccer would be harder to solve. Maybe you’d have to resort to probabilistic estimates; given these two teams at this stadium, the chance of the Red Sox winning is 78.6%, because the model can’t predict which direction some random air gusts will go. I guess this is no worse than having Nate Silver making a betting model. But on the individual level, it’s still a combination of your (well understood) genes and (well understood) training regimen.

Hedge funds already have some of the best weather models in the world. There's alpha there right now. Or at least there was; I don't know how much has been anti-inducted away. The god AI will certainly be able to do at least as well. It will probably make our current best models look like a mere Magnus Carlsen. And if there's alpha in taking a more minute view, scoping the model in to a particular stadium, why can't it do that? Where there is alpha in the AI getting information, the AI will go there and get the information. It will be able to massively reduce the error bars. And all you need to get rid of war is reduce the error bars enough to get to a negotiated agreement. There's tons of alpha there, so there they will go. Until that alpha has been anti-inducted away, and we're right back in the paradox.

All this is at the "suppose" level. Yes, the god AI will be able to do this. IF it exists. I say the god AI will not exist.

And if there's alpha in taking a more minute view, scoping the model in to a particular stadium, why can't it do that? Where there is alpha in the AI getting information, the AI will go there and get the information.

It's an AI, a brain in a box. Even if it's really fucking smart, it has no advantage over us meat-brains in going places and getting information.

I say the god AI will not exist.

This is sort of the crux. I happen to agree with you. The point of my comment was to investigate the tenets of a group of folks and see what the implications are. I think that if one adopts a position like in that Scott quote, then the implication is something like the end of war.

Agreed, at least until the prices of solar panels and drones drop by a factor of 10-100.

Yes.

While I'm here, I'd suggest that so far in real life people index so hard on the intelligence that they overlook how little data the available AI can process at one time, which is a big limitation on IRL usefulness, or at least a speedbump. I have some professional experience with this (albeit mostly secondhand) and from what I can tell it's kinda like if you're dealing with a very smart intern with the memory of a goldfish. It can process data blazingly fast but you have to spoonfeed it one bite at a time. Which makes the blazing speed a bit underwhelming.

Now, this gives it definite advantages relative to all-human employment but you also have to hold its hand everywhere.

Note that I'm not making any predictions or claims, just noting my IRL understanding, and I know that context windows continue to be able to be expanded regularly - but AI ain't gonna be able to take over the world or even my job if it can't watch the entire Star Wars trilogy in one sitting.

I know that context windows continue to be able to be expanded regularly - but AI ain't gonna be able to take over the world or even my job if it can't watch the entire Star Wars trilogy in one sitting.

I just checked, and the current leader has 100 million tokens ("equivalent to 750 novels"), while non-specialized models are in the 100k-1M range. You're going to have to update your arguments (then update them again in a few months when AIs meet your new standards, then update them again...).

When we use them in practice we have to cut up the content that we feed them because we have much more content (gigabytes worth) than they can handle.

As I said, I think this is a solvable problem. But a lot of AI enthusiasts are, in my impression, just using them as personal assistants and not necessary engaging with them in more strenuous real-world use cases.

In my experience ‘real’ context is 5k to 15k max, even for the big models.

The large context windows are based on ‘can it retrieve very specific information from 1M tokens ago’ not ‘will it naturally remember that this information exists and how it might be relevant to whatever it’s doing at the moment’.

How is the alleged 100M model on RULER / NiaH? I find myself skeptical based on their blog post, the lack of concrete info therein, their choice to build a custom benchmark instead of using the industry standard ones (like RULER), their claims of having 100x'd context over publicly available SOTA, and their choice to name themselves "magic ai".

Your point does stand with gemini's 1M context window though - that one is the real deal, although the real killer will be a large-context reasoning model (without the ability to meaningfully process the things they retrieve from their context window, long-context LLMs don’t have much of an advantage over RAG).

I'm not familiar with that model (I just found it by searching), but I wouldn't doubt if they were simply Goodhearting their way into some flashy claims.

One thing to keep in mind is that these models are the worst they'll ever be. Give it a year or so and someone (either one of the big companies or someone building off their work) will release a model with both early-2025 level quality and >=100M context.

Agreed. Though I suspect progress on any concrete performance metric you care to predict will advance about as fast as you expect, and real-world practical uptake will be much slower than you expect (at least of you're Situational-Awareness-pilled, which is the vibe I get), because going from 5% to 95% on one of the benchmark tasks has limited practical value.

Isn't that basically the premise of Laplace's Demon?

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats