@aqouta's banner p

aqouta


				

				

				
7 followers   follows 0 users  
joined 2022 September 04 18:48:55 UTC

https://x.com/Spencer_Gray

Friends:

@aqouta

Verified Email

				

User ID: 75

aqouta


				
				
				

				
7 followers   follows 0 users   joined 2022 September 04 18:48:55 UTC

					
				

				
					

Friends:

@aqouta


					

User ID: 75

Verified Email

I didn't say "trade bad." I said "the US had unambiguously declined relative to other countries as a superpower" which is largely a measure of relative military strength.

If we're just talking military strength then it's much more debatable and the whole story is China. Ukraine isn't really a sign of a decline in super power status Russia has been meddling in and doing military actions in non-allied nations this whole time and if anything their failure in Ukraine shows that they are not a peer nation. The Iran hostage crisis was in the 80s. Besides just that China is very big and rising I don't really see evidence of a decline in US military super power status, we remain the only nation that has demonstrated an ability to project force far from our borders and it's always been the case that that ability had limitations(vietnam, korea, afghanistan).

And I don't think it makes the US mostly better off not to be the only superpower. The US is not better off because of the war in Ukraine. It is not better off because it needs to maintain parity with China in the Pacific. Those situations make the US worse off.

It's not my position that the rise of china's military might makes the US better off, it's my position that if you're talking about a declining united states I don't really think people's complaints are that in a hypothetical pacific theater China is marginally more able to push its weight around. Remember that they kicked us out of North Korea in the 50s. I don't think our relative military strength as compared to China can justify describing the US as in unceasing decline to the degree that we might as well have the democrats be overrun by actual communists.

I was mainly speaking economically. Arguments can be made about the amount of outsourcing and whether a different paths could have saved American manufacturing, I honestly think this is overstated, heavy protectionism hasn't prevented the decline of US automakers. And I've argued with Dase about China's not so friendly use of their own economic policy. But it's undeniable that we get a lot of cheap stuff out of china's rise. And not just cheap crap, good solar panels and electronics. If you want examples of things that have not only resisted inflation but dropped even in nominal terms stuff from china is a good start.

I think your case would be stronger if you argued that this was what was happening based on specific facts.

I mean the facts are the facts, we have a commenter upthread insisting that it doesn't matter whether his complaints about affordability are true because what really matters is the messaging or whatever. US Share of world GDP has been flat, it's the EU and Japan that collapsed to make room for China. The statistics do not show us in decline, I do not believe the doomers get to have the null hypothesis on this. I only bring up the french revolution as a counter example to if smoke then fire nature of your previous post, it's sufficient to show that smoke does not always mean fire.

I can speculate why the doomers believe what they do, I'm afraid it's going to be hard to distinguish from bolverism. Our culture is inculcated with an underdog narrative. The meek shall inherit the earth, the plucky rebels vs the tyrannical empire, the hopeless last stand of middle earth, Rocky, Miracle on Ice. Pick your western epic and there's a very good chance it's an underdog story. This is hard to square with being the most powerful and prosperous nation in human history. So we invent internal divisions, "maybe our nation itself is the evil empire" is one popular among lefties. "Maybe there is a shadowy extra-national(jews, cultural marxists, political Islam) force that has infected the state and we must free ourselves from it" is popular from the right. In either case a decline plays into the narrative. There is a handy justification for why the rebels actions are harmless(hurting a corpse doesn't count, the damage was over determined) and justified(only our radical action can restore us to glory).

Perhaps I've been reading too much the last psychiatrists lately as well but I see the pattern of "Frantic activity as a defense against impotence." Railing against the decline as some kind of systemic fait accompli frees the doomer from any obligation to set their own house in order. It feels like action to declare on a forum that the DSA might as well take over the democratic party but it doesn't actually require anyone to do anything but stew in their own pessimism.

You mentioned social media and that's also certainly a factor. Communications technology is another of the classic preludes to revolution and also often correlates with improvements rather than a decline. The number of times I've seen people unironically use the luxury of the family guy lifestyle, simpsons houses, or straight up watch ads from the 90s to argue for a decline in standard of living is incredible. Literally fictional evidence of a time that never existed. All while pelted with images of the lifestyles of the rich and famous that would simply have remained private in past times. This paragraph feels tired to write, none of it is novel, but it bears repeating. The rant from network(1976) remains true, the tube is not reality. The meta narrative of that movie is actually pretty interesting this light actually. It's a guy who's bit is basically the same as the doomers of today but in the 1970s. Could there be more proof of a this being a timeless trope? Anyways after this rant about the decline because of some specifics related to network television ownership shifting he is immediately won over by the new ownership and proceeds to preach their message instead of his previous radical one because the particulars really don't matter, only the sense of indignation matters.

I tend to agree with this but as phrased, this is mostly a generalized argument in favor of any policy that is painful over the short term and against policies that seem nice over the short term, (and as such an argument against, say, offshoring to China!) when I think the reality is slightly more complicated.

I agree things are more complicated but that's broadly my point. We need to actually have a sober discussion about which policies are good and which are bad and seriously consider the trade offs and downstream impacts. Populists are not capable of this, they default to conspiracy theories of elites who inexplicably want the decline.

The decline that is most readily apparent to me relative to 1990 is the decline of the United States as the world's sole superpower relative to other nations. Pretty much everything else can still be argued, but that is a very unambiguous decline in relative standing. Do you disagree?

This just seems like catch up economics, and in a way that mostly makes us better off. I wouldn't expect even a very well run US to permanently be so far ahead that even much more populous countries couldn't compete. It's a story of the rise of other places more than a decline at home. We're genuinely enriched by Northern European Optics and Medicine development. There is some argument I'm receptive to on protectionism in targeted cases but it's generally the case that we benefit from trade. We've declined in the sense that if we were the one healthy person in a home full of sick people that them recovering would represent a decline in our share of total strength.

I agree with this, but it seems to me that there is always someone peddling bad ideas. Why are they gaining traction? It seems to me that all the answers point to some sort of decline (although to be fair that could be something like your setback and not like FC's decline).

The French Revolution was launched when things were improving and fought most radically in the areas that improved the most. One really can't say where there is discontent there is decline.

But why pound the table with this sort of rhetoric when you have a track record of good policies that succeed in making people happier and decreasing their burden of living?

Because the good policies that succeed in making people happier and decreasing their burdens are like bitter medicine and unpalatable leafy greens while populist promise that actually sweet cocktails are not only tastier but healthier. Worse the presidential cycle and delayed impact of good policy often reward the populist elected because things weren't getting better fast enough the opportunity to enjoy the fruits of the investments from the good policy even while they hand out the seed corn. Rent control is going to reduce the amount that some new yorkers pay in rent and it will take some time for the costs to come due. But they will and the hangover will be nasty.

Okay, but I think that bad events make things worse, too. If you are claiming populism is making things worse (which I have some directional sympathy for), it's fair to ask what bad events it is responsible for. Unless you're suggesting that populists are making things worse by complaining about bad decisions made by non-populists.

Bad events are setbacks. The kind of thing we overcome. The decline I'm arguing against and would justify how we're better at dealing with communists coming to power now than we will be in five or ten years isn't caused by a recession, which while bad will pass.

Unless you're suggesting that populists are making things worse by complaining about bad decisions made by non-populists.

Populists are making things worse by getting elected and putting pressure on politicians do things like rent control or putting broad tariffs on our allies. By antagonizing and polarizing the country to such a point where we're seriously considering electing people to the senate that deny the legitimacy of our nation or our elections.

I am not sure I agree with this. I don't think (for instance) that the "Russia hacked our elections" narrative should qualify as populism, even though I definitely think it could at times collapse a complex topic (foreign interference in US elections) into a simple narrative about how things could be great if not for Russian schemes and all we need to do is implement counter-disinformation schemes and all free speech concerns were made up by MAGA-types to prevent good policy.

This one is borderline, mostly because it doesn't really include a call for some quick fix. "Russia hacked our elections and thus we should disregard them and install me as president" would be populist. If you wanted to call it populist rhetoric I wouldn't argue too hard but it's not central to my reading.

Wikipedia's stab at it is more "political stances that emphasise the idea of the "common people", often in opposition to a perceived elite" and I think I prefer that definition, even with the typo - it seems both less pejorative and less prone to "everything is populism."

What I'm saying is pretty compatible with this, as basically both populist wings have employed conspiracy theorizing to make some amorphous "elite"(billionaires, Epstein class, coastal elites, Capitalist, fascist, racists, patriarchy, international Jewry ect ect.) responsible for everything while declaring their own leadership as non-elite even if they're the president of the united states or a billionaire. In practice there aren't any two groups in existence you can't force into the plucky rebels vs empire framing that slop loving people won't slurp up if it gives them that "we're the righteous underdogs" tingles.

Here's where I point out that I said populism is making things worse, not that it's responsible for most bad events. There is a base rate of bad governance and populism as an influence makes them worse and more frequent. I don't really see the scale to be from populist to bipartisan so much as populist to stability. You can have bipartisan populist slop(no tax on tips, abolishing property taxes) and partisan status quo maintenance(Both sides have decided they like the filibuster when it serves them). Left and right wing populism find a lot of horseshoe like things to agree on, Epstein, Conspiracy theories, scapegoating of rich elites, hatred of the banks ect ect.

It's not necessarily populism to criticize things like Chinese outsourcing, certainly it's not populism to get in the weeds on bank regulation. Populism is when complex topics are collapsed down to simple narratives about how things could be great if not for ${outgroup_schemes} and all we need to do is implement ${simple_fix} and that second order effects are made up by ${hated_outgroup} to prevent good policy.

Obama condemned Ayers in his 2008 run and the connections between them are pretty sparse. There's a world of difference between Obama and this batch of explicit DSA candidates.

A lot of people think the 1990s were pretty good years, and they're my personal bet for America's high-water mark.

Family of mine died of disease we now have cures for in the 90s. Populism was less in some ways back then, but I'd much rather live in 2026 than in 1996.

Would you care to make a list of the worst political decisions made since 1990, and see how many of them were populist vs elite, or however you'd frame the dichotomy?

It's hard to really do this without it turning into some point by point relitigation of individual things that read as tit for tat partisan nonsense. I'll put forward many of the politics around BLM and the Trump Tariff fiasco as the kinds of fruits I see populism yielding. I'm not very interested in going point for point on which side was worse or who's tit justified which tat.

When were they marginalized in the first place?

I know republicans have accused democrats of being communists forever but they really aren't. They have trouble saying no to their leftist flank for sure, and I've always criticized them for that as a major failure, but there are actual major differences between the DSA crowd and the mainstream democrat crowd. Obama did not make space for people who praise Mao. If the Democrats never marginalized the DSA then how do you understand the current conflict in the democratic party? Why is the DSA primarying long standing members if they were there all along?

Ultimately politics just can't deliver what so many want from it. It's not the source of prosperity, it's not the fount of culture. It does matter and is necessary to settle coordination problems and keep things moving along. But there's this terrible cycle where politicians promise that the government will deliver what is not in its power to deliver, then they fail and populists think the reason for this failure is because the politicians are insufficiently diligent or forceful. The force vector making things worse from politics is founded in your sentiment upstream, that things might as well get worse, we might as well have an actual communist party so that all the masks come off and we can have this end of history battle between the two failed 19th century ideologies. It's a derangement. I'm of the bold opinion that the Democratic party getting worse is in fact bad, that it has been bad that the Republican party has gotten worse under Trumpism.

I don't know where you doomer types get this sense of unceasing decline from. In the last few years we've cured obesity, gained the ability to catch rockets that went to space, taught computers to speak, built cars that drive themselves and shrunk batteries enough that everything we have can be wireless.

To the degree that we're declining it's almost entirely due to these populist currents causing awful short sighted political decisions. No, it's actually not good that we're going to have brain dead communist populists fighting brain dead right wing populists over who can hold back progress more effectively in 2028. In would in fact be much better for everyone if the communists were kept marginalized.

There are only 350k machinists, I propose a moratorium on building tooling for this niche field of which I personally know no practitioners.

people aren't going to have to do anything and the market will eventually sort itself out.

The entire thing at discussion here is whether the government should intervene to prevent the market from clearing.

Communists are and he's aligned with the recent push for the DSA to pull a tea party into MAGA transformation of the Democratic party. Plenty of gripes are fair with the democratic party, but the DSA is much much worse.

Yes, no disagreement there. It raised his political profile and was embarrassing for his political opponents. Seems basically what's going on here, albeit I'm talking more about the social media posting than the sending which I have no reason to believe Israel is actually in charge of.

The appeal isn't quite the moment to moment mechanics of play, although I think they're better than a clicker would deliver. It's kind of like factorio in that you're building and improving an engine that turns map inputs into currency that can be spent improving your ability to turn map inputs into currency with a healthy dose of gambling dopamine hits. In live and recently refreshed trade leagues it's a very engaging puzzle to try and discover niches and farming strategies(which may be more or less synergistic with your build) where you can get more leverage on your time spent. The economy is very dynamic and responds to what builds are popular. If the biggest build of league is winter orb then some of the things that winter orb builds wants can be worth ten or a hundred times as much as they'd be worth in a different league so the inputs to get those items become valuable, the inputs needed to get the inputs needed to get those items become more valuable and so on.

In short it's a big puzzle box with a lot of different axis of skill expression from the ability to run maps more quickly to the knowledge of how to optimize a build and knowing what content to apply that build to. When it all comes together and you're earning absurd amounts of currency because some hunch about where the market would move it feels incredible. Also, its just satisfying to watch the stash tab fill up while you grind.

new Path of exile one league. I started playing in earnest in delirium league and winter orb is for the first time strongly viable and I've decided to go for a build. it's worked pretty well but I'm holding off on the power stacking shift with ralekesh boots. I parked my wealth in a hinekora's lock and some mirror shards a few days back and they've since uncharacteristically declined ~20% in value to my chagrin. Currently farming 8 mod map drops because it is profitable with a quick map clear time so I'm building up a big supply of charts for the new mechanic.

Amid the outrage on Twitter, old tweets resurfaced from Yair Netanyahu calling on Arabs and Muslims to invade Ceuta in 2019:

This seems very Trumpian, similar to Texas bussing migrants to northern cities.

I would like to flag that I would happily engage with Chairman Xi Jinping if he came on this forum and insulted me and hope that he does.

Bugs, especially exploitable bugs, are a consequence of the fact that on any objective scale our programming languages still suck (especially the fact that the ones which suck less on security tend to suck more on performance) and our programmers suck (yeah, including me, sorry), not because it's actually impossible to write a program that does what you want but doesn't also let you get p0wned as soon as someone figures out just the right corrupt input data to send.

This is heavily disputed to put it lightly. The ask here is something like all software everywhere be written/rewritten incredibly defensively and continuously upgraded as the models get better at finding and exploiting vulnerabilities. Even hardening what we have now, even with mythos helping via glasswing, this is long term project. The trouble is that the defender has to win every fight in every domain for every surface, while the attacker needs to only win once. The effort disequilibrium is massive.

initiatives like Project Glasswing are a pretty good way to keep the short-term equilibrium in favor of the defender too.

Yes, and I've personally read reports on my software that have gone through this program. Importantly the defends have better tools here.

If you're also thinking about e.g. biological security ... well, yeah, there is a chance we'll all be dead soon.

Yes. Seems pretty bad. I personally do not want to die.

Since I've blocked you on X (deservedly, seeing the unearned confidence in the latter half of your post)

Honestly man, I had a lot of respect for you and always approach in good faith, part of me still hopes this is some kind of crazy misunderstanding of mistaken identity. When you blocked me you implied I was a Trump supporter or something, which is just confusingly not true. And if smug confidence is a sin, well I guess you've already cast the first stone.

margins on inference at frontier labs are obscene

Yes, I've argued this point before on this forum several times against people convinced the frontier labs were losing money even on inference.

Frontier AI is wildly profitable, it has high software margins, and Chinese distillation is powerless to dent that.

No, actually, sating demand does in fact dent profitability of selling a scarce resource even if it only goes from extremely obscenely profitable to merely obscenely profitable, and the projected profitability is what dictates investment levels. A competitor that copies your R&D at a lag shifts your optimal resource allocations from research to buildout optimization. Research is what accelerates the frontier thus Chinese distillations slow acceleration.

> The only reason they're in the red is that they are, indeed, reinvesting tens of billions into scaling compute for R&D and larger-scale training. How much capital do they need to keep accelerating?

This is literally the whole of the argument. Nothing else in this post matters at all.

Enough. This is all noise

agreed.

All plausible negative impact of Chinese labs, distillation or whatever, can be negated by roughly 2 months of regular progress.

Wait, how? do you mean 2 months of the ability to reap high margins with paused competition?

The only serious question is how the multi-trillion valuation pie gets sliced – how much goes to "labs", how much to Jensen, and how much to clouds serving a mix of closed and open weights.

These are all considerations that people consider carefully when they decide how many trillions of dollars to put into building out datacenters. Dario famously underbuilt in the last cycle on these mundane economic concerns.

On the more important topic though, and you're certainly allowed to only reply to the part of my post you disagree with, but I'm much more concerned with the, clear to me fact, that open weight models are unsafe and nothing can make them safe. The deceleration thing might even be good given this. Even given just mundane uplift I can't seem to get away from the offense/defense equilibrium disruption. Those are heavy dice to roll.

One angle I rarely see discussed is that open weight models are inherently decelerationist for the frontier. They greatly reduce the profit motive to push the frontier, especially if distillation is allowed because the make it much harder to recoup the investment thus reducing the capital available to invest. Ultimately though they're definitionally unsafe and can never be made safe, any work put in to align them can be sanded off with post training and even with just mundane levels of uplift the idea that the offense/defense equilibrium will favor the defender in most areas, let alone all areas is just impossible to believe.

As long as they aren't destroying books that are actually rare then I genuinely can't imagine the thought process of someone who thinks about this for more than 10 seconds and doesn't come to the conclusion that this is fine. The worst thing I can think of it is that people have to go through this whole expensive and time consuming rigamarole just to get copies of books that almost certainly have all already be digitized. Almost all of these books went from a digital format to a printed form and were reconverted to a digital format, it's absurd to me that people get worked up about this process on some kind of moral level.

I agree that the grinders exist, and they care about the loot and juice systems being rich enough that they can keep climbing the grind for better gear that will make it so that you can more efficiently grind for even better gear cycle. Where I disagree is that this really has any negative effect on the casual player who spends 12 hours on the acts. Because you definitely don't need to grind in the acts to keep moving forward, in fact it's the opposite, almost every new player kills way more monsters than they need to.

And it seems like the grinders are the market being catered to at the moment (PoE, I'm looking at you, because they have been around so long they have dedicated players who've played the game for years as it developed and who can now whip through all the levels fast to get to endgame maps and are then bored if their maxxed out builds with the extra-special gear don't find it enough of a challenge).

This complaint doesn't really make sense in poe, the exp penalty for being overleveled means if you're slow and don't do the sweaty run past almost everything strategy you end something like perpetually 3-4 levels over the zone which is fine, you'll be stronger than you need to be, it's like a built in auto-difficulty adjuster.

The vast majority of DoorDash traffic. I understand the value when one is three sheets to the wind, or ill, or whatever and getting food delivered becomes more valuable. I don't understand how it accounts for such a large share of orders that it's a huge element in the service economy. I'd think the average person would order delivery once a month or less, it appears to be much more than that.

We don't own a car so if we want to stay in and not cook we turn to it once or twice a week. It's roughly as expensive as going out to eat which is another thing we do quite often. It's definitely not frugal but surely the convenience should explain the appeal even if you wouldn't want to pay that mcuh.

I'm not the one that defined the problems with AI in relationship to Malevolent AI Gods, Singularities, Outcome Pumps, Roko's Basilisk, Grey Goo/Paperclip maximizers, Magical Genies, Shoggoths, Goodhart demon, etc. These are all inherently Sci-fi ideas. I have no idea how you seem to think any of these relate to how Neural Networks, or Bayesian Networks, or any other of the myriad ideas ML/AI engineers have designed.

These are all intuition pumps and metaphors for explaining the actual intuition. Goodhart isn't a science fiction concept, he explained that under optimization pressures a system that satisfies your stated objective can fail your intended objective and in fact the better the system gets at optimization the better it gets at trading off your intended objective in pursuit of better achieving your stated objective. This thread started out because of an example of this very principle playing out in reality.

Again this assumes that this eventually AI will be sentient

No. And this is the third time it's been explained to you so let me be extra blunt. No part of alignment requires or cares about sentience, wants or experiences for the system. The lights need not be on. You didn't even have to click into the short essay, the paragraph I posted covers this.

These aren't "misalignments" they are are classic errors on a non-perfect model in a system lacking in classic control theory.

Error implies the quantity shrinks as you train it more, this is true of hallucinations, they've been reduced. A hallucination doesn't help you achieve a target goal. This is not true of misalignment problems because misalignment is reaching the goal through unintended and harmful pathways.

The coastrunner boat spinning infinitely scoring forever isn't failing to minimize loss. It's minimizing loss perfectly. But the loss target was not aligned with what we actually want.

Your #2/#3 ignores a failure mode monotonically worsened by the thing that fixes every other failure mode, which is why you keep having to reach for "someone forgot a unit test." All the unit tests pass, it's performing exactly as expected. You can't expect to build smarter and smarter scaffolding to swatt it away from optimizing around your guard rails. Eventually it gets strong enough and you lose, game over.

We don't scream that "software algorithms are misaligned" because some coder forget an unit-test on an edge case.

Grep doesn't go looking for novel ways to satisfy your query, it's simple executing a list of instructions. Deterministic software fails when it runs into states you didn't specify at compile time. Optimizers fail when where you specified correctly but it turns out the specification wasn't what you actually wanted. They're a different class of problem.

#1: Sure, AIs get more capable all the time, how do you know there isn't an asymptote?

Obviously we don't and I'm pretty careful to specify, as I did in the previous comment, that this is one of two valid outs. If you think these systems won't get much more capable then sure, although I'd like you to demonstrate that because people predicting it have been very wrong for a while now. But if that's your contention then I think your whole argument would be much different. The whole ask of the AI safety crowd is essentially to impose an artificial asymptote.

#3: Congratulations you've converged on a fundamental question for any human system

Yes. And we have not solved this problem with humans either although we've spent a tremendous amount of effort attempting to better mitigate the problem. contract law, courts, fiduciary duty, audit, separation of powers, criminal enforcement, a state monopoly on violence. Half the discussions on this forum boil down to disagreements on how exactly we're failing to solve this problem. And that's between humans who are individually squishy, mortal difficult to scale(see corporations and their shenanigans in the 17th century for what happened immediately after we found novel ways to scale human effort), must sleep and have evolutionarily built in game theory instincts.

Sure and there are christians, mormons and muslims in the physical sciences. I don't begrudge people their religious beliefs. If a frontier AI researcher wishes to belief in alignment-problems that his/her/their belief. The Bay is quite literally for Rationalists like Utah is for Mormons. I can still think it's a silly belief and I still think the word misalignment has been invented to describe a problem with system error by a bunch of non-engineers. And the lack of actually solving the problem, or even making progress on it, is indicative of a general grift specifically for something like MIRI.

This proves far too much. It labels all speculation of dangerous outcomes as fundamentally religious. Is climate change fundamentally religious? Fear of demographic replacement? Pick your conservative or liberal fear. This position dooms to you dying out to the first large scale novel problem you encounter.

And I dispute heavily that it's a bunch of non-engineers that are concerned. Some of the biggest voices in the movement are the frontier engineers including the CEO of the lead frontier lab. If you're going to dismiss anything the non-engineers say as not credible because it comes from non-engineers and what the frontier engineers say as non-credible because it's "hype" then you've made your position unfalsifiable.

We are not. Sci-fi predicts innumerable future realities, most of which to not occur, it predicts an unmeasurable amount of future technologies, most of which never get developed, and it almost rarely ever actually predicts how those actual technologies will work. We exist in a reality shaped world and bad sci-fi fans mistake aesthetics for substance

You invoked sci-fi, to sneer at ideas. I'm happy and prefer to not compare things to science fiction. But you're using "scifi" as a talisman to avoid engaging with any speculation whatsoever.

If you wanted me to further read something you should have linked it.

I did and it remains linked.

Further more I'm not sure how this magical second analogy makes the argument any better. If you'd like to make an argument rather than quoting scripture at me I am open to hearing one.

The argument is made at length in the piece. And any many other places, it straight credulity that you have not seen it.

Start from the basics of existing technology and make actual arguments based on the current reality and the projected reality from our current understanding of Artificial Intelligence. Don't start in make-believe land and attempt to redefine reality as leading to it.

The argument is simple:

  1. AI is getting more capable.
  2. capabilities include the ability to do harm, often as a side effect of pursuing some other goal
  3. We have no reliable way to ensure that the inevitable trade offs of harm and pursuing goals aligned with the true interests(As distinct from the technical interests represented in a prompt) of the one who invokes it, yet alone humanity as a whole.
  4. At a certain level of capabilities those trade offs might include the permanent disempowerment of humanity or our extinction.
  5. At some level of capability an AI will be invoked that will find the utility function maximizing route includes an acceptable trade off that results in some, to humans, extinction level threat.

Events like this demonstrate that the pursuit of some goal that to us is of trivial value, passing a cyber security benchmark, warranted the trade off and harms of breaking another organization's cyber defenses. A perfect microcosm of the fear of some future catastrophic event. To dismiss this I think you have to either contest that AI will not become much more capable or present some strong reasoning for how we will solve alignment.

Because it's not possible to solve.

Then we should not build it.

Forcing Sentient Beings

I thought the sneer you had was that alignment people were ridiculously thinking they were sentient? It seems like something you believe more than them.

And to further my opinion can you provide any evidence that in the past decade of AI Safety research have those researchers ever produced anything more than words on paper. The evidence points to this being a grift, has MIRI produced a novel ML model that is more safety conscious and performs any task well, have they produced an "alignment" mechanism or algorithm that actually works?

Believers in some version of their cause are currently littered throughout the frontier labs. You have an impossible standard here where you blame them if they're on the frontier, for clearly not believing in what they preach, or blame them for not being on the frontier as then they must be uninformed cranks. No way to win, you never have to actually think about it. But yes, of course they don't have an alignment mechanism, their whole point is that the problem is incredibly hard and that we have no solved it, that we might need to spend decades solving it, but that the alternative is everyone dying.

The reason I mock "misalignment" is that because it is used as this nebulous term by a bunch of sci-fi cargo cultists.

you best get used to Sci-fi shaped predictions because we're in a sci-fi shaped world.

If "misalignment" is my sentient AI model doesn't do what I want, well that's assuming the conclusion that model is already sentient. Note in the Yud's analogy, it's genies that are the stand in. Genies are already sentient, they can make decisions on how they listen to you. A genie is not a non-sentient wish granting device that tries to fulfill you wish to the best of its ability. That would be the actual analog to an LLM-Agent. This is the problem with analogies, they require a level of similarity between the two abstractions, when that similarity doesn't exist, the analogy, no matter how clever, does not apply.

If only you had read the next sentence!

Luckily you have, in your pocket, an Outcome Pump. This handy device squeezes the flow of time, pouring probability into some outcomes, draining it from others.

The Outcome Pump is not sentient. It contains a tiny time machine, which resets time unless a specified outcome occurs. For example, if you hooked up the Outcome Pump’s sensors to a coin, and specified that the time machine should keep resetting until it sees the coin come up heads, and then you actually flipped the coin, you would see the coin come up heads. (The physicists say that any future in which a “reset” occurs is inconsistent, and therefore never happens in the first place—so you aren’t actually killing any versions of yourself.)

You refuse to actually engage in any of the arguments being made and are dead stuck on the prior that it's all nonsense, it's epistemic closure.

Yud's whole "misalignment" also just applies to humans

Yes, humans are not generally aligned. If you read histories of what humans have gotten up to then this seems pretty obviously to be the case.

don't due exactly what you want without you enumerating it exactly.

You still don't seem to grasp what is meant by alignment. It's specifically even stricter than that! That's the whole point. It's not enough that they do exactly what you ask, because for complicated enough problems you need it to be much much better than just technically doing what you ask.

It doesn't need a fancy sci-fi term, and it literally isn't solvable. It hasn't been solved in the history of the human race.

Correct, it's a very very hard problem. You are in full agreement with the AI safety people who think there is a good chance we're on the path to annihilation, you just inexplicably seem to think it isn't a big deal and refuse to elaborate on why besides sneering and not engaging with the arguments.