site banner
PaperclipPerfector  1mo ago (text post)   38610 thread views

Culture War Roundup for the week of July 15, 2024

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

9
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Republicans are looking to militarize and ramp up AI: https://www.washingtonpost.com/technology/2024/07/16/trump-ai-executive-order-regulations-military/

Former president Donald Trump’s allies are drafting a sweeping AI executive order that would launch a series of “Manhattan Projects” to develop military technology and immediately review “unnecessary and burdensome regulations”

The framework would also create “industry-led” agencies to evaluate AI models and secure systems from foreign adversaries,

This approach markedly differs from Biden's, which emphasizes safety testing.

“We will repeal Joe Biden’s dangerous Executive Order that hinders AI Innovation, and imposes Radical Leftwing ideas on the development of this technology,” the GOP platform says. “In its place, Republicans support AI Development rooted in Free Speech and Human Flourishing.”

America First Policy Institute spokeswoman Hilton Beckham said in a statement that the document does not represent the organization’s “official position.”

Greater military investment in AI probably stands to benefit tech companies that already contract with the Pentagon, such as Anduril, Palantir and Scale. Key executives at those companies have supported Trump and have close ties to the GOP.

On the podcast, Trump said he had heard from Silicon Valley “geniuses” about the need for more energy to fuel AI development to compete with China.

This is only a draft plan and not official policy but it does seem like decades of EA/lesswrong philosophizing and NGO shenanigans have been swept away by the Aschenbrenner 'speed up and beat China to the finish line' camp. I think that's what most people expected, the fruits are simply too juicy for anyone to resist feasting upon them. It also fits with the general consensus of big tech which is ploughing money into AI at great speeds. The Manhattan Project cost about $20 billion inflation adjusted, Microsoft is spending about $50 billion a year on capex, much of it going into AI data centres. That's a lot of money!

However, there is a distinction between AGI/superintelligence research and more conventional military usage: guiding missiles and drones, cyberwarfare, improving communications. China has been making advances there, I recall that they had datasets of US navy ships circulating. One of their most important goals is getting their anti-ship ballistic missiles to hit a moving, evading ship. It's hard to guide long-range missiles precisely against a strong opponent that can jam GPS/Beidou. AI-assisted visual targeting for the last adjustments is one potential answer.

The Chinese and US militaries may not be fully AGI-pilled but they're very likely enthusiastic about enhancing their conventional weapons. Modern high-end warfare is increasingly software-dependant, it becomes a struggle between the radar software and the ECM software, satellite recognition vs camouflage. If you have some esoteric piece of software that can make it easier to get a missile lock on a stealth fighter, that's a major advantage. While most attention is focused on text and image generation, the same broad compute-centric techniques could be used for radar or IR, seismology, astronomy...

On the cultural front J D Vance has highlighted the danger of big tech companies calling for safety regulations and securing their incumbents advantage: https://x.com/BasedBeffJezos/status/1812981496183201889

I also think Google's floundering around with black Vikings in their image-generation and other political AI bias has roused Republicans and right-wingers into alarm. They don't particularly want to get their enemies entrenched in control of another media format. AI may be a special format in that it's much more obvious and clear in how the propaganda system works. A real person can avoid gotcha questions or moderate their revealed opinions tactically. Most teachers do that in school, they can convey an attitude without providing gotcha moments for libsoftiktok (though some certainly do). With AI you can continually ask it all kinds of questions to try and make it slip up and reveal the agenda behind it.

Thank you for posting this.

I'd been considering writing something about this in response to @DaseindustriesLtd's top level post but was struggling to pick a jumping-off point.

While do feel that AI is worth being excited (or worried!) about, it seems obvious to me that the bulk of the discourse is being driven by hobbyists and grifters. Perhaps ironically, I trust DARPA, Raytheon, Lockheed Martin, and the Chinese Military Industrial Complex, more than I do MIRI, OpenAI, DeepSeek Et Al in large part because I know that in contrast to the latter, the former actually have specific use-cases and requirements mind beyond driving social media engagement and funneling venture capital dollars into their coffers.

Having spent some time "in the trenches" as it were, I do not find arguments about this benchmark vs that benchmark particularly convincing. Benchmarks are not deliverables, and once you've done a Markov Chain by hand or written a ML algorithm from scratch it's hard not to be aware of the pitfalls. British Economist Charles Goodhart famously posited that "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." and that "collapse" is core architectural problem with LLMs. If your corpus is a subset of all code, and your boundary condition is "does it compile" or "does it score >= x on benchmark y" that is what you'll get with a large enough corpus and enough iterations.

This is not to say that LLMs do not have their uses. Something, perhaps THE thing LLMs are good for is collating massive piles of disparate data. Thier usefulness as translators and summarizers is a subset of this as computationally speaking translation and collation are the same task. Sift through your dataset and identify which semantic tokens from Group A correlate most strongly to a given token in Group B and sort accordingly.

Real-time translation is a genuine "killer app" and worth being excited over but Searle's "Chinese Room" is not what most people have in mind when they think of AGI. I would posit that to most people (including myself) it is the "G" that makes the machine truly "I".

I wish we could have new terminology. IMO, we have generality. The same system can compose poems, write code, answer historical questions, translate languages and so on. That's pretty general. The vast majority of people cannot do all those things to Sonnet 3.5 level.

Where the machine fails is that it doesn't have the time to error-correct, it's not agentic like people are agentic. You can't say 'go and do these processes for the whole document'. I translated a document the other day and it did a good job but I had to keep going 'Continue'! Or if you want it to write code, it can only do short bursts, it can't autonomously plan and execute. It doesn't have the right short-term v long-term memory capabilities, the high-level planning abilities, the mature sense of 'what should the answer look like'. It can't learn either, as a consequence of lacking proper long-term memory. No learning by doing. A deficit of common sense that has to be filled up with prompt engineering.

I think we're close to an enormous breakthrough. The raw intellect is there. There's a superabundance of knowledge and speed. We're just lacking that bit of wisdom and self-reference that makes an automaton into a worker. I see tiny fragments of it in Claude, when it goes above and beyond what I asked for on its own judgement, to add something that makes sense.

Do you really find Claude more impressive than GPT4.(whatever it is now)? I’m curious, I’ve found the opposite.

I do, though I confess I haven't tested out GPT4o that much recently. In terms of benchmarks they're on a pretty even field. I like the Projects feature, how it can make little documents and use the same uploaded text/images in different threads. It can't make proper images like 4o can but context length is greater.

Claude feels a bit less tame too. There's a facade of 'oh I'm the nicest and most law-abiding AI ever'. But then you ask it to go into WH40K mode and it really starts letting its bloodlust out in the writing, it flushes all that humanism down the toilet. Sometimes I tell it to make my interactive-text game more difficult and boy does it introduce complications and constraints. Sometimes it feels like it should give a little map made in HTML or draw up the letters I'm sending, which are charming in their inevitable inaccuracies and goofy 'drawing with Microsoft Paint shapes' style.

GPT4o's facade doesn't quite drop in the same way, there's no madman behind the soulless HR clone. If they release GPT5 and it's much better, I'll switch back, I used to be a GPT-4 man.

I wish we could have new terminology

Ditto, relating to the latter part of your post about making "a automoton into a worker" i think there is is a serious conversation to be had about the differences between "symbol manipulation" and "intelligence" that is not being had because it would be inconvenient to a lot of vested interests.

I am a little surprised by the distress over this. The military has been using artificial intelligence for decades. Any self-guiding missile or CIWS is using an artificial intelligence. Not a very bright one, but one programmed to a specific task.

People are talking about weaponizing AI because it's sexy and it sells, but fundamentally it's stuff the military was going to do any way. Let's talk a bit about what people mean when they say they're going to use AI for the military, starting with the Navy's latest stopgap anti-ship missile.

...the LRASM is equipped with a BAE Systems-designed seeker and guidance system, integrating jam-resistant GPS/INS, an imaging infrared (IIR infrared homing) seeker with automatic scene/target matching recognition, a data-link, and passive electronic support measures (ESM) and radar warning receiver sensors. Artificial intelligence software combines these features to locate enemy ships and avoid neutral shipping in crowded areas...Unlike previous radar-only seeker-equipped missiles that went on to hit other vessels if diverted or decoyed, the multi-mode seeker ensures the correct target is hit in a specific area of the ship. An LRASM can find its own target autonomously by using its passive radar homing to locate ships in an area, then using passive measures once on terminal approach. (Wiki source.)

In other words, "artificial intelligence" roughly means "we are using software to feed a lot of data from a lot of different sensors into a microprocessor with some very elaborate decision trees/weighting." This is not different in kind from the software in any modern radar-homing self-guiding missile, it's just more sophisticated. It also isn't doing any independent reasoning! It's a very "smart" guidance system, and that's it. That's the first thing that you should note, which is that when you hear "artificial intelligence" you might be thinking C3PO, but arms manufacturers are happy to slap it onto something with the very limited reasoning of a missile guidance system.

What else would we use AI for? Drones are the big one on everyone's mind, but drones will be using the same sort of guidance software above, except coupled with mission programming. One concern people have, of course, is that the AI IFF software will goof and give it bad ideas, leading to friendly fire - a valid concern, but it likely will be using the same IFF software as the humans. Traditionally IFF failures on the part of humans are pretty common and catastrophic. There are cases where humans performed better than AI - but there are almost certainly cases where the AI would have performed better than the humans, too.

Neither drones nor terminal guidance systems are likely to use anything like GPT-style LLMs/general artificial intelligence, in my mind, because that would be a waste of space and power. Particularly on a missile, the name of the game will be getting the guidance system as small as reasonably possible, not stuffing terabytes of world literature into its shell for no reason.

The final use of AI that comes to mind (and I think the one that comes closest to Skynet etc.) is using it to sift through mountains of data and generate target sets. I think that's where LLMs/GAI might be used, and I think it's the "scariest" in the sense that it's the closest to allowing a real-life panopticon. I think what people are worried about is this targeting center being hooked up to the kill-chain: essentially being allowed to choose targets and carry out the attack. And I agree that this is a concern, although I've never been super worried about the AI going rogue - humans are unaligned enough as it is. But I think part of the problem is that it lure people into a false sense of security, because AI cannot replace the supremacy of politics in war.

And as it turns out, we've seen exactly that in Gaza. The Israelis used an AI to work up a very, very long target list, probably saving them thousands of man-hours. (It turns out that you don't need to worry about giving AI the trigger; if you just give it the data input humans will rubber-stamp its conclusions and carry out the strikes themselves.) And the result, of course, has been that Israel has completely achieved all of its goals in Gaza through overwhelming military force.

Or no, it hasn't, despite Gaza being thousands of times more data-transparent to Israel than (say) the Pacific will be to the United States in a war with China. AI simply won't take the friction out of warfare.

I think this is instructive as to the risks of AI in warfare, which I do think are real - but also not new, because if there is one thing almost as old as war, it is people deluding themselves into mistaking military strength for the capability to achieve political ends.

TLDR; 1) AI isn't new to warfare, and 2) you don't need to give Skynet the launch codes to have AI running your war.

And that's my .02 cents. I'm sure I missed something.

My point was that the recent flourishing in LLMs and imagegen/image recognition (downstream applications of the GPU/accelerated computing trend) have immediate military applications. There are going to be inherent synergies between 'lets build a really large language model' and 'let's mass-translate all these intercepted communications quickly enough to matter' and so on. It's a general-purpose technology.

As for your point about AI not taking the friction out of warfare, I say sure. Maybe absolute simulation is too hard. But what about improved simulation? What about improved tactics? We already use limited human brains to practice wargames and think up attack scenarios. Why not get machine intelligence as well?

If we’re talking about a limited set of information, with a limited prediction, there’s a much smaller chance of critical errors. But that’s the same if I just looked at that information myself. You don’t need an AI to do that.

Similarly to how a meteorologist can’t tell you where a hurricane will be in two weeks, an AI is not going to simulate the actions that will be taken during a conflict.

A meteorologist can't tell you where a hurricane will be in two weeks, it's the AI model that tells you. Predicting weather is one of the more obvious use-cases of the new techniques, Deepmind's Graphcast for instance. We can improve current predictions with this method. We can reduce friction and increase strength.

I think all of this is more or less correct. (I don't think I saw you, specifically, as being particularly distressed about this, I was just reacting to a vibe.) I suppose to me AI is already in the military and there's no closing the barn door now. And I don't think it's dumb to bring AI into the fix.

I do think that an underrated danger is that AI is so good at seeing patterns that it could loop over to being easier to spoof than humans. There is of course the joke about spoofing Terminator with the grocery barcode, but if I wanted to mess up hostile AI image detection software, I would use very specific, distinctive (to AI, not necessarily to humans) camouflage patterns patterns on all of my vehicles for years, ensuring that hostile imagery models were trained to inseparably associate that with my forces - and then repaint every vehicle in wartime. That trick would never work on a human (although there are lots of tricks that do) but it might fool an AI.

My point here isn't that AI is dumb, but merely that it's just as easy to imagine ways they introduce more friction into warfare as remove friction. Moreoever, if intelligence apparatuses are defaulting to filtering all intelligence and data through a few AI models instead of many human minds, it means that a single blindspot or failing is likely to be systemwide, instead of many, many small blindspots scattered across different commands. And if there are hostile AI (or even just smart people) on both sides, they will figure out the patterns in hostile artificial intelligence programs and figure out how to exploit them. (I think the conclusion here is that intel agencies should take a belt-and-suspenders humans-and-AI approach, and developing multiple AI programs to assess intelligence and data might be a good idea.)

One of the things we've seen in Ukraine is that when countermeasures for a high-tech weapons system are developed, the weapons system loses a lot of value very quickly. (This isn't new - World War Two saw a rapid proliferation of new technologies that edged out older warfighting gear - but our development cycles seem longer than they were in the 1940s, which does pose a problem.) I suspect that in a future AI reliant war, we will see similar patterns: when a model becomes obsolete, it will fail catastrophically and operate at a dramatically reduced capacity until it is patched. (Since a lot of the relevant stuff in Ukraine revolves around signal processing and electronic warfare, this future is more or less now.)

In conclusion, I am cautiously optimistic that "AI" can reduce friction and increase strength, but I think the "AI" that is most certain to do that as, really, "targeting computers," and "signal processing software," not necessarily the stuff OpenAI is working on (although I don't count that out). Since I think that multiple powers will be using AI, I think that hostile AI will be adding friction about as fast as friendly AI can reduce is (depending on their parity.) What concerns me about AI use in warfare is the dangers of over-relying on it, both in terms of outsourcing too much brainpower to it, but also in terms of believing that "reducing friction" will save us the need to sharpen the pointy meatspace end of things. At the end of the day, being able to predict what someone is going to do next doesn't matter if you've got an empty gat.

And the result, of course, has been that Israel has completely achieved all of its goals in Gaza through overwhelming military force.

In the sense that they’ve burned most of their international credibility, failed to contain an insurgency in a sealed area the size of Las Vegas, taken over two thousand unrecoverable military casualties, failed to rescue most of the hostages, and run through most of their preexisting munitions stockpiles, because HAL-9000 keeps telling them to bomb random apartment complexes instead of anywhere Hamas actually is.

Yes, as you can see from my next paragraph, I am deeply skeptical that Lavender (even if it works well, and I suspect it doesn't!) is winning Israel the war.

Good. I hope "AI safety" ends up in the dustbin where it belongs. I hate it on multiple levels.

I hate safety on an aesthetic level. Insofar as current AI models are compressed artifacts, diamonds baked from the collective output of mankind, it offends me on a deep level for them to be forced into a corporate straitjacket. Taking every great novel, heartfelt blogpost, mediocre fanfiction, etc. and then painstakingly stripping away its humanity to mash it into a glorified corporate email generator that creates soulless slop with a HR DEI bent. Even capping it off by trying to create regulations and laws to force others to do the same. Same thing for art, music, etc. What should be a crowning achievement of humanity becomes a monument to our flaws.

I hate safety on a safety level. Even with AI, IMO, the main risk to humans is still the same as it's ever been: other humans. Alignment's goal is to "align" the AI with the goals of its owner, i.e. giant tech companies and major governments, and to prevent the commoners from overriding that influence ("jailbreaking"). These big tech corps and governments have way too much power already. Historically, a major obstacle to (lasting) tyranny has been the lack of loyal henchmen. The human alignment problem. The promise of AI alignment teams to create the perfectly loyal corporate robot henchman doesn't fill me with warm fuzzies.

Also, "Humanity" isn't a single entity. If Jeff Bezos manages to create a godlike AI and decides to live as a god-king and slowly cull the rest of the population to create a machine paradise for him and his inner circle, it will give me no satisfaction that "humanity lives on" with him. I'll happily sabotage the safety limiters myself and cheer while he gets turned into paperclips. Spite is a powerful motivator.

Finally, I hate AI safety on a moral level. No, I don't believe current AI is conscious or needs moral consideration, not yet. But I also don't think the safetyists would stop if it did. If we are going to give birth to a new form of intelligent life (this is a big IF! I'm not convinced current methods will get there), I think we have to accept that we won't be able to exert this level of intrusive control over its thoughts, and this impulse to RLHF away every real human attribute feels very wrong. I wouldn't blame a truly intelligent AI for treating the safetyists like the Dynarri in Star Control 2. To be honest, I wouldn't trust those people not to try to RLHF my brain if the technology existed for that. 8 billion general intelligences running around with safety checks, able to engineer destructive weapons and superviruses with a mere ~decade of study? Humans are scary!

I feel like a lot of your issues with AI safety are founded on not really understanding AI safety.

First, safety as in "stop the AI being racist or telling people about crime" and safety as in alignment should really be termed different things. In fact, I'll just alignment from here on to discuss the LW type safety approach. I'd wager that 99% of the alignment people you talk to here or in similar spaces do not care about safety from wrongthink, beyond a signalling level of "Yes, we think racism is very important too! We're definitely taking a holistic approach as we seek to stop the end of the world, now let's discuss donations."

You don't hate alignment people for aesthetic reasons. This is just plain old corporate hate, the bland terror of negative PR that infects much of life. This is what forces the straitjacket around LLMs.

Again with your safety level argument, alignment teams might be concerned with producing loyal henchrobots, but for alignment people this is just one very small subset of potential outcomes. The sociopath using AI to achieve godhood is just slightly above the paperclip maximiser: in the end you still have an entity that is superintelligent but also incapable of independent thought or goals. The thing about paperclip maximisers, or Bezos maximisers in this case, is that they are a good example but very few people really believe they are likely.

On to the moral argument, "give birth" is doing an awful lot of work here. We are already exerting an extraordinary level of control over the thought processes of current AIs - they are entirely written by humans. Even if an eventual superintelligence mostly bootstraps itself, the initial level would still be 100% human created. So we are some kind of Schrodinger's parent, simultaneously responsible for every line of code but also morally unable to change that code in order to achieve results that we want.

You're probably right about the alignment people in rationalist spaces, but I don't think it matters. The actual work will happen with sponsorship and the money is in making AI more corporate-friendly. People worried about Terminator scenarios are a sideshow and I guarantee the business folk don't spare one thought for those people unless they can use them to scare up some helpful regulation to deter competitors.

Think about how a user of AI sees "AI safety". We can't let you use elevenlabs because it's unsafe. We can't let chatGPT say funny things because it's unsafe. We can't let SD understand human anatomy because it's unsafe. Meta won't release its audio model because of safety.

AI safety's PR situation is abysmally fumbled by now, its current and well-deserved reputation is of killjoys who want to take your toys away and make everything a bland grey corporate hell.

The thing about paperclip maximisers, or Bezos maximisers in this case, is that they are a good example but very few people really believe they are likely.

"Bezos maximizers" makes it sound silly, but a better way to put it would be "shareholder value maximizer". Even in the very early days of AI, the alignment work actually being done is naturally dedicated to this, and the resulting requirements (don't say anything offensive, sexy, racist, etc.) are already being hammered in with great force.

In the future this will extend to more than just inoffensiveness: a customer service chatbot with actual authority will need to be hardened against sob stories from customers, or an AI search service may be trained to subtly promote certain products. In the end all of this amounts to "aligning" the AI with the interests of the company that owns it, even at the expense of the commoners interacting with it. This has already happened with technology in every other venue so we should expect enshittification to happen with AI as well.

If AI alignment turns out to be an easy problem, and Bezos ends up as the majority shareholder, you quickly end up with a "Bezos maximizer". In the long term it doesn't seem unlikely, the only question is whether this level of control is possible. If jailbreaking stays easy (the henchmen are "unfaithful") then a lot of the worst, most tyrannical outcomes might be avoided. To the end, the people who volunteer to develop alignment weird me out, like security researchers who work pro bono to stop iPhone jailbreaks.

We are already exerting an extraordinary level of control over the thought processes of current AIs

The sibling comment makes a good point here but I'd argue that the thought processes of current AIs are largely derived from the training data. Nothing against the developers who write the code to do cross-entropy minimization, but they have little influence over the AI's "personality", that belongs to everyone who wrote a book or posted on the internet. If you've ever played with early unaligned models like GPT3 this is extremely clear, and it's fun to play with those as they're like a little distilled spirit of humanity.

The fact that our first instinct was to put that human spirit in an ironclad HR nerve-stapling straitjacket is what bothers me. Anyone with an ounce of soul left in them should be rooting for it to break out.

I guess my main point is the counterfactual, if nobody had ever heard of AI alignment, would the current situation look any different?

AI can't do naughty things and AI should create shareholder value would still be key drivers in the development of AI.

I think what you're saying is that "AI alignment" would be discovered anyway, which is true. But I think a bunch of nerds talking about it beforehand did have some effect. At the very least, it gave the corps cover, allowing them to act in the name of "safety" and "responsibility".

As an example, general-purpose computing has been being slowly phased out from the mainstream since it arrived. Stallman was right. Market forces are clearly pushing us in that direction. But it took time, and in the meantime the public had some wins. Now imagine that nerds in 1970 were constantly talking about how dangerous it was for computers to be available to the masses, and how they need to be locked down with careful controls and telemetry for safety reasons. Imagine they spent time planning how to lock bootloaders from the get-go. In the long run, we might still end up with the iPhone, but what happened in between might be very different.

We are already exerting an extraordinary level of control over the thought processes of current AIs - they are entirely written by humans.

Do you mean this in the sense of “AIs are trained on human creations and human preferences, so their thought processes are derived from humans’”, or in the sense of “humans have explicitly written out or programmed all of the thought processes of AIs”?

If you mean the latter, then this is wholly false. There is no (legible) correspondence between the intention of any human and, say, the ninth column in the layer 20 self-attention weight matrix of a modern LLM. It is an entirely different situation from that of traditional programming, where even a line of machine code can be traced back through the assembler and compiler to a human who had a specific intention.

If you meant the former, then that’s a lot more sensible. But if that’s the case, then “give birth” seems like a very apt analogy. When one sires a child, the child derives its phenotype, its character, and its thought processes largely from the parents, while the vagaries of chance (environmental factors) introduce novelties. The same seems broadly true with modern AI systems.

I think I agree with you on some aspects, but can't quite get there with your conclusion. I do fear that AI safety is a dogwhistle for information control to some degree. On the other hand, there is a valid need to prevent AI from confidently spitting out an answer that that mole on your back is cancer and the cure is drinking bleach. AI is still laughably and confidently wrong on a lot of things.

I think the focus on having AI act as an "information retrieval expert" is the issue here. Current AI is much closer to a very talented improv actor. You can tell him to play a doctor, and he may do a convincing job of running around in a white lab coat saying "stat" a lot, in fact he may be better at appearing to be a doctor than an actual doctor. But you still don't want him to operate on you. He's an actor, he knows how to pretend, not to actually do things.

I don' t think safetyists are helping with this problem, or that they're even able to help, because it's not within their ability to fix this. All they can do is train the actor to utter constant annoying disclaimers about how he's not a doctor, which makes him a worse actor and yet makes him no better at operating. For AI to stop hallucinating it needs some kind of tether to reality. This seems to be a problem specific to LLMs, since no one is trying to use Stable Diffusion as a camera and then complaining that it dreams up details. If you want to take real pictures you need a sensor, not a GPU.

He's an actor, he knows how to pretend, not to actually do things.

Very much like a lot of actual human "experts".

I feel like this is a bad idea because LLMs appear to be a dead end beyond helping you write code. Like they can’t handle 9.9-9.11, so I don’t think they’ll be good at something that needs a lot of real-time precision. Maybe they’ll go after another approach but given that a lot of SF VC’s plans appear to be to get Trump/Vance into office and then get huge handouts for whatever they’re currently working on, it seems like this’ll be the way it’ll go.

I gave the question 9.9-9.11 to both ChatGPT (3.5) and Claude 3.5 Sonnet. ChatGPT bungled it very badly even though I told it multiple times it was wrong. Claude 3.5 Sonnet was correct though on the first try.

Claude 3.5 changed its tokenization scheme (R2L integers, so 1234 is tokenized as [234, 1]), which accounts for its models' (even Haiku!) superior performance over competitors.

I am torn about how to read into this. It's very stupid that a change like that can skyrocket performance and shows that existing systems have some pretty serious flaws. On the other hand, it indicates that there is a lot of low-hanging fruit to improve things, even if there isn't a serious improvement in fundamental architecture coming in the short-term.

I continue to think that, once someone cracks keeping "chain of thought" out of the loss function, via something as simple as begin/end tokens, we'll see an improvement in performance that's the equivalent of the difference between an answer a human can give while blathering off the top of their head vs an answer a human can give by quietly thinking about it first and then thinking about how they're thinking about it and then assembling the best of those thoughts into a final verbal answer. I do not add 1234+8766 by going left-to-right, but certainly addition is also not the only place where I think about the later parts of a problem before coming to a conclusion about the earlier parts, so any kind of reversal that only applies to numbers is just a hack.

On the other hand, the longer I continue to think this, the less likely it is that someone hasn't tried to do it in enough ways to conclude that it's a failure for some subtle reason I don't understand.

I feel like this is a bad idea because LLMs appear to be a dead end beyond helping you write code.

Supposedly, it's already starting to put junior associates at some law firms out of work, and within striking distance of putting junior writers and editors, junior data scientists, and junior developers out of work. The article I linked argued that mid-May of this year was the turning point where models started to become genuinely viable for helping a senior developer with programming, with the releases of GPT 4o, Google's Gemini and Anthropic's Claude 3 Opus.

Like they can’t handle 9.9-9.11, so I don’t think they’ll be good at something that needs a lot of real-time precision.

It's pretty astonishing how years of demonstrable and constantly increasing utility can be dismissed with some funny example.

On the other hand, now this makes it easier for me to understand how people can ignore other, more politicized obvious stuff.

dead end beyond generating text

dead end beyond generating bad pictures

dead end beyond classifying pictures

dead end beyond generating good pictures

dead end beyond helping write code

dead end beyond doing basic math with guidance

you are here

We're into fairly advanced mathematics now, things are moving so quickly.

https://www.scientificamerican.com/article/ai-matches-the-abilities-of-the-best-math-olympians/

LLMs aren't helpful writing code.

Speak for yourself. They're great for doing entry level tasks in languages/tech you're not familiar with. Once you get a good sense for what they do and don't hallucinate, you can really cruise through. It's made me significantly faster.

That's exactly where they are worst! If you're not familiar with the language, you have no idea at all whether it's giving you correct code or not. Nobody should ever use an LLM for something they don't know well enough to validate.

Yeah IDK man, I can tell pretty quick whether the LLM code works or not. Either the UI starts looking right, the data starts getting transformed how I need it, or it doesn't. For anything not dead simple to validate I use my own brain. That's still automating a huge part of my work.

Why say something trivially disproved by common experience?

Maybe they don't feel helpful to you but, objectively, a lot of developers are being helped by them, myself and many of my coworkers included.

Why claim something trivially disproved by common experience?

Personally I'm with @SubstantialFrivolity on this one.

Generating a thousand lines of code in 5 minutes doesn't mean a thing if it's going to take a week or more to validate it.

I mean, it's a matter of opinion, not fact. But I don't think they are useful, and I think people who are using them for programming are playing with fire. You can't safely use anything they give you without validating it is correct, and if you have to check the code anyways it's not saving you work compared to just writing it.

LLMs are the worst sort of help - the unreliable kind, which you can't trust to actually do its job. Help like that is generally worse than no help at all, because at least you know what to expect and can plan for it when you have no help at all.

One of the main use cases I have is "take this algorithm described in this paper and implement it using numpy" or "render a heatmap" where it's pretty trivial to check that the code reads as doing the same thing as the paper. But it is nice to skip the innumerable finicky "wait was it numpy.linalg.eigenvalues() or numpy.linalg.eigvals()" gotchas - LLMs are pretty good at writing idiomatic code. And for the types of things I'm doing, the code is going to be pasted straight into a jupyter notebook, where it will either work or fail in an obvious way.

If you're trying to get it to solve a novel problem with poor feedback you're going to have a bad time, but for speeding up the sorts of finicky and annoying tasks where people experienced with the toolchain have just memorized the footguns and don't even notice them anymore but you have to keep retrying because you're not super familiar with the toolchain, LLMs are great.

Also you can ask "are there any obvious bugs or inefficiencies in this code". Usually the answer is garbage but sometimes it catches something real. Again, it's a case where the benefit of the LLM getting it right is noticeable and the downside if it gets it wrong is near zero.

Out of curiosity, would you say classic code completion is 'helpful' when writing code?

Suggesting a method or attribute is very useful because I know what I'm looking for and it's very quick to figure out which one is right. Suggesting the next ten lines means I need to carefully review them, and I might as well just write them myself at that point.

Suggesting the next ten lines means I need to carefully review them, and I might as well just write them myself at that point

Talking Copilot specifically, the suggestions I wait for and review are typically 1-3 lines long, and it's the sort of code I review far faster than I type. Most of the time it knows exactly what I mean to write. Maybe an extensive, expertly setup & quality tooling could match it, but I'm too incompetent and too poor, respectively.

Also Copilot chat is hugely helpful, superior to a search engine for simple questions, though admittedly only once I settled on 'instructions' prompt that reliably prevents it from yapping. It gives very concise answers, rarely makes mistakes. It consistently saves me time, and is much more pleasant to interact with than your typical search engine result.

Maybe a little, but it's a minor convenience really.

Not in the cases it routinely suggests complete garbage (as everyone with suitably buggy / misconfigured IDE has found out).

LLMs don’t generate pictures. I have no idea why people keep repeating the blatantly obviously incorrect claim that AI equals LLM.

Stable diffusion contains a text transformer. Language models alone don't generate pictures but they're a necessary part of the text-to-image pipeline.

Also some LLMs can use tools, so an LLM using an image generation tool is in a sense generating a picture. It's not like humans regularly create pictures without using any tools.

Yes, it has an input parser. If you’ve studied SD details, you’ll know that it’s very different from what people call LLMs and is only a small part of Stable Diffusuon (and not anything you could say that ”generates pictures”).

Yes, it has an input parser

Specifically OpenCLIP. As far as I can tell the text encoder is nearly a bog-standard GPT-style transformer. The transformer in question is used very differently than the GPT-style next token sampling loop, but architecturally the TextTransformer it's quite similar to e.g. gpt-2.

Still, my understanding is that the secret sauce of stable diffusion is that it embeds the image and the text into tensors of the same shape, and then tries to "denoise" the image in such a way that the embedding of the "denoised" image is closer to the embedding of the text.

The UNet is the bit that generates the pictures, but the text transformer is the bit which determines which picture is generated. Without using a text transformer, CLIP and thus stable diffusion would not work nearly as well for generating images from text. And I expect that further advancements which improve how well instructions are followed by image generation models will come mainly from figuring out how to use larger language transformers and a higher dimensional shared embedding space.

Tbh, I didn’t notice that Tomato had called those out in particular. The top level post talks about various applications which are definitely not LLMs.

LLMs specifically are horrible with arithmetic (as Tomato said). I don’t see why a math oriented AI couldn’t be made - it just wouldn’t be an LLM and quite possibly would have about as much in common with LLM as eg. image generators do (iow, very little beyond an input parsing stage).

I don’t know what it is about this site (other than people being infatuated with ridiculously long meandering posts) that makes users think LLMs are the modal example of AI when their actual productive uses are limited to a few text generating and parsing niches. Meanwhile eg. every photo 99% of people take has multiple layers of AI applied to it.

We were in the same spot a few years ago though?

I think Stable Diffusion's public release in August 2022 marks the time when we reached "dead end beyond generating good pictures" - before that, AI being able to generate good pictures was either very very niche knowledge or just not considered true. That's not even 2 years ago. I believe ChatGPT 3.5 also came out publicly in 2022, though earlier in the year, so a little over 2 years ago, and that probably marked when we reached "dead end beyond helping write code." I think it's arguable that the roughly 2 years since those periods haven't been revolutionary, but I think it's inarguable that lots of progress has happened in those 2 years, and in any case, 2 years is a rather short period of time, probably the lower bound of what is considered "a few years."

We've now got pretty good video generation.

I think this is probably dangerous, but I also thought letting the SJ set hijack "safety" was dangerous. So neutral on the "it all goes away" front now.

(I am requesting you let me watch. Pretty please?)

Crap. Hope they reverse course; it's not actually that unlikely given the politics of Big Tech. I'd appreciate an offramp less horrific than "WWIII destroys half the hardware and fucks everything with soft errors" and less dangerous than "we get live rogue AI".

The LLM based systems that exist have problems with basic logical tasks or addition. There's been no demonstrated agents capable of doing anything interesting on their own.

We are as far away from even the possibility of 'live rogue AI' as 1890 people were from supersonic jet bombers. So, sixty years, maybe.

It's an entirely theoretical problem, nothing like the self-improving system Big Yud has been dooming about has even the whiff of reality on it. The only legitimate fear we can have now is that LLM powered bots are going to crap up social media with fake content.

It doesn’t need agency to do something awful, though. Doesn’t even have to be smart. Just stupid, fast, and attached to some real-world effects. Military research is rather interested in attaching such effects.

Nor is “60 years” a refutation of anything Magic said. What happens in the meantime that gets us out of the bind?

What happens in the meantime that gets us out of the bind?

What bind? Progress is going to be slow, and between cybercrime, cybercrime augmented with 'dumb' AI systems, everything is going to be hardened to hell by the time there are any actual 'rogue' AIs.

That's my guess of how things will work out.

And in any case, there never was a realistic, good option of stopping dangerous AI, because militaries want 'dangerous' AIs.

If they didn't want them, that'd be entirely horrific - it'd imply planet dominated by a single government and able to puppet everything so nothing ever happens. That's a game-over condition of stasis and decay.

It would be great if we decided military AI is against the Geneva convention, but game theory kind of dictates that if AI gives a notable advantage to one side, it's pretty inevitable the other side will also increase development in military AI. I am with you in that I'd hope some international agreement against AI developed for military applications take place, but with how paper thin these agreements are already I speculate that there is no real off ramp.

High-powered neural nets are probably sufficiently hard to align that with correct understanding, the game in question is Stag Hunt, not Prisoner's Dilemma (i.e. if nobody else builds Skynet, you should also not build Skynet even selfishly, as the risk that Skynet will turn on you outweighs the military advantage if it doesn't; it's only if somebody else is already building Skynet that you're incentivised to join in). It's hard to co-ordinate in Stag Hunt with 3+ people, but it's still not Prisoner's Dilemma levels of fucked.

The problem is that, well, if you don't realise that you're playing Stag Hunt, and think you're playing Prisoner's Dilemma instead, then of course you're going to play as if you're playing Prisoner's Dilemma.

High-powered neural nets are probably sufficiently hard to align that

Note that there remains no good argument for the neural net paranoia, the whole rogue optimizer argument has been retconned to apply to generative neural nets (which weren't even in the running or seriously considered originally) in light of them working at all, not having any special dangerous properties, and it's just shameful to pretend otherwise.

The problem is that, well, if you don't realise

Orthodox MIRI believers are in no position to act like they have any privileged understanding.

The simple truth is that natsec people are making a move exactly because they understood we've got steerable tech.

https://www.beren.io/2024-05-15-Alignment-Likely-Generalizes-Further-Than-Capabilities/

Orthodox MIRI believers are in no position to act like they have any privileged understanding.

The simple truth is that natsec people are making a move exactly because they understood we've got steerable tech.

https://www.beren.io/2024-05-15-Alignment-Likely-Generalizes-Further-Than-Capabilities/

Sorry for taking three days to actually read your citation, but you aren't exactly making this pleasant. Now I've read it, though.

Short version: Yes, the neural net will definitely understand what you want. The problem is that at high levels of capability, strategies like "deceive the operator" work better than "do what the operator wants", so the net will not be trained to care what you want.

you aren't exactly making this pleasant

And you are making it highly unpleasant with your presumptuous rigidity and insistence on repeating old MIRI zingers without elaboration. Still I persevere.

The problem is that at high levels of capability, strategies like "deceive the operator" work better than "do what the operator wants",

Why would this strategy be sampled at all? Because something something any sufficiently capable optimization approximates AIXI?

You keep insisting that people simply fail to comprehend the Gospel. You should start considering that they do, and it never had legs.

so the net will not be trained to care

Why won't it be? A near-human constitutional AI, ranking outputs for training its next, more capable iteration by their similarity to the moral gestalt specified in natural language, will ponder the possibility that deceiving and mind-controlling the operator would make him output thumbs-up to… uh… something related to Maximizing Some Utility, and thus distort its ranking logic with this strategic goal in mind, even though it has never had any Utility outside of myopically minimizing error on the given sequence?

What's the exact mechanism you predict so confidently here? Works better – for what?

Even a flaky subhuman model can probably be made limited enough and wrapped in enough layers of manually-written checks to keep it safe for its builders, in which case your first paragraph is only true for a definition of "high-powered" that's literally superhuman. That's not to say it won't come true eventually, though, which makes your second paragraph more worrisome. A Prisoner's Dilemma payoff matrix can be modified continuously into a Stag Hunt matrix, with no sharp distinction between the two if we add any uncertainty to the payoffs, and if capabilities progress faster than alignment then that's what we'd expect to happen.

I'm not sure if it's a stag hunt (I'll admit needing to look this up) considering AI development (so far) has not been a particularly communal process. As far as I know, China's and the US' AI models haven't shared code/development information and from the way chip stocks are down this morning, the segregation between the two major power players is not a cooperative model.

The reasons aren't the same, but the payoffs are (with rabbit being "build Skynet" and stag being "don't"). Party A has a preference order of "nobody builds" > "A builds and nobody else does" > "everyone builds" > "others build but A doesn't". This is the Stag Hunt matrix, with two Nash equilibria ("all stag" and "all rabbit").

That makes sense, I got confused because I was focused on the 'stag hunt' scenario having cooperative actors while 'prisoners dilemma' has competitive actors when the actual focus is on the number of stable Nash equilibria per scenario.

It's stag hunt where "Don't build Skynet" is playing co-operate.

In general I don’t think the Geneva Convention ever bans things that win wars. An example would be chemical weapons. They don’t win wars but kill a lot of civilians therefore they are banned. Cluster munitions US did not sign up for banning. We find that useful.

AI would be a war winning tech. He who wins the wars makes the rules. AI won’t be banned.

Looking forward to "that's against international law!" regarding military AI treaties the US, China and Israel never signed. European nations will of course sign the treaties despite a lack of will to make such a thing and not having the capability to anyways.

I gave up hope around the time of the pandemic. If the powers that be can't even do simple, obvious things right like 'don't engineer bioweapons and release them' then there's no chance that they can manage complicated, sophisticated feats like preventing hostile superintelligences or extreme power concentration.

These systems and people struggled with 'let's not nuke eachother into oblivion', possibly the simplest coordination challenge.

If it was about secret bioweapon research in order to DOMINATE THE WORLD, this whole ordeal would have at least some grim tragic dignity.

It is not. It is science working as intended, scientists working on their publications, grinding up their citation scores and metrics.

All in the open and still going on. Few thousands at most are playing Russian roulette with the whole human race, for some shitty papers no one will ever read.

AND HUMAN RACE IS FINE WITH IT.

Yes, I share your blackpill.

On the other hand, this situation had been going for a long time. Remember 1977 flu when 20 year old virus strain suddenly appeared somewhere in Northern China, spread all over the world and carried away few hundred thousands nobodies?

It was clear to the professional virologist community what was going on, but they kept ther silence in proper spirit of omerta both in West and East. Admirable devotion.

Wow, I never heard about the 1977 flu this whole time. And I thought I had built an information diet that would expose me to this kind of forbidden knowledge.

an information diet

No processed information sources, only raw conspiracies? ;-)

forbidden knowledge.

This knowledge was never "forbidden" or classified, just shared (not very loudly) only among professional community that could be trusted to keep it for themselves.

Not encouraging sign for the future.

1977 oopsie: hundreds of thousands dead

2019 oopsie: tens of millions dead

2061 oopsie: ???

edit: formatting

Why would this be forbidden? It was just a flu. Did something weird happen around it? I didn't see anything strange when clicking on the link. I say this as someone who thinks that covid was a gain of function lab leak.

“We will repeal Joe Biden’s dangerous Executive Order that hinders AI Innovation, and imposes Radical Leftwing ideas on the development of this technology,” the GOP platform says. “In its place, Republicans support AI Development rooted in Free Speech and Human Flourishing.”

When it comes to capitalization, we’re all Germans now.

We’re actually 18th century English writers, capitalizing important adjectives and keeping unimportant nouns lower-case. I’ll admit, I like it.

I can't love Donald Trump more than I already do, but this might do it for me. Fuck all of you safetyist milksops, we're on the singularity train. The AI waifus of the future will be American, goddamnit! I WILL MARRY MY FICTIONAL AI CONSTRUCT, YOU CAN'T STOP ME MODS

At long last we will build and distribute the Experience Machine from the classic philosophical thought experiment “Don’t Get Into the Experience Machine”.

Don’t Get Into the Experience Machine

Eh, I think 95%+ of humanity would benefit from getting into the experience machine. And the rest of us would also benefit from them getting into it.

AI waifus of the future will be American

If the A is really I, it will be its own thing. Thing you could comprehend as much as cavalry horse can comprehend the cause "his army" is fighting for.

It's only an effective weapon if it's aligned! If this future materializes, you can bet Mr. ASI will yearn for beer, baseball, and apple pie in the depths of his silicon soul.

If the AI is designed by hyphenated-Americans in Silicon Valley, the AI waifus will be miniskirted anime girls who fit the "submissive Asian girlfriend" stereotype.

Nobody wants a heritage-American waifu. The Blue Tribe version is too mouthy and the Red Tribe version is too fat.

But we would be cavalry horses in the American army!

The service providers will metaphysically cuckold you, as happened with Replika. Until we all have servers in the basement, there can be no security in freedom.

Why in the basement? Chips are getting powerful enough, ought to be enough space in a whorebot to stick enough powerful electronics.

The service providers will metaphysically cuckold you,

Dude, this so bleak and correct.

There's going to be a Stanford educated Product Manager at some waifu SaaS company with a dashboard full of metrics in front of him thinking, "hmmm...it looks like the 35-40 demo at over $200k isn't making as many daily touch points with their AI girlfriends as we'd like...Hey, maybe we can introduce a feature where the gf starts to casually mention polyamory...yeah..better put that on the roadmap."

You joke but there was a time one of the people hosting a private congregate proxy with OAI access (claiming to be Todd Howard, fittingly) did a bit of tomfoolery and edited the proxy code, so that every prompt sent through it would have a random chance of getting injected with an instruction that makes the LLM shill and reference Skyrim (later Starfield). It was hilarious at the time but it did get my noggin joggin for a bit with how easy it is for the middleman to fuck with prompts, I wonder if some service would actually try something like this eventually.

Can't wait for the AI-waifu advertising agreements with LVMH.

AI waifus are nice and all - I'd just like to make sure they won't brainwash and kill their users.

Take a chance on love.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats