This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
At the risk of producing frustrated groans from everyone, I find it hard to get too worked up about any civilisational issue with a timeline longer than 20 years because it seems extremely likely to me that we'll have superintelligent AI by the mid-2030s (that's me being conservative), and at that point all bets about capabilities and risks are off. While I'm not a committed AI doomer, it looks from every angle to me like we're in terminal-phase human civilisation. What follows could be very good or very bad for us, but whatever "it" is, it won't be subject to the same logics and power structures as our current global socioeconomic order.
I drafted a very long comment to this effect in the discussion about declining TFR and dysgenics last week which I failed to post due to user error, but I think the point applies to climate change too. Optimistically, I think it's not unlikely that ASI will get us over the line on nuclear fusion and related tech, allowing us to transition entirely away from carbon economies in fairly short order and easily offset any residual carbon footprint with direct carbon capture. Or maybe it'll allow us to conduct low-risk geoengineering at scale. Or (more pessimistically) maybe it will secretly deploy nanoengineered pathogens that will wipe out most of humanity. Either way, I don't think climate change will be a problem that we (or whichever of us are left) will be worried about in 2050.
I asked you before, what concrete actions are you taking when you strongly believe that we will have utopia/apocalypse in 10 years? Do you have any bonds with longer than 10 year maturity? Do you find it stupid to invest in any new whisky with a plan for aging it for more than 10 years? Along the line with your demographics skepticism - do you consider people stupid for having kids now, if they won't matter in 10 years? A this point I am really curious.
More options
Context Copy link
Sidetracking the thread a little bit, but given that OpenAI and its competitors hit a brick wall in progress recently, what keeps you optimistic about ASI? Admittedly I'm not following the field very closely, but are there any interesting breakthroughs that I've missed that you think get us closer?
EpochAI can see another 25x https://x.com/Simeon_Cps/status/1826134542857499005
More options
Context Copy link
Fair question, but no, I don’t think OpenAI have hit a brick wall. GPT-3 was June 2020 and GPT-4 March 2023, so even if the next leap took the same time to train up (obviously it’s not that simple) we wouldn’t expect a similar leap in performance for another couple of years. On top of that, the GPU supply chain is creating short-term bottlenecks for training runs. We might see glimpses of true next-gen performance from competitors before then, but I expect most of the buzz for the next 18 months or so to be dominated by increasingly agential models and better multimodal capabilities. There’s also the long-delayed rollout of ChatGPT’s voice upgrade, which is a bigger deal both technically and in terms of social effects than most people realise.
Zooming out, AI now benefits from a forcing economy in a way that was never true of previous AI summers. Outside of specialist applications, there wasn’t much money to be made in AI until comparatively recently, especially for generalist systems like LLMs. But in the wake of ChatGPT you have real AI revenue streams, and every nerdy 18 year old wants to study machine learning (some of them will even get jobs). While we might have a short-term AI bubble as Capex grows out of all proportion to revenue run rates, it’ll be a temporary blip. There’s still gold in them hills, and we’re only scratching the surface of what’s possible in terms of AI products even using existing tech. Most big non-tech firms are still figuring out their AI strategy and paying OpenAI and Microsoft service fees for dumb off-the-shelf products. A lot of the real commercial impact of AI in the short-term is consequently going to come from last-mile products that invest time and energy in tailoring the better open-source models to specific business use cases.
Zooming out even more… look, humans aren’t that smart. We’re the dumbest possible species capable of building an industrial civilisation. Our intelligence is limited by a bunch of very contingent factors like caloric consumption, the size of the birth canal, and the fact we’re layering a System 2 architecture onto a 600 million year old foundation. Even if these constraints didn’t apply, evolution is just not that great of a search algorithm in design space. Take eusociality in insects for example. This is an incredibly successful strategy, with roughly three quarters of insect biomass today coming from eusocial species. But evolution stumbled across eusociality pretty late, only really getting going around 150 million years ago (compared to 400 million years for insects in general). It’s not because it requires large brains, but because evolution is just a crappy blind algorithm for finding optimal equilibria and human ingenuity can do a lot better. Nor is there any reason to think that anatomically modern humans constitute some kind of upper bound on intelligence; the massive intelligence differentials just among humans provide good evidence of that.
So to summarise: OpenAI is going about as fast as we might reasonably expect, the economic fundamentals of AI development have shifted in a way that is likely to accelerate long-term pace, and the goal we’re reaching for isn’t even that hard.
Is there any good theory basis for this claim? It seems to me just as likely that "intelligence" is more like large-scale Bayesian inference, and that for a given quantity of sensory input the possible predictive performance is quite bounded, and potentially even grows logarithmically such that billions of times more input data may only marginally improve the output.
But I will admit I'm somewhat spit balling here and not familiar with the existing literature.
The “dumbest possible species” claim is mostly a soundbite and truism, but the basic idea would be (1) that we see increasing encephalisation (especially in the neocortex) and increasing behavioural sophistication in the Hominins all the way up to Homo sapiens and Homo neanderthalensis, and (2) a small minority of the very smartest humans in very recent history (the last 1000 years out of the 300,000 or so of our species) were required to make the necessary move from agrarian societies to industrial society. Of course they were building on indispensable social, political, and economic foundations, but if you drop the IQ of Europe by 1SD for the second millennium AD I think it’s unlikely we’d get the Industrial Revolution at all.
Regarding the idea of Bayesian limits to intelligence, that applies well to cases where the dimensionality is fairly constrained, notably perception. The space of cognition (“possible good ideas”) by contrast is much more open-ended, and applies at multiple levels of scale and abstraction (because we need heuristics to deal with any large scale system). I don’t see any reason to think we’re even close to “topping out” in cognition, and the outsize contribution of the smartest humans compared to merely very smart humans provide some evidence in this regard.
I will admit that the emergence of AI may finally give some interesting answers and maybe closure to philosophical questions about how introspective and abstract philosophy and mathematics are. As much as (some) math claims to be proof pulled from the ontological ether, can the concept of, say, prime numbers be explained to an intelligence with no real-world sensory inputs? Does the notion of counting make sense in an absence of things to count?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
OpenAI has a 100b valuation per latest funding rounds.
To me that suggests:
They don’t think they can build a moat, OR
The technology is plateauing.
To build on what @Incanto said:
Even without taking OpenAI's charter into account, I have no motivation to buy stock in a company that's trying for singularity. Leaving aside the question of whether I think that company is instead going to end the world (and I do), assuming for the sake of argument that they'll succeed in getting controllable superintelligence...
...what process exactly is paying me my dividends? Control of a singularity event, in a relatively-short timeframe, gives sovereignty. Yes, the law says that I can replace them if they don't pay me, but the law will have no power over them if they succeed, because singularity rapidly means they outgun the government. They can just turn around and stiff me and I've got no recourse; only direct control of the AI matters.
More options
Context Copy link
They also have a weird semi-charity corporate structure that may make investment riskier and bring down the valuation - but it's true that markets don't seem to be pricing in a near-term singularity.
More options
Context Copy link
More options
Context Copy link
There's more to AI than chatbots. GDM had its IMO proof solver and AlphaProteo, which, in normal times, would be potential candidates for technology of the year. The only reason they're not is that they are somewhat obvious next steps in existing research programs, coupled with lots of engineer time and compute.
I do agree with you that we may be in the plateau of the transformer S curve; we'll know more when OAI releases its next model. In the coming weeks, as they say.
More options
Context Copy link
More options
Context Copy link
I struggle with these sorts of things too. But I don't think superintelligence is a given, just yet. There's also the possibility that superintelligence happens, but it just doesn't even care about our problems. How much time do we spend worrying about ant habitats?
In the mean time, I think there's real value to geoengineering in the here and now. For relatively little, we can prevent the Earth from warming any more. Why wouldn't we want to do that?
Not much, but we also don't especially trouble ourselves not to destroy them incidentally if, say, we want to build a road over them. And there's instrumental convergence to worry about - for most possible goals, you can do a lot better if you use the atoms in the apes for something else.
You'll also go through a hell of a lot of unliving atoms before using ape atoms is going to be worth the effort, unless you're already so all-powerful that it's 0 difference to you.
Yeah, the most likely reason that AI would want to KILL ALL HUMANS is that they see humans as a threat. I think this idea might come from the movie "Terminator" in which the computer becomes self aware and humans try to pull the plug, forcing the AI to fight back.
But, as we've seen, AI safety research is sloppy and compromised. When the AI becomes self-aware, no one will pull the plug.
And besides, there won't be an obvious "oh shit" moment. More likely, as AI progresses, it will entwine itself into every bit of human society, making it all but impossible to remove. Entirely dependent on AI, we will be no more threat to it, then the chimps in the zoo are to us.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link