This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
The problem with corpos being made up of humans is similar to trying to make ever better computers without changing transistor size. You can optimize the layout, cooling, etc, but you'll forever be bound by the size. Corpo capabilities and architecture are chained by their components. They would be a lot more dangerous if they could produce better humans at scale (compare the performance of Jane Street vs retail investors, or special forces vs green Army grunts), or produce a new part to do mental and social work (AI).
Isn't the whole point of the argument that AI will be such a threat because it will, by virtue of being more intelligent than us, be able to breezily figure things out (like self-improvement) that we simply couldn't because of our inferior intelligences? If that's the case it doesn't seem to matter that much that corporations (or as pointed out below, any form of supra-human coordination, states, political parties, etc.) have certain limitations at the outset, because their 'superintelligence' ought to allow them to overcome those limitations in short order. After all the self-improvement scenario also assumes that AI is limited at the outset but rapidly transcends these limits.
Right, but corporations that are staffed by humans aren't smarter than humans and can't become smarter than humans. "Being a corporation" doesn't remove the scaling limits that constrain the human brain in specific. If you remove that limiting factor, then yes, corporations are scary too.
A corporation (really, any human organization--I think I'll just say that going forward) is smarter than any individual human that comprises it, by virtue of being comprised of many different intelligences. Likely, any (or at least most) human organization is smarter than any individual human on earth, since it is the sum total of all the human intelligences that make it up. This is comparable to the oft-repeated hypothetical where AI bootstraps by copying itself many times over. So I think it is fair to describe a human organization as a "superintelligence" in the same sense meant by AI x-risk proponents.
I think intelligence as a single axis really breaks down here. Well-run organisations can beat humans in specific ways — better parallelization, less likely to get bored/tired, wider and deeper expertise — but often not in the ways that are really interesting. (If von Neumann joined as an entry-level employee at some megacorp today, would the organisation become smarter than him in any reasonable sense?)
Orgs seem good at gluing together boring competencies and shoring up human shortcomings, but we haven't figured out the interesting stuff yet — we have no idea how to assemble 1000 mediocre writers into a Steinbeck or 1000 mediocre physicists into a Feynman.
So I think "superintelligence" is the wrong word for orgs. "Superhuman", yeah, in the more limited sense that a horse or a plane is superhuman in some capacities. But we're not at the point (yet) where we've cracked the alchemy of coordinating lots of human intelligences into an organisational superintelligence. So I think that's the critical difference between orgs rn and actual x-risk from superintelligences
I would say so. "It" could do/understand anything Von Neumann could do, by virtue of von Neumann being able to do it on behalf of the corporation, and also do/understand anything that any other person or persons employed at the corporation could do that von Neumann happened to be unable to do. I would say that counts as "smarter than Von Neumann."
From my experience in creative writing workshops I believe a number of mediocre (or at least non-optimal) writers working together will be better than any one of those writers working independently. Maybe not as good as Steinbeck, but if Steinbeck was a member of the workshop, then sure.
I don't really know anything about Richard Feynman's life or work but I would feel safe in saying that the collection of Feynman and his colleagues, research partners, etc. was 'smarter' than Feynman was on his own.
Lots of people have told me "humans aren't perfectly coordinated" or "corporations are made-up of humans" but this just seems to shift the argument from the danger of "superintelligence" to "the danger of superintelligence at some arbitrary level."
Okay, but what it's using von Neumann for in this instance is inputting numbers on a spreadsheet. There's not an unlimited number of positions at the company where it's getting full use of von Neumann's intelligence, and moving the best people to those positions is also limited by competence. And it only gets the one von Neumann, so while in principle it can reach von Neumann tier competence in any capability, it cannot necessarily reach von Neumann tier competence in every capability. And it still can't inherently beat von Neumann's peak output in any particular skill he possesses.
I think the better framing here is that you can use cooperative ventures to remove limits and drawbacks that prevent people from reaching their peak performance. But I don't think you can get the aggregate to exceed the individual's performance that way.
More options
Context Copy link
But an organisation can't do all the things its component employees can.
Toy example: Alice is a stubborn science expert; Bob is a stubborn humanities expert; as individuals, they are capable of answering questions about their respective areas of expertise, but as AliceBobCo they squabble and can't come to an agreement on any question. The company is dumber than the sum of its parts.
(The key point is that even though the company has employees that possess the skills it needs, it lacks sufficient structure to yoke "the will of the organisation" to that individual's skills.)
I agree that a Steinbeck + lackeys is more competent than just a Steinbeck! — just not in a way that merits "superintelligence". It's not a qualitative step up; you don't get Steinbeck^2, you just get Steinbeck with some minor amps. There seems to be a power cliff of generating a great writer in the first place, and our only recipe for adding that capability to an organisation is "employ a great writer".
I think you're underestimating what people mean by "people aren't perfectly coordinated". It sounds like you think they mean the intelligence of an org scales like O(ln(N)) or something (i.e. making an org 100 times bigger makes it only 10 times smarter). I think it's more like: making an org 100 times bigger probably makes the org as a whole dumber, but capable of massively more work and some related benefits (that are sorta intelligence-related but not the whole thing). I.e. it's not "orgs are superintelligences but beneath the danger threshold", it's "orgs are subintelligences connected to massively powerful but dumb machinery".
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link