stoatherd
1yr ago
I think intelligence as a single axis really breaks down here. Well-run organisations can beat humans in specific ways — better parallelization, less likely to get bored/tired, wider and deeper expertise — but often not in the ways that are really interesting. (If von Neumann joined as an entry-level employee at some megacorp today, would the organisation become smarter than him in any reasonable sense?)
Orgs seem good at gluing together boring competencies and shoring up human shortcomings, but we haven't figured out the interesting stuff yet — we have no idea how to assemble 1000 mediocre writers into a Steinbeck or 1000 mediocre physicists into a Feynman.
So I think "superintelligence" is the wrong word for orgs. "Superhuman", yeah, in the more limited sense that a horse or a plane is superhuman in some capacities. But we're not at the point (yet) where we've cracked the alchemy of coordinating lots of human intelligences into an organisational superintelligence. So I think that's the critical difference between orgs rn and actual x-risk from superintelligences
But an organisation can't do all the things its component employees can.
Toy example: Alice is a stubborn science expert; Bob is a stubborn humanities expert; as individuals, they are capable of answering questions about their respective areas of expertise, but as AliceBobCo they squabble and can't come to an agreement on any question. The company is dumber than the sum of its parts.
(The key point is that even though the company has employees that possess the skills it needs, it lacks sufficient structure to yoke "the will of the organisation" to that individual's skills.)
I agree that a Steinbeck + lackeys is more competent than just a Steinbeck! — just not in a way that merits "superintelligence". It's not a qualitative step up; you don't get Steinbeck^2, you just get Steinbeck with some minor amps. There seems to be a power cliff of generating a great writer in the first place, and our only recipe for adding that capability to an organisation is "employ a great writer".
I think you're underestimating what people mean by "people aren't perfectly coordinated". It sounds like you think they mean the intelligence of an org scales like O(ln(N)) or something (i.e. making an org 100 times bigger makes it only 10 times smarter). I think it's more like: making an org 100 times bigger probably makes the org as a whole dumber, but capable of massively more work and some related benefits (that are sorta intelligence-related but not the whole thing). I.e. it's not "orgs are superintelligences but beneath the danger threshold", it's "orgs are subintelligences connected to massively powerful but dumb machinery".
More options
Context Copy link