Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.
naraburns nihil supernum
1yr ago
(text post)
3627 thread views
Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.
Jump in the discussion.
No email address required.
Notes -
Interesting. Most people seem to think once we get AGI we’ll speed to super intelligence very quickly, why the 15 year gap? You don’t believe in self recursive improvement?
I believe in it, I just don't believe it will be that fast, especially at the "median human" skill level that I posited for the weaker of the two AGI definitions.
I'm having a somewhat hard time serializing my thoughts on this into a well-written argument, having drafted and re-drafted the rest of the post about three times now, so I'm sorry but I'll just give you some badly connected bullet points.
It's curious how the cutting edge of AI wound up being driven by computationally extremely costly black boxes that imitate human behaviour. Because of that, a lot of the standard intuitions about how AI will have a much easier time self-improving than humans because it will be a neat algorithm running on silicon are actually invalidated - a GPT-like AGI will operate as an inscrutable binary blob that inputs and outputs at bounded speed in an imprecise medium (text in human language), and resists parallelisation and other attempts at speeding up a single instance due to a myriad of superlinear factors and hardware barriers that are stumping teams of the world's top physicists, engineers and probably lab technicians holding illegible metis about how many times you have to bang the ASML machine with a wrench to get error-free DRAM, depending on the day's weather. I'm not convinced that generating more mediocre-human-tier ideas to try out is the primary bottleneck in this process, as opposed to something like "try changing the number of times you bang the machine, and if you break it, wait 6 months for ASML to build and deliver a new one" or "get the capital and paperwork lined up to even be allowed to try building a better chip".
There are billions of approximately average people, who are getting nowhere near innovating on cutting-edge AI or chip design. The weak AGI will probably require some pretty fancy high-end hardware to run; I don't think a billion copies of it will be available that soon after the 100 or 1000 or so it's first deployed on are. Due to the aforementioned curious nature of our current best candidates, N instances of them will probably scale a lot more similarly to "N humans" than to "an Nx as smart human". Changing the scaling will require solving a problem that so far has not proven easy for N mediocre humans to solve; drastically increasing the N will require physical-world stuff that the AGI can't do and therefore will continue advancing at meat human speed.
With current hardware, training times for new models are counted in weeks or months. Even if the million-mediocre AGI cluster generates a valid idea to slightly improve itself, it will take this long before it can start taking the million-slightly-less-mediocre AGI cluster online. This comes out of the same finite pool of resources: if it uses its numbers to perform "grad student descent" and generate 10 ideas of which one works without understanding one, this will take 10 times as long. We have no evidence that anything near our level can do better than grad student descent (i.e. identify some number of strategies/hyperparameters/? of which one randomly amounts to an improvement), and grad student descent is done with grad students who are much better than mediocre.
Nothing I've seen even from people near the top 10^-n fraction of the current human intelligence distribution has suggested to me that returns to intelligence are that insanely high. Von Neumann (even 1000 of him) could probably not have snapped his fingers and built GPT-4 in a year. Otherwise I think we would be seeing more divergence between countries right now than we do. Therefore I think that even as we approach the strong end of the weak-strong scale, acceleration won't be that great.
The sort of fiction that is enjoyed and written by our community betrays our biases on this matter: we always fantasize that if only we were 10% more intelligent, had access to that perfect motivator pill, had perfect memory or a magic notetaking and spaced repetition system, or some consciousness-forking powers, then the problems that we face day to day would just evaporate and turn the real world into a joyful, snappy, hairless RTS experience. (The positive framing of this, I believe, is called "internal locus of control".) A lot of strong AGI prognosis, in my eyes, winds up being coloured by some sort of bitter projection of what remains of the same fantasy: yes, one begrudgingly concedes, I will never become like Rationalist Harry and it was immature of me to think so - but the AGI will be just like Rationalist Harry, and it will leave you wishing you had gotten me as Rationalist Harry instead.
More options
Context Copy link
More options
Context Copy link