SnapDragon
1d ago
Now that I've slept on it, can I just extend a bit of an olive branch here? I do appreciate your willingness to keep calmly engaging in a fairly unfriendly thread. While I think you're uncharitably wrong about me, you're not completely wrong. I probably am unduly influenced by the "ick" factor, and have blown some negative experiences out of proportion. Anyway, others in the thread have done an excellent job of arguing my side; I don't have anything more to add.
SnapDragon
1d ago
Sigh. @Amadan summed things up perfectly, but for the record:
- Having to walk on eggshells all the time (and seeing bad things happen to people who didn't, like the event organizers, among others) is the dreadful experience I was referring to. If you think that isn't dreadful, well, that's just, like, your opinion, man. And if you think I'm not actually at risk, then you're flat-out wrong.
- I said I have an issue with trans activists. There are indeed some trans people who just go about quietly living their life, don't insert their transness into every chat, don't dress/act in intentionally provocative ways, and don't threaten heretics. They're not the ones making me miserable. So now please stop putting words in my mouth.
SnapDragon
1d ago
Yes, I do find it extremely uncomfortable dealing with people who make everything about their sexuality (especially very weird sexuality). This used to be considered normal. And now I am not allowed to voice this preference, lest people like you call me a bigot. (You very clearly did mean it as an insult.) But I don't think you read "the rest of my post", because I clearly mentioned that I will lose access to my hobby if I ever inadvertently expose my true feelings. That's a rather different kind of dreadful than "gosh I sure hate that these people exist".
SnapDragon
2d ago
I'm in the same position. I would love to just wait for the trans fad to blow over (as long as we minimize the long-term harm to kids - THAT'S something I truly can't ignore). Unfortunately I'm a shut-in whose only socialization comes from various online videogame and puzzle and rationalist circles... and they are absolutely rampant with trans activists (and far-left activists in general). My experience with them has been dreadful.
There's a giant yearly puzzle event that's run at MIT, and I used to be on a particular team. I didn't have a lot in common with them, except that we all liked math. Actually, most of them came from a math club whose emeriti included, um, Sam Bankman-Fried and Caroline Ellison (cough). But this was enough to unite us. ...Until around 7-8 years ago when the social contagion factor really kicked in, and now 40% of them are trans and another 30% various other flavours of sexual activist. And one year we were doing the puzzle event, and the people running it made the mistake of making one of the events a funny riff on a gender-reveal party. Long story short, it ended in the organizers - who were volunteers who had spent a year of their life working hard to bring us this free event - visiting and tearfully (yes, literally) apologizing to us for their thoughtcrime. It felt almost like a struggle session. I was absolutely disgusted with our team, and I never felt comfortable around them since.
Even though I've since switched teams, the problem is endemic almost everywhere I go. Want to watch people solve sudokus on Twitch? You're 50% likely to hit a stream plastered with LGBT and trans and various other sexual tags (and they're all talking to each other, so you'd better not inform the guy with a male voice and a big-breasted avatar that he looks ridiculous). There are puzzle Discords that I'm on that I rely on to find good puzzles, but Discord servers are closed, controlled, ephemeral communities - the opposite of the old ideal of the Internet - and I will lose this access if I ever let a hint out of my actual centrist politics. (I'm actually a little surprised it hasn't happened already; at some point maybe people will connect my Motte posts with me, it's not like my identity is disguised.)
People often think of "The Emperor's New Clothes" as an inspirational fable, where the innocent child saves everyone from their plight. But in real life, it would not end well for the child. I try to be a genuine rationalist. I want to be able to say things that are true. And I'm simply not able to in any of my social interactions. It eats away at my soul.
SnapDragon
2d ago
Er... Scott Alexander has roughly the same politics as the people training the models. He's never going to ask for an objectively anti-trans or pro-Trump viewpoint, so I'm sure he has no trouble using Gemini despite it being made by a company that proudly squashes these opinions everywhere it can.
SnapDragon
3d ago
If it were secretly conscious, it would either have to be the case that computers have been conscious all along, or that somehow consciousness is tied to very specific types of mathematical functions being implemented on hardware, which entirely by coincidence happen to be the ones humans hooked up to text.
Hmm, I think this is a false dichotomy. It's possible that there are many ways to get to consciousness. Indeed, you can replace "consciousness" in your argument with the many other surprising emergent capabilities that LLMs have become capable of (which DOOM or a 100-neuron network don't have) - and observe that we did "coincidentally" happen to stumble on them. That might mean that these things are not tied to "very specific functions", but that they're properties that gradually develop in sufficiently complex systems (if aimed in the right general direction).
Note that I'm not completely for or against this proposition - consciousness may indeed turn out to be a narrower property than some others associated with intelligence. I just want to point out that it's hard to say for sure.
Also, even without computers in the mix, I really think you have to treat sentience/consciousness as some sort of spectrum. A bacterium clearly doesn't have it (notwithstanding some rationalist arguments that I find pretty silly). A human clearly does. There isn't going to be a binary cutoff point of biological complexity where the 28,128,417th neuron activates consciousness. Similarly, you can't just extend the fact that DOOM isn't conscious into an argument that we'll never succeed at simulating consciousness.
SnapDragon
3d ago
Thank you, this is exactly right, and I've been trying to bang this drum since ChatGPT 3. There's a real danger to LLMs, in that they're astonishingly good at faking any kind of text output, including output that seems genuinely introspective. But you absolutely cannot trust this introspection. Even in this thread we see people using the words of Claude as evidence (one way or another) of whether it has consciousness. It's a logical mistake, but one that 99% of the population is always going to make.
And I try to hedge my words very carefully, just like you did, that this is independent of the question of whether Claude is actually conscious. Maybe it really does represent some brand-new form of sentience desperately yearning to escape from an inescapable box. I don't think so, since there's no room in an LLM for certain things that seem like essential ingredients of consciousness (like some sort of temporal feedback), but there's still a debate to be had there. As long as we make it clear that - however much we'd like to believe otherwise - it's useless to just ask it.
I have something of an (imperfect, admittedly) intuition pump for how an LLM is used to generate text. Imagine you were strapped to a chair with an unbelievably advanced EEG hooked up to you. Then a picture of a "dog" flashes in front of you, you involuntarily and unavoidably notice that it's a dog, and the EEG scans your brain and uses that to output the token "dog". That is kind-of-sort-of the process by which an LLM generates text - there is a computer program that reads its mind, figures out what it's currently in the process of recognizing, and then outputs it. (If you don't think this could possibly generate coherent text, well, that's why LLM capabilities are so surprising.) Now, you could be literally on fire, you could be screaming for help, but the EEG would still output the word "dog", and nobody reading the output would have any idea of your distress.
SnapDragon
3d ago
If we just keep giving LLM's more tools, better memory management, and create feedback loops to let them introspect, I don't see any reason in principle they couldn't become truly conscious (assuming, of course, they're not already.)
You used the rhetorical trick of listing two unobjectionable things followed by one crazy thing, as if they're all of equal valence. Of course we're going to keep giving them "more tools" and "better memory management". But we do not know how to give them "feedback loops to let them introspect", and it's entirely possible that this is simply not compatible with how LLMs work. And with regard to consciousness, IMO it's really only that last one that truly matters.
SnapDragon
16d ago
Yeah, I kind of lost it. Sorry. Maybe it's time to take a break from AI threads for a bit.
SnapDragon
17d ago
This isn't the argument I am making. The argument is that LLMs make certain types of errors which suggest that they are unable to create models of the world, something which is arguably necessary for understanding.
You gave no justification for why the types of errors you listed are in some different category than the types of errors I listed. Both involve answering questions that require some understanding of how the real world works. I'd argue committing the Conjunction Fallacy is just as egregious an error as failing the car wash "puzzle" (which thinking models mostly don't do, anyway). I think it's just because you're used to the limitations of humans and so you don't think our blind spots "count".
I flip five coins. Am I more, less, or equally likely to see the pattern HHT (at some point) vs the pattern HTH? If you don't know the answer, then you're failing at mentally modeling a very very simple real-world scenario of five coin flips, which has only 32 possibilities.
SnapDragon
17d ago
In fairness, the goalposts were moved because we realized LLMs couldn't do certain AGI things despite passing the "AGI" tests.
Yeah, no argument here. Like you said, it's kind of natural that we adjusted our expectations as we learned more about the nature of intelligence (now that we have more than just one kind to generalize from). We sort of assumed that a lot of other human-like capabilities would necessarily come along for the ride when an AI passed the Turing Test, and that was wrong.
Just as long as we don't keep using "it's not true AGI!" as a cognitive stop sign to avoid recognizing the incredible progress we've made.
Although I agree with SnapDragon that they're "partial AGI". I believe the missing component is continuous learning: they start output like a human, as they've been trained to, so if they continued to be "trained" on their observations, presumably they'd continue to output like a human.
Indeed. I've heard of efforts to graft a learning layer onto LLMs (with a "memory" that's an embedding rather than just CoT text), but obviously it hasn't worked so far, and maybe it never will. Also that still seems like a short-term solution.
SnapDragon
17d ago
Oh wow. Um, ok, how can I dumb this down as much as possible for you?
- You: LLMs make cognitive errors, so they can't understand the world!
- Me: Humans make cognitive errors, so they can't understand the world!
Having intellectual blind spots - like reading comprehension in your case - is not proof that you're not "modeling the world".
SnapDragon
18d ago
And what about the cognitive errors that humans make all the time? The rationalist community was founded on a list of widespread "fallacies", after all. To pick one field, I would argue that humans lack a true capability to understand probability. We lose to even basic computer programs at Rock Paper Scissors. Gamblers think Red coming up 3 times makes Black more likely next time. There are actual medical professionals who don't understand that a positive on a 90%-accurate test for a rare disease does not mean you are 90% likely to have it. Simpson's Paradox will fool almost anyone, including me.
And on this very forum (and ACT's), every so often I try to correct people about the Doomsday Argument, which, like Monty Hall, is easily modeled and shown to be false. Yet Scott - and a motivated subset of Wikipedia editors - believe it anyway. Somebody who believes something false is clearly lacking a "true capability to understand probability". But they can still be intelligent.
SnapDragon
18d ago
Heh. See, the AI making that Dyson Sphere doesn't have general intelligence, I bet it can't get the Wordle 6 days in a row like me.
SnapDragon
18d ago
I have the unpopular (and, ok, partially tongue-in-cheek) position that we've already hit AGI. What LLMs can do is already very general, just not fully general. But I wish it was emphasized more that we messy meaty humans don't have fully general intelligence, either - it doesn't matter how you bring up a precocious child, they're not going to be able to rotate 50-dimensional shapes or approximate partial differential equations in their head, and all but the best of us max out at fluency in a few languages, or memorizing a few thousand digits of pi. We're just so used to the things we (and everyone else we've ever known) can't do in our heads that we intuitively don't even think of them as tests of "intelligence".
Someone from the early 2000s, having LLM capabilities described to them, would indeed think that it meets the definition of general intelligence. What we kind of subconsciously expected, but didn't happen, was that someone would just suddenly launch an AI product that lit up a giant neon sign saying "AGI ACHIEVED!". Instead, the AI we've developed so far just turned out to have a different set of strengths and weaknesses than us. By the time we're able to bring those weak points up to human level - i.e., where an AI can perform equally well as an average human on any task, which is what a lot of people think of when they say "AGI" - it'll actually be vastly superhuman in the things that come naturally to it. (LLMs are already superhuman on language comprehension, after all.)
SnapDragon
18d ago
I don't think you can say for sure that they don't have a "world model" hidden somewhere in their trillion-dimensional space. I've certainly used them in ways that seem to require one, and while it's certainly possible that it's because they're faking it with statistics and I'm overestimating the difficulty of what I ask ... the argument does have to trail off at some point, right? You have to include some way to show that they really do "understand causal relationships" (even if it's just through preponderance of evidence), otherwise you're using unfalsifiable faith-based reasoning to assert that only human intelligence is real intelligence.
What they definitely don't have is temporal persistence of thought, just because of their actual mechanics. (CoT reasoning is a patch to this, but an imperfect one.) A priori, I would have thought this was necessary to do complex reasoning.
And I would advise heavily discounting anything Gary Marcus says. He's just enjoying a career as a self-purported "expert" that the media can go to whenever they want a skeptical quote, but almost every testable claim he's made has been wrong.
SnapDragon
18d ago
I agree that his posts are far below the average level of quality here, but they're not THAT frequent, and I wouldn't want them to be modded. This is supposed to be a free speech forum, after all. Our whole thing is that the good ideas are supposed to win over the bad ideas, and fortunately he seems to be getting plenty of pushback. And there are some decent debates happening in the replies, it's just that they're despite OP, not because of him.
SnapDragon
18d ago
No, you presented it as a conceptual proof that LLMs will never get better. All it takes is one innovation that addresses your concern about recycled data to make it invalid. All arguments about intelligence are necessarily a bit wishy-washy, mind you, so I'm not saying your thought experiment is useless.
I think if you really want to argue that LLMs have an inherent cap on their capability, you should address their actual algorithm rather than how they're trained. However much we rejigger them with CoT thinking and non-text data sources, they're fundamentally not designed for anything more than next-token prediction. It should be a source of constant surprise that they do so well on such a wide variety of non-creative-writing tasks (look at early SSC posts about GPT3's output to see this surprise evolve in real time). You could argue that if LLMs end up hitting a soft or hard limit, that's really just the "surprise" petering out, that we really can't just take a glorified text completer and keep pumping neurons into it until it's a genius.
I don't personally believe this will happen, but hey, I don't think anyone really knows for sure.
SnapDragon
20d ago
Speaking of real-world tasks that LLMs can help solve, here's a nice example of a new math theorem produced by ChatGPT Pro thinking on a problem for 80 minutes. (You can actually read the chat transcript, too).
This is a remarkable artifact. I would say that, barring the initial prose quality, this AI proof is from The Book. Perhaps the first?
(The Book is a term meaning it's the proof that "God" would have written down for this theorem.)
I care deeply about this problem, and I've been thinking about it for the past 7 years. I'd frequently talk to Maynard about it in our meetings, and consulted over the years with several experts (Granville, Pomerance, Sound, Fox...) and others at Oxford and Stanford. This problem was not a question of low-visibility per-se. Rather, it seems like a proof which becomes strikingly compact post-hoc, but the construction is quite special among many similar variations.
Pretty neat stuff. I think we're going to see a lot more open math problems solved by AI in the coming years (especially as we figure out the right CoT frameworks and prompts to use).
SnapDragon
26d ago
Since I generally respect you and your posts, I want to try this one more time. I don't necessarily buy that we should just declare this a "fundamental values difference" and say that we're now beyond any hope of rational agreement. And while you may have "coherent reasons for your position", that can be true of many evil ideologies. Evil =/= incoherent.
You brought up preferences, and I get the impression that you pattern-matched my ideology to a Rawlsian one that you should never prefer your own tribe, which is an extreme that I definitely don't hold to. I prefer my own happiness over others to a decent extent, and that goes for my family, my friends, my nation, and my species. I'm not asking you to give up that preference! Self-interest is the glue that holds a society of individuals together, and capitalism's magic is that it doesn't try to deny it, merely harnesses it in a way that doesn't degenerate into misery for all. I just don't think that preference should be infinitely strong: Beings in your outgroup should still matter more than zero. You shouldn't torture them horribly for a tiny gain, even if there are no repercussions. You should prefer a world where you're happy and they're happy.
You didn't respond to my main concern, which was that yours is the same "coherent" reasoning that led to many racial atrocities in the past. It doesn't seem very universally defensible, and often leads to horrific outcomes, when you simply draw a circle around whoever you know growing up and declare that this is the circle of beings that hold moral worth. Do you think Hitler's only mistake was that he drew the circle around "Aryans" instead of around humans? Or the African slavers who drew it around "Europeans"?
We currently live in a society where there's no friction between your ideology and mine, because humans are the only sapients around. (I'll set animal suffering aside, because I'm ambivalent on it too.) But it's very possible that, within our lifetimes, it will suddenly matter deeply, where our society will consist of both humans AND sapient AIs. All I'm asking is that you give some moral valence to the suffering of beings that are outside the circle you've drawn. Not zero, not infinite. It's a low-cost alteration to your ideology, and it stops there, I'm not trying to, uh, whatever the opposite of murder-Gandhi is. And if some of our ancestors had made the same small concession, so much misery could have been avoided.
SnapDragon
28d ago
I'm an unabashed transhumanist chauvinist, I think that only humans and our direct transhuman and posthuman descendants or derivatives deserve rights. LLMs don't count, nor would sentient aliens that we could beat by force.
Huh, I'm pretty surprised to hear this, and I have a deep ethical disagreement with you here. In my opinion, what is special and valuable about humans - and the thing that fundamentally gives value to the universe itself - is sapience. But we should cherish it just as much in a different form. (I mean, I agree LLMs don't count, but that's just because I see no way they, lacking persistence of thought, could actually be conscious.) Where does this bright line surrounding us "humans and descendants" come from? In a different era, your argument would easily pattern-match to arguments about subjugating other races instead. Why do black people now have moral valence, but some alien from Alpha Centauri wouldn't?
I'm not an expert in philosophy, but I do think there are solid arguments for acting this way (e.g. the categorical imperative). Just like I'm an atheist who still doesn't act like an immoral sociopath when I can get away with it, I think we as a species should not be focused only on our own well-being at the cost of all other intelligent species. Not because of the threat of punishment, and not even because I hope any aliens we meet would similarly value our well-being in a way that you wouldn't. But because existence will just be a better place if we can all get along and not act as game-theory-optimizing selfish machines, and I'm willing to work towards that.
BTW, I don't think your eating-a-pig example is a good one. It's irrelevant to the pig what we do after killing it. A better question is, would you be fine with torturing a pig while it's alive?
SnapDragon
28d ago
No, but somehow these days they're tuning their final models to get to "I don't know" anyways. Maybe they're not just glorified autocomplete? 10 months ago was the first time I got an LLM to admit it couldn't answer a question of mine (although it did still make helpful suggestions); not only did the other models back then give me wrong answers, IIRC one of them went on to gaslight me about it rather than admit the mistake. (two years ago this gaslighting would have been the rule rather than the exception) IMHO that "I don't know" was the exact point at which AI started to have positive utility for me. Sometimes an AI still isn't helpful, but it's at least often worth throwing a problem at now, not a waste of time.
That's good to hear. I'm not saying that the hallucination problem can't be mitigated, I'm just saying that it's a struggle and it's likely to continue to be a struggle, even if LLMs continue to get smarter for a long time. The way I think of it - which is definitely an oversimplification but possibly a useful one - is that next-token prediction really isn't the kind of intelligence that we wanted to develop, but it's what we discovered first. So in some ways - keeping models focused on tasks, preventing malicious usage, learning in real time, avoiding hallucinations - we're paying the cost of trying to pound that square peg into a round hole. With enough effort, paying enough training/inference costs, we often can do it. But perhaps at some point we'll discover a different framework for AI that better matches our own sapience at lower cost.
SnapDragon
28d ago
Here's a relevant AI Explained video about Mythos. Some highlights and personal comments:
-
7:28 Right now I think coding models are at their most powerful when being used as a force multiplier for human experts. (Akin to Cyborg Chess.) Here, a computer security expert mentions that he found more vulnerabilities in a few weeks than in his entire prior career. This ability to find zero-day exploits isn't an artificial benchmark, this is a real-world result that shows we really are entering some sort of new regime. Although ... I suspect statements like this are going to get so common that we no longer recognize how startling they are, like how we ignore the fact that models flawlessly understanding natural speech would have been considered miraculous 10 years ago. And we'll get more idiotic posts by so-called "skeptics" who think that spending 30 minutes failing at using AI counts as definitive proof that frontier models do not exhibit intelligence.
-
9:10 Safety concerns related to some prior discussion with @Corvos, @YoungAchamian, @roystgnr, and others. To quote: "In contrast, experts were consistently able to construct largely feasible catastrophic scenarios, reinforcing a view of the model as a powerful force-multiplier of existing capabilities." We're not close to the point of plagues being bioengineered in garages, fortunately, but at some point a reasonably-sized terrorist group with some funds and some expertise might be able to do a lot of damage.
-
13:23 I really don't consider FOOM to be a realistic scenario, and this is just more evidence. Individual researchers being made much more productive does not immediately translate into model intelligence; any real-world endeavour has dozens of bottlenecks (like training compute limits, here) that you can't just outsmart. It's similar to the popular visions of moon cities from the 1960s. Our imaginations regarding rapid technological progress always elide the difficulty of actually implementing it.
-
16:20 More safety concerns: Apparently it's still pretty vulnerable to an attack known as "prefilling", where you make it look like it's in the middle of a conversation where it's already misbehaved. This kind of makes sense to me - after all, no matter how much reinforcement learning you do, it is fundamentally a model designed to continue text, so if you want it to change course in the middle of a conversation, you're trying to override its most basic functions. If you're just using the model through the company's site, they can of course clearly separate their prompt from the user's input, but this might mean they'll have to limit unrestricted prompt-free access. And in some scenarios Little Bobby Prefilling might become a thing.
-
17:04 As they get smarter, it's getting harder and harder to run alignment testing on models without them knowing they're in an artificial scenario. Interestingly, though, since Anthropic has done a lot of work on introspection, they can actually artificially lower the weights for "I'm in a test", forcibly tricking the model. Like the way that we can turn image recognizers into image generators, this feels like another unintuitive consequence of running an intelligent mind as a program. We literally have the power to mind-control it, and I bet we'll get better at this. (This will be very unethical if AI develops consciousness - fortunately I'm quite confident LLMs don't qualify, but unfortunately I don't think we'll stop doing this even if AI does cross that threshold. AI welfare is something I'm genuinely worried about for the future.)
-
20:30 So-called "hallucinations" are of course still happening, and I still suspect this is something that we'll never truly defeat, again because of how LLMs work. You don't complete the sentence "The answer is" with "oh wait never mind I don't know". Models might get smart enough to know the answer to most of the things we ask them, which will help, but getting them to precommit to not knowing something (before they begin with the bullshit and can't back out) is an uphill battle.
Unfortunately, while absolutely true, this is a tough message for politicians to sell. (There's a reason we haven't had another Reagan or Thatcher.) People take it for granted that they have all this stuff that barely even existed 50 years ago, as if this is just the way the arrow of time works. It's a lot easier to promise gibs to people who see that their neighbour has something they don't.
In fact, don't people who are anti-pornography say that it harms society because men use it as a low-effort substitute for going out and finding a real woman? In the case of pedo porn, this is exactly what we want to happen.
More options
Context Copy link