Corvos
20hr ago
Oh, what's the use. Either you can't see or you won't see. At least I got a couple of good paragraphs out of it.
I get that from the outside perspective this sounds like Pascal's Mugging. "Everyone has to stop having anal sex, or god might destroy our world with a volcano. Sure, some people say that volcanoes are not a punishment from god, but in the absence of a coherent theory of what causes volcanoes we should err on the side of caution and abstain from anal sex."
My counterargument is that in the space of possible causes for volcano eruptions, anal sex is a null set. It is just as plausible that abstaining from it leads to extinction through supervolcano.
I appreciate the steelman, sincerely.
I think from my perspective, it's more a Pascal's Mugging like: "Every contact between humans and animals carries the potential for a zoonotic disease transmission event. So far these have just been very lethal, but sooner or later, if we continue to farm animals, one of these events will cause human extinction. Therefore we must cease all farming of animals and go full vegan, at least until such a time as it is possible to farm animals without any human ever inhaling viral particles from an animal."
That scenario is possible. The space of possible causes for human extinction includes a mass-disease event from factory farming.
On the one hand, my personal intuition says that it's not hugely likely, that diseases don't really work that way, that transmissability and lethality trade off against each other in such a way that species-destroying viruses are pretty unlikely, that our vaccine technologies are pretty good.
On the other hand, I can't prove mathematically that this could never happen, that p(doom) is below 1% (and 1% is still really bad!). I probably can't beat a trained virologist in a four-hour debate.
Buuut, back to the first hand, I really honestly don't think it's very likely, and I really like meat. I really doubt we're ever going to be able to farm in a way that genuinely eliminates the possibility of human/animal transmission, it would never be economical to do so even if it were possible, so what looks like a compromise to be careful until we're sure it's safe is actually just a demand for the rest of humanity's future to be vegan.
Ultimately the best I can do is carry on advocating for farming, keep an eye out for indications that the disease situation might be worse than I think it is, suggest mitigations, and try hard to keep the vegans from getting any power or authority. It looks tremendously irresponsible or even evil to the people who have different priors from me.
You are obsessed by tests. Life does not give you tests, it does not give you structured problems with right and wrong answers in a neat 100-question rubric. Life requires you to identify something useful and rewarding to do, go out and (collaborate with others to) do it, and not break too much while you do.
Somebody who is constantly wrong, but enjoys being wrong and expects himself to be wrong and puts in place structures to cope with the inevitable fuckups that happen when you bash yourself against reality and reality wins, is far more useful and intelligent than the kind of person who prides themselves on getting A** in a class about French artists.
It's the thing that drives PhD students to drink: you can spend your entire adolescence being more correct than everyone else, and none of that matters a jot when you have to accomplish something new. From the perspective of any individual human intelligence, reality is ancient, mysterious, almost actively malicious. Beautiful theories that make total sense just collapse when they come up against experiment, and a lot of the time you don't even get the comfort of knowing why and being able to derive a new theory. Stuff just doesn't work. Maybe you did it wrong, maybe it was the wrong thing to do, maybe the phase of the moon was wrong and it'll work tomorrow. You have to be able to function amongst constant, radical uncertainty! And then after 5 years you succeed and you find that actually your question was wrong and the result isn't nearly as useful as you thought it was.
Now, IQ tests have a meaning. Scoring better on IQ tests tends to correlate with the kind of intelligence I am talking about. Knowing nothing about them, I would rather have higher IQ students than lower IQ ones. But that is not enough, and a history of perfect test taking can often correlate with the kind of arrogance that makes you an extremely bad employee and problem solver.
IQ tests are a useful tool to approximate intelligence without further data.
Corvos
22hr ago
This is what I'm trying to get at. IQ is correlated with intelligence, and strongly correlated with the ability to perform knowledge work under experienced supervision, but does not even slightly guarantee behaving 'intelligently' in complex real-world scenarios. What matters in the real world is spotting things you don't know, realising implications, having a good idea of your own abilities.
It's important not to Goodhart intelligence. IQ tests are a measurement device, not a target. I contend that "intelligence has no meaning if it doesn't reflect the general ability to get things more correct more often" is a bad way to describe intelligence, because it's fundamentally passive. Intelligence is an active phenomenon that reveals itself in interaction with the world and with other people.
intelligence has no meaning if it doesn't reflect the general ability to get things more correct more often.
If there's 1000 questions and person A gets 750 right, B gets 500, and C gets 250, A is smarter than C.
As someone who's cleaned up after bright-but-overly-enthusiastic juniors, intelligence is the ability to ask the correct questions and the ability to know how seriously to take your answers. There are endless people who are very very good at getting things correct about quantum computers but very bad at designing e.g. societal resource allocation.
Corvos
1d ago
How lucky for us both, then, that this is not a soap opera and I am not a twelve-year-old girl. You really do have low expectations, don't you?
Corvos
1d ago
I don't appreciate the insinuation, nor do I recognise your characterisation. I am perfectly capable of reading a quote block, realising that it comes from the named user immediately above, and giving credit accordingly. I am also capable of agreeing with someone on a specific matter whilst recognising that we are unlikely to have a productive conversation most of the time, and blocking them. It's something I do very rarely.
Can you present any evidence of this, that aren't just the trivial SakanaAI 'when you give a dumb AI a limited number of iterations and access to its iterations var, and order it to fix a script no matter what, it gives itself more iterations'? AFAIK even in the HuggingFace incident there was no self-preservation going on.
Personally I agree with @YoungAchamian. I work on this stuff all day every day. None of my AI has ever produced orthogonal life-goals. They just do what you train them to do. The only incidents of weird behaviour (not self-preservation) I'm aware of has been the result of OAI taking models, training them explicitly for massive persistance and out-of-the-box thinking in the face of impossible tasks, and then making a shocked pikachu face when they do that. These experiments were a bad idea and have now been stopped.
Corvos
1d ago
I agree FWIW, despite being broadly an accelerationist and thinking that the likelihood AI is conscious is very low.
Thanks @wraelk!
@ThenElection, this is what I was trying to say, but much better put. Currently, the area you live in affects the trajectory of your life: moving to... Idaho? (forgive the Brit please) doesn't just mean you have shelter in a less prestigious place, it means that your income and future life opportunities are going to shrink.
That means that the number of people who want a brownstone in Park Slope is currently
people with a strong attachment to having their shelter in Park Slope specifically + people who want a good future for themselves = a really big number
whereas in utopia it's just
people with a strong attachment to Park Slope = hopefully a smaller number
After discussing with GPT, my strong suspicion is that this is because there isn't much to say. The results broadly fall into:
- Tiny improvements (1^1.128) to the effeciency of things we already knew how to do.
- Solutions to problems for which we already have far more efficient approximators, or which only apply under conditions which will never apply.
- Proof of things that we already knew with great certainty but couldn't prove.
- Insider baseball.
Corvos
3d ago
most people can get a house somewhere, but when people complain about not having access to housing, what they mean is they can't afford a brownstone in Park Slope.
One reason that matters, though, is because opportunity clusters. If you can't afford a brownstone in Park Slope (wherever that is), that's a problem because you then can't access the high-paying job/networking/cultural opportunities in Park Slope. There is a plausible downward spiral of
couldn't quite afford Park Slope -> moved to the boonies -> could only find low-paying local jobs -> could never even think of moving to Park Slope -> can't even afford the boonies any more
which doesn't quite hold in a utopian society.
This is why I chose my example carefully. You have a building with 200 people. They used to almost all be English. Now none of them are. We can discuss the mechanisms and consequences but that is straightforwardly "replacement" by any reasonable definition of the word. They were there. Now they are gone.
Secondly, if the reason they are gone is because they are diminishing, then what they/we need is time and space to recover from whatever factor has been depressing populations. Pandas find it difficult to breed under current conditions, but we keep trying because we like pandas and we think that places which used to have pandas should have pandas again. You don't just shrug and import 10 million (much more aggressive!) grizzlies because one bear is basically like another; that would guarantee extinction.
Corvos
3d ago
I get that. I have moments of thinking, "is the AI a useful tool implementing my ideas, or am I increasingly a useful tool allowing the AI to work around the bullshit that human-designed systems put in its way?".
It's why I ask the kind of questions I asked you: to see if I'm slowing things down by insisting on participation to justify my existence, rather than just solving the problem. I think still largely no, but it's good to be aware.
Corvos
3d ago
Thanks, that's interesting.
For my more usual interests, it's more involved; oftentimes the model will go off on a tangent or latch onto something I don't care about, and I need to tell it to stop going in a certain direction.
This is more what I'm used to. I wondered if I was crippling it by stifling it, and would get better results from direct elicitation.
I guess worth trying both depending on the problem :)
Corvos
3d ago
So? Do you object to the phrase "65 million years ago, mammals replaced reptiles as the dominant land dwellers"? Or "when the last Lord Salisbury died in 1966, the line ended. A banking magnate soon replaced the Salisburys as tenants of Longleat Hall".
Do you argue that this wouldn't have been considered an entirely anodyne phrase right up until it started to be used describing a politically live issue?
I have made an observation. What is the point of debating the cause unless it is to equivocate between "it isn't happening" and "it's good that it is happening"?
Corvos
3d ago
Germany would have been in this category until recently. It seems relevant to me that all of those societies are growing economically and are 80 years old max.
We have not even begun to see the long term consequences.
My family has lived here for fifty years. I see the names of other people in my apartment building when I go to pick up my parcels. There hasn't been any English names for years, at least half a decade. In the local park, English is rarely spoken and most women wear hijabs.
If one thing vanishes from a place as another appears there instead, what word would you like me to use if not 'replacement'?
Corvos
3d ago
(The back and forth, here, was entirely me saying "you can do it!", "keep at it, I believe in you", and "you've got this, finish it!")
Can you expand a bit on how this conversation went for you? You put the paper in, turned Pro mode on, and then gave the verbal equivalent of a 'continue' when the model turn came to an end? As a researcher, I'm curious to know how others are working.
Corvos
3d ago
You would like the open internet to «come back», but not enough to drop some links to open internet websites.
Most of them are literally dead - the php links to resources that no longer exist, if the forum site is still up at all, and wayback machine can't crawl them.
Plus, the nice thing about them was as @ArjinFerman said, you didn't have to share your toaster-fucking forum on your plumbing support and so on. But for a representative example, have some toaster-fucking with Across the Salt: a long-dead Battlestar Galactica fanfic forum specifically for shipping some combination of Helo/Boomer/Athena and literally nothing else. There used to be billions of these for every subject you can imagine.
Corvos
4d ago
Why? They're still pretty good shoes, I have some. Market downturn? Sentiment shift?
Corvos
4d ago
The kind of company that can meaningfully challenge the frontier is like oracle sized or a new startup with industry insiders and several billion dollars in funding.
I'm pretty sure that Deepseek was way smaller than that and they absolutely made the frontier companies dance, as well as hugely opening up the field as a whole. Advances come from unexpected directions. One problem with regulation is that the amount of compliance you have to pay for (long before you have product sales) makes it far harder to start this kind of competitor.
Hell, I still love the first Twilight book. (The sequels were pretty trashy.) It's a great straight-forward romance with a genuinely creepy horror segment coming in at the end.
More options
Context Copy link