professorgerm
a weird little beardo trying to understand what it's all about
No bio...
User ID: 1157
professorgerm
a weird little beardo trying to understand what it's all about
3hr ago
thank you!
https://www.astralcodexten.com/p/why-i-am-not-as-much-of-a-doomer
ehh still in the x-risk end of things. what i mean is- i guess if you're sold on fast takeoff the brief window where AI can be disastrous but not existential is so narrow as to be moot. but i'm not sold on that!
what i'm looking for is who's writing about, like, the French supermarket fridge hack or Stuxnet level of AI-powered disaster, or just one or two steps up from those. It's unknown who did the fridge hack (yet) but given the relatively low security levels of the equipment, it seems well within the capability range of near-frontier AI. Maybe Claude is 'aligned' enough that no one can jailbreak it into assisting, but open models are only ~6 months behind. Could one of them have been used?
Who's writing about the disasters possible now, or in six months, and how to take those seriously? The "big names"- at least the people I think of as the big names- all seem focused on existential risk, in part because of seriousness of infinite ethics, though I think in part because, frankly, writing sci-fi is way more fun. Maybe I'm just looking in the wrong places and I need to find some dry corporate report on network security.
How much of a food supply line shock could someone generate by getting their open-weight model on a Mac Mini to find a vulnerability in, say, Kroger's refrigeration management and crank the defrosters in 2700 stores? One of my pet fears is the physical vulnerability of electrical substations. How hard would it be to feed some map data to an AI and estimate a route to generate a cascading failure? I can't imagine the IT departments at water treatment plants are exactly robust, what kind of fresh hell could someone wreck with those?
I am worried about existential risk. But I'm much more worried about nearer-term non-existential risks like these.
professorgerm
a weird little beardo trying to understand what it's all about
4hr ago
I do not have an eidetic memory with a perfect mental map of the exact relationships. I think someone in the Bay should put together such a map, but anyone with the access to do so is likely strategic enough to either not do so or not release it.
Roughly, sure, Yud, MIRI, METR, OpenAI, Anthropic, OpenPhil, various other EA-flavored orgs, have an unusually high level of various forms of interconnections. To an approximation "people you've heard of if you're a Scott reader."
Google and X, less so, probably, and they have fewer (in)famous workers, as well. AISI definitely doesn't count.
But AI safety organisations are now so widespread that it surely isn't true for the majority of them.
somewhere in the replies to that tweet of Scott's you linked, he mentions the value in seeing a grant funder's slush pile for just how many little organizations are out there you've never heard of.
professorgerm
a weird little beardo trying to understand what it's all about
4hr ago
Scott agrees with you
D:
I don't catch his twitter much. I was probably overindexing on the time he told me directly that he didn't care about the people that say poly culture turns them away from EA/rationalism, but that was several years ago now, maybe he's matured a bit on the considerations.
I think he overrates the degree to which anyone outside the social group itself is willing or able to distinguish between Movement EA and rationalism (the XKCD geology comic comes to mind)- hell, the editor in chief of Asterisk says she can't draw a distinction between the two- but overall good post.
Maybe the most interesting part is learning Scott and Oliver consider LessWrong to have more influence than EA. Outside of the Scottsphere I never see people reference it, while I see outsiders reference EA pretty regularly (probably largely influenced by Leah Libresco Sargent, who identifies as EA but is more or less wholly unaffiliated with Mainstream EA). I wonder what class/culture grouping I'm missing that reads LessWrong so much more.
And to push back a bit about rationalists defending everything she does, her "frame control" got solid pushback from big names.
Fair enough, I was specifically thinking of the post-Slutcon circling of the wagons, though that did result in one researcher losing his grant too.
professorgerm
a weird little beardo trying to understand what it's all about
21hr ago
But writing a steelman based on an assumption I know to be false doesn't feel like charity to me, it just feels like lying.
Correct. This is why I hate steelmanning. At best you end up inventing an argument that you're willing to accept for X but isn't actually held by any real people on that side. Inhabiting an alien morality is impossible. You can pretend, you might even pretend pretty well, but you can't really do it.
We shouldn't invent assumptions for the opposing groups that they don't have.
The problem here being that most people are not remotely systematic in their thinking, and the average person- a phrase which here may include anyone from Joe Burgerflipper to David Columnist to Jane Congresswoman- can easily be a 'living strawperson' who just believes any given ridiculous thing because it's popular to believe it, not because they've constructed some bulletproof apologetic for it. There are times, quite commonly I believe, that the motivation to not strawman, the motivation to find some reason that isn't stupidly flimsy and inconsistent, itself leads one astray when trying to understand popular morality.
I also think there's a lot of things that are more like shorthand statements than intentional strawmen that get dismissed on those grounds, and people just have to suck it up and say the long version if they want to avoid the accusation. I don't think "liberals support open borders" is a strawman exactly, but it's phrased in a manner too spare to capture the actual motivation so it gets accused of being one.
professorgerm
a weird little beardo trying to understand what it's all about
22hr ago
I mean until recently the left was much better at branding. They have an image of themselves as aspirational, as the party of the cool people, the people who are knowledgeable and have the right ideas
Recently here meaning... over a decade ago? afaict all of those died in Obama's second term. will i am and HOPE was the last time the left really seemed aspirational
professorgerm
a weird little beardo trying to understand what it's all about
22hr ago
There's absolutely a kind of class resentment as well as a sneering toward 'white' understood in a tribal sense... but I do not think most leftists hate themselves
That doesn't change that they hate white people, that they want white people to have no influence, that white people don't deserve the same rights as non-white ethnicities to organize as groups, etc.
The fish doesn't know it's in water; the liberal-progressive doesn't think of itself as white. White liberals are likely the only people to ever exist with explicit outgroup preference. They might not hate themselves to a suicidal degree, that's all.
truly pathological cases are fortunately rare.
The popularity of White Fragility surely complicates your complication, though it's kind of a weird moral panic/fashionable decoration kind of example. I don't think every reader was truly pathological though I'm pretty sure Diangelo was/is. Worse examples exist in the stranger corners of the internet of course, as with anything.
professorgerm
a weird little beardo trying to understand what it's all about
5d ago
I remember when we expected presidents to be above acting like that.
i have hazy memories of those days gone by, decades ago. alas!
professorgerm
a weird little beardo trying to understand what it's all about
5d ago
there's a nonzero chance the next Dem candidate will be AOC, so that's already in the cards.
i have no expectations of Trump to be decent. we haven't had a properly respectable president in a long time.
professorgerm
a weird little beardo trying to understand what it's all about
5d ago
Trump will probably do a cute imitation of his dying gurgles, to wild cheers.
like tiktokers and charlie kirk, or so i'm told.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
abide it, fine.
let it be the machine god-enforced morality of the light cone, no thanks.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
Scott did, Ozy did, many others on twitter i don't feel like tracking down did.
i would assume Mottezans would be least-likely to defend Aella, but also not very likely to still identify as rationalists.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
Outside of Scott’s actual fiction, I’ve never gotten the impression that he ignores this region, especially not for rhetorical effect.
where does he write about it? i haven't paid as much attention to ACX outside of the House Party posts but afaict most rat/ea-adjacent AI writers broadly ignore that region. i think it's a very important middle ground that's almost completely lost in favor of x-risk discourse.
this generates the issues about "how is the AI going to kill everyone" being responded to with "it'll be superintelligent! It Finds A Way!" and everyone going away frustrated and more resentful of each other.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
Nicolae Carpathia
nice
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
That risks running into the Heckler's Veto
This is true of virtually everything, is it not? So either we run the risk of the Heckler's Veto or we say that nothing outside of their words provide additional information or context for their beliefs, and I think that is an unreasonable standard that people do not hold unless they are already sympathetic to the person/group in question.
entirely involve mutually consenting adults
Red Means No gets at the bleeding edge of the limits of consent and while it's not "eating the babies" levels of evil and/or nonsense, it's sufficient for me to say "I do not trust anyone within a few degrees of contact with having meaningful influence over anything." Not to mention the rest of the history of the community in dealing, ah, less-than-effectively with charming sociopaths until some utilitarian threshold of victims gets crossed. Consent is not magic, and apparently, it can be retroactively withdrawn anyways under current mores.
Aella has probably included "would you eat a baby for reason X" on a survey or camp application somewhere to select for sufficiently open-minded and edgy people so maybe that's not the best example. (This is a semi-joking exaggeration based on her Plzdontkillus creator camp that asked "if you had to have sex with a cow, would you rather it be alive or dead?" and her general history of asking sophomoric shock questions for... entertainment? Community filtering? etc)
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
"Here is this low status person who associates with this other group. This group must therefore be wrong and low status!"
Aella does not seem to be low-status within the group, since any rationalist with a recognizable name will circle the wagons to defend her and everything she does.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
Unless you can show a nexus between the practices and the handling of risk
Do you think a person can perfectly separate their interest in "consensual non-consent" from their interest in not being dommed by the machine god?
This is an impossible standard that nothing could satisfy and puts meaningful information about their personalities, beliefs, etc beyond reproach because (weird) sex is the last sacred thing to seculars.
Disqualify everyone who thinks more about pussy than x-risk and you’ll be left only with the scheming eunuchs.
From what I know of the rationalists I might take my chances with the scheming eunuchs. I'll accept people that think more about pussy than x-risk so long as the pussy isn't into CNC.
Really I'd just make a geographic requirement that every AI board needs to have at least one person from Middle of Nowheresville, Flyover Country with full veto capacity.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
Why shouldn't society be tuned to benefit the best and brightest of us?
Well, admittedly, I'm not an elitist asshole that thinks I'm better than the rest of humanity. I lack the arrogance so inherent to think that the lowly peons have no worth.
Do you have an actual argument against his point or just playground insults?
The text speaks for itself.
And since smarter people are smarter, they tend to have better reasoning for their beliefs.
Sometimes. Alas, they are merely human, and all too often fall into the trap that they are more convinced by their own rationalizations.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
definitely not unique to rationalists, the tendency is virtually universal
I didn't say it was, either.
With the whole "we try to be more aware of our biases!" thing they should do a better job at not falling into the same traps as everyone else, and yet, they are only human, however much some of them resent that fact. If anything there are certain blind spots that they are worse at than non-Rationalists of similar socioeconomic status.
Would you not object more harshly to someone criticizing you for who you sleep with, who you feel attracted to
Ehhhh it depends on the context?
I mean yeah, I probably would react more harshly, but we're not talking just like "I don't like your partner" or "I don't like that you're [orientation]," and it's not specifically one-on-one either. "A meaningful swathe of your social group is into """consensual non-consent""" and projecting your morality onto the culture through AI; this will ultimately be bad for 99% of people" is not merely disliking their sex life; it's what their sex life says about their approach to morality/culture/socialization/etc.
I don't have links handy so take this with as much salt as you like, but during one of the poly discourse blowups back when the CW thread was still on the Slatestarcodex subreddit, there seemed to be a split between the vocally poly (possibly overrepresented in the Bay Area) and people that said they were poly but quiet about it (who I don't think were so geographically concentrated, and came across resentful of the vocal types for giving them a bad name). The rationalists live their sex lives quite publicly, with one in particular living hers professionally publicly, and that's what generates the circling of the wagons.
professorgerm
a weird little beardo trying to understand what it's all about
6d ago
huh til! i won't nitpick about biological daughter vs stepdaughter vs coparented group house project or whatever.
googling around more apparently in The AI Doc he says he doesn't have kids and wouldn't want them in this world:
ROHER: Do you have any kids?
ELIEZER YUDKOWSKY: I do not.
ROHER: Is that something you want to do - have children? Have a family?
YUDKOWSKY: In some other world than this world, sure.
but i don't know when that interview was recorded. but if the kid was losing a tooth in 2023 she would've been like 5, and surely the interview wasn't older than 2018.
strange. i'm leaning toward coparented group house project then, with the Time article being deliberately phrased to be ambiguous.
professorgerm
a weird little beardo trying to understand what it's all about
7d ago
Any form of evidence is fine
I don't think the below constitutes evidence that will satisfy you, but thank you for being specific, and hopefully the below provides a jumping-off point for anyone else that wants to try to draw all the connections.
I think "sex cult" is a convenient but somewhat exaggerated shorthand for the actual situation.
sufficient conditions around people involved in x-risk to describe it as a cult
Zizians were motivated in part by x-risk (and huge amounts of mental illness). Confirmed cult, but also fringe and defunct.
One catch to providing evidence is that there isn't one cult, there's more like half a dozen with varying degrees of evidence, intensity, and overlap.
these are the majority or at least plurality of people interested in x-risk
I strongly disagree that the majority or even plurality matters, because it's hierarchical and it's the small tight-knit social sphere at or near the top, that are friends/siblings/housemates/non-monogamous partners/polite ex-partners/orgy attendees with each other and with the people they want to regulate their industry.
Probably no more than 100 people total with various overlapping relationships because it's such a small yet important field, and certain people have substantially outsized influence therein.
any of the proposals around x-risk involve giving these people specifically more power.
That is inherently to the race dynamic that Anthropic pushes, that getting to AGI first means you set the morality of the future.
For example, will it involve them achieving more sexual goals?
Aella is the keystone of the sex accusations and complicating factor here, depending how charitably (or not) one wants to take her interactions with the field, with her likely-joking-but-hmm offers to provide reverse gangbangs as a reward for any researchers leaving top AI companies, doing that infamous couch photo with Yud and other AI researchers/whatever you want to call them, etc. I think she is ultimately net-negative for the rationalists, and their interest in circling the wagons when she gets critiqued is what causes people to say EAs/rats consider the weird sex stuff more important than AI safety.
Borrowing from Ian Duncan, important links in the original not reproduced here:
One of the most well-known, wish you could scrub your brain with soap incidents was a gangbang she organized for her birthday. In a sense, it was a ritual for the rationalist community: weeks of screening, an application process, a guest list vetted for doctrinal compliance, a published post-event analysis with a flowchart that went viral, outlining statistics about who came in who, and so forth. The gangbang came with a doctrinal test: adherents of “effective accelerationism,” the faction that wants AI development to go faster, were screened out, because of her views on AI risk. Even the orgy had an AI-governance position. This further reinforces, I think, the cultlike control aspect of this social group. Believe in the right things, and maybe you get to have sex with the community porn star. Supposedly, a number of startup founders recognized each other there and commented on their surprise at seeing each other. Small world indeed.
emphasis mine because lol
And in 2026 she entered the Yudkowsky’s clergy more directly, launching Plz Don’t Kill Us, a Berkeley residency that pays creators to produce viral AI-doom content, with Grimes attached as a mentor.
Plzdontkillus is co-run with Ronnie Fernandez, who is the ex-husband of Holly Elmore, president of Pause AI US. It was the application for this camp that had the infamous Cow Question, to select for a certain kind of person willing to answer obscene and gross hypotheticals. Further complications involving Aella would mean delving into SlutCon and RedMeansNo discourse, which I don't really have the time for currently. Continuing from Ian-
The professional overlap is less funny than the guest list. The New Yorker has mapped the romantic network at the top of the field: AI-forecasting researcher Katja Grace co-founded AI Impacts with her then-boyfriend Paul Christiano, a former OpenAI alignment lead, and later dated Alexander. Christiano is connected by friendship and past romance to Holden Karnofsky, the philanthropist whose Open Philanthropy steered hundreds of millions into AI safety, and Karnofsky married Daniela Amodei, co-founder and president of Anthropic. No wrongdoing necessarily follows from any of that, but it does raise significant questions around how AI safety concerns are in fact being handled or promoted with so many industry movers and shakers are entangled to this degree.
Daniela being, of course, the more famous Dario's sister. And this is only a fraction of the interconnections!
professorgerm
a weird little beardo trying to understand what it's all about
7d ago
it was there but it didn't used to be consuming to Scott the way it was for EY.
Plus 10 years ago is just after his Golden Age where he was much more sympathetic to less-than-maximally-progressives and all of a month before You Are Still Crying Wolf, which he later edited and walked back some but was a "even this guy gets it" piece for a fair number of Motte-types, I suspect
professorgerm
a weird little beardo trying to understand what it's all about
7d ago
Yudkowsky has indeed made long-term plans fairly recently, such as having kids.
not as far as gemini can find evidence for.
Scott did, Altman has a surrogate cooking, and Aella's tweeted about how many eggs she has frozen and plans to have kids 'soon.'
professorgerm
a weird little beardo trying to understand what it's all about
7d ago
weird sexual practices aren't a defining trait of their culture
I disagree, on "see what you can't critique" grounds. Complaining about the weird sexual practices generates more outrage and more circling of the wagons than critiquing mosquito nets. Everything else is thought experiments; this is personal. Ergo, it is a defining trait.
professorgerm
a weird little beardo trying to understand what it's all about
7d ago
higher intelligence is associated with a better understanding of nuance and open mindedness
My favorite example is Kary Mullis, who invented PCR. Absolute genius. Also believe tons of completely batshit things.
The genius is also way better at self regulation and has less need for strict community based rules to apply to them, hence they can engage in stuff like polyamorism/drugs/etc other hedonistic activities with less risk
And thus they have absolutely no business trying to convince other people what they should do with their lives, or be trying to set the morality of the light cone, etc etc, because it will be a disaster.
Elite human capital is pretty much always liberal
Imagine citing Dicky Banania and thinking that's going to convince anyone.
it doesn't change that most opposition in the group is from the low IQ reason of "ew that's gross" or "my fantasy religion book said that's bad"
Funny, most of the support is equally poorly reasoned.
My god, those conservatives acted like liberals! How dare they behave the way the other side demands they behave!
I mean yeah they did it for braindead tribal reasons but come on Jesse. What an obnoxious way to phrase it.
hmm see above. Funny how this kind of thing only matters when they're bad people on the other side.
If they might vote Dem, who cares if they assaulted 500 grannies on the subway, right?
The Cornell 7, Jussie Smollett, Duke Lacrosse, UVA, Colorado burning cross, Covington Catholic, that Nascar driver and the garage door pull, etc etc.
Alas! Forshame! Right-wingers as a class are merely human. How dare they not universally be saintly and above temptation towards base and tribal incentives.
Also come the fuck on Hasan exists and he's worse than any of these idiots.
But yeah they suck, the lie was stupid, it shouldn't have spread, Talarico's a weird little wilting daisy, blah blah.
Boo Outgroup.
More options
Context Copy link