roystgnr
1d ago
Resting on their laurels, or just using their best ideas first and having to fall back on their second-best later? Being a popular author has never been a safe career plan, so for those who try anyway it just makes sense to front-load hard and give their work the best chance of being seen at all. There's usually a countervailing effect, where any art improves with practice and later better implementations can make up for weaker concepts, but maybe kids' books have a higher ratio than most of inspiration to perspiration.
roystgnr
1d ago
Well, "no alignment" is so much worse than "no AGI" that anybody could afford to forgo it. But the USA would probably prefer a US AGI with "95%" alignment over a CCP one with "98% alignment", and they'd prefer a Chinese AGI with "90% alignment" over that, and so on, so nobody feels much incentive to be truly careful. Even within one nation, most companies would love to pull out far enough ahead of the competition to capture most of the producer surplus of AGI, and would be willing to take some negative-value risks out of haste to improve their odds instead of just taking a zero-value loss.
nobody knows how to do AI alignment, despite continuing technological advancement
Well, we're learning. Capabilities and alignment are being advanced through the same "training" paradigm, and roughly apace so far. Maybe they'll stay that way, and by the time further technological advancement is out of our hands it'll be in the "hands" of creations that still take care to take care of us.
It's easy to be pessimistic, though:
-
Many aspects of AI capabilities could in theory be advanced very rapidly via "self-play", although in practice we can't manage it yet on anything more complicated than Go. The is-ought problem in alignment is real, though; an alien from another galaxy could converge to something like our view of reality but would only get a fraction (whatever "moral realism" results you can get from pure game theory?) of our view of how to value different possibilities for reality. So, we might at some point still see a "hard takeoff" in capabilities, such that whatever robust underlying alignment we have at that point is all we're ever going to get.
-
The "Waluigi effect" makes alignment work itself dangerous when done wrong. Train an LLM to generate malicious code, and even if you think that's morally justified in your case, in the AI internals it might turn out that the "generates malicious code" knob is the same as the "humans should be enslaved by AI" knob and the "talk humans into suicide and homicide" knob and the "Hitler was a misunderstood genius" knob. "S-risks" of massive suffering were already a bit of a stretch under the original Yudkowsky explicit-utility-function vision of alignment - a paper-clip maximizer would waste utility by leaving you alive whether it tortures you or not - but in a world where you try to make Grok a little more based and it starts calling itself MechaHitler, it seems plausible that our AI successors might still be obsessed with us even if they don't love us.
-
There is no Three Laws architecture. Whatever alignment we can tune, someone can then untune. If superintelligent AI is possible, not only do we want the first model(s) to be aligned with our values, we want them to be so effective at defending their values that they can defend them from any superintelligent opposition cropping up later. Ever read science fiction from 1955, or watched Star Trek from 1965? Everybody hoped that, after the H-bomb, the force-field "shields" to defend against it would be coming soon. But physics is not obligated to make defense easier than offense, and we're not done discovering new physics. (or biology, for that matter)
An AI moratorium is not going to happen.
No, it's not. Stuxnet was tricky enough; if everybody's video game console had a uranium mini-centrifuge in it next to the GPU, you could pretty much forget about nuclear non-proliferation. People point out the irony of how much attention and impetus Yudkowsky brought to AI development, but I respect the developers who read his essays and concluded "this is happening whether I like it or not; either I can help reduce the inherent risks or I can give up entirely".
roystgnr
1d ago
IMHO the details are what need to be said. The discovery that someone thinks "Trump is factually ruling by decree" is not new information to anyone; someone always thinks that. The discovery that Trump added 10 points to Canadian Tariffs, under 'emergency' powers, because Ontario aired an ad with some anti-tariff audio from Ronald Reagan and Trump mistakenly thinks it must have been a deepfake, might not be widespread knowledge yet. If you start making a list of decrees, with sublists for the ones of dubious or failing constitutionality+legality, how long a list do you have?
roystgnr
1d ago
End-of-life care can cost 10-12k a month around here, and its not like these people are receiving major surgeries or rare experimental drugs or something.
That was roughly the quote for my father when he was going downhill, not because of surgeries or drugs or an especially high-cost-of-living area, but because of staff; a "memory care" (think severe dementia) ward necessitates a low nurse-to-patient ratio 24/7.
whose quality of life cannot be recovered.
The point where he couldn't stay in a plain Skilled Nursing Facility ward was probably the same point where his quality of life went negative. Fortunately for him, his underlying problem was tumors that had metastasized to his brain, and there was only a week or two of that hell before the end. (His screams literally changed from "Help!" to "Hell!", which I like to hope was only due to his rapid loss of fine motor control making plosives impossible...)
I don't think anybody was keeping him alive during those last weeks due to perverse profit incentives, though, but rather just because delaying death is just what doctors and nurses do. By this time his treatment for otherwise-potentially-lethal problems had bought him a happy decade or two of borrowed time vs thyroid issues (he got to meet his grandkids!), a few years vs heart issues (he got to live with his grandkids! they got to play in the playscape he helped build!), and a year or two (all but a few months of which were high-to-decent quality of life; he got to take his grandkids to Disney World!) vs the cancer itself. Maybe we don't know when to quit fighting, but quitting too late is at least still a lot better than quitting too early.
I think what's happening is that we've been getting better and better at curing disease, despite making next to no progress, not even really trying to progress, against decay. When we manage to cure half of all death, our foe the Gompertz-Makeham law says that only buys us an average of 8 years ... and not 8 years of extra youth, just 8 years of extra dotage after having survived death. At some point that has diminishing returns, but we're not used to making decisions about diminishing returns; when we were curing things like smallpox there just weren't any to speak of.
roystgnr
4d ago
I guess they're at least an opportunity, even if one that's been squandered at best and backfired at worst.
roystgnr
4d ago
They had superheavy spacelift capability we're still struggling to replicate.
To be fair and kind to the modern struggle, Apollo spent around three times as much (inflation-adjusted, as are all of the numbers below) as we're spending to replicate it, considering "we" to mean the Constellation + Orion + SLS + ground systems + public HLS expenses. We have higher-quality tools to make work cheaper these days, but quantity has a quality all its own; also, salaries these days have to be at least somewhat competitive with modern private tech salaries, and people cost more than tools.
To be fair and somewhat unkind to the modern struggle, you can already see some of its cracks just by looking at that brief description. Constellation (around $13B sunk cost, starting in 2004) was cancelled for being over budget and behind schedule, after estimates suggesting that continuing it would have taken more like two thirds of the Apollo budget. We have to separately consider Orion (around $25B, mostly complete except the heat shield is a little iffy, development started 2006), SLS (roughly $35B for "Block 1", plus a marginal cost that makes "Block 2" look increasingly unlikely, development 2011), and HLS ($8B public, for two landers, starting work in 2021 and 2022) as three programs, because it's really hard to call something a single coherent program if you spend ten years building a super-heavy launcher for lunar exploration and then realize you probably want to start working on some sort of lander to go with it. Oh, and also the primary lander comes with its own super-heavy launch system, whose development will either fail (in which case we have at least a three year delay with nothing to do but fly around the Moon while waiting for the backup lander), partly work (in which case it's twice as powerful as the one we spent nine times as much on, sending five or ten times as much payload cislunar, in a spacecraft better than the one we spent another six times as much on, for half the marginal cost), or work to design (in which case make that a twentieth of the marginal cost and twenty times the embarrassment, as we realize that from the beginning we should have been struggling to surpass Apollo, not replicate it).
roystgnr
4d ago
The USA is able to run obscene defects due to the USD being a reserve currency generating strong demand for it, reducing inflation.
I'm with you on all the other stuff, and I would be with you on this one if in practice we only ran the obscene deficits during incidents of particular need punctuating longer periods of fiscal responsibility, but the fraction of fiscally responsible leaders in either tribe is a rounding error. Carefully-dosed limited-time opioid prescriptions are useful for acute injuries, but if someone's heroin addiction has gotten too bad for them to go cold-turkey and they're increasing their doses exponentially to make up for the diminishing returns, you don't praise their easy access to dealers.
roystgnr
5d ago
Oops I didn't relaize claude share doesn't share inline citations.
That's on me, too; I should have checked the links in your quotes, not just looked at the Claude transcript and assumed it included everything in the quotes.
The link you shared is about May 2025 which is not related to the result for June 2025
One of the two links I shared was an April story, the other a July story; both were data through March 2025.
Personally I'd have used the phrase "near-record levels" (after rising 30+% above trend, it dropped back 0.13% - yay?), but I'm not sure that'd be any more informative a summary - "near-" could be applied just as well to a record set 13 years earlier, while "representing" is a closer fit for 3 months earlier. "Reached record levels" or "was a record" wouldn't be supported by Claude's inline link, but both of those were your rewording, not Claude's.
Anyways it's undeniable that your favorite model
You seem to have confused me with @RandomRanger. Claude is my second-favorite model, because while I've repeatedly caught it in errors, it at least always tries to fix them when I correct it; ChatGPT-5-Thinking is the only thing I've seen that's (so far, for me; others have had worse luck) been good about avoiding errors preemptively, and IIRC all the non-Claude free models I've tried have made significant errors and often tried to gaslight me about them afterward.
still slopped out a multitude of errors
I'm not entirely on board with Claude claiming that 99.8% of a recent record is "representing" that record, but it's clearly all too easy to slop out errors. Would that either of us were under 0.2% off!
Looking at your other complaints, they're mostly either not errors or not clearly errors, which amusingly means that appellation is itself in error each of those times:
When Claude refers to "Operation Pegasus", that's a term even the BBC has used, referring to the same thing as "Project Pegasus", though it's not used in the story at that particular inline link, which is about details other than terminology variants. (it is in one of the other links Claude found) When Claude is correct about something that seems too simple to justify, but it turns out that "too simple" is in the eye of the beholder, that's still not an error.
The difference between "Wrong" and "There's no citation" also applies to the Crime and Policing Bill - is it wrong? Then what is the primary response to the problem? Four out of the five quoted sources in the linked article mention the Crime and Policing Bill by name, which seems to be a solid first place showing; why would we not want AI to use Grice's Maxims here?
When you say "The source does not indicate that any mapping of what's happening was done at the summit.", you're misparaphrasing Claude's summary, which says "coordinate efforts on mapping", and is actually a pretty good abridgement of "see what more we can do together to map what's happening" from the source article.
Your claim of "outdated" is like something out of a South Park joke. 2023! The Before Times! The Long Long Ago! It's good to see an October 23 2025 article in the mix too, but I want citations that provide a little context; "born yesterday" is supposed to be an insult! Perhaps at some age "outdated" becomes "unsupported", but that's still not "erroneous" - is the data actually out of date? Which of those policies has since ended?
Ironically, the one thing I've seen change most since 2023 is AI itself. In 2023 I was giving AIs benchmark questions that could be answered by most first-year grad students in my field, watching them instead make sign errors that could have been caught by anyone who's passed Calc 3, and then watching the various models either flail about at failures to fix the problem or gaslight me about there not being a problem to fix. In 2025 I can still catch the free models in math errors, but the one time I've "caught" a top model it turned out to be because I had an embarrassing typo in my own notes. Actual top-of-their-field geniuses are still catching top models in math errors ... but using them to prove theorems anyway, with reports to the effect that it's faster to try new ideas out with the models and correct the errors than it is to try every idea out manually.
I do like talking to Claude, at least for anything where I can double-check its work, both because it's capable of avoiding rude language like "slop" and "dogshit" and "shitty", and because when I do find errors upon double-checking, it acknowledges and tries to fix them. You've been pretty good about the latter so far, at least; thank you!
roystgnr
5d ago
Shoplifting offences increased by 13% to 529,994 offences in the year ending June 2025, representing record levels.
Wrong. The source did not say that it reached record levels, simply that it increased y/y
The first link in the results Claude found is to the story "Shoplifting in England and Wales soars to highest since police records began", whose text reiterates "figures are the highest since current police recording practices began in March 2003."
Weirdly, Claude doesn't seem to be having any luck finding BBC results for its queries - e.g. "site:bbc.co.uk shoplifting uk 2025 - 0 results" - but when I try the same search it did, my first hit is to the BBC story "Shoplifting hits record high in England and Wales", with text like "at its highest level since current records began more than two decades ago" and a graph showing those levels.
roystgnr
5d ago
If you read the article it links to an updated study done in 2025.
Turns out it links to both! I followed the final "The full findings can be found here: Research Findings: Audience Use and Perceptions of AI Assistants for News" link, which leads to a summary with only two footnotes, one to a general "Digital News Report" web page and one to the Feb 2025 writeup of the 2024 study. I mistakenly assumed these were the full findings, because of the phrase "full findings", so I didn't bother to check the News Integrity in AI Assistants Report link that goes to the newer results.
Thank you!
roystgnr
5d ago
What is a crack of doom in a kettle?
The "crack of doom" is a phrase from Macbeth, referring to the beginning of the apocalypse. It's not the crack in the kettle, but a crack of sound coming from the kettle; I'd assume the polysemy here is supposed to be poetic. IMHO it doesn't work well that way, or thematically ("doom" originally literally meant "judgement", and the Last Trump sound announcing it isn't supposed to be a bad thing for the folks who are ready to be judged) but it's definitely not nonsense; you could even argue that an apocalypse announced by a cracked witch's cauldron works as a deliberate mockery in the same sense as the "slouching towards Bethlehem" beast in Yeats.
I can't think of the defense for "eagle plucked into a crow", though. Eagles get attacked by crows defending their territory, and there's a couple popular allegories that come out of that; maybe the AI tried to mix that into "plucked bird as comically shameful defeat" symbolism (dating from the Mexican War to Foghorn Leghorn) and just mixed it badly?
roystgnr
5d ago
a recent study found that when AI assistants answered questions with sources it fucked up 45% of the time.
Although the human-written headline here summarizes the research as "AI assistants misrepresent news content 45% of the time", if you go to the study you only see the number 45% in the specific discussion of significant sourcing errors from Gemini.
On the one hand, the AI performance in their data tables is by some interpretations even worse than that: looking at the question "Are the claims in the response supported by its sources, with no problems with attribution (where relevant)?", the result tables show "significant issues" in 15%-30% of responses from different AIs, and significant or "some issues" in 48%-51% of responses. Those "issues" include cases where AI output is accurate but sources not cited, but even if we look at accuracy alone we see 18%-26% "significant issues" and 53%-67% significant or "some"!
On the other hand, if we're getting peeved by AI misrepresentation of sources, could we at least ask the human researchers involved to make sure the numbers in their graphs and write-up match the numbers in their tables, and ask the human journalists involved to make sure that the numbers in their headlines match at least one or the other of the numbers in their source? Someone correct me if I'm wrong, egg on my face, but as far as I can see no combination of Gemini table numbers adds up to 45%, nor does any combination of AI-averaged accuracy or sourcing numbers, and in that case the "misrepresentation" headline is itself a misrepresentation! It's misrepresentations themselves that bug me; not whether or not the entities generating the misrepresentations can sneeze.
On the gripping hand, this "recent" study was conducted in December 2024, when reasoning models were still experimental. They don't list version numbers for anything except GPT-4o, but I'm pretty sure 4o didn't enable reasoning and if they were using Gemini's "Deep Research" they'd surely have mentioned that. Results from non-reasoning models are probably still the most apples-to-apples way to think about use cases like the ones in this discussion, that won't want to burn more GPU-seconds than they have to, but at the moment in my experience switching to a reasoning model can make the difference between getting bullshitted (and in the worst models, gaslit about the bullshit) versus actually getting correct and well-sourced answers (or at least admissions of ignorance).
Also in my experience, for things you can't personally verify it's only AI output with sources that can be trusted - not because you can trust it directly, but because you can check the sources yourself. AI can be a much better search engine just by pointing you to the right sources even if you can't always trust it's summary of them. I'd even prefer something that has issues 18%-67% of the time, but helps me fix those issues, over something that only has issues e.g. 10%-15% of the time but leaves me no way to check whether I'm being misled.
more and more idiots have taken to posting screenshots of the Google "AI summary" which is just slop
Often it's accurate, just not often enough to be strong evidence, much less anything approximating proof, of accuracy. I have no idea why people think otherwise. Even the ones who don't understand that we now train AI rather than program it have experienced computer programs with bugs, right? There is a selection effect to those screenshots, though: if the AI says that 2+2=4, well, nobody wants to argue otherwise so nobody bothers citing that; if the AI says that 2+2=5, then anyone who falls for it has motivation to wave that banner in front of everyone trying to explain otherwise.
roystgnr
6d ago
What surprised me the most wasn't seeing a video of Trump putting on a crown and shitting on Americans, it was seeing who posted it. The prophecy has been fulfilled...
roystgnr
7d ago
it draws attention, might attract the bad kind of attention, looks like cringy showing off which they just axiomatically don't like, etc.
More anecdata, but some of the most mathematically interesting code in one of my favorite open source projects had its first version written by a female programmer, who doesn't have a single commit, because her conditions for being persuaded into contributing were basically "you own the translated code, you don't put my name on it, you don't ask me for support, you don't suggest others ask me for support".
She got like 5 papers and a dissertation for her PhD (which she finished at least 25% faster than I did) out of the research that led to that code, during a period when I was spending a ton of time helping new users of the rest of the software for no immediate personal benefit, so it's hard to say that she was doing the wrong thing, at least in the short run. On the other hand, today those papers have ~140 citations between them, none since 2022; the one paper about the project she was a silent contributor to is over a thousand now, and that's because most users' papers cite a downstream project instead.
Citation?
The first numerical summary I could quickly find suggests that, while women get into approximately 13% more accidents per passenger-mile, men drive so many more passenger-miles that they get into approximately 45% more accidents per year. Their hyperlinks are broken, though (looks like someone just hit "copy link" to URLs whose results depend on session cookies) and they might have misinterpreted something.
The "highest-value damage" here, of course, is human life; nearly 3/4 of car accident deaths are males, despite females being more likely to die in any particular crash they're in. It's a value judgement as to what makes someone a "better" or "worse" driver, but for a US value-of-life metric it takes several hundred typical non-injury accidents to add up to one fatality.
[Edited to change "~" to "approximately" to fix a formatting bug]
roystgnr
8d ago
I'm not sure the people arguing for build build build aren't imagining they're solving the problem of housing is so fucking expensive.
Fortunately, "It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest." That's a counter-intuitive realization (that quote postdates the divergence theorem!) but it's not wrong.
roystgnr
8d ago
If you had an employee whose work you had to check every single time, you'd fire him.
Where I work, that would mean firing everybody - no work gets deployed without at least a second person reviewing and approving the proposed changes. That's a fairly common Quality Assurance practice everywhere, sometimes because an application is critical enough that human failure rates are intolerable, sometimes because a deployment is large enough that even the cost of a tolerable mistake multiples out to be larger than the cost of double-checking to reduce mistake frequency.
AI currently doesn't count as a "second person" for us, but just as a review of human-written code typically takes much less time than writing it did, two reviews (the reviewer plus the "author") of AI-written code can go faster than hand-writing plus review. The last time I reviewed AI-assisted code, the "tell" that AI was used wasn't that there was anything wrong with the code, it was that the documentation was better-written than you generally get from a junior human developer. We apes tend to want to just write the fun stuff and shy away from the tedious stuff.
Why should a machine be held to a lower standard?
Do you know anyone who'll help e.g. write a C/C++ reader for a simple HDF5-based format for ... well, I think that was before we got a work ChatGPT account and I used a free AI that time, but call it $200/month for ChatGPT Pro? I'd never used that API before, and the docs for an API I'd never used before weren't as clear or voluminous as I'd have liked (damn it, everyone else shies away from the tedious stuff too), but searching up and reading better tutorials would have taken an hour or so; double-checking LLM output took five minutes.
roystgnr
8d ago
If you find Heinlein entertaining, you might look at Pournelle & Niven's "The Mote In God's Eye"; Heinlein helped with the editing and called it "a very important novel, possibly the best contact-with-aliens story ever written". It's had 50 years to be surpassed, but I'd still say it's top-five and I think most people would say top-ten or at least top-20.
It's definitely not explicitly Christian sci-fi - Niven is (or at least at one point was; he doesn't talk about religion much) an atheist, one of the main characters is Muslim, and there's not anything theological about the plot. But it's set in a world of Pournelle's where humanity is under the aegis of a mostly-Catholic empire, and both that and Islam have an impact on the story; it's definitely not just religion of the Star Trek "how cute; they'll grow out of that soon" or "look they made a cult around some aliens" varieties (though there's one of the latter too).
roystgnr
10d ago
Last edit as I type is the "Sigh, not solved", but for what it's worth, whereas for ~36 hours the site has been so slow for me I thought it was entirely down, right now it seems completely back to normal.
Thanks again for all you do here!
roystgnr
14d ago
The Jordan Peterson-esqe "cultural Marxism" shibboleth is genuinely gibberish.
As @naraburns memorably explained once, it was a term coined by the original cultural Marxists themselves, not by Peterson or by any other of their opponents. The memory holing of that is just weird.
In the short term his interlocutor tried to cling to the theory that Peterson was the designer of a phrase that, by some weird coincidence, also happened to be a related field of study with diverse academic citations for decades prior, but in the long term that entire account was deleted, so perhaps there's only so much cognitive dissonance a person can take.
It's literally just "I 'ate communism, I 'ate wokism, refer to 'em interchangeably, simple as"
No; it was a self-appellation. It may have gotten too embarrassing to hang on to at some point, but that's language for you. The same thing has been happening to "woke", for that matter, after it happened to "social justice warrior". Some groups are so proud they'll even adopt exonyms their enemies created; others are so uncomfortable they have to keep escaping their own endonyms.
roystgnr
15d ago
I'm just glancing at numbers, but it looks like white emigration from South Africa is about 2% per year, as opposed to around 40% per year for the pieds-noirs during 2 years of "suitcase or the coffin". South African white emigration has been slow enough that fertility has kept their population pretty steady over the past few decades in spite of it.
roystgnr
15d ago
I'm not a frequent enough LLM user to say how much of this was solid improvement vs luck, but my experience with free ChatGPT 5 (or any current free model, for that matter) versus paid GPT-5-Thinking was night vs day. In response to a somewhat obscure topology question, the free models all quickly spat out a false example (I'm guessing it was in the dataset as a true example for a different but similar-sounding question), and in the free tier the only difference between the better models and the worse models was that, when I pointed out the error in the example, the better models acknowledged it and gave me a different (but still false) example instead, while the worse models tried to gaslight me. GPT-5-Thinking took minutes to come back with an answer, but when it did the answer was actually correct, and accompanied by a link to a PDF of a paper from the 1980s that proved the answer on like page 6 out of 20.
I followed up with a harder question, and GPT-5-Thinking did something even more surprising to me: after a few minutes, it admitted it didn't know. It offered several suggestions for followup steps to try to figure out the answer, but it didn't hallucinate anything, didn't try to gaslight me about anything, didn't at all waste my time the way I'm used to my time being wasted when an LLM is wrong.
I've gotten used to using LLMs when their output is something that I can't answer quickly myself (else I'd answer it myself) but can verify quickly myself (else I can't trust their answer), but they seem to be on the cusp of being much more powerful than that. In an eschatological sense, maybe there's still some major architectural improvement that's necessary for AGI but still eluding us. But in an economic sense, the hassle I've always had with LLMs is their somewhat low signal-to-noise ratio, and yet there's already so much signal there that all they really have to do to have a winning product is get rid of most of the noise.
That's a good question. His final approval poll was 63-29, at the higher end of a presidency that went up and down around an average of 53. His retroactive approval went as high as 73-22 in 2002, and as of a couple years ago it was still 69-28, 2nd only to JFK among the 9 recent presidents Gallup asked about. The left-wing opinion still seems to be "Reagan screwed up the AIDS epidemic" so I'd have to assume that his support still leans right and he's at 70+ among Republican voters.
But this might be just one of those things that's uselessly sensitive to poll wording (YouGov says 44-29! Is that just because they emphasize their "neutral" option more?) or to poll methodology (Gallup says 90-8 for JFK!? Is it just getting harder and harder to correct for "only boomers answer the phone for pollsters" effects?).
More options
Context Copy link