I think that in theory, you are correct. Software could be designed so that it is provable that e.g. there is no possible TLS traffic which will allow an attacker to reconstruct the server's private key any faster than just cracking the public key.
For the moment, just about none of the software we use has such proofs, however. As you mention, even the mathematical foundations of existing crypto primitives rarely offer such guarantees -- more often it is just "we looked into that problem for three decades, and it seems really hard" (e.g. integer factorization). Concrete implementations without formal verification likely have implementation bugs as well. Mythos did not discover any exploits which would have been impossible for humans to discover, it was just that nobody had spent that much human eyeball time on auditing the software. This is what I meant by "token pissing contest".
Furthermore, actually specifying what theorems should hold to keep your system secure is itself hard. If you have a TLS server which will provably never leak its key, but is happy to use it to sign and decrypt on behalf of the attacker, that is still a broken system. If you have a larger system, then completely specifying what you do not want an attacker to be able to do seems difficult.
Also, the software we care about does not run on Turing machines, it runs on physical hardware. In everyday use, most computer hardware behaves as an idealized model. Billions of people use DRAM every day, and it just works. Except that there are corner cases where it will not behave as advertised.
I can not speculate if an ASI could build a system which even a much stronger ASI could not penetrate, and what the performance costs would be. But I think it is unlikely that human-level intelligences subject to design pressures besides security will build hardware and software systems which ASI's will not be able to penetrate.
OpenAI will want to hype up the capabilities, not the misalignment.
I mean sure, crying rogue AI is a good way to draw attention to their model. But they are not some tiny startup who needs the publicity.
If I had a model with advanced intrusion capabilities, I would hype it up by offering pentesting as a service. This has the advantage that if becomes known that I prompted the model, I will not go to federal prison and not completely destroy the reputation of my company. Find a HuggingFace-sized company or three who are willing to take a free pentests in exchange for acknowledging that you found critical vulnerabilities in their stack.
The people who believe that it is all just empty hype about "stochastic parrots" would obviously claim that it is all fake, but these are unlikely to become your customers in any case.
So AI has reached a new milestone (writeup by theZvi, who is usually diligent and excellent about AI news reporting. If you read anything, reading his analysis is probably better than whatever I am writing. It has meme pictures, too!)
Basically, OpenAI was testing the 'cyber' capabilities of their Galaxy model, so they ran it in a mode with decreased security and told it to do some benchmark called ExploitGym which presumably tests exploit finding and executing ability.
Their model decided that the best way to do this would be to gain network access, hack Hugging Face (an LLM and tool hosting platform, as I understand it) and obtain the answers it needed to ace the ExploitGym benchmark. Apparently it discovered and chained quite a few zero days in the process.
There are multiple takes on this. I will focus a bit on the politics and conspiracy theories, which seems appropriate for this forum.
"It is all just a PR stunt by OpenAI"
I mean, sure, AI labs will hype up their products. Marketing by alignment worries is definitely a thing. Oh, our latest model is so smart and powerful, we are really scared about it.
Personally, I am disinclined to believe it because it would require a conspiracy between OpenAI and Hugging Face. It seems unclear what the incentives for Hugging Face (or a few rogue employees) are.
Obviously it is impossible to rule out that someone leaked the relevant sources of Hugging Face's business to OpenAI and then OpenAI employed some human IT security researchers to find exploits and make it look like the model had done all the work on its own.
But I do not buy that. It would require quite a few people to commit crimes for which they would go to prison for a very long time if caught (or until pardoned). Obviously people will go over all of the steps the model took with a very fine comb, and "Was it a reasonable guess that this attack might work without inside information?" is a question which will be on their mind.
We also have the data point that Mythos was (very likely) able to find new exploits. (Yes, mostly with access to the source code, and for all we know Anthropic spent a billion in token costs. But "cutting edge LLMs are able to find exploits even in well-audited software" is a reasonable claim.)
"OpenAI was sloppy and did not sandbox their model properly, so we just need better sandboxes to solve this"
I mean, obviously their sandbox was defective, no shit. But the idea that the next time OpenAI will just invest 20% more effort and build a sandbox which is ASI-proof seems utterly optimistic.
Air-gapped systems are a PITA to run, which is why they did not test their model air-gapped. And even with an air-gapped system, there is no guarantee that a sufficiently smart model would not be find a way to get some peripheral to send signals. Nobody wants to really put their system, power generator and operator in a Faraday cage in some deep mineshaft for every test. (Unless someone mandates it.)
"This will completely overturn cyber security -- you will need good LLMs to watch for attacks by bad LLMs"
It might be right that it will overturn 'cyber' 'security' (my scare quotes). However, I am with Zvi in that I do not think there is a reason why this should favor defense. After all, an attacker could spend a whole lot on tokens while your defensive LLM is sitting on limited infrastructure -- at least if you are sufficiently paranoid not to hand the AI labs the key to your kingdom. And even if you trust the cloud, there is the problem that your budget might not have room for winning all LLM-vs-LLM token pissing contests.
Perhaps it will lead to new paradigm -- attackers spinning up thousands of copies of very good security professionals might well lead to an era markedly different from when humans were in the loop between the explore and exploit phase. But in the grand scheme of things, it feels like worrying about the future of Our American Cousin in the aftermath of the 14th.
"Cutting edge models are obviously misaligned. DOOM!"
This seems to be a very common LW take. As somewhat of a doomer myself, I find myself agreeing. For being intrinsically unfalsifiable, the prediction record of the doomers seems not bad so far.
The model clearly knew that it was not doing what the prompters had wanted it to do. It just did not care, because it was trained to do whatever it took to ace it tasks. This has implications way beyond IT security.
An ASI in this mode is basically an evil genie. "Oh, you wished that your wife would never fall out of love with you. So obviously I killed her, it was the only way to be sure."
The appropriate response would be to send the marines to the AI labs to stop the development of frontier AI models at least until we figure out what adequate safeguards are (and possibly until we solve alignment, though we would want to coordinate with China about that).
If we had a president Obama or even GWB, there was some chance that a crackdown would happen. But with Trump and his cronies, I doubt that there are any who both understand the severity of the situation and have any incentive to manipulate Trump to do something about it.
Oh well, how is the other side of the culture war reacting to this significant increase in p(doom)?
"Iran warns of ‘eye for an eye’ response if US follows through on Trump’s threats to destroy infrastructure
I mean, not entirely. Hidden between Democrats need to hammer Trump on his unprecedented corruption and Why many Black Americans were rooting for Argentina to lose the World Cup , there is OpenAI’s rogue agents are a wake-up call to risks posed by artificial intelligence.
The article is not that bad. The author seems EA-affiliated and is clearly aware of the doomer arguments, but has diluted to an almost homeopathic level as to not alienate his blue tribe friends:
This week’s incident should serve as a wake-up call, forcing us to ask an uncomfortable question: should we really be building dangerous systems that we can’t control?
But it is the 41st headline or so on that website.
I think the best thing we can hope for are some incidents which unaligned AI which will be impossible to ignore even for the CW-fighting media before we come to the point where we will no longer detect any incidents.
I fully believe that by 2030 global warming will be largely solved and by 2040 the issue will be essentially forgotten like the hole in the ozone layer.
I would share your optimism if the half-life of atmospheric CO2 was a year or two. Sadly, CO2 is a very stable molecule.
Google's AI claims that 50-70% will be absorbed by oceans and plants over 50-100 years. To get rid of the rest, the carbon needs to be transported into the deep oceans or even back where we took it from and this will take even longer.
I could as well say:
Good news, while the propelling charge has been accelerating the projectile to dangerous speeds in the past, the pressure in the barrel is actually rapidly decreasing. Soon the bullet will exit the barrel, and at that point the gunshot problem will be solved because friction will take over and stop the bullet.
Meanwhile, my daughter's pre-K teacher has invited herself to my house. She called today, and went on about a "mandatory home visit", next week, with highly inflexible weekday hours.
The operative question would be mandated by whom? State legislature could do that (and states have certainly passed stupider legislation than that). Still, if it is about child welfare, I would rather let CPS handle it. On the other hand, I would be surprised if any courts found that individual schools had the right to mandate home visits.
Also, having a teacher visit every kid at home seems really personnel intensive. Back when I was a kid, if a teacher wanted to speak to the parents of an individual kid, she would tell them to come to her office (but also had no mandate to make them come, though my parents certainly did).
I will grant you that the Floyd riots were left-wing, and that about that many people died as a result. (Though my impression was that most were people murdering each other for the usual reasons in areas which had been "liberated" from the police.)
I do not think that most of deaths qualify as terrorist-related. By contrast, Bin Laden et al were plotting to kill as many Americans as possible to accomplish some (vague) political goal.
All of the pipeline which Rubio talks about also did not play a role in the Floyd protests. There was no large donation drive, training and recruitment effort, especially not on an international level.
There are things which contain politically motivated violence (e.g. the Floyd protests, J6, the US independence war, the Confederacy), which can be quite bad (as some of these are) but that does not make them a central example of terrorism.
Yawn.
So the Sunni Jihadists had to do 9/11 to claim their title of top terrorist threat. If I were OBL watching from hell, I would be very upset. After all, he killed thousands to earn his spot, and now Antifa is getting the focus after shooting a few Feds and (very allegedly) sabotaging the Reflecting Pool?
"Left wing terrorism" is roughly like "right-wing hate groups". To the degree that they are widespread (e.g. celebrating the death of Kirk, or perhaps Floyd), they are covered by 1A. To the degree that they go beyond what is protected by 1A, they are not actually widespread.
Today’s far-left terrorists can raise money in one country. They can host their communications in a second country. They can receive training in a third country.
Translation:
(1) You should not be able to raise money for causes we do not like. More debanking.
(2) You should not have access to untapped communications.
(3) You should not be able to go to other countries to meet with people for a political cause we do not like.
Sounds like a pile of horseshit to me.
That seems plausible. TIL.
This seemed more true in the early 2000s, when the desktop ruled supreme and most desktop software used the Win32 API.
These days, decidedly less so. Most software seems to be based on some toolkit which offers cross-platform support. Desktops lost a lot of relevance to both mobile and web apps. Antitrust bureaus are just waiting for Microsoft to try to sue WINE. (Well, the EU at least. The current US administration would probably be ok with them doing whatever as long as the bribes kept coming.)
This has been going on forever. What, you want to use assembler? Kids these days can't even be bothered to remember their opcodes!
The truth is that we have a larger demand for software developers than our pool of people who can write raw x86, asm, C, and even Python.
Obviously there is a correlation between competence and choice of language. If I have a field where half the pre-prints are written in LaTeX and half the pre-prints are written in Word, I would likely think "Uh-Oh" when I open a pre-print and see Arial.
Still, sub-par programmers can write sub-par code in any language you give them. I have seen physicists without a CS background trying to write C++, and generally go please for the love of God just use Python already.
Personally, I did ignore Python for my first decade of programming -- no Ce-sque syntax, indentation matters, much less dense than Perl were enough to convince me that it is obviously not a cool kid language.
Later I came around. There are tasks where it is the right tool for the job. And if you want to self-express, you can always import functools or modify the methods of an object on the fly. (And yes, sometimes you run into cases where Python's "lol, don't worry about the implementation details" approach leads to stuff the programmer would not have expected, hiding details is a lossy process.)
Recently, I have tried to use LLMs to add features to open source projects.
Making an LLM write a PR which I would be happy to attach my name to is actually hard work. Sure, the agentic loop will generate code which does the job, but then I need to check corner cases. At the moment, I am still trying to figure out if there is not a neater way to implement what I want with less overhead, after all I do not want to settle the project with more technological debt than required.
On the other hand, I could never be arsed to write tests, and this is something where I do not care too much about code quality. If two tests could be combined in one to save a couple of lines, who cares. Even if there is the odd bug in a test case, this is still no worse than no test case.
Documentation is also something the LLMs have much more patience for than I do. Though I will need to think more carefully of how to name my examples when specifying the feature because these names tend to make it into the documentation.
And of course, there is a lot of code which does not need to be optimized for being maintainable. I have seen PhD students use LLMs write utilities which would seriously impress me (given my baseline expectations) if they were hand-written.
Of course, how much maintainability matters depends a lot on one's timelines. Still, submitting PRs full of AI slop with the excuse "don't worry about tech debt, the AGI will clean up the code really soon" seems like really bad style for interacting with human-maintained projects.
I would frame it like this: for a single use FFP2 mask, the manufacturer has tested that the mask will last a shift. So from a purely technical compliance point, you are correct.
However, this is similar to the claim that you need to exchange the oil in your ICE every six months or 8k kilometers, whatever comes first. Sure, it is what the car manufacturer recommends, and if your car is part of some life-critical infrastructure (e.g. a patrol car), it might be easiest to do this rather than trying to justify deviating from it. But a lot of people get decent mileage out of older cars despite replacing the oil much rarer than recommended.
The general public was never trained to wear FFP2 masks the way professionals (like people working with asbestos) hopefully are. Realistically, a huge fraction of the public was not getting the full benefit of 95% particle filtering due to imperfect use.
In the "flatten the curve" era, the idea was not that we could eradicate COVID-19 by making sure that everyone inhaled less than 5% of particles in public. Instead, the idea was "particle filter masks likely offer some marginal benefit over medical masks or textile masks even if worn imperfectly, we might as well try to benefit from that".
Basically, we are in the realm of cost-benefit analysis, not strict compliance. I can not say I was really surprised by this study claiming that some masks remain very effective even after forty hours of use (though I did not anticipate that they would even survive washing that well). I am sure that there were employees in the ministry of health who shared my instinct that most of the exposure would come from ill-fitted masks and a lack of compliance in the population, with overlong reuse being a distant third. The average citizen likely wore his mask to the supermarket, so during an acute shortage, you might want to order two for him, which should offer some protection over two months, and once they arrive you can reassess the market situation.
Add to that some basic economic literacy. The CDU donors obviously did not produce their FFP2 masks in their own factories, they procured them from China, same as everyone. If China is shipping masks they will have masks to sell to the taxpayer at some hefty markup, but this also means that other people will also be selling Chinese masks. And if China is not shipping masks, then they will not have masks to sell at any price either. CDU donors are not the Spacing Guild, they do not have a monopoly on transportation.
If Spahn had ordered enough masks to cover hospital shifts for a few months from 3M or some other medical or PPE company even at a hefty markup, nobody would have cared much. Instead, he ordered colossal quantities from his grifter friends.
This is what @StableOutshoot also said.
I agree that "imprison" sounds like it refers to police conduct. Of course, if this was the case, then any policeman who arrests a foreign official (say, because he just shot up a school) is criminally liable under this title, which is likely not what the legislature intended.
--
As an aside, it might just be cultural bias, but I find the way laws are written in the US rather opaque, like there is no expectation that citizens should even try to understand what the law is (at least pre-LLM). 18 USC 878 seems a prime example. First off, it is very niche, criminalizing threats of a list of offenses (which themselves are special cases of more general offenses against members of the public instead of foreign officials) -- which I presume is in turn a special case of a more general statue, because it seems unlikely that the US is fine with people threatening other federal crimes.
Then there is the fact that 18 USC 112(1) reads like the result of a brainstorming session without any attempt at deduplication. One would think that the verb "assault" also covers "strike" and "wound". Or the fact that it then includes the local definitions of another section in a way which would never pass any code review if it was software.
The idea seems to be much "let's just frame things broadly and rely on prosecutorial discretion".
If I were a criminal, I would much rather do crimes in Germany where I then have to deal with the Strafgesetzbuch (and the Nebenstrafrecht, sure) rather than doing crimes in the US and deal with a labyrinth of federal and state-level norms. In the long run, this will put the US at serious disadvantage attracting criminal talent.
The federal, state or local authorities of the United States shall not impose any impediments to transit to or from the headquarters district of representatives of Members or officials of the United Nations
TIL. Now I am wondering if any hostile foreign head of state (e.g. an Ayatollah or North Korean) was ever willing to bet his freedom on the US sticking to the letter of the law.
Upon further digging, the WP article on the Rome Statute quotes Article 98(1) as the following:
The Court may not proceed with a request for surrender or assistance which would require the requested State to act inconsistently with its obligations under international law with respect to the State or diplomatic immunity of a person or property of a third State, unless the Court can first obtain the cooperation of that third State for the waiver of the immunity.
So as long as Bibi only came to NY to visit the UN headquarters and not spend his holidays there, he would likely be fine.
My reading of 18 USC 878 in connection with 18 USC 1201 is that it refers to illegal kidnappings only. The key operative word in 1201 is "unlawful".
Arrests by police are almost never so blatantly unlawful that we prosecute the police for kidnapping.
More generally, threatening to have someone arrested is not considered threatening a crime.
Even the suggestion that the city that hosts the UN would seriously consider arresting guests of the UN should be insanely problematic.
I think that the arrest of foreign officials wanted by entities outside the US is obviously a federal matter, and absent specific legislation from Congress entirely up to the executive. Obviously there are tradeoffs between diplomatic standards and enforcing international norms.
But if Trump did an 180 tomorrow and told Netanyahu that he is not welcome in the US and would be shipped to the Hague in he appeared on US soil, that would be entirely within his power.
I agree that it should be right up the alley of the FDP. A market-based solution favoring individual rights over fears of individual incentives leading to globally suboptimal decisions.
Though them favoring economic liberty over social liberty has happily brought them to the brink of irrelevance these days, of course.
Sure, the strongest-held believe of the CDU/CSU is that the Bundeskanzler should be of their party.
With regard to the CDU being just right of the median voter, I think that they are a bit more to the right than that. A 2022 study I found claims that 41% are for legalizing surrogacy while 24% are against it.
But yes, I agree that they do not have any hills they are actually willing to die on. Not that I think that this is something which sets them apart from most other parties. The SPD would probably agree ("with belly aches") to anything up to legalizing hunting the working class for sport if they felt that this was required to maintain the coalition.
Culture war in Germany: former COVID-era minister of health Jens Spahn has finally resigned as the leader of the parliamentary group of the conservative CDU/CSU.
Spahn (born 1980) got elected to the Bundestag in 2002 and served as a backbencher. He came out as gay in 2012, some 11 years after Wowereit had firmly established that gay politicians were a thing in Germany.
Under Merkel IV, he became minister of health. I surmise the reasoning was that he was a loyal party member and the health ministry was not particularly important, so the damage he could do there was limited.
Then, COVID hit, and health policy became kinda important. Oops.
I will credit that he was no RFK Jr, but mostly did what the experts recommended (whatever you think of that), apart from having meetings with donors when he had just warned against gatherings (like many other politicians did).
But he certainly saw an opportunity for grift and took it. This was the so-called Maskenaffäre. In the summer of 2020, it became increasingly clear that COVID-19 was airborne and particle filter masks (US: N95, EU: FFP2) would offer some protection. This lead to a dearth of protective masks in Europe. So Spahn funneled money taxpayer money to entrepreneurs with close ties to the CDU/CSU to procure such masks.
Now, it was obviously reasonably for the ministry of health to acquire some face masks even at inflated prices common for medical equipment -- at least for people working in hospitals, but he overestimated the amount required by orders of magnitude. Per WP, he bought 5.8 billion masks for an average price of about an Euro per mask. That is 70 masks per person in Germany, most of which were later destroyed when they expired.
As someone very willing to wear masks when recommended (at least until I got multiple doses of the vaccine), I estimate I went through perhaps 10 or 15 masks in total. While there was a supply shortage at a time, it only lasted a few months before Chinese masks flooded the market. The average price I paid in retail was certainly less than a Euro.
The Maskenaffäre cemented the image of the CDU/CSU as a hive of grifters, but despite them losing a few regional elections, it did not lead to Spahn getting driven out of politics. He did lose his ministry to Karl Lauterbach (SPD, physician, COVID hawk) and did not manage to become the heir of Merkel.
In 2025, he became the leader of the CDU/CSU parliamentary group, which is probably not in the top 5 jobs in political Berlin but likely in the top 20.
That is all just background info, not while he was now resigning.
The CDU/CSU has always been a force of social conservatism. Merz famously voted against criminalizing marital rape in 1997. The CDU was not the only party to represent the interests of the industry (the FDP also existed), but it was (for a long time) the main party to vote for if you did not like LGBT rights.
It is thanks to Spahn's CDU/CSU that surrogate pregnancy remains illegal in Germany today. So while Spahn and his husband did not break any German laws when they contracted with a surrogate mother in the US to get a baby, it was generally taken as a defection from the values endorsed by the CDU/CSU and Spahn personally.
Another casualty of Spahn's retreat is the satire news website der Postillon, which in the past years published dozens of articles on Jens Spahn based on his mask procurement which depicted him as both clueless and corrupt. Semiconductor crisis over: Spahn buys chips for more than 3 billion euros was a rather representative example. After he resigned, over 700 employees specializing in such articles reportedly did lose their livelihood.
All of potential right wing donor cash will be instanly appropriated by grifters and scammers.
That feels very much not true. Considere AIPAC. Sure, notionally bipartisan, but I will call support for an ethnostate with a far-right government enacting far-right policies a right-wing cause area.
And quite effective, too. I think there was recently a vote where only about a quarter of the representatives were willing to vote for cutting funding for Israel. I doubt that this reflects the electorate.
(White nationalists, you heard it here first: don't waste your donations on the KKK, MAGA or Neonazi groups, when AIPAC can offer you ten times the blood on your hands per dollar spent.)
Different states have different laws. If only there was some entity with the power to regulate interstate commerce, this would all be so much simpler.
Personally, I think it is (mostly) reasonable to expect people to follow the laws of the state they are in. Nor do I have a problem with having them stand trial for something which is criminalized by both states if they commit a crime remotely: if you send a mail bomb to a different state, shipping you to that state for your trial seems fair enough.
If your state (and federal laws) did allow you to ship weed over state lines, then it should not have to worry about how the state of the recipient -- or any states on the package's route, for that matter -- might feel about it. Likewise with shipping or serving porn. If Utah does not want porn to move through their fibers, that should between them and any telcos who own fibers in Utah, not people running servers.
On the other hand, as a member of the rest of the world over whom the US would totally claim jurisdiction I see some poetry in the equality in injustice between an European who has to keep US laws and a Pennsylvanian who has to keep NJ laws.
Should we also execute Russian draft dodgers?
Also, should we generally enforce other countries laws? Send back the North Korean defector, or the girl who fled an abusive marriage in Afghanistan?
I agree that this is an excellent reason, but is contingent on the outcome of the war.
I do not think that this is a common objection.
Providing a better alternative to anyone living in shitty conditions is generally not considered ethnic cleansing, especially not on part of the party providing the alternative (unless it is also responsible for creating the shitty conditions).
"We can't take you, because then we would be complicit in ethnic cleansing" is a reasoning on par with "I can't unlock your shackles, because then you might resist and I would be responsible for a violent rape". I do not recall seeing either in the wild.
So my take is that Dario really likes the Biden era AI plan where there would be a small number of officially sanctioned companies who worked with government on regulation and all other AI vendors would be effectively banned.
If this was the case, the other AI vendors would be much more on board with the idea.
The truth is that there are only ever going to be a small number of companies developing cutting-edge LLMs due to economic constraints: model training is very expensive.
Most proposals of the safety crowd I have read actually center on training new very large models. I do not think that Dario wants to use regulatory capture to prevent startups from spending a couple of millions on model training.
He doesn't seem to grasp that the Trump admin isn't going to give him any control over government policies.
Anyone with more than a passing acquaintance with reality can see very well that that Anthropic is on the MAGA shitlist, and I strongly suspect that this includes Dario. Him writing AI policy for Trump is about as likely as Trump getting his Nobel.
They are philosophically opposed to a setup where the elected President has to follow the dictates of a specific citizen with no constitutional authority.
I would expand this to say that they are philosophically opposed (though I can't imagine any of them using that phrasing) to any checks on presidential powers, including but not limited to domain experts, the SCOTUS, foreign states insisting on keeping their territories, international agreements, and broadly physical reality.
- Prev
- Next

It appears that some life-forms on the motte are still bitter about the Great Oxidation Event, when the gas of life was largely replaced by unnatural toxins such as oxygen. Glad to see that someone is pushing back against the Eukaryote chauvinism so common here!
FWIW, I agree that life on Earth will be fine whatever the CO2 levels are. As the saying goes, there are no bad environmental conditions, only ill-adapted species. With regard to whiny polar bears and corals, it is really their own fault for having picked ecological niches which are going away. Still, in a few hundred million years we will probably have at least as much biodiversity as we have today. It is not Earth's first extinction event, and it will not be Earth's last either.
Even humans as a species will be fine -- as reflected in our Least Concern vulnerability rating. Sure, individual human populations might die, but realistically humans die all the time. If we started caring on that level then we would have to wet our pants every time a world war happens. Preposterous. We might as well start caring about individual humans while we are at it.
More options
Context Copy link