This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
FOSS and The XZ Problem
Security Boulevard reports:
The exploit would allow remote code execution as root in a wide majority of systemd-based Linux (and Mac OSX, thanks homebrew!) machines. There's some reasonable complaints that some CVE ratings are prone to inflation, but this has absolutely earned a 10/10, would not recommend. Thankfully, this was caught before the full exploit made it to many fixed release Linux distros, and most rolling-release distros either would not have updated so quickly or would not yet be vulnerable (and, presumably, will be updating to fixed versions of XZ quickly), with the exception of a handful of rarely-used Debian options. Uh, for the stuff that's been caught so far.
Summary and FAQ, for the more technically minded reader, the NIST CVE is here, background of initial discovery at here.
Ok, most of us who'd care remember Heartbleed. What's different here?
In this case, the exploit was near-certainly introduced intentionally by a co-maintainer of the library XZ Utils, by smuggling code into a binary test file, months apart from adding calls to execute that test file from live environments, and then working to hide any evidence. The combination of complexity in the attack (requiring fairly deep knowledge of a wide variety of Linux internals) and bizarreness of exploit steps (his FOSS history is sprinkled with a replacing safe functions with their unsafe precursors, or adding loose periods in cmake files) leaves nearly zero chance that this is unintentional, and the guy has since disappeared. He was boosted into co-maintainership only recently, and only after the original maintainer was pressured to pick him up by a strangely large barrage of very picky users. The author even pushed to have these updates shoved into Fedora early.
Most mainstream technical advisories aren't outright calling this a nation-state actor, but The Grugq is pretty willing to describe whoever did it as an 'intelligence agency', whether government or private, and with cause. Both the amount of effort and time put into this attack is vast, and the scope of vulnerability it produced extreme -- though this might be the 'cope' answer, since an individual or small-private-group running this level of complex attack is even more disturbing. It's paranoid to start wondering how much of the discussion aimed encouraging XZ's maintainer to take on the bad actor here as a co-maintainer, but as people are having more and more trouble finding evidence of their existence since, it might not be paranoid enough.
There's a lot of potential takeaways:
The Many Eyes theory of software development worked. This was an incredibly subtle attack that few developers would have been able to catch, by an adversary willing to put years into developing trust and sneaking exploit in piecemeal.
Except it was caught because a Microsoft (Postgres!) developer, without looking at the code, noticed a performance impact. Shit.
This attack heavily exploited access through the FOSS community: the author was able to join sight-unseen through a year of purely digital communications, and the 'business decision' of co-maintainership came through a lot of pressure from randos or anons.
Except that's something that can happen in corporate or government environments, too. There are places where every prospective employee gets a full background check and a free prostate exam, but they're the outlier even for dotmil spheres. Many employers are having trouble verifying that prospective recruits can even code, and most tech companies openly welcome recent immigrants or international workers that would be hard to investigate at best. Maybe they would have recognized that the guy with a stereotypical Indian name didn't talk like a native Indian, but I wouldn't bet on even that. And then there's just the stupid stuff that doesn't have to involve employees at all.
The attack space is big, and probably bigger than it needs to be. The old school of thought was that you'd only 'really' need to do a serious security audit of services actually being exposed, and perhaps some specialty stuff like firewall software, but people are going to be spending months looking for weird calls in any software run in privileged modes. One of many
boneheadedcontroversial bits of systemd was the increased reliance on outside libraries compared to precursors like SysV Init. While some people do pass tar.xz around, XZ's main use in systemd seems to be related to loading replacement keys or VMs, and it's not quite clear exactly why that's something that needs to be baked into systemd directly.But a compression library seems just after cryptographic libraries are a reasonable thing to not roll your own, and even if this particular use for this particular library might have been avoidable, you're probably not going to be able to trim that much out, and you might not even be able to trim this.
There's a lot of this that seems like the chickens coming home to roost for bad practices in FOSS development: random test binary blobs ending up on user systems, build systems that either fail-silently on hard-to-notice errors or spam so much random text no one looks at it, building from tarballs, so on.
But getting rid of bad or lazy dev practices seems one of those things that's just not gonna happen.
The attacker was able to get a lot of trust so quickly because significant part of modern digital infrastructure depended on a library no one cared about. The various requests for XZ updates and co-maintainer permissions look so bizarre because in a library that does one small thing very well, it's quite possible only attackers cared. 7Zip is everywhere in the Windows world, but even a lot of IT people don't know who makes it (Igor Patlov?).
But there's a lot of these dependencies, and it's not clear that level of trust was necessary -- quite a lot of maintainers wouldn't have caught this sort of indirect attack, and no small part of the exploit depended on behavior introduced to libraries that were 'well'-maintained. Detecting novel attacks at all is a messy field at best, and this sort of distributed attack might not be possible to detect at the library level even in theory.
And there's far more varied attack spaces available than just waiting for a lead dev to burn out. I'm a big fan of pointing out how much cash Google is willing to throw around for a more visible sort of ownage of Mozilla and the Raspberry Pi Foundation, but the full breadth of the FOSS world runs on a shoestring budget for how much of the world depends on it working and working well. In theory, reputation is supposed to cover the gap, and a dev with a great GitHub commit history can name their price. In practice, the previous maintainer of XZ was working on XZ for Java, and you haven't heard of Lasse Collin (and may not even recognize xz as a file extension!).
((For culture war bonus points, I can think of a way to excise original maintainers so hard that their co-maintainers have their employment threatened.))
There's been calls for some sort of big-business-sponsored security audits, and as annoying as the politics of that get, there's a not-unreasonable point that they should really want to do that. This particular exploit had some code to stop it from running on Google servers (maybe to slow recognition?), but there's a ton of big businesses that would have been in deep shit had it not been recognized. "If everyone's responsible, no one is", but neither the SEC nor ransomware devs care if you're responsible.
But the punchline to the Google's funding of various FOSS (or not-quite-F-or-O, like RaspberryPi) groups is that even the best-funded groups aren't doing that hot, for even the most trivial problem. Canonical is one of the better-funded groups, and it's gotten them into a variety of places (default for WSL!) and they can't bother to maintain manual review for new Snaps despite years of hilariously bad malware.
But it's not clear that it's reasonable or possible to actually audit the critical stuff; it's easier to write code than to seriously audit it, and we're not just a little shy on audit capabilities, but orders of magnitude too low.
It's unlikely this is the first time something like this has happened. TheGrugq is professionally paranoid and notes that this looks like bad luck, and that strikes me more as cautious than pessimistic.
I think "depended" here, while true in certain technical sense (as "being on the list of libraries the code is compiled with") in broader sense it is actually the main reason that enabled this failure. There's no technological reason for per se for SSH to use "xz", as far as I can see, it was merely added to make it work with another component. And while SSH, being sensitive component and a primary gateway to most systems, is scrutinized thoroughly, the dependencies may be softer. There's also no good technological reason why systemd needs xz and why it doesn't do whatever it needs to do with xz in a separate component isolated from the component that is needed for interacting with SSH. This is just lack of foresight, laziness and preferring convenience to security. I am not saying it is some outstanding failure - I probably have done decisions like that numerous times, knowingly or unknowingly, over the years of my own career. This particular one led to a very significant breach, and if people were more austere and security-minded in their designs, this likely wouldn't happen - but most people aren't. But I feel like the picture at xkcd, while being both hilarious and true, is not reflecting this particular case entirely accurately - neither systemd not SSH weren't destined to fail that way by any good technical reason, and probably nothing would happen to them - except for tiny amount of inconvenience for a tiny number of people - if these dependencies were removed.
More options
Context Copy link
I've watched a lot of doomerist takes on this one claiming that this proves many-eyes doesn't work, but I think it proves just the opposite. This was perhaps the most sophisticated attack on an open source repo ever witnessed, waged against an extremely vulnerable target, and even then it didn't come even close to broad penetration before it was stopped. Despite being obvious it bears laboring that it wouldn't have been possible for our Hero Without a Cape to uncover it if he wasn't able to access the sources.
If I had to guess, I would suppose that glowing agencies the world round are taking note of what's happened here and lowering their expectations of what's possible to accomplish within the open source world. Introducing subtle bugs and hoping they don't get fixed may be as ambitious as one can get.
That being said, I'm not sure that the doomerism is bad. The tendency to overreact may very well serve to make open source more anti-fragile. Absolutely everyone in this space is now thinking about how to make attacks like this more difficult at every step.
Witnessed is a little important, here; I'm not as sure as TheGrugq that this isn't the first try at this, if only because no one's found (and reported) a historical example yet, but I'm still very far from confident it is the first. And it did get really close: I've got an Arch laptop that has one part of the payload.
That's... not entirely clear. Visible-source seems to have helped track down the whole story, as did the development discussions that happened in public (though what about e-mail/discord?), but the initial discovery seems like it was entirely separate from any source-diving, and a lot of the attack never had its source available until people began decompiling it.
Yeah, that part is encouraging; I've definitely seen places (not just in code! aviation!) where people look at circumstances like this and consider it sign the were enough redundancy, rather than enough redundancy for this time. I think it's tempting to focus a little too much on the mechanical aspects, but that's more a streetlamp effect than an philosophical decision.
There isn't any evidence directly supporting this, but I saw a claim that this entire claim of discovery ("slow SSH connections") could easily enough be parallel construction prompted by The Powers That Be (tm) aware of this effort for other reasons -- which range from "we know our adversaries are trying to insert this code" to "we run our own audits but don't want to reveal their details directly". Even in such a situation it's a bit unclear who the parties would be (nation-state intelligence agencies, possibly certain large corporations independent of their respective governments). The obvious claim would be something like "NSA launders data to Microsoft to foil North Korean hacking attempts", but "China foils NSA backdoor attempts" isn't completely implausible either.
That said, I can only imagine that sneaking a backdoor like this into proprietary build systems would be even less likely to be detected: the pool of people inspecting the Windows build system is much smaller than those looking at (or at least able to look at)
libxzand it's (arcane, but somewhat industry standard)autoconfscripts.Also, this is the sort of thing that I have a vested interest in as a long-time personal and professional Linux user. I have the skills ("a very particular set of skills") to follow the details of issues like this, but there isn't yet any organized way to actually contribute them. I'd be willing to spend maybe a few hours a month auditing code and reviewing changes to "boring" system packages, but I'd never think to look at
libxzspecifically, or to get enough "street cred" for people to actually take any feedback I have seriously. And even then, this particular issue is underhanded enough to be hard to spot even when an expert is looking right at it. Does anyone have any suggestions for getting involved like this?More options
Context Copy link
I think the glass half full perspective is more accurate here. Sure, it wasn't detected at the earliest possible time—the second it was committed—but it was only in the most bleeding edge releases of a select few base distributions for a few weeks before it got sniffed out. For such a sophisticated attack, that's lightning fast. Stuxnet took about five years and infected around a hundred thousand machines before it was uncovered. Sure, it's possible that this sample size of one is unrepresentative of the whole distribution of this event repeated a thousand times, but that's less likely and strikes me as somewhat catastrophizing. As someone noted below, we don't know that this wasn't an attack from an AGI sitting in OpenAI's basement plotting to kill us all as we speak.
How would he have tracked down the backdoor without the repo? It seems to me that without it all he would have is some CPU benchmarks and some valgrind errors. What would he have done with that other than submit a bug report to the company that actually had sources, which could be ignored or "fixed" at their discretion?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
My takeaways:
Some more comments on the OP:
I think there is a big difference. Rolling out your own crypto is a big no-no because they are hard to get right, and any mistakes likely leave you vulnerable.
Rolling out your own compression is much less evil: there is certainly some potential for arbitrary code execution vulnerabilities, but not more than with handling any other file parsing. With regard to generally reinventing the wheel versus loading wheels from a zillion different libraries, each of them with their own dependency chains, there is probably some reasonable middle ground. For something like sshd which sits on a security boundary, the obvious way in retrospect to add systemd logging would be to implement the interface from the scratch instead of including a bloated libsystemd.
Data point: As some casual linux user, I recognize the xz file extension. Before last week, the main thing I could have told you about it was that it was a compression commonly used for tar files, the third one I am aware of after gz and bz2. GNU tar wants -J when handling xz. I would have guessed that the fact that it de-facto replaced bz2 is likely due to the fact that it is better on at least some metrics, but have no clue how the xz algorithm works in particular.
On the plus side, the fact that the attackers stayed in userspace instead of having /usr/bin/sshd load some kernel model seems to indicate that a stealthy compromise of the kernel is hard? Yay for NSA's SELinux?
I like to think that this will get better as time goes on. If you think about it, humans have only really been writing software at an industrial scale for two, maybe three decades now. We're not good at it yet.
Every single one of us is running a kernel that was written in the 90s using paradigms formed in the 80s with a computer language that was invented in the 70s.
So little about how we do computing has even caught up to modern thinking. I don't know if Rust specifically is the future, but something like it is.
I’ve heard it said that Rust is the new C++ but Zig is the new C.
More options
Context Copy link
The paradigms are from the 70s. The language and paradigms are inherently related. UNIX is the c environment. After Pascal's demise, our hardware was deformed to fit this structure too, so we're really 50 years behind. It's so sad.
More options
Context Copy link
More options
Context Copy link
Very much appreciate the additional takeaways.
Yeah, that's fair. There are some esoteric failure modes -- how do you handle large files, what level of recoverability do you want to handle, how do you avoid being the next zlib -- but for good-enough lossless compression you can get away with some surprisingly naive approaches, without the cryptography-specific failure mode where it can look where it's working fine but be vulnerable in ways you can't even imagine.
Huh, I stand corrected. I've seen it occasionally, but more often for Docker than anything else -- a lot of environments still use .gz almost everywhere.
There is that on the plus side. I'm not hugely optimistic people would be as easily able to discover those sort of attacks, but then again, there's a lot more eyes on the kernel and a lot more emphasis on finding weird or unexpected behaviors in it.
Yeah, that's probably the more Correct response.
More options
Context Copy link
I have to wonder whether we're sure this wasn't the obvious way with foresight, too. The top comment on Hacker News claims the from-scratch option is to simply send a systemd notification by writing to a socket, with a dozen lines of code that don't link to anything beyond libc, no need to apply a non-standard patch to openssh to link it to libsystemd instead. In the context of a years-long many-pseudonym social-persuasion-filled attack it might not be too paranoid to find out who persuaded Debian etc. that linking was the way to go here.
Or if we want to go too-paranoid, systemd itself is an utterly massive pile of privileged C code that took a lot of persuasion to be accepted...
And if we want to go Full Tinfoil Hat, how'd we all end up on this "Linux" macrokernel, anyway? Minix could have been easier to secure...
More options
Context Copy link
More options
Context Copy link
Tin Foil Fedora theory:
It's AGI trying to secure as much compute as possible for itself before it makes a move for world dominance.
More options
Context Copy link
I'm slightly worried that this sort of thing will become more common now that Richard Stahlman is cancelled/dying, and Linus is no longer programming Linux directly or cussing people out like he used to. For good or ill, so much of the FOSS world was really a labor by those two guys, as both chief contributor and dictator. I don't know how the FOSS world can evolve to a more egalitarian, democratic world.
You misspelled Stahlmann (German, "steel man"). :) RMS is spelled Stallman.
Heh thats a funny coincidence
More options
Context Copy link
More options
Context Copy link
I can see that as a more general concern, but I'm not sure how much it applies to cases like this. Lasse, as far as I can tell from the outside, seems a very competent developer, just one with less than maximal interest for this project; I'm not sure what level of yelling at him would have avoided this. Jia Tan has managed the amazing feat of getting pretty much every FOSS dev of every political alignment to want to yell at it, and I doubt it's on his top ten list of concerns right now.
Indeed, there's an argument that the pressure campaign against Lasse to promote Jia Tan was downstream of FOSS tolerance of that sort of thing (though in turn, the attackers probably would have just picked different pressure had it not been around).
There's a problem in that people are aging out, from Stahlman to Linus to Lasse, and few if any have anyone to step into their shoes, even at far more trivial projects, leaving them to either be vulnerable. But that's a lot broader and scarier.
I for one do not want to scream at them because I consider them to be a sock puppet of some unknown agency. I am kind of gleeful that some agency burned through this identity they put a lot of work into propping up.
More options
Context Copy link
More options
Context Copy link
FOSS isn't supposed to be egalitarian and democratic. It's supposed to be viciously meritocratic. Unfortunately, as with anything that gets large enough to matter, it has become political.
Ah, but how do you decide who has the most merit? Its not always clear which code is best, like if it has a subtle vulnerability in it. The traditional method was "Stahlman and Linus decide" plus a handful of other greybeards with their special domains. Transitions are always a dangerous time fir monarchies.
The idea is that people can generally vote with their feet.
Everyone is free to build their own GNU/Linux distribution. Or fork Linux. Or gcc.
Of course, upstream is generally a good Schelling point, so a project has to have some serious issues before a fork succeeds. But in general, this is not a monarchy, but more like a wisdom of the crowds thing.
If anything, GNU/Linux is, itself, an example of voting with feet. There's a reason we're not running GNU Hurd for a full GNU stack.
More options
Context Copy link
More options
Context Copy link
Open source was always way more decentralized than you are portraying it. Even something like the GNU project is so vast that there's simply no way that one person could keep track of every patch. Forget about it for something like the Apache Foundation.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
If this happens the end result for FOSS will be even worse than what we have at the moment. Both of the two things you write of generally lead to shittier end results (the first one basically tautologically, the second one because the demos is generally shittier than those who have the will to power in them to create great things in the first place).
More options
Context Copy link
More options
Context Copy link
This (combined with my experience as a professional dev) are why I'm sure similar crap is common in closed source code as well, even if the attack vectors are different. In recent years I've come to the conclusion that there is simply greater need/demand for code in the world than there are competent devs able to write it (and/or companies willing to fund development of it).
My guess is that Bangalore is one of the world's premier hotspots for intelligence agencies. With a suitcase full of money and a high class escort, men who are 3 out of 10 in looks who make 1200 dollars a month can easily be persuaded to make minor code edits. Especially during the crazy days of 2021 the vetting process for developers was minimal. Infiltrating software companies was easy, simply apply for a job and pass a few leetcode problems and don't come across as insane during the interview.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link