site banner
sockpuppet2  1mo ago (astralcodexten.com)   2771 thread views

Scott: Come On, Obviously The Purpose Of A System Is Not What It Does

astralcodexten.com

This made me reflect that I hadn't actually thought critically about the phrase (at least, commensurate to how often it's used). For fun, if you think the purpose of a system is what it does, write what you think that means, before reading Scott's critique, then write if you've updated your opinion. For example: I think it's a useful way of re-framing obviously dysfunctional systems, so as to analyze their dysfunction, but Scott is persuasive that it's not a good means of understanding systems, in general, so people should be more cautious about adopting this framing and using the phrase, rhetorically.

(Spoilers go between two sets of "||")

13
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

This seemed like a particularly bad and uncharitable post by Scott. The examples he chooses at the top worded in what seems like an intentionally ridiculous manner, e.g. characterizing it as "the purpose of a cancer hospital is to cure two-thirds of cancer patients," rather than "the purpose ... is to cure as many cancer patients as possible, constrained on the available resources and laws, which happens to be two-thirds of them." Or "The purpose of the Ukrainian military is to get stuck in a years-long stalemate with Russia" rather than "the purpose ... is to defend Ukraine, constrained on the available resources and international politics, with a years-long stalemate against Russia an acceptable result."

To the point at hand, I always saw the phrase as something riffing on the same sort of concept as "sufficiently advanced incompetence is indistinguishable from malice." All systems have both intended consequences and unintended consequences, this is obvious. But what's troublesome is, all systems have unforeseeable unintended consequences, this is also obvious; as such, it's incumbent on the people who design systems to include subsystems to detect and react to unforeseen unintended consequences. And if they didn't include a subsystem like that or didn't make such a subsystem robust, then we can conclude that the purpose of the system included being entirely tolerant of whatever unintended consequence is at hand. And in practice by my observation, it often tends to involve, as one of its primary purposes, the designers of the system feeling really good about themselves and their conscious intentions, versus having the purpose of actually accomplishing whatever they consciously intended to accomplish.

But what's troublesome is, all systems have unforeseeable unintended consequences

I think what's troublesome is that a lot of systems have foreseeable "unintended" consequences, and the debate over POSIWID is whether failure to prevent a foreseeable consequence means that it must have been an intended consequence.

the debate over POSIWID whether failure to prevent a foreseeable consequence means that it must have been an intended consequence.

I think you aren't wrong, but I also see it a little differently, in the context of failing to prevent a foreseeable consequence. Rather, is such a failure an indication that part of the purpose of the system is to cause those foreseeable consequences unintentionally? As they say, you can't make an omelet without breaking a few eggs, and if "breaking eggs" refers to causing meaningful harm, it can feel really bad to intend to do it. But omelets are delicious, so why not create a system where eggs get broken without you having to intend it?

Then that gets into question of what "intent" even means, and whether someone's "conscious" intent is their "true" intent.

I agree with you here. I was kinda expecting Scott's article to get into the question of just what it means for a broad society-wide institution to have a "purpose" which would likely get into issues like your last sentence, but he never went there.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats