This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
IQ tests sometimes contain questions with multiple right answers, or multiple logical approaches, though some believe that the better-designed ones eliminate these, while others think that approaching a problem from multiple angles is a better judge of intelligence than doing the right answer fast, or simply knowing the format of the test. And that's a persistent problem in many of these tests, where even a passing familiarity with the type of question asked often affects performance. Like, if you grew up in a household with certain puzzle books, which not all people do, your brain has already been primed to perform a few of these tasks, and will spend less time on processing overhead like trying to understand the instructions. Which, by the way, oh yeah, many IQ tests also time you and factor this into their final scores. This factoring already introduces a designer-specific subjective judgement regarding the relative weight of time vs accuracy, which is yet another reason why the "right and wrong answer" paradigm is not quite accurate. Like, if you had to choose between taking 8 seconds on a task and doing it right the first time, 10 seconds at a task but tried 2 approaches in that time, 12 seconds on a task but being extra sure you are correct, and 4 extra seconds to understand the question but 6 seconds to complete the task, which one is the most intelligent child? Maybe some of these average out across many questions, and you still get a measure that's "pretty good", but did you miss something? Yeah, maybe so! You miss more if you weren't very careful in creating the question and didn't test it. But wait! You focus-group tested the question and it was fine, but how about the focus group's representativeness? Okay, maybe that last one I'm being pedantic and petty, but just wanted to illustrate how small biases can stack up if you aren't careful.
Another common example of bias for critics of IQ tests is how verbal sections often contain things that rely on specific word exposure, or can be contorted by vocabulary size. You grew up in a house where everyone talks like they are 18, vs you had college professor parents, your vocabulary exposure is radically different. Are questions that lean on that really a measure of innate intelligence then? Some people say yes, but others say (especially if your desired interpretation is about genetics) no -- a classic case of where design assumptions of the test itself can only show up much, much later, and someone might dismiss as cosmetic and irrelevant but is actually a fundamental failing. It's been a while since I took a specific factual look at some of the industry-leading tests, but this kind of thing was true for a long time. Maybe the field has advanced a lot?
With that said, you're absolutely correct that you can still rank participants within a study using IQ scores, but I want to really, really drive this home: the distance between the subjects even in the same test is not predictable nor uniform nor interpretable without stating explicitly the assumptions, and many times the assumption step is skipped. So if you give 10 kids an IQ test and the final scores are ranked (let's ignore for now any relevant objections above), let's say without a time component, then yes the IQ test will rank the students and you're fine. But if you use your normal-transformation technique, and get one child at 85 and one at 100, you can't make the interpretation about about how big that gap is without considering how the test normalization process was done. It's not a gap between the students, because the test wasn't calibrated on them. If for example the test was calibrated on Chicago high school students, then your difference is one standard deviation in the normalized test results of a Chicago high school student. If it's one standard deviation worth from a theoretical population that you, um, forced to be perfectly normal in the first place... well, maybe you can see the issue that pops up, the math equivalent of begging the question. I hope this answers your point about comparing people within the group. Yes, you can compare, but the issue in this particular case is one of interpretation, which is what we in a thread like this are usually interested in.
I will grant you that in probably many cases, this is probably not going to make a massive, massive difference. But there will be a difference, and your test's rigor will be degraded, and if you make a habit of hand-waving these things away, at some point you actually will have a mountain of waived assumptions that will come back and bite you. Plus some of these logical leaps can actually torpedo entire analysis, they aren't always small effects. My perhaps wrong general impression of the IQ testing landscape does lend me the impression that hand-waving is a professional pasttime. It's possible the IQ testing companies and researchers have carefully considered all of these issues in detail. I will confess I'm not all that motivated to look and find the super rigorous approaches because as described elsewhere I do think that the whole concept is both unnecessary and more the equivalent of smart people masturbating to their own superiority more than a genuine desire to understand how intelligence works, but that is more about my overall perception of the overall state of rigor in the field. It's not about the concerns I raised, because those are actually very potent concerns that come up often and need to be addressed.
Like just as a brief look, I guess the WAIS-IV is popular? They seem to have tweaked it so there a few sub-index scores, there's still perhaps a bit of test bias though extent is debated, they did a better job indexing and calibrating the test, but also put a strong speed emphasis on some sections which as I mentioned is a big hang-up for some. Some concerns like the multiple-answer and verbal paradigms I can't evaluate quickly. And, worst of all, who makes and develops the test? Pearson. Fuck. Potentially, a whole lot of issues there. Did you know that one company peddled a screening test (actually used in many schools!) that a major study found could only determine whether a child was a grade level reader or not only 3ish percent better than a COIN FLIP? Still mad about that one. This podcast miniseries about that whole debacle was super interesting that you might find a fun listen.
I consider my own critique as one more based in methodology and "remembering the fundamentals" than a takedown of the whole field of psych, but if you see it as easily becoming such, I don't think that's too much of a stretch, so I get that and would respect it. Personally, I think it's more the field of education specifically that is pretty piss-poor. Enjoyed our convo for sure! I guess my ideal scenario is to get people to assign much lower weight to IQ scores and be more careful when using the info, or more broadly, making sure we're using and evaluating statistics more contextually, remaining aware of the fundamental limitations. Like for HBD stuff, if I were to boil it down and exaggerate for effect a bit, the problems in the paradigm of design racist test -> get racist results -> the poorly performing races must be dumber is one that needs a bit of consideration, rather than jumping straight to "why are you doubting all my fancy numbers". Or not even bringing race into it, if a hypothetical bad test rewards general knowledge a lot over some "pure skill" kind of questions, and then we use it immediately to gauge genetic stuff with "pure skill" connotations, we shouldn't be surprised pikachu face that we get weird results, or even if we get normal expected results, we have to go for fuck's sake we weren't even testing for that so the interpretation doesn't work. Anyways, rant over. Cheers :)
More options
Context Copy link