This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
I don't see how it's easily gamed, but it does select for niche things. If these niche things are high quality to those that it appears to, isn't that fine? If less than 20% enjoy jazz, the better conclusion is "Of people who like Jazz, this one album is really good", rather than "Only 19% of all people like this album". Everything with a wider appeal has less depth, there's a sort of trade-off. I'd go as far as saying that everything good is niche. There's more people towards the middle of every standard distribution, but the best things (which are still popular enough to survive) are a few standard deviations to the right. And, if you allow those outside the niche to change what's inside of it, through the power of numbers, they will just destroy it or turn it into what they already like (which is plentiful everywhere). Hence why communities (like this one!) protect themselves with gatekeeping and rules and try to stay under the radar of outside political pressure.
I think the reason that votes aren't visible for a while on here is exactly to avoid starting a feedback loop (this one is the social one where people are influenced by other peoples votes). I also think that comments are sorted by "new" by default rather than by "best" (but I could be wrong), and that the "controversial" rating exists because the alternative is that the first decent comment to be made on a thread ends up being #1 simply because it started its exponential growth earlier.
Would your proposed weighting account for these things? (I don't know much about bayesian weighting)
You game it by controlling who gets access/early access to the game.
You can also review bomb other new games released close to your own release.
Given the quantity of games released this sort of score manipulation effectively turns that particular metric into a view of what has been released very recently, what is sufficiently niche to not attract non-fans and non-shills and what has most ratio manipulation behind it.
Controlling who gets access to the game (and therefore reviews) immediately devaluates reviews. I don't feel like this is an unintuitive exploit that we should guard against, but something so obvious that it goes without saying.
I wonder if text-only review would help the next problem. Instead of saying "this game is good/bad", you'd have to give information about the game. So if a game has 10 hours of gameplay, you'd write "I don't like that it's so short, only 10 hours!" but any reader who prefers shorter games would see it as a positive. Reviews like this would describe how the game was, and allow readers to judge the description, rather than merely access the judgement of the previous person.
Isn't this only if a new product gets a single positive review, putting it at 100% (perfect score)?
I do agree with the "niche" thing, but the opposite problem (niches being labeled bad because they don't appeal to a large amount of people) seems harmful as well, because it selects for watered-down content which is inoffensive and all-around unremarkable. If this sounds confusing, think about spicy foods: Those who love what is spicy wants the most spicy food available, but it's necessarily a small minority which enjoys this food, so any global rating would judge this food to be unappealing. If you place games (or other works) in a thousand-dimensional space, then all the edges and corners are maximums, and gives people who enjoy X the most X available. But across all people who judge the contents of this space, the highest scores will be biased towards the middle or possible the surface-area of the shape within the space. Less than 1% of music being listened to is Jazz, does this mean that it's universally hated, or that it should be banned? But that's the argumentation being used against controversial which has less than majority-support, for its removal is justified with the word "democracy". This is a bit of a rant, but I want to challenge the assumption that popularity is a measure of quality, and I think that rating systems may be inherently limited by the wrong assumption that there's one objective measure of good. I'm just theorycrafting, don't feel pressured to engage if it doesn't interest you!
The point isn't that it's too hard to figure out but that it effectively ruins that sort of filtering given the very large (and increasing) amount of games released every day. You might as well sort by new.
What you want is positive/negative reviews weigthed by how many reviews it has and how long it has been released, possibly with some downweighing very early reviews, both positive and negative. Text based reviews are of course also good but those are also possible to game, as seen with both smaller and larger games on steam, and it's quickly getting much worse with the advent of LLMs. This isn't a problem affecting a small amount of games but something that tons of developers do and it's increasingly an arms race, just like with regular search. You practically have to do it to be discoverable.
Ultimately you, as a consumer or a storefront, either need someone you trust to review the game or a very large amount of reviews, preferably released over a longer period of time, from paying customers so that it's effectively impossible to game.
I agree with what you're saying about niche interests but the problem of gaming the reviews still exists there and I feel like the solution is still weighted results but filtered by genre. Perhaps some amount of the featured games could be using genre filtered results so as to not get drowned out by the 800 pound gorillas and get some discoverability.
I feel like you just keep describing systems the Steam review score already has.
It a positive/negative aggregate which only counts steam purchases (hard to game review numbers when each review requires giving money to Valve), weights the descriptive categories based on the total number of reviews, has a recent reviews subcategory to downweight early reviews.
Since it only counts paid Steam purchases, it works especially great with niche genres. The steam review percentage doesn't correspond to what fraction of people like the game, it corresponds to what fraction of people who looked at the game and thought it interesting enough to spend money on liked it. (it works less well when the game has a divisive feature that doesn't neatly cleave across genre lines, such as any game with timers getting like a -10% to the review score)
My one big issue with the steam store is not with the reviews, but that recommendations of similar games seem to weigh popularity way more than similarity. Though I don't think there's a magic fix to discoverability, there is just too games coming out for that.
I'm not claiming these are some new revolutionary ideas, I'm pointing out why a simple ratio of positive/negative reviews don't work in a competitive and saturated environment like this.
This is a hard problem to solve and Valve has clearly thought about this quite a bit and their system is decent.
When there are a lot of purchases, sure, but initially you can spend part of your marketing budget gifting keys to sympathetic people, which people do.
Gifting keys to sympathetic people is free, you can just generate keys for your own game. But those don't count towards the review score. Review manipulation requires buying the game from dummy steam accounts with actual money and probably obfuscated payment methods. Which is not to say it doesn't happen, but there is a pretty clear line between marketing and probably steam tos violating review purchasing.
When there is pretty much 0% chance of discovery there isn't much of a difference. This is rampant on all major online marketplaces with user reviews as well as social media.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The main gaming that happens is review bombing. Even a few negative reviews can push your game out of the profitable peak and into the dead tail, so there’s lots of opportunities for bad actors to threaten devs into submission.
Does that actually happen, though? It seems like it's just an excuse that companies make when their games fail. Many negative reviews come quickly whenever a company does something people don't like and posts about it go viral.
Anyway, your point is valid, bbut if a million newish accounts with a low amount of money spent just starts reviewing games negatively, that's quite easy to spot. My steam account has spent more than 1000$ on games and is more than 10 years old and it's an active account. You can't fake that. You can even collect stats about how many reviews are made by "certainly legit" accounts, and if one game suddenly has a lower ratio than many other games, where it previously didn't, you know somebody used bots. I made it easy for myself by choosing Steam as an example, but the problem doesn't sound very difficult in general
I am assuming much lower numbers of reviews. Most people don’t review to start with, and the ones who do are statistically freaks, which is not necessarily promising.
If you assume a non-triple-A game has 20 4-star reviews, then a single 1-star review drops its average from 4 to 3.85. If you’re competing against hundreds of other games that’s potentially the difference between profit and no profit. Like AirBnB. So a legitimate user can make credible threats.
Even on the larger scale, review bombers, like Twitter cancellers, can accumulate easily to destroy something for some perceived wrong or just for money.
Ratios are a valid signal but I don’t think you can use them in isolation.
Reviews don't always need text and deep through, you can sometimes just click Like or Dislike or give it an amount of stars out of 5 or 10. I would suggest that only people who brought a product could review it, but didn't as I noticed that would drop the amount of reviews low enough that you could harm a million dollar game by buying 5 copies and making bad reviews.
Twitter cancellers can easily destroy things because they can disconnect with reality. They can accuse of things which aren't true, and they can get people involed which don't actually care about the product in question nor knows anything about it. Remember in 2016 when Hillary was predicted to win at 99% certaincy? The polling existing in a biased bubble which wanted to make people believe that he would lose, while the actual physical voting was the "reality" which corrected the delusion afterwards. When people contol the narrative they control how reality appears, but they cannot control reality, so any method we can use to ground a thing firmly in reality will make it harder for deceptive people.
Anyway, your point is valid and I'm a little less certain of ratios now. I think absolute votes are worse, though. 500 up and 400 down is a net karma of 100. 80 up and 2 down is a net karma of 78. And 100 karma in 24 hours will appear above something which gets 50 karma in 2 hours. Finally, even money is not a valid signal anymore. Many DEI games lose money, but they win the war by losing the battle. Some even suggest that companies are being subverted so that they can be made to fail and then bought cheaply by competitors
I think @Ioper is right, some kind of weighting is good. Or provide the option to sort by either - more customisable algorithms would be wonderful.
I always wonder how people handle this. Some sites provide a‘trending’ tab, but I don’t think total reviews is usually weighted by time. And if it were, that would result in a good game’s score decaying as it aged.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link