We Spoke To Developers About Steam User Reviews

Recent changes to Steam reviews, which filter out reviews from keys that weren't purchased directly through Valve's digital store, have caused all sorts of worry and concern. The intent is to remove false positives in the form of reviews exchanged for keys and the like, but legitimate reviews are also affected. Games that were Kickstarted no longer have their backers' assessment contributing toward the rating Steam displays at the top of the page, and people buying through Humble Bundles or elsewhere are similarly excluded by default.

We contacted a variety of developers and publishers, including Larian, Stardock and Mode 7, to hear if they thought the move might stamp down on unfair practices, or whether it would end up hurting rather than helping.

Loyalty, abuse and manipulation

Developers who have run successful crowdfunding campaigns were among those who showed immediate concern about the changes, and Swen Vincke, founder of Larian (who ran successful Kickstarters for Divinity: Original Sin and its sequel) captured the feeling of many who got in touch with me:

Well, these are your most loyal fans and so you’re cutting them away from the sampling pool. That doesn’t really feel fair. We have 42K backers whose opinion will be hidden behind a filter and not be accounted for in the score. If we sell say 500K units (which would be a success for a top-down turn-based RPG), that’s almost 10% of our audience. I don’t know what the statistical significance will be - it doesn’t seem to have made a big difference for Divinity:Original Sin 1 - but I can imagine that in the early days, when the opinions of early adopters are important, it might hurt us that the adopters with the strongest interest in the game won’t be heard, at least not in a way that matters. And since initial opinions often seem to set the trend, that feels like a loss for us. But as I said, I do understand why they’re doing it. A score only matters if you can trust it and I can imagine there was a lot of abuse and manipulation going on.

Not everyone has to imagine the abuse and manipulation. Brad Wardell, founder and CEO of Stardock, reckons the changes are for the good as they'll help to fix one of the main problems with Steam reviews.

I really like the changes. It's been one of the worst kept secrets that some studios were gaming the review system by handing out copies to super fans so that they could launch with 99% review scores. What will be interesting is when people start to notice those who were doing this.

The Steam review system is still very punishing to certain types of games. Particularly games with high hardware requirements as most PC gamers think their 5 year old machine is still state of the art and won't accept that a game plays slowly on it. That could be solved if reviews by players who were refunded weren't included in the rating. But overall, the Steam review system is still the best system currently available in my opinion.

Sean Colombo of BlueLine Games says this gaming of the review system is a trend that he's noticed recently, the studio having first released a game on Steam in early 2014. The way he describes it, studios are approached by people willing to provide reviews in exchange for keys, and he understands why some developers might be tempted:

There is a new trend that Eastern Euro/Russian players will contact you in hordes to offer Steam Reviews of your game if you give them a free copy. As long as they don't promise a positive review (which is super sketchy), that actually might be worth it for a lot of devs since a small pool of reviews is dangerous... until you have a bunch of total reviews, one bad review puts you in "Mixed" territory which will basically stone-wall your game's growth.

Throwing out the data with the bathwater

Founders of Mode 7, Paul Kilduff-Taylor and Ian Hardingham, who are currently working on Frozen Synapse 2, also recognise that Valve needed to deal with "a real problem". Hardingham appreciates concerns that the solution causes its own problems though:

Valve is unusually flexible with allowing developers to register as many keys with their system as you like - allowing us to sell hundreds of thousands of Steam keys through a Humble Bundle for instance. It also allows us to send out many promotional copies and run closed betas in any way we see fit. This is incredibly developer-friendly in comparison to some other distributors: I would never want Valve to change it.

Exempting non-Steam user reviews from the overall score is a response to a real problem where developers are giving out free games in return for good reviews. I would much rather Valve did this than change their open approach to external keys. However I do think you could look to remove specific accounts which have questionable reviewing/activating behavour and possibly get 90% of the benefit without losing the large amount of valid data that's being thrown out with the bathwater here.

Kilduff-Taylor thinks similarly:

It's frustrating that the idiotic behaviour of a few unscrupulous people has, once again, forced changes to a system which affects everyone. I definitely appreciate that Valve need to find a way of shutting down this type of fraud, though.

There is certainly an issue here for developers who want to release alphas on their own site, then make the jump to Steam at a later stage. The players that get hold of a game early on often tend to be some of the most passionate and vocal: they now won't get to contribute to the game's overall score. It's a small thing - and it's great that they will still be allowed to leave comments - but it could potentially have an impact.

Ultimately, everyone should be striving to make games which are well received throughout their life-cycle by different types of player: as long as developers stick to that, they shouldn't have anything to worry about here.

Navigating the mountain of shovelware

Developers are worried though and many of those worries seem to come from young studios and those that were already worried about issues on Steam, including visibility and the difficulty of not being buried beneath the "mountain of shovelware". That's a phrase used by Adam Blahuta of Jellyfish Games, who was originally excited by the prospects of Steam Early Access, but now feels it might not be the best place for his game:

...Steam's Early Access program excited me when it launched. The two most obvious benefits were early feedback and early revenue. The former would challenge our assumptions about the experiences our game provides and the latter would allow us to improve the game's quality and marketing material prior to launch.

At first, only the most eager of early adopters bought these unfinished games. They generally came with an understanding of what they were getting for their money and would frame their reviews accordingly. The more Early Access became popular, the more players started to flood in. There was an obvious shift in tone as the generally constructive pool of reviews started to get overshadowed by obnoxious negative reviews based on a misunderstanding of what one should expect of a game prior to release.

What made matters worse was the mountain of shovelware that made it through the now relaxed Greenlight approval process. As these titles started to garner almost as much attention as the quality games in development, this created a perception that most Early Access games were garbage.

This means that anybody who buys a game in alpha with incorrect assumptions or coming at it from a negative mindset is more likely to post a negative review. If the review's criticism is undeserved, this will unfairly affect sales, which will turn away people who would enjoy the game. Once things start going in the wrong direction, it can be very difficult to come back from that, no matter how much work you put into your subsequent updates. This makes it incredibly important to come up with a sales strategy that mitigates the impact of these undeserved reviews.

Colombo, at the digital boardgame developer Blueline mentioned above, has experience with a launch that backfired, although not for entirely predictable reasons. His story highlights how another recent change to user reviews - the introduction of a recent reviews filter - can help when a game stumbles out of the gates. Particularly since players very rarely update their reviews, even when developers update their games:

Users will sometimes update reviews, but not often. This means you have to be really careful when you launch. We launched Simply Chess for free on Steam (mostly because I love Chess and our board-game engine is really powerful... didn't think many people would play it) with a "Premium" version (which is practically a donation). ...our two-person team tested it, but we didn't do Early Access or anything.... then we had a couple hundred thousand users in the first couple of days. Not only do this many users find way more bugs than we found, but they also caused issues that we had never seen in our other games (eg: more than 1,500 concurrent games made the list of in-progress-games really laggy). We understandably got negative reviews, but even though we were able to get almost every reported bug solved through several updates within the week, very few of the negative reviewers updated their reviews.

When Steam released their "recent reviews" feature, this gave us a second-chance since it was a strong signal to new users, that the game was worth their time even though the Launch was rough.
It took many months to get the amount of eyeballs on the game to outweigh the initial bad reviews. Some users update their reviews which helped a lot, but many more didn't want to spend the time looking at a free game again.

And then there are those developers who seem to see this as yet another potentially confusing and hazardous aspect of an already baffling marketplace. Roger Valldeperas, developer of Flat Heroes, a game our Graham praised just yesterday, is still learning to navigate Steam as a dev and seeing the layout change feels like a setback:

...overnight we lost over 2/3 of our reviews (it's still not much in real numbers because we don't have many reviews yet, but way worse because that same reason, we didn't have many and we have even less now), and that left the game in a worse position than it already was, because when you release a new game and are not a known dev you rely almost entirely on press, streamers and giveaways, since pretty much nobody will buy a game that has not appeared on the press and has no reviews on Steam.

From these 3, press and streamers receive way too many e-mails to cover half of the game (among many other sites we sent RPS a press release with a key) so many devs are left with no other choice than trying to give some keys to random strangers hoping they'll like the game and will tell their friends, and maybe add a review (although I'm completely against asking for reviews in exchange of keys).

That said, even if in our specific case it was kind of unfair because of the timing (we had worse release conditions than the others before us), I do think it is a good measure overall, something had to be done with all the keys-for-reviews market, which was getting worryingly huge (like many devs, we received lots of e-mails asking for keys in exchange of good reviews).

Don't panic?

To those worried developers, Failbetter founder-gone-freelance-writer Alexis Kennedy has a brief message. Valve, he reckons, have been here before, and they'll steady the ship if it springs a leak.

My thoughts: DON'T PANIC. Valve's modus operandi is to make a swingeing change with no warning, watch the explosions, and then calmly recalibrate. You can call this heartless, you can call it far-sighted, but it's what they do. It's horrible that honest developers with strong communities have suffered from this. But making good games and having a strong community is the best kind of future-proofing there is, if you can ride out this one.

The important thing, as Danny Day has pointed out, is to give them feedback so they can know what to change. Valve needs us too; even they have competitors; and they think about the long term.

Eugene Hopkinson of VoxelStorm, creators of AdvertCity, doesn't call the changes "heartless", as in Kennedy's quote; he sees them as "an aggressive market control move":

...any reviews that came from bundle sales have of course been removed from the count as well - and if a backer pays $2 for a bundle of a dozen games late in their life cycle, they're far likely to judge the games less harshly than if they paid the full Steam $20 price early in the same game's release history - an effect that's compounded by the fact that early purchasers were more likely to encounter bugs that were ironed out in later iterations.

The fundamental problem, from our perspective, is that the system is rewarding the (proportionally more negative) opinions of early purchasers and disregarding the (predominantly more positive) perspectives of those who have obtained their key through another kind of interaction than a full price Steam purchase; for a game like AdvertCity with a very steep learning curve, the Kickstarter backers (for instance) are far more likely to play the game for longer, and our data has shown that with this game, play time is directly correlated to whether we receive a positive review from them or not. The net effect has been that this game's overall rating on Steam has dropped from an unreserved "positive" down to a "mixed" overnight. It seems as though this is really an indirect way for Steam to punish bundles, third party stores and so forth, as well as dissuading the developers who use them - an aggressive market control move.

Bundles and benefits

Some developers have seen very different results regarding Bundle sales though, saying that eliminating reviews from external sources has actually boosted their Steam rating. Here's Michael Molinari, developer of Choice Chamber:

I'm happy about Steam's review change, as it's brought my reviews back to Very Positive, from Mixed. I had less than 10k units purchased through Steam, but being in the summer Humble Bundle sent total units to over 110k, bringing with it lots of negative reviewers who likely didn't even want the game to begin with.

Peter Willington of Auroch Digital tells me the same is true of some of Auroch's titles, though also says that "this change basically nerfs the voices of the strongest advocates of projects borne out of crowdfunding":

...the change has actually been beneficial for us so far: we've seen the scores of some of our titles slightly rise.

We think this is something to do with those games being somewhat niche. The people who buy our games directly from Steam tend to be passionate and vocal and really appreciate that we're making the kinds of games they like. But we've also had these games appear in bundles, which puts them in front of players who might not be the target audience, and they subsequently mark the game down in a review for not being the kind of mainstream game they expected. Those bundle reviews now don't count towards the rating, pushing our score higher.

Simon Roth, developer of Maia, has created a Patreon, partly as a result of these changes, which he reacted too with a degree of despair. He told me how he thinks Steam reviews could be improved as a whole:

...the issue with Steam reviews is it's built in a way that doesn't really benefit anyone. The system needs to focus on giving detailed information about a game to players and demand that from reviewers. Reviews should address aspects of the game and guide the reviewer with some structure to maximize its worth. A system like this would remove all the "funny" reviews, the low effort ones and also reduce the issues of scammers as it would no longer be a viable business model for them. A level of Valve managed light community moderation on top of that would filter out the rest of the chaff. I know Valve have very little interest in putting any manpower into such things, but not everything can be automated away.

Users should be given the choice to directly message the developers before writing a review. So many reviews are just tech support problems in disguise. Using the user's survey data to present what machine they played on would also allow purchasers to know if a performance or tech issue might affect them.

I'll leave the final word to Blahuta, whose closing paragraph to me captured what a lot of indies - particularly those without established names or communities - were saying to me:

I'm an indie developer who's been working on a passion project for a little over 2 years and I'd really like to start selling an alpha build soon. If I go with Steam Early Access, I'm taking a risk that I'll drag any early negative reviews with me all the way to full release which will hurt my sales in the most critical period. If I do my alpha release outside of Steam (Itch.io, the game's website, etc.), I'll still get the benefits of alpha while only having Steam reviews from the final version, when they are most likely to be positive. However, with Steam's latest changes to user reviews, any champions of my game won't be able post visible user reviews on Steam launch day because they'll be using activated keys they were given as part of the sale outside of Steam. So, what do I do?