On our recent GEAR:30 podcast, Shop Talk: Pulling Back the Curtain on Gear Testing (episode #172), we discussed (once again) some of the best and worst ways to test skis — and many of these worst practices still take place in the ski industry today.
In response to the episode, a number of people wrote in to ask what we thought about blacking out top sheets in the interest of creating a more “objective” ski test.
We addressed this question several years ago, and thought it would be worthwhile to repost and to revisit the topic.
Let us know what you think.
This past Saturday, we got tagged by LINE Skis in a social post (along with SKI, Powder, and Freeskier magazine), about a “blacked-out” top sheet snowboard test that Snowboarder Magazine put together.
The post said this:
“Well done Snowboarder Magazine! Completely even playing field. No favorites, no tuning wars, no advertising dollars from competing brands and no sponsored / influenced testers. This would be a great idea for skiing. Do you agree? SKI Magazine, FREESKIER Magazine, BLISTER, Powder? We think adding something like this is a great idea.”
So since we were asked to weigh in, we did, and we replied that, while this was certainly an interesting experiment being done by Snowboarder Magazine, here at BLISTER, we don’t actually believe that blacking out top sheets would affect the results of our testing at all, and that there were far more important factors when it comes to creating a “completely even playing field.” Such as:
(1) Not accepting advertising dollars from any of the companies whose products are being reviewed — which we don’t.
(2) Not having sponsored skiers reviewing skis — which we don’t.
(3) Testing product over multiple days, and in multiple conditions — which we do.
(4) Finding skilled reviewers for our product testing, not just skilled skiers. (Because being really good at skiing does not mean that you are necessarily really good at teasing out and articulating the nuances of how a given ski performs.)
(For more on how and why we do things the way we do here at BLISTER, you can check this out.)
And so, assuming that the 4 criteria above are in place, do you still think that blacking out top sheets would make for a much more objective test?
Because we could certainly start working to put such a test together.
The more time you spend thinking about this, the more the logistical issues start to pile up. For example:
(1) It would be incredibly easy for us to identify certain well-known skis just from their shape, even if you blacked-out the top sheets. This seems like a really big problem.
(2) For this to be an apples-to-apples test, you’d need to pick a category of skis. It would be pretty useless to review a blacked-out 118mm-wide pow ski vs. a blacked-out 93mm-wide carver vs. a 105mm-wide charger. So you’d have to pick a category. Which is fine, except then it would seem that you would pretty quickly encounter the first problem we listed — some of the shapes would be obvious.
(3) So to counter that, you could have a bunch of companies all make a brand-new ski — e.g., either a touring ski, or a jib ski, or a directional charger. But they’d have to (a) want to do that; (b) have the time and resources to do that; (c) they’d need time to develop prototypes and test the new ski so that it didn’t perform like garbage — so we’d have to wait something like 6-18 months to do this test. And then (d) I guess they’d all then release for sale to the public all of these production skis in the same category?
Maybe they’d want to do this?
But given all of these logistical complexities, this is why we believe that there are far simpler, more straightforward, and more impactful steps that can be taken to trying to create a level playing field. And we have gone to great lengths to create such a level playing field — and have turned down a lot of advertising money from the ski and bike and running shoe and apparel companies in order keep this playing field level. We do this because we believe that this is the right way to do things if the goal is truly to provide the most accurate product information for consumers.
And taking all of this into account, the more it seems that the ‘blacked-out top sheets’ test makes the most sense if (a) you are taking money from advertisers (and therefore you need to find ways to remove the financial conflict of interest), and / or (b) your reviewers are working for (or are sponsored by) ski companies — so you need to make sure they aren’t being homers for their products, and haters on everyone else’s).
But again, we don’t think you should be doing (a) or (b) in the first place if you’re really trying to create a level playing field.
Furthermore, we train our reviewers to be as accurate as possible when testing a product. Our job is to talk about what a product does or doesn’t do well; identify where it excels and doesn’t excel, performance-wise, compared to its direct competitors; and help people understand which products will — and won’t — work best for them.
Another factor of our testing that helps to weed out any would-be bias is the product-comparison work we do. It’s possible that a tester might come in thinking, “I really tend to like skis from this company.” But then they have to go A/B that ski against several other skis in that category, and report how it compares on groomers, in moguls, on steeps, in powder, in crud, etc.
And then we often have multiple reviewers testing those same products in the same conditions, so we have other reviewers confirming or challenging any potentially undeserved praise or criticism that might come from a reviewer’s personal bias.
Given all of that, does it still sound to you like blacking-out the top sheets is a really compelling way to get more accurate / less biased ski assessments? Worth doing? Not worth doing? Thoughts on the numerous logistical issues?