Recently, the RPG social media sphere reheated one of the classic controversies du jour: Should RPG critics write a review of an RPG product they have not played?
Some insist that playing a module / an adventure / a zine / a supplement is a minimum requirement. Not just a minimum requirement, but a bare minimum requirement. How could it be any different? How could you write a review of a music album without listening to it?
Is it reasonable to write a review of an RPG supplement without playing it first?
Reviews I like
I've mentioned before that I like Between Two Cairns.
In the most recent episode, Brad and Yochai were joined by Amanda Lee Franck to review The Reach of the Roach God. The Reach of the Roach God consists of three adventures and supplementary text (I learned from the review).
Brad had run the first adventure, a Quiet Lake (using Spenser Campbell's Slayers? Haha!).
Amanda had run the second adventure, Spider Mountain Temple. She regretted that she jumped straight to this adventure instead of running the first one first.
Nobody had run the third adventure, City of Peace. Everybody on the podcast highlighted they thought this adventure seemed the most challenging to run and they weren't sure how they would start to go about it, although they liked certain things about it.
As always, they gave an overview of the product and talked about what they liked and didn't like in the product. Subjects of discussion included:
- the quality of the art
- the amount of art
- the position of the art on the page related to other content
- the NPCs
- the boxed text
- the procedures
- the maps
- the keying of the maps
- the (custom!) font
- the layout of the pages
- the terseness and humor of the writing
- the relationship of the art to the text, etc.
(This content seems to be the same regardless of whether or not the hosts have actually played the product or not.)
Was this a reasonable review of The Reach of the Roach God?
Reviews I do
I have tried to review a few RPG products. I tried really hard to do a good job. It was a difficult task to do well (and I'm not sure I did do well). So I do not do RPG reviews.
I do have a job where reviews are a regular part of my job.
Product reviews
- Is this feature aligned with the overall strategic goals of the company?
- Does the feature solve the actual needs of a specific user profile? Does it balance the needs of multiple user profiles? What are the tradeoffs?
- Is this the right level for an MVP?
- What metrics will be used to determine next steps?
- Are there bugs in the MVP?
- Are these bugs launch blockers?
- What is the correct positioning for this product?
Design reviews
- Are the CTA buttons the right size?
- Do the navigation buttons make sense?
- Are elements that look interactable actually interactable?
- Is the design responsive to screen size?
- Is the UX copy primed for translation?
- Is the color contrast accessible?
Technical writing reviews
- Are the instructions consistent?
- Are the screenshots up to date?
- Are unhappy paths documented?
- Do code samples exist?
- Do code samples have proper formatting and highlighting?
- Will the sample code provided actually run if you copy and paste it?
- Is the language in active voice?
- Are the headers descriptive?
Because my job is in software, I am not an end user. I do not do the job that our end users do. Nobody I work with uses the product that my company produces on a day-to-day basis.
Is it reasonable to provide reviews for a product I do not use?
User testing and feedback
For all of the above, something that is missing is the feedback from actual users. Did we get it right?
- Are we providing value to our customers?
- Is the UX delightful?
- Is the documentation helpful?
We spend a lot of money and a lot of time to try and get these answers out of real users. Our business depends on it.
Does the the need for user feedback invalidate the other types of reviews provided during the product development process?
Is user feedback helpful in the same way as other types of reviews?
Very good points. I did a few reviews on my blog, but it is a very tough and unrewarding task. A thorough playtest + review of a module, past the "yeah its good" or "the 12HD giant was a bit tough on my group" is like uncredited/unpaid editing.
ReplyDeleteIt is very time consuming to play someone else's material. Many times you yoink an element from a source for your home game. Is it then fair to pass judgement based on that sample?
Given the semi-amateur nature of niche/indie/OSR RPGs, even candid feedback by attentive eyes are a very good filter. So, without having listened to the podcast, I trust they did a good overview.
Your analogy with the music album is not really accurate. I think reviewing an RPG product is more like reviewing instruments or music sheets. How the music sounds will heavily depend on the skill of the purchaser, the room they perform in (acoustics, etc), their energy when performing, etc.
Thanks for the post! Some good ideas to mull over