A critic is someone who enters the battlefield after the war is over and shoots the wounded.
—Murray Kempton

What makes a good review? This is my take, and is unapologetically opinion. There are plenty of useful reviews out there that fail to hit all these notes. But these constitute my ideal. This post was prompted and informed by several Google Plus conversations (here and here, at least).

Constructiveness

Reviews of tabletop RPG products generally concern finished products, so why bother with thinking about improvement? The ship has already sailed. First, focusing on constructive feedback forces the reviewer to really think about why something is a problem rather than just following feelings. Second, writing constructive reviews helps avoid snark. Third, providing constructive feedback reveals reviewer priorities and biases concretely. Simple criticism is often underspecified. For example, if a reviewer says that the monsters were too generic or too simple, the reader has to infer the meaning of generic or simple from past experience or make their own best guess. If the reviewer says that the monsters would be improved by adding attack routines and combat weaknesses, then the reader knows that the reviewer is specifically looking for tactical complexity in monsters and can weight the review accordingly.

Thesis

The best reviews use the review form to express a more fundamental idea, rather than simply evaluate a product. I imagine this criteria may be somewhat controversial. I understand preferring a focus on functionality and simple facts. However, reviews have audiences beyond just immediate buyers. They also inform future creators, including creators other than the original product’s author. For example, when I discussed Courtney’s Megadungeon zine, I framed my review around the idea of presenting a megadungeon piecemeal using the zine form. I make no claims about how insightful that particular thesis is, but that is what I mean by more fundamental idea. Someone else might run with the baton you provide. Reviews shape the form in addition to describing and evaluating.¹

Focus on value

Price is what you pay, value is what you get.
—Warren Buffett

Apart from ease of use, which is in some ways a measure of future opportunity cost, when I review something, I try to pay attention only to the value. I do not see the role of a reviewer as to judge whether product X is worth Y dollars. First, who knows what the cost will be in the future? For example, maybe a product will get new pricing (such as becoming pay what you want), be included in a bundle, or go out of print and become scarce. Second, I lack the info to evaluate how the value I see might match up to any particular reader’s resources. One person’s extravagance is another’s impulse purchase. On reflection, while writing this post, I decided that it may be useful to include how much I paid, and when, for commercial products, as that makes it easier for a reader to make a judgment and also makes the source of the product clear. So when I get a chance I may go back through my reviews and include that information. And that serves as a nice transition to considering free copies for review provided to reviewers by producers, and the attendant incentives.

Absent conflict of interest

The best reviewers do not accept complimentary copies. I believe Bryce buys all the modules he reviews (“I bought this stuff and read it so you don’t have to”). Consumer Reports buys the stuff they test (“Our shoppers pay full retail and purchase all the products we test to generate our ratings from the same places consumers do; we accept no sample products for testing”). I would never review something I was given a free copy of for purposes of facilitating a review. A commitment to this principle has been in the about section of Necropraxis since before it was called Necropraxis. I think this helps to maintain editorial independence.

For a given individual reviewer, of course ethics can outweigh incentives, but in the ideal case the incentive will be absent. I need to know more about a reviewer to have a sense of whether the incentive will matter compared to a case where there is no incentive. Similar concerns justify why people often prefer to pay for a fee-only financial advisor, because conflict of interest is less likely. I don’t think this is a categorical taint, but it is a factor that must have an effect in aggregate. I believe a norm against accepting free copies for review would be beneficial.

The scene is small enough that there will always be some degree of nepotism in reviewing materials created by friends. This problem is different than the appeal to material self-interest inherent in free review copies, but is related, and is probably harder to systematically combat in a small hobby scene, but we can still try. I disclose in a review if I feel this is a potential factor. All I can say here regarding this principle is to do your best and perhaps try to put on a slightly more critical hat when reviewing something by a friend or acquaintance.

Absent numerical ratings

Numbers, or star ratings, when applied to reviews, are heuristics, designed to help the reader avoid deliberative thinking. Tabletop roleplaying products are way more complex than Uber rides, and Uber rides are about the most complex product or service that I think might benefit from using such heuristics. I prefer drawing attention to particular strengths and weaknesses, with explanations, rather than just providing numbers. Numerical ratings provide only an illusion of objectivity, unless they are tied to specific definitions, but even then I think ratings are more likely to get in the way of actual consideration.

Absent snark

Good reviews are entertaining, because who wants to read hobby material of any kind that is boring? Trashing someone’s work for the shit-stirring is the low road to entertainment, and can be particularly tempting when writing a review, as the mindset of reviewing is generally critical. As a writer, snark can be easily confused with wit. Unless you are Nietzsche or Wilde though, your snark is probably not witty.

Further, the Internet attention economy and social media feedback loops incentivize snarky reviews. In the short term, and maybe even in the long term, snark will probably get your blog more views. It might even be a road to low-rent celebrity. However, I would rather be respected by a small number of people for civility than known by a large number of people for being a clown. Of course, good reviews will honestly point out shortcomings or problems and avoid pulling punches, but there is always a way to do that without snark. Given the social dynamics and the technological affordances of the Internet, I am probably spitting into the wind here, but mean spiritedness and snark are the bane of thoughtful discourse.

Basis: play versus perusal

For RPGs in particular, it is also helpful to be explicit about whether the review is based on using the material in play or simply reading the product. Reading a product for inspiration or entertainment is a perfectly legitimate use of a game product, but the review reader may be looking for something else.

Consideration of format

I am a book snob. I like the physical objects, and poorly bound books make me sad. This means that, generally, I avoid print on demand books for anything other than small booklets or pamphlets. If it is hardcover, I want stitched bindings. I realize that others may have different concerns, but if I was reading a review, I would always like to know whether a hardcover book was produced using a traditional print run using offset printing and whether the book has a stitched or glued binding.

One of the more thoughtful professors I have had so far in grad school provided a template for writing reviews of academic articles in one of his classes. Though the task of reviewing an academic article differs from reviewing products in several ways, I think this template could still be a reasonable starting point. A functional review can often handle each of these points with a single paragraph.

1. Description. Restate the core idea or main function of the work in your own words. Mostly avoid evaluation here. This ensures that you understand the author’s goal and potentially reveals misunderstandings earlier rather than later. In an RPG product review, it might be reasonable to foreshadow the overall evaluation either in a topic sentence or final transition sentence, but minimize judgment. If you have an idea around which you are organizing your review, here is where you provide an overture.
2. Praise. List strengths. If there are no strengths, why are you even bothering? Deciding to review a product at all is a form of curation and publicity. There are no RPG products that are so bad that trashing them is a public service.
3. Major shortcoming. Discuss the biggest problems. If possible, explain potential fixes. Keep this to one, or at most a handful, of issues, and prioritize.
4. Minor shortcomings. Here is where you can unload on everything else. Again, explain potential fixes. This often works well as a list with bullet points.
5. Conclusion. Tie everything together and summarize the overall judgment. Here is where you can best most effectively unfurl your opinion flags.

^{1. Tangentially, I think the same thing is true of good session reports. A good session report has a thesis beyond just a chronological list of what happened in play. Grognardia was particularly good at this. But that is a topic for another post. ↩}