Espresso review journalism is seriously broken. How do we fix it?

There is an obvious demand for impartial, expert and informative reviews of espresso equipment and coffee in all price ranges. But, the demand is not being met for several reasons:

1. Even expert consumers typically have meaningful experience with a very limited number of machines. By "meaningful" I mean long-term daily use. All the modifiers we use, such as "wonderful," "great," "poor" and "shoddy" are implicitly comparative and informed by our experience (or lack of experience) with other machines.

2. Most amateur reviewers are in love with their own machines or need to justify the money they spent on them. At best they give their pride and joys the benefit of the doubt and at worst look at them through rose-colored glasses.

3. Too many reviewers are strong on mechanics and engineering and weak on the palate, often because they have limited access to top-tier espresso bars. This leads to tech-heavy reviews with suspect conclusions (or no conclusions) about the cup. (God bless the reviewers who are talented in both areas).

4. Too many reviews are of the nearly-useless "fresh out of the box" variety. People do not wait long enough or continue their reviews long enough to reveal the problems that show up in a month or two (much less a year or two) later.

5. Finally, too many of the most qualified reviewers, who also have the best access to multiple machines for long-term comparative reviews, are hamstrung by editorial policies that avoid offending advertisers or by personal ties that prevent offending friends and clients. Instead of much-needed expert negative reviews, there is silence. This applies to print journalism, as well as on-line.

I know that more than one of these criticisms applies to me, so I am not just pointing fingers. What is the solution?

I am guilty of at least a few of those. Having access to enough machines over a long period of time is both time consuming for the reviewer and expensive. If the machines are supplied through some channel then there is at least the appearance of the reviewer serving the needs of the supplier.

I have dealt one company who manufacturers a coffee product who is not as friendly with me as they previously were after I had reviewed one of their products honestly as I always try to do, and they apparently felt the review was not enough in their favor (I assume) and our relationship seems a lot colder than it use to be... icy, actually. I pointed out the positive and negatives of the product, and did so fairly, i thought. So for most reviewers they know that as soon as their reviews are perceived by suppliers as not as flattering as they would like, fairly or not, the supply side will dry up and then they have nothing to review. It can be a razor-thin edge between keeping your job and being honest. Trying to serve two masters is difficult, balancing the two sides so as to be transparent by featuring the positives and negatives.

The "Out of the Box" review syndrome is a matter of getting your review published before someone else does— first counts, or at least that is the perception. For an amateur like me with limited sources for equipment and an even more limited bank account, it is a rare occurrence when that happens, and I have to really be careful about keeping my excitement from tainting the review, one way or the other.

I think the only real solution is throwing $$$ at the problem by either finding a deep-pocketed reviewer who has the cash to buy machines on a regular basis, and has the financial resources to not worry about a "real" job and can spend the time doing the job right... Should I post my PayPal ID, my shipping address, or both???

And as far as publications, there are precious few which deal with top-end equipment, and since they all depend heavily on advertising dollars, don't expect to see anything to change in that venue. back in the early '80's there was a fellow named Joe Minton who worked for Rider Magazine (a motorcycle mag out in the foothills near you, Marshall). He was their technical editor. He did a detailed review of after-market oil coolers. He used digital logging thermometers and rode the same course on the same bike with different coolers, and he adjusted the graphs for ambient temperature. He found one of the coolers actually made the bike run a bit hotter because of its design of using round tubing with no internal fins (round tubing has the lowest surface-to-volume ratio of any shape). The manufacturer was a major advertiser in the mag and wrote a very terse letter to the mag, and Joe not only published the letter but gave the scientific reasons why the cooler did not work. Joe lost his job there, and shortly thereafter the manufacturer changed the design of the cooler. That was some 25 or 28 year ago.

In this day and age, what with home financing and job loses and the economy and mouths to feed and threats of lawsuits a writer has to be careful.

I would say that impartiality is a myth in any field. It's an ideal to which we aspire, but it's ultimately unattainable. The closest you can get to impartiality, really, is when you are open about your results, methodology, and also your connections, motivations, etc.: e.g., the way that home-barista acknowledges its sponsors for each review, and also, frequently, the backstory behind how the sponsorship/equipment acquisition came about.

Ultimately, on some level, performance is an opinion. Any evaluation of any machine is going to have a complex mix of qualitative and quantitative data and analysis. Heck, judging reviews as good or bad is an exercise in subjectivity itself.

I would counter your claim that espresso review journalism is broken by opining that, from my perspective, espresso review journalism is a mixed bag. There are ways in which it could be improved, certainly (such as more transparency), and it would be AMAZING to see a third party with an unthinkable amount of money sponsor reviews by professionals that would free them from responsibilities such as "being nice" about their sponsors' equipment, but ultimately, it seems like your points, Marshall, exemplify the "grains of salt" with which we evaluate the opinions and work of others.

It seems to me that such a skill (evaluating where other people are coming and judging their perspective from your own) is something that we have to do every day, in any hobby or work that we pursue. It seems to me that your list is really just a criticism of consumer reports type journalism in ANY category, be it hi-fi audio equipment, sports cars, etc., tuned to the specific issues that are manifest in the espresso world.

I suppose the best way to improve this scene, though, is for you to invest some money into sponsoring some 'fair,' wide-ranging reviews of equipment people might be interested in. The way to 'fix' a 'problem' of this nature is to compete in the market of reviews. You need to find an 'impartial' funding source, and 'impartial' reviewers. You need to develop an audience that appreciates your 'impartiality,' which of course, once developed, will make it increasingly easy to find 'impartial' funding.

I'm still a little confused though, as I was in A Plea for Intelligent, Adult Machine Discussions. I feel that Home-Barista is slowly working towards solving many of the issues that you bring up, with the qualification of the fact that Dan tried to do it 'for reals,' and hence it is subject to its own shortcomings. I feel similarly about machine discussions. I have my own 'short list' of people whose opinions I respect a lot, others who don't seem very helpful, and still others who annoy the crap out of me and that I usually ignore. I expect most people develop this kind of "network" of people for themselves, and I believe that, owing greatly to this site's excellent community, great Guidelines for productive discussions, and fair, even-handed moderation, this forum at least functions quite well. Yes, people have their own darlings, opinions, prejudices, but we all know, usually, what they are, and for the most part seem willing to discuss that and work around it. I am very pleased with this community and the quality of reviews that it generates, and I think it's amusing to read claims that it's 'broken.'

Just my 2¢, though.

shadowfax wrote:I would say that impartiality is a myth in any field. It's an ideal to which we aspire, but it's ultimately unattainable.

Absolutely.

I have my own 'short list' of people whose opinions I respect a lot, others who don't seem very helpful, and still others who annoy the crap out of me....

Definitely a quotable quote! I am considering having that tattooed on my back.

...hope that last bit wasn't TOO annoying....

shadowfax wrote:I would say that impartiality is a myth in any field. It's an ideal to which we aspire, but it's ultimately unattainable.

There is a difference between being impartial and having no prior opinions. Judging fairly and judging unfairly. Merely having biases and wallowing in them. Try telling your kids the next time you settle a fight between them that impartiality is a myth.

I don't disagree with you (about the difference between being fair/unfair, etc.), but I don't see the point you're trying to get at. None of us on this forum is one of your children, either, nor should they be treated as such. My point before is that (unlike an immature child) I don't need to be convinced of someone's 'impartiality' in order to find their review useful. I'd rather be convinced of their openness (among other things).

Random Thoughts . . . with, first, a bit of background.

As some may know, I spent some 35 years in various facets of the wine trade, including (relevent to this discussion) many years as a professional wine journalist, and many as a professional wine judge at various competitions. That means absolutely nothing, of course, when it comes to judging espresso, but more on that later . . .

To say that "espresso review journalism is seriously broken," and ask how it can be fixed implies several things, not the least of which is that people care. No one does.

OK, before you start screaming, understand that a) I have learned an enormous amount from the people who both run this site and post on its various discussion boards, and that b) were it not for this site, I'd still be happily pulling "decent" shots on my Gaggia, and no clue that there was "more" out there for the home user. (I am extremely hesitant to call myself a "home barista," but I shall forever remain indebted to this site -- more so than to my credit card companies.) I also know that I have a great deal more to learn, that I am "weak" on precisely some of the fundamental points Marshall addresses in his initial post.

But even the most casual participant on this board (let alone the "regulars") -- even the lurkers too timid to registar and participate for whatever the reason (all too often, it's a fear of feeling foolish, but that's another question: "How to welcome more new posters?") -- are but the tiniest fraction of one percent of the espresso-consuming population. The other 99+ percent of the public do not care, do not know about "espresso review journalism" (and wouldn't care even if they did).

As a result, I would disagree with Marshall's contention that, "There is an obvious demand for impartial, expert and informative reviews of espresso equipment and coffee in all price ranges." I would say, "No, no one really gives a $#!+."

At least, no one outside the relative handful of people who participate here on a regular basis.

The people who visit this site AT ALL are already interested enough in espresso to seek out a site such as this, but they are extremely small in numbers. The people who register here are even fewer, and the regular participants are fewer still.

I have no hard numbers, but I am willing to bet that Home-Barita.com is no different from the wine sites I participate on: the majority of posts are made by a handful of people; the majority of people post very little. AND, especially with regard to this site, a sizeable number of posters come onto this site looking for specific information/opinions about buying a new machine and/or grinder, and once they get this information, they disappear never to be heard from again.

Rule #1: Remember at all times you are preaching to the converted.

Marshall is correct when he says, "Even expert consumers typically have meaningful experience with a very limited number of machines"; "Most amateur reviewers are in love with their own machines or need to justify the money they spent on them"; and, "Too many reviewers are strong on mechanics and engineering and weak on the palate." But this is true in every field where there is sufficient cost involved.

But "espresso review journalism" differs from wine journalism in one very important way: the writer who reviews espresso is reviewing TWO very different things. One is the machine, the "hardware" if you will, and virtually every machine out there (at least in the "prosumer" and "commercial" classes) is a solid contender for producing very good-to-great espresso, along with the occasional "god shot." The second is the drink itself, the "software," and this has more variables than you can shake the proverbial stick at. And, since virtually every machine is a capable one, this latter "software" review is largely irrelevent to the review of the machine (the "hardware") itself.

Off the top of my head, I cannot think of any publication other than Car and Driver, or some similar automotive publication(s), that "hang onto" a model long enough to do the sort of "in-depth" analysis (as opposed to "the nearly-useless 'fresh out of the box' variety) that Marshall decries. Problem. That takes money. Or patience. Neither one of which people typically have in adequate supply.

As far as the need to "avoid offending advertisers or . . . offending friends and clients," as long as one takes advertising (or has friends), that's bound to happen. It comes with the territory. And until one has a magazine, or a web site, as large as the Wine Spectator, for example, people can and will get offended and cancel their advertising and/or sponsorship when they get hammered by an unpleasant review.

The "Speculator" gets around this, by the way, because no one in the trade a) takes them seriously, and b) can live without them (once they have achieved sufficient size). For example, does anyone think a micro-winery takes them seriously? They don't, and the wines are always sold out anyway via the "exclusive" mailing list prior to the publication of the review -- even if the review sucked, it would be too late. And while large-scale operations may get hammered by a bad review, and see a slight sales decline, overall they get more good reviews than bad, and the ads continue to be seen by millions of people . . . "doesn't matter what they say, as long as they spell my name right!" Furthermore, in terms of bad reivews, even the self-anoited "Wine Advocate," Robert Parker (who takes no advertising, by the way), doesn't publish reviews below 85 points on a 100-point scale. Hard to offend if you never publish a bad reivew!

OK, back to coffee/espresso.

What possible good does it do anyone to give a new (e.g.) GS3 or Vibiemme Double Domo to someone for (e.g.) six months for a thorough "bench test"? First of all, the machine costs "big bucks.
Even at the cost to the vendor, it's expensive. How many machines would 1st-Line, Chris' Coffee, Stefano's Espresso Care, etc. have to sell to recoup the cost of that machine? (The fact that they have ever done this at all should be loudly applauded!) What is to gain, and what is to lose? A bad review of the hypothetical new triple-boiler machine from Oedipal Engineering will kill -- or at least seriously dent -- sales. A great review may convince a couple of people who were on the fence between buying the Oedipal machine or the one from Old Simon, but how many units are we really talking about? These suckers cost $$$$ -- we're not talking about selling 10,000 units a year, after all.

So, IMHO, it would seem that -- barring some billionaire (with a desire to become a millionaire) funding a publication to go out and actually buy machines, and run them daily in a Consumer Reports-type testing lab, I am afraid we are "stuck" with having people like you and me write their opinions -- regardless of how qualified we may (or may not) be to do so as measured by some outside standard.

I know this was not his intent but, within limits, Marshall is questioning the entire existence of this site. But as long as one has a forum open to the public (even one requiring registration), one must be prepared for a wide variety of opinions as well as a wide variety of expertise in the field.

I am sure each of us "[has their] own 'short list' of people whose opinions [we] respect a lot, others who don't seem very helpful . . . " That only makes sense. And I am also sure there is some overlapping consistently in our lists, but no universal consensus.

So while I do think Marshall has raised some interesting points, I am not sure that -- realistically -- there is any solution. We shall continue to read posts where various people make this or that claim, based upon little or no experience; posts where the author unwittingly may allow a vested interest (e.g.: the purchase of ____________) to affect his or her judgement; and posts that require an ample number of grains of salt.

Feel free to email me your addresses, so that I may send you the salt needed for my own posts now . . . .

Cheers,
Jason

zin1953 wrote:Feel free to email me your addresses, so that I may send you the salt needed for my own posts now . . .

For what it's worth, I have 40 lbs. of rock salt in a closet, purchased for my water softener and most of which will never be used. Save yourselves the trouble and send me your address for salt.

Hi Marshall,

You have made this point a number of times and I think that it is a very good one. Without wanting to sound arrogant, I have had a fair bit of experience with many commercial and prosumer machines. Whilst I certainly haven't had the benefit of a long-term review of most, it is quite apparent to me that some machines are plain worse than others and that, in comparison with others, some machines deliver so little for the price at which they are sold that I would never buy them. I often cringe inwardly when I read that someone has bought a particular machine that I think is crap and then turns around and trumpets its virtues online. Conversely, I make a conscious effort to avoid 'naming and shaming' because I know many people in the industry who sell many of these machines. I did write one review for a machine sold by a company with which I had relations and my experience in doing that was basically the same as Randy's.

Frankly, I think that there must be one or two machines that represent the best value at any given price point and everything else is not worth buying.

As for getting there, as others have said, I think that the most workable solution is expressed in this site's guidelines:

Be open and honest. Many people rely on opinions presented in these forums as part of their purchase decision. The source of the information you present and basis of the opinions you express are as important at the statements themselves. For example, you should make it clear whether you speak from first hand experience, what you read elsewhere (and if appropriate, link to the source), or conventional wisdom.

By providing this background information, people can make their own decisions on this:

I have my own 'short list' of people whose opinions I respect a lot, others who don't seem very helpful, and still others who annoy the crap out of me....

I don't think that there's much utility in having a peer ratings system for reviews. All that will happen is that people who have no experience with the subject of the review will stay silent and people who have bought the machine that is the subject of the review will rate the review highly, to the extent that it vindicates their purchase.

The lack of experience that most people have with machines other than their own seems to lead to regurgitations of how good that machine is, raising it to mythical status. This was the case for the Silvia, but, lately, as people have sold their silvias for the next big thing, a more realistic appraisal of the machine seems to be taking place. In Australia, the ECM Giotto was really the trailblazer for the prosumer machine market and I think that it enjoyed legendary status as a result. I'm told that economists call it "first to market advantage." This phenomenon is part of the reason why a lot of commercial operators don't really participate in these forums.

Personally, I don't put a lot of stock in most reviews that I read.

Hi Jason,

Regarding this:

But even the most casual participant on this board (let alone the "regulars") -- even the lurkers too timid to registar and participate for whatever the reason (all too often, it's a fear of feeling foolish, but that's another question: "How to welcome more new posters?") -- are but the tiniest fraction of one percent of the espresso-consuming population. The other 99+ percent of the public do not care, do not know about "espresso review journalism" (and wouldn't care even if they did).

As a result, I would disagree with Marshall's contention that, "There is an obvious demand for impartial, expert and informative reviews of espresso equipment and coffee in all price ranges." I would say, "No, no one really gives a $#!+."

At least, no one outside the relative handful of people who participate here on a regular basis.

I think that you miss the point that the relevant market is not the general public, but people who would purchase a prosumer espresso machine. I'd wager that a significant number of people who decide to pony up a few grand for an appliance would probably at least punch the name of the thing into google.

Furthermore, in terms of bad reivews, even the self-anoited "Wine Advocate," Robert Parker (who takes no advertising, by the way), doesn't publish reviews below 85 points on a 100-point scale. Hard to offend if you never publish a bad reivew!

Good point, but the review is still valuable as long as it is accurate. I think that it would be interesting to see the reviews for some real stinkers, even if their names were withheld.

virtually every machine out there (at least in the "prosumer" and "commercial" classes) is a solid contender for producing very good-to-great espresso, along with the occasional "god shot."

I don't think that that is the case. The various pstat settings, thermosyphon restrictors, steam tips, injectors, etc. can be changed on many machines to get the performance that you want, but that isn't something that you can really expect a consumer to do. So some machines come out of the box performing worse than they should. There are a few machines on the market that I just can't imagine would ever really be great to use, nor do I think that what they offer at the particular price point ought to compel people to buy them - even if they were OK.

Cheers,

Luca

PS. That was about espresso machines. There are much fewer obstacles to presenting useful and impartial reviews of coffee and I think that the emergence of new "coffee critics" in the vein of coffeereview.com and coffeecuppers.com would be beneficial to the specialty coffee market.

Great post, Jason !!

Sadly, we've become too much of a 'no risk' and 'tell me what to [buy / think]' society. Only the immature / insecure / inexperienced put blind faith in 'reviews'. The vast percentage of the participants here have been around the block more than a few times. Sadly too many don't do enough research first, swallow the swill on commercial sites and end up here avant eBay.

Luca's contention that some machines offer more value is valid, but there are other factors besides raw performance. Aesthetics may count more for some purchasers. No one set on the gleaming chrome of an e61 is likely to be swayed to a Vivaldi.

No one who does a modicum of research here is at any risk of purchasing a turd, except by willful stupidity. What more could we ask?

No guts? No glory!