I would say that impartiality is a myth in any field. It's an ideal to which we aspire, but it's ultimately unattainable. The closest you can get to impartiality, really, is when you are open about your results, methodology, and also your connections, motivations, etc.: e.g., the way that home-barista acknowledges its sponsors for each review, and also, frequently, the backstory behind how the sponsorship/equipment acquisition came about.

Ultimately, on some level, performance is an opinion. Any evaluation of any machine is going to have a complex mix of qualitative and quantitative data and analysis. Heck, judging reviews as good or bad is an exercise in subjectivity itself.

I would counter your claim that espresso review journalism is broken by opining that, from my perspective, espresso review journalism is a mixed bag. There are ways in which it could be improved, certainly (such as more transparency), and it would be AMAZING to see a third party with an unthinkable amount of money sponsor reviews by professionals that would free them from responsibilities such as "being nice" about their sponsors' equipment, but ultimately, it seems like your points, Marshall, exemplify the "grains of salt" with which we evaluate the opinions and work of others.

It seems to me that such a skill (evaluating where other people are coming and judging their perspective from your own) is something that we have to do every day, in any hobby or work that we pursue. It seems to me that your list is really just a criticism of consumer reports type journalism in ANY category, be it hi-fi audio equipment, sports cars, etc., tuned to the specific issues that are manifest in the espresso world.

I suppose the best way to improve this scene, though, is for you to invest some money into sponsoring some 'fair,' wide-ranging reviews of equipment people might be interested in. The way to 'fix' a 'problem' of this nature is to compete in the market of reviews. You need to find an 'impartial' funding source, and 'impartial' reviewers. You need to develop an audience that appreciates your 'impartiality,' which of course, once developed, will make it increasingly easy to find 'impartial' funding.

I'm still a little confused though, as I was in A Plea for Intelligent, Adult Machine Discussions. I feel that Home-Barista is slowly working towards solving many of the issues that you bring up, with the qualification of the fact that Dan tried to do it 'for reals,' and hence it is subject to its own shortcomings. I feel similarly about machine discussions. I have my own 'short list' of people whose opinions I respect a lot, others who don't seem very helpful, and still others who annoy the crap out of me and that I usually ignore. I expect most people develop this kind of "network" of people for themselves, and I believe that, owing greatly to this site's excellent community, great Guidelines for productive discussions, and fair, even-handed moderation, this forum at least functions quite well. Yes, people have their own darlings, opinions, prejudices, but we all know, usually, what they are, and for the most part seem willing to discuss that and work around it. I am very pleased with this community and the quality of reviews that it generates, and I think it's amusing to read claims that it's 'broken.'

Just my 2¢, though.

I don't disagree with you (about the difference between being fair/unfair, etc.), but I don't see the point you're trying to get at. None of us on this forum is one of your children, either, nor should they be treated as such. My point before is that (unlike an immature child) I don't need to be convinced of someone's 'impartiality' in order to find their review useful. I'd rather be convinced of their openness (among other things).

Random Thoughts . . . with, first, a bit of background.

As some may know, I spent some 35 years in various facets of the wine trade, including (relevent to this discussion) many years as a professional wine journalist, and many as a professional wine judge at various competitions. That means absolutely nothing, of course, when it comes to judging espresso, but more on that later . . .

To say that "espresso review journalism is seriously broken," and ask how it can be fixed implies several things, not the least of which is that people care. No one does.

OK, before you start screaming, understand that a) I have learned an enormous amount from the people who both run this site and post on its various discussion boards, and that b) were it not for this site, I'd still be happily pulling "decent" shots on my Gaggia, and no clue that there was "more" out there for the home user. (I am extremely hesitant to call myself a "home barista," but I shall forever remain indebted to this site -- more so than to my credit card companies.) I also know that I have a great deal more to learn, that I am "weak" on precisely some of the fundamental points Marshall addresses in his initial post.

But even the most casual participant on this board (let alone the "regulars") -- even the lurkers too timid to registar and participate for whatever the reason (all too often, it's a fear of feeling foolish, but that's another question: "How to welcome more new posters?") -- are but the tiniest fraction of one percent of the espresso-consuming population. The other 99+ percent of the public do not care, do not know about "espresso review journalism" (and wouldn't care even if they did).

As a result, I would disagree with Marshall's contention that, "There is an obvious demand for impartial, expert and informative reviews of espresso equipment and coffee in all price ranges." I would say, "No, no one really gives a $#!+."

At least, no one outside the relative handful of people who participate here on a regular basis.

The people who visit this site AT ALL are already interested enough in espresso to seek out a site such as this, but they are extremely small in numbers. The people who register here are even fewer, and the regular participants are fewer still.

I have no hard numbers, but I am willing to bet that Home-Barita.com is no different from the wine sites I participate on: the majority of posts are made by a handful of people; the majority of people post very little. AND, especially with regard to this site, a sizeable number of posters come onto this site looking for specific information/opinions about buying a new machine and/or grinder, and once they get this information, they disappear never to be heard from again.

Rule #1: Remember at all times you are preaching to the converted.

Marshall is correct when he says, "Even expert consumers typically have meaningful experience with a very limited number of machines"; "Most amateur reviewers are in love with their own machines or need to justify the money they spent on them"; and, "Too many reviewers are strong on mechanics and engineering and weak on the palate." But this is true in every field where there is sufficient cost involved.

But "espresso review journalism" differs from wine journalism in one very important way: the writer who reviews espresso is reviewing TWO very different things. One is the machine, the "hardware" if you will, and virtually every machine out there (at least in the "prosumer" and "commercial" classes) is a solid contender for producing very good-to-great espresso, along with the occasional "god shot." The second is the drink itself, the "software," and this has more variables than you can shake the proverbial stick at. And, since virtually every machine is a capable one, this latter "software" review is largely irrelevent to the review of the machine (the "hardware") itself.

Off the top of my head, I cannot think of any publication other than Car and Driver, or some similar automotive publication(s), that "hang onto" a model long enough to do the sort of "in-depth" analysis (as opposed to "the nearly-useless 'fresh out of the box' variety) that Marshall decries. Problem. That takes money. Or patience. Neither one of which people typically have in adequate supply.

As far as the need to "avoid offending advertisers or . . . offending friends and clients," as long as one takes advertising (or has friends), that's bound to happen. It comes with the territory. And until one has a magazine, or a web site, as large as the Wine Spectator, for example, people can and will get offended and cancel their advertising and/or sponsorship when they get hammered by an unpleasant review.

The "Speculator" gets around this, by the way, because no one in the trade a) takes them seriously, and b) can live without them (once they have achieved sufficient size). For example, does anyone think a micro-winery takes them seriously? They don't, and the wines are always sold out anyway via the "exclusive" mailing list prior to the publication of the review -- even if the review sucked, it would be too late. And while large-scale operations may get hammered by a bad review, and see a slight sales decline, overall they get more good reviews than bad, and the ads continue to be seen by millions of people . . . "doesn't matter what they say, as long as they spell my name right!" Furthermore, in terms of bad reivews, even the self-anoited "Wine Advocate," Robert Parker (who takes no advertising, by the way), doesn't publish reviews below 85 points on a 100-point scale. Hard to offend if you never publish a bad reivew!

OK, back to coffee/espresso.

What possible good does it do anyone to give a new (e.g.) GS3 or Vibiemme Double Domo to someone for (e.g.) six months for a thorough "bench test"? First of all, the machine costs "big bucks.
Even at the cost to the vendor, it's expensive. How many machines would 1st-Line, Chris' Coffee, Stefano's Espresso Care, etc. have to sell to recoup the cost of that machine? (The fact that they have ever done this at all should be loudly applauded!) What is to gain, and what is to lose? A bad review of the hypothetical new triple-boiler machine from Oedipal Engineering will kill -- or at least seriously dent -- sales. A great review may convince a couple of people who were on the fence between buying the Oedipal machine or the one from Old Simon, but how many units are we really talking about? These suckers cost $$$$ -- we're not talking about selling 10,000 units a year, after all.

So, IMHO, it would seem that -- barring some billionaire (with a desire to become a millionaire) funding a publication to go out and actually buy machines, and run them daily in a Consumer Reports-type testing lab, I am afraid we are "stuck" with having people like you and me write their opinions -- regardless of how qualified we may (or may not) be to do so as measured by some outside standard.

I know this was not his intent but, within limits, Marshall is questioning the entire existence of this site. But as long as one has a forum open to the public (even one requiring registration), one must be prepared for a wide variety of opinions as well as a wide variety of expertise in the field.

I am sure each of us "[has their] own 'short list' of people whose opinions [we] respect a lot, others who don't seem very helpful . . . " That only makes sense. And I am also sure there is some overlapping consistently in our lists, but no universal consensus.

So while I do think Marshall has raised some interesting points, I am not sure that -- realistically -- there is any solution. We shall continue to read posts where various people make this or that claim, based upon little or no experience; posts where the author unwittingly may allow a vested interest (e.g.: the purchase of ____________) to affect his or her judgement; and posts that require an ample number of grains of salt.

Feel free to email me your addresses, so that I may send you the salt needed for my own posts now . . . .

Cheers,
Jason

zin1953 wrote:Feel free to email me your addresses, so that I may send you the salt needed for my own posts now . . .

For what it's worth, I have 40 lbs. of rock salt in a closet, purchased for my water softener and most of which will never be used. Save yourselves the trouble and send me your address for salt.

Luca, I agree with most of your post, although I have much less confidence than you that there would be such clear winners at given price points among espresso machines. As Ian points out, there are factors that go into high-end machine purchases, and, as the Titan Grinder Project really drove home for me, when you want to talk about taste and results in the cup, and then you want to say that there are clear winners in x category, you really have to ask, whose taste?

luca wrote:I don't think that there's much utility in having a peer ratings system for reviews. All that will happen is that people who have no experience with the subject of the review will stay silent and people who have bought the machine that is the subject of the review will rate the review highly, to the extent that it vindicates their purchase.

One other thing. I just want to clarify strongly that I fully agree with you here, and I am not totally sure if you quoted me to disagree with me. Just to be clear, I do not want a "poster" rating system, and I don't think that would help me or anyone else--I don't even KNOW who's voting in those ratings, so why should I trust them? I was referring to my own personally developed short list, a list that I have developed by doing massive amounts of reading in this forum and all the sundry places I find via links, Google searches, etc. I have my own criteria for picking people whom I respect on the internet (and elsewhere), and I expect everyone develops this as well. The information is out there. You have to do the work. That was what I was getting at. And in that vein, I wanted to QFT:

cafeIKE wrote:No one who does a modicum of research here is at any risk of purchasing a turd, except by willful stupidity. What more could we ask?

This is my opinion of Home-Barista in a nutshell, and a pretty funny nutshell at that.
Of course, there's always room for improvement, but I don't think that the status quo is anything like 'broken.'

Reviews are important, and being honest and open, charitable and thorough is important. But the fact that available reviews are a 'mixed bag' in terms of these criteria does not make the scene broken. Reviewing will never determine the best machine (in a class, or just period), because espresso is a matter of taste, and more than that, it's a journey. We all (with few exceptions) started with crappy machines and coffee. We all improved through a combination of experience and education. And even as we get into the high end, out tastes change--they sharpen and evolve as our exposure broadens. New origins, new roast levels, new doses...

Espresso is the hands on foodarthobbythingy. Who is here looking for the answer? I thought we were all enjoying the ride. Espresso is a highly active method of food preparation. And it's fitting that, to get good results, you need to be as active in your decision making process as you do in hammering down your skills and broadening your horizons after you pick your setup.

And no, I have not and will not compare my opinion to a Vivaldi II.

zin1953 wrote:But even the most casual participant on this board (let alone the "regulars") -- even the lurkers too timid to registar and participate for whatever the reason (all too often, it's a fear of feeling foolish, but that's another question: "How to welcome more new posters?") -- are but the tiniest fraction of one percent of the espresso-consuming population. The other 99+ percent of the public do not care, do not know about "espresso review journalism" (and wouldn't care even if they did).

As a result, I would disagree with Marshall's contention that, "There is an obvious demand for impartial, expert and informative reviews of espresso equipment and coffee in all price ranges." I would say, "No, no one really gives a $#!+."

At least, no one outside the relative handful of people who participate here on a regular basis.

luca wrote:I think that you miss the point that the relevant market is not the general public, but people who would purchase a prosumer espresso machine. I'd wager that a significant number of people who decide to pony up a few grand for an appliance would probably at least punch the name of the thing into google.

Ah, but Luca, my point is that this is exactly what is already happening. The (relative) handful of people worldwide who have decided to "pony up a few grand" already ARE googling for information, already are stumbling across this site (and other similar ones), etc., etc. My point about the market being a tiny fraction of the general, overall market was made in terms of funding some sort of professional magazine/review website along the lines of what Marshall was calling for (long-term reviews, "journalistic" reviewers, etc.).

If you broaden Marshall's scope -- or rather, my scope, as Marshall actually did say, "There is an obvious demand for impartial, expert and informative reviews of espresso equipment and coffee in all price ranges" (emphasis added) -- if you speak of ALL espresso and coffee machines, from the inexpensive moka pot, the Mr. Coffee automatic drip machines (with the built-in timer so the coffee is made before you awake in the morning), and the Krups $99 "espresso" machine, through the Technivorms, the Clovers, and the Synessos of the world . . . THEN you might have the basis for a publication with a wide-enough consumer base. I just don't think the "prosumer" market is large enough.

zin1953 wrote:virtually every machine out there (at least in the "prosumer" and "commercial" classes) is a solid contender for producing very good-to-great espresso, along with the occasional "god shot."

luca wrote:I don't think that that is the case. The various pstat settings, thermosyphon restrictors, steam tips, injectors, etc. can be changed on many machines to get the performance that you want, but that isn't something that you can really expect a consumer to do. So some machines come out of the box performing worse than they should. There are a few machines on the market that I just can't imagine would ever really be great to use, nor do I think that what they offer at the particular price point ought to compel people to buy them - even if they were OK.

Luca, I will defer to your greater experience in this area. I was not trying to suggest that all machines are equal, nor that some are not clearly better than others; rather, I was attempting to say (admittedly in a less-than-accurate way) that the differences between (most) prosumer machines is relatively slight, even if important, and that -- as you quite rightly said -- tweaks can be make, but "that isn't something that you can really expect a consumer to do."

I would expect the participants here -- whatever their expertise -- to comment upon and/or advise potential buyers on the ease of various machines with which they have some experience, and I also expect the readers of such comments to apply the appropriate grains of salt . . .

Finally . . .

zin1953 wrote:Furthermore, in terms of bad reivews, even the self-anoited "Wine Advocate," Robert Parker (who takes no advertising, by the way), doesn't publish reviews below 85 points on a 100-point scale. Hard to offend if you never publish a bad reivew!

luca wrote:Good point, but the review is still valuable as long as it is accurate. I think that it would be interesting to see the reviews for some real stinkers, even if their names were withheld.

The key to this is "accurate," but not (I don't think) in the way you mean it. Whether it is Robert Parker, James Halliday, Jancis Robinson, or any other wine writer (including me!), how can it be accurate in any real, objective way?

I have my own taste buds in my mouth, not Robert Parker's, not Jancis Robinson's, not yours -- and with the not-so-obvious exceptions for things like volatile acidity, 2,4,6-tricholanisole, and other flaws in wine -- opinions differ. Not everyone agrees that Château Cache Phloe 2009 is the best Cabernet ever made, or that the 2009 Jean Deaux Vineyards Shiraz is outstanding! Reviews of wines, movies, restaurants, and the like are fundamentally subjective, not objective, and intelligent minds can disagree in matters of taste.

In reviewing the performance of a prosumer HX or DB machine, one can objectively report upon the size of the boiler(s), the wattage, the recovery time, etc., etc., etc. . . . one can log "performance" issues, as one can with a car (0-60 mph in x seconds, average MPG, etc.). But ultimately, "styling" comes into it, and people chose the Ford Mustang rather than the Ford Edsel . . .

I honestly don't know, Luca, how different machines affect different coffees. I don't have access to five "prosumer" machines -- all set up side-by-side, all PID'd -- with x number of pounds of coffee roasted 48 hours ago, to pull shots and understand how, at _____℃, this machine really brings out the ______ qualities of the beans, but when you adjust the temperature to _______℃, it is actually that machine that really shines with these beans . . . nor do I know how switching to a different blend of beans will affect my opinions of the five machines in terms of which I think is best.

I know how to get espresso from my machine that pleases me and my palate. (Fortunately, it also pleases my wife; makes things much easier!) But I have no idea if you would like it -- or, at least, like it as much as I.

The interesting thing about great coffee (rather than great machines) is that, while we can (probably) all agree that this is a "sink shot," not all will agree that this is a "god shot"!

Cheers,
Jason