This is all very interesting, and I don't know if I have much to add, or if I'm qualified to do so. My equipment is a very bottom-end spring lever I bought at a low price, and an antique manual lever I was given (plus a few $10 hand grinders). One thing at least, I don't need to justify my expenditures (although i am in love with the machines and the grinders!).

Human subjectivity is a difficult mountain to climb - some will say they pull lots of "God" shots, and others, equally-skilled, will say they seldom do.

I suppose the efforts of those here on HB and other forums to review machines are as good as human nature (and a volunteer system of time and effort) will allow. And if you start paying the reviewers, well that's another story. I'll echo others to say that a "newbie" who reads these forums should be able to make as informed decision about a "home appliance" as anyone can reasonably expect.

Not to say that Marshall's question is valid, and that we shouldn't continue to improve the state of things.

PC

Ike, I don't see much of a parallel between buying wine and buying an espresso machine. I can sample lots of wines by the bottle or even by the glass before deciding which ones to buy by the case. When that case arrives, I can be reasonably certain it will taste like the bottle I sampled. I can also be reasonably certain that the wine will not develop chronic mechanical failures a month after I take delivery.

As for my own decisions, I have access to a network of dealers, technicians, top-tier baristas and others that is not available to average consumers. Some of their advice begins, "Don't repeat this, but ...." I also have better hands-on access. I'm just looking for a way to get more of the same information into the hands of consumers, so they can make better-informed decisions.

And yes, I do believe a well-informed review by a journalist with integrity serves a useful purpose. One journalist friend read my original post and wrote to complain that I even referred to on-line reviews as "journalism."

luca wrote:Regarding clear winners at price points; surely it must at least be possible to create a short list, leaving a few machines that make espresso with different taste characteristics. For example, there is one prosumer machine on the Australian market that I wouldn't wish on my worst enemy at any price point - it has a ridiculously low shower screen, so you can't fit much coffee into the portafilter, it isn't put together so as to allow for easy servicing, it has a dual-walled steam wand that simply doesn't work - in fact it retains heat and, using fresh coffee and a large commercial grinder of some description, I was totally unable to get a shot that didn't start gushing alarmingly quickly as it finished. OK, so *maybe* that machine can be tweaked to make it better. Fine. That's the vendor's job, not the consumer's.

Unquestionably, although clear winners may not be easy to pick out, it's often very easy to pick out clear losers from a group. Technical inferiorities such as the ones that you point out also stand out to most competent reviewers--Such shortcomings would be immediately pointed out by any Home-Barista review. But what about this category that Marshall is referring to? The mystical "second machine" land of $1000-2000 (give or take), mostly HX, mostly E61 machines (with a number of exceptions, including the low-end double boiler machines)? E61 machines come in a seemingly infinite number of flavors. Price-wise, they seem more or less bracketed by the Expobar Lever ($1000 last I checked) and the Vibiemme Domobar Super (getting close to $2000 by now if it hasn't passed that). What is to separate these machines, tastewise? Well, surprisingly much. The Expobar has cheaper components, and it runs a lot hotter (from my reading), making its flushing routine much more irritating, and I'm sure that all of that causes less consistency for the machine. Granted. But I would say that the Expobar's shortcomings are well-known, and that it doesn't take a surreal amount of trouble to know that it's a machine that can make fine espresso, but with a lot more blood/sweat/tears than would be required from a VBM, or, for that matter, a double boiler machine. I believe that the reviews as they currently exist tend to find out these technical flaws and document very well the trouble you are likely to have making espresso with the machines in question.

I really think that quality control issues are, generally, well documented. Not for all machines, mind you. There are a lot of machines that are much less reviewed than others, and buyers take a more significant risk getting machines that don't have many or any professional reviews. Can you blame professionals, though? Heck, who wants to individually review ~200 E61 machines? What is that going to be but an incredible amount of work with the primary conclusion that 95% of them are just 'more of the same' with few if any distinguishing features for a few of them. The good, professional reviewers out here don't have the money for that. On a limited budget, professional reviewers look for machines that they think will be good, that already have somewhat of a following/interest, because they want their review to be able to steer users towards good machines, rather than away from crappy ones. But again, back to the real point: Espresso Review Journalism is not broken; we don't need to fix it; that said, it could probably be improved--with better funding to professional reviewers.

Finally, as a bit of a segue into the mixed bag that looking at results in the cup:

Marshall wrote:By the way, a stock Silvia in skilled hands makes pretty good espresso. The endless list of Silvia modifications and special techniques are the result of 1,000 obsessives (like us) focusing on it like a laser beam and putting our minds to ways to improve it. I still recommend it (with a PID) to beginners.

For a differing, professional opinion of the cup results, look at Jim Schulman's comment in Do you suffer from upgraditis?

another_jim wrote:More controversially, a recent stint with a PIDed pair of Silvias as a two group test machine left me much colder than I expected. The Silvia gives shots with the body of a commercial espresso machine, but with very poor taste clarity, worse than many other home machines. I would not recommend it as a starter machine for people primarily interested in coffee taste. I'd spend the extra few hundred and go straight to an Expobar, Bezzera, or Oscar as a starter machine.

Marshall wrote:One journalist friend read my original post and wrote to complain that I even referred to on-line reviews as "journalism."

I'm surprised more on-line reviewers weren't offended.

I have four points to make on this:

1. The espresso review community is us. There are some reviews in newspapers, but I think most people looking for machine reviews will end up at one or more of the forums.
2. We have improved tremendously over the last ten years. Back then, the top end was a Gaggia Classic and MDF, and espresso shots at cafes outside Seattle were uniformly undrinkable. In order to taste the range of good espresso, one had to make a pilgrimage to Seattle. Now many of us have commercial equipment, and access to some the best shot pullers on the planet. This has obviously raised the bar on our understanding of how good espresso can be.
3. One thing hasn't changed in ten years. Whenever a well known member of the community upgrades to more expensive equipment and asserts that this has improved his or her espresso experience, it sets off a storm of anger. "My machine is as good as yours," "I bought such and such based on your recommendation, now you tell me it isn't that great." Yes, this is disconcerting; but it's also humorously revealing. The complaints are not about the taste as such, instead, the complainers want to serve three star shots, not just solid home cooked ones, and are suddenly being confronted with the loss of a star.
4. But when it comes to 3 star espresso, the machine exists in the context of the person's skills, the coffees used, and the grinder. If I were using a medium dark roasted Brasil/Harar blend with a Rocky or MDF grinder, there wouldn't be much difference between a Gaggia Classic or an NS Aurelia (my current "as good as it gets" reference machine). If I'm using an Ecafe DP Yrg with a Robur, there is going to be a huge difference. In practical terms, I guarantee Marshall would have seen far less improvement if he were still using the Mini rather than the Max, or was pulling Monkey blend roasted to a rolling second in an off the shelf home roaster.

Our reviews are based on the pursuit of godshots and 3 star espresso. But the overall commitment and context this requires has forced me to reconsider what I would tell a non-hobbyist. My advice to people who don't want to be hardcore, but who do have some taste and some money to spend, is to get a good quality super auto and then use the very best coffees available. While they still don't achieve close to the mouth feel or flavor density of real espresso, they are beginning to do justice to the nuances of good coffees.

Unfortunately, none of the vendors has as yet sponsored a superauto shootout. I would have no problem doing such a test, since I think it would be the most honest and relevant piece of espresso journalism I could do for most of the buying public.

Marshall wrote:Ike, I don't see much of a parallel between buying wine and buying an espresso machine. I can sample lots of wines by the bottle or even by the glass before deciding which ones to buy by the case. When that case arrives, I can be reasonably certain it will taste like the bottle I sampled. I can also be reasonably certain that the wine will not develop chronic mechanical failures a month after I take delivery

IMO, the parallel is just that : A few years back I picked up a couple of cases of an excellent Barolo. The corks were crap and 1 in 3 went down the sink. Ditto '89 BV Latour Reserve. Great promise at release, but 50-50 after a few years. If wine were as reliable as my Vibiemme, I'd be happy as the proverbial pig.

another_jim wrote:One thing hasn't changed in ten years. Whenever a well known member of the community upgrades to more expensive equipment and asserts that this has improved his or her espresso experience, it sets off a storm of anger.

How true. When Mark Prince dumped his Silvia for a Livia, I felt so, so ... used!

another_jim wrote:One thing hasn't changed in ten years. Whenever a well known member of the community upgrades to more expensive equipment and asserts that this has improved his or her espresso experience, it sets off a storm of anger.

That's a very, very sad commentary.

Marshall wrote:1. Even expert consumers typically have meaningful experience with a very limited number of machines. By "meaningful" I mean long-term daily use.

I'd suggest that there is only a very, very small group that will develop a 'meaningful' experience as you describe it, with anything more than a handful of machine in their lifetime, and very few reviewers of any electromechanical kit develop a 'meaningful' relationship with the kit being reviewed/

2. Most amateur reviewers are in love with their own machines or need to justify the money they spent on them. At best they give their pride and joys the benefit of the doubt and at worst look at them through rose-colored glasses.

Uhm... yup...

3. Too many reviewers are strong on mechanics and engineering and weak on the palate, often because they have limited access to top-tier espresso bars. This leads to tech-heavy reviews with suspect conclusions (or no conclusions) about the cup..

Ehm, with the variations in beans, technique, water, humidity, elevation, roasts, the list goes on, it would be rather less than useful to have a reviewer tell me how this beans blended with that bean on that day post roast at this elevation with this grind and that humidity tasted with this much flush and that much pre-infusion with the variation in temperature because he's at 118V and you've got 112 at your house....

The technical aspects, features and performance of the machine are tangible. They won't vary much from place to place, from user to user. The taste of the cup is dependent on far too many other variables to suggest that the reviewers results are going to be your results.

5. Finally, too many of the most qualified reviewers, who also have the best access to multiple machines for long-term comparative reviews, are hamstrung by editorial policies that avoid offending advertisers or by personal ties that prevent offending friends and clients.

That's simply not journalism. That's advertising. Whole different (and far less honest) enterprise.

Psyd wrote:That's simply not journalism. That's advertising. Whole different (and far less honest) enterprise.

There's a difference

A trio of Mark Twain quotes on journalism :
"So I became a newspaperman. I hated to do it, but I couldn't find honest employment."

"I am personally acquainted with hundreds of journalists, and the opinion of the majority of them would not be worth tuppence in private, but when they speak in print it is the newspaper that is talking (the pygmy scribe is not visible) and then their utterances shake the community like the thunders of prophecy."

"If you don't read the newspaper you are uninformed. If you do read the newspaper you are misinformed."

There's a difference?

Slight. The journalist has the sheen and the veneer of honesty and a lack of bias.
The ad man has no such compunction.

"Advertisements contain the only truths to be relied on in a newspaper." -- Mark Twain

Marshall wrote: 5. Finally, too many of the most qualified reviewers, who also have the best access to multiple machines for long-term comparative reviews, are hamstrung by editorial policies that avoid offending advertisers or by personal ties that prevent offending friends and clients.

Psyd wrote:That's simply not journalism. That's advertising. Whole different (and far less honest) enterprise.

The rule is that we don't speak ill of other coffee people. If a coffee or piece of gear isn't good, we don't review it. This is neither advertising, which is praising things regardless of their quality; nor is it journalism, which is reporting other people's reactions, regardless of their or the reporter's expertise.

Mostly nowadays, I find myself not speaking ill of the level of people's ethical judgments.

More random thoughts:

Marshall wrote:Ike, I don't see much of a parallel between buying wine and buying an espresso machine. I can sample lots of wines by the bottle or even by the glass before deciding which ones to buy by the case. When that case arrives, I can be reasonably certain it will taste like the bottle I sampled. I can also be reasonably certain that the wine will not develop chronic mechanical failures a month after I take delivery.

As I was the one who first brought up a wine analogy, let me simply say the analogy I made was to reviewing espresso/coffee, and not the machine itself. (Marshall, I would refer to #3 in your original post.)

* * * * *

luca wrote:Regarding the differences between prosumer machines; 'slight' is relative. I agree that most people would probably struggle to pick differences between many of the machines on the market, but I think that many people who will spend the money on a prosumer machine are very interested in espresso and will continue to improve their palates and barista skills to the point where the 'slight' difference is significant to them. Unfortunately, working backwards, I think that this means that most people probably don't have the experience to really select the best machine for them when they buy a prosumer machine, because much of that experience will be gained through buying the machine. All speculation, but I hope that it at least makes sense. If I'm right about that, I can't really see any solution beyond either working in an espresso bar for half a year before buying a machine or being open to the possibility of revisiting the market a few years down the track and seeing if it is worthwhile upgrading your 'prosumer' machine.

We agree completely.

luca wrote:Regarding subjectivity of reviews; yes, it's a very good point, but I think that that applies to anything. The utility of Parker's reviews isn't diminished by their subjectivity, as long as he is consistent in his scoring.

Again, we agree completely! I have found that my palate and Parker's agree on relatively few things overall, and yet, I find his reviews can be of great use to me precisely because Parker is consistent! Other writers are of use to me only in so far as a) they are consistent, and b) I can "calibrate" my tastes in wine to theirs.

luca wrote:I guess that the difference between reviewing wine (or coffee) and reviewing espresso machines is that the former is something that the consumer will purchase enough of to be able to start to understand what the critic is on about and, so, to learn how to get the most out of the reviews, whereas the consumer is unlikely to buy many of the latter.

Again, we agree. Keeping in mind, as I pointed out above, I was limited my "wine analogy" to coffee reviews, rather than machine reviews -- or at least that was my intention -- this is a flawed analogy at best. One doesn't review the grapes in a vineyard, one reviews the finished wine. But Ken Davids (and others commenting on the blends of and by various roasters, but I will use Ken as an example) does not comment upon the final "wine." Or rather, they do, but the "wine" is made a shot (or a double) at a time, but thousands of different individual "winemakers."

If I have raved about (e.g.) David Schomer's "Dolce" blend from Espresso Vivace, it is because I enjoy the espresso that I can make at home from it (or I enjoyed the espresso I tried at Espresso Vivace itself). Reading about Ken Davids giving the ______________ beans (single origin or blend) roasted by ________________ a score of __________ is not only no guarantee that I will agree, it's no guarantee that I'll produce anything like the espresso he was consuming . . . different person operating the machine, different machine, different grind, different temperature . . . different drink in the cup! As a result, I might hate the very same beans Ken's site gave a (e.g.) "92." This is true even if it turned out that, if Ken had pulled the shot for me, it might be the greatest "god shot" I ever had! But with a different "barista" . . . yuck!

In other words, reviewing beans is at the very least one step "removed" when compared to reviewing wine.

luca wrote:Regarding objective factors; fantastic ... to the extent that they are useful.

Again, yes -- absolutely right. But at least there is something "concrete" to compare, even if people do think "bigger = better" when that it not automatically so . . .

Cheers,
Jason

It's interesting to me that this discussion has been so focused on the problems with reviews of machinery. While I can certainly be as technology focused as the next guy or more so (obviously), and people certainly come looking at the reviews when it comes time to buy the next toy, it seems to me like the average consumer would benefit more from more focus on coffee, as opposed to coffee equipment, journalism.

Perhaps it's just me, but most people I know drink coffee that makes me sad for them, and trying to inform people about good machines strikes me as very much secondary to trying to help them to find shops they can try and beans that can come out great either from a Silvia or a Synesso. Posts from the regulars here about beans that are currently being offered (as opposed to reviews of blend X from 3 seasons ago, or an SO from 3 months ago long since sold out) have been among, if not the, most useful "coffee journalism" I've seen... perhaps an irrelevant aside as to improving the espresso equipment review publication process, but there you go. Then again, consider what might generate more traffic (and thus ad revenue, interest from manufacturers in providing machines, etc.), equipment reviews that people obsess over and then never return to again after buying a machine, or coffee updates that people could read regularly for new things to try along with a smattering of regular equipment reviews...

JimWright wrote:It's interesting to me that this discussion has been so focused on the problems with reviews of machinery. While I can certainly be as technology focused as the next guy or more so (obviously), and people certainly come looking at the reviews when it comes time to buy the next toy, it seems to me like the average consumer would benefit more from more focus on coffee, as opposed to coffee equipment, journalism.

Perhaps it's just me, but most people I know drink coffee that makes me sad for them, and trying to inform people about good machines strikes me as very much secondary to trying to help them to find shops they can try and beans that can come out great either from a a Silvia or a Synesso. Posts from the regulars here about beans that are currently being offered (as opposed to reviews of blend X from 3 seasons ago, or an SO from 3 months ago long since sold out) have been among, if not the, most useful "coffee journalism" I've seen...

And then you run into all of the subjective problems I have attempted to detail . . .

While not all together useless, reviewing coffee itself can be an exercise in frustration . . . what is more helpful is/can be reviews of the coffee roasters that do a great job: [i]______________ Coffee Co. is known as a specialist in SO beans; ___________ Roasters has a way with Indo-Pacific beans[/i, etc., etc. -- in other words, names of top-quality roasters one should try. But to say that "this" coffee is great raises all the problems Luca and I were discussing . . . .

I agree that reviewing coffee is perhaps even harder than reviewing wine in that varying tastes, and additively, varying equipment and technique, along with batch variation, bean age, etc. cause potentially large differences in perception of quality (or, to put it more subjectively, and/or perhaps address a slightly different but more important concept, "goodness").

Still, for my own part, I have found the indisputably subjective and varying taste impressions of coffee aficionados to be very valuable, and wonder if this wouldn't be of more use to the average coffee drinker (or even average enthusiast, that is to say, someone willing to even try making espresso at home) than more attention to the equipment reviews. (A GS/3 or Technivorm can't make a great cup from crappy beans - the reverse is not always true...)

cafeIKE wrote:IMO, the parallel is just that : A few years back I picked up a couple of cases of an excellent Barolo. The corks were crap and 1 in 3 went down the sink. Ditto '89 BV Latour Reserve. Great promise at release, but 50-50 after a few years. If wine were as reliable as my Vibiemme, I'd be happy as the proverbial pig.

If you have to reach "a few years back" for an example, you are talking about an outlier, not the norm. I will stand by my statement that people investing in high end home espresso equipment (or low or mid-level, for that matter) are in need of informed, impartial guidance, notwithstanding your Barolo tragedy.

Consider the following scenario: "I wonder if that espresso machine is really worth my shelling out $2,500?" "Really? Espresso Machine Quarterly has it in a three-machine comparison by the U.S. Barista Champion?" "Nah, I don't care what he says." "Why not?" "Well, I heard Ian's Latour Reserve sucked after a couple of years."

Marshall wrote:Consider the following scenario: "I wonder if that espresso machine is really worth my shelling out $2,500?" "Really? Espresso Machine Quarterly has it in a three-machine comparison by the U.S. Barista Champion?" "Nah, I don't care what he says." "Why not?"...

"Because the test review was run on doubles at an interval of a few minutes at a temperature several degrees hotter than what I like. I make singles every coupla hours. I can't stand the roast style the guy prefers : In Seattle, Salt Lake or Shanghai they always taste like coffee flavored BBQ sauce."

Perhaps I'm overly jaded on 'reviews' having endured the 'Transpistor' and 'Digital' revolutions. Too many reviews are written on how much better the NewWhizBang is because it has more power and less full power THD, ignoring Group Delay, Transient Intermodulation, Crossover Notch, Thermal Modulation, Quantization, Jitter and a host of other CLEARLY AUDIBLE problems.

"Reviewers" are often selling themselves and far too often have insufficient technical credentials.

There's no free lunch. Do the research. CAVEAT EMPTOR.

another_jim wrote:I have four points to make on this . . .

(Edited for the sake of bandwidth; they are right above for people to refer to.)

Jim, I agree completely with your four main points. To continue with the wine analogy just briefly, the average quality of "table" wine produced in the US today is far higher than it was in the 1960s, when I first got into the trade; heck, it's much higher than in the 1980s!

another_jim wrote:Our reviews are based on the pursuit of godshots and 3 star espresso. But the overall commitment and context this requires has forced me to reconsider what I would tell a non-hobbyist. My advice to people who don't want to be hardcore, but who do have some taste and some money to spend, is to get a good quality super auto and then use the very best coffees available. While they still don't achieve close to the mouth feel or flavor density of real espresso, they are beginning to do justice to the nuances of good coffees.

Unfortunately, none of the vendors has as yet sponsored a superauto shootout. I would have no problem doing such a test, since I think it would be the most honest and relevant piece of espresso journalism I could do for most of the buying public.

Well, this is the first time I can recall anyone "recommending" a superauto. While I agree that a "superauto shootout," would make interesting reading, I am not sure the results would make a sizeable portion of the participants here run out and by the "Gaggseccapresso 9000 . . . B"

However, I do know people who own superautos, and are quite happy with them! (I take it as a compliment to my own very meager-and-in-much-need-of-improvement skills that they have voiced a preference for my own espresso, however none have rushed out to dump their superautos in favor of an HX or DB model . . . . )

That said, there certainly is a market for superauto machines -- if not, Gaggia, Saeco, Jura-Capresso and others wouldn't be making them for the home-market. (Airports, automobile showrooms, roadside gas stations and Starbucks are a different market entirely.) And a number of new-to-espresso consumers and as you say, those not wanting to become "hard-core," are asking about superautos on this and other sites.

At least superautos take one "M" out of the equation. Perhaps superautos should be looked at (again) as a way of tasting coffees for review . . . not "cupping," per se, as you'd want to use your own setup for use in your café or when roasting, but rather as a method of attempting to "standardize" the techniques of preparation for someone to taste various coffees for consumer reviews à la Ken David.

* * * * *

One more problem with reviewing coffees that I have yet to bring up: the date of the review.

In wine, just because the 2009 Cabernet Sauvignon from Château Cache Phloe might be stunning doesn't mean the 2010 or 2011 vintages will be. In coffee, it's the same thing. Being an agricultural crop, sometimes Mother Nature gives up better beans than in other years. A roaster may be able to compensate for this in a blend, just as producers of non-vintage Champagnes can, but those who offer single types of beans (e.g.: Columbia) will have a more difficult time, and just as with single vineyard wines, single origin coffees from a single farm can vary significantly from crop-to-crop.

But there are far too many roasters and beans to review them all each and every single year. Thus, some reviews will be out-of-date, and thereby give the consumer false or misleading information about currently available coffees. Institutional hazard, and one I know NO WAY of avoiding.

Cheers,
Jason