This is all very interesting, and I don't know if I have much to add, or if I'm qualified to do so. My equipment is a very bottom-end spring lever I bought at a low price, and an antique manual lever I was given (plus a few $10 hand grinders). One thing at least, I don't need to justify my expenditures (although i am in love with the machines and the grinders!).

Human subjectivity is a difficult mountain to climb - some will say they pull lots of "God" shots, and others, equally-skilled, will say they seldom do.

I suppose the efforts of those here on HB and other forums to review machines are as good as human nature (and a volunteer system of time and effort) will allow. And if you start paying the reviewers, well that's another story. I'll echo others to say that a "newbie" who reads these forums should be able to make as informed decision about a "home appliance" as anyone can reasonably expect.

Not to say that Marshall's question is valid, and that we shouldn't continue to improve the state of things.

PC

Ike, I don't see much of a parallel between buying wine and buying an espresso machine. I can sample lots of wines by the bottle or even by the glass before deciding which ones to buy by the case. When that case arrives, I can be reasonably certain it will taste like the bottle I sampled. I can also be reasonably certain that the wine will not develop chronic mechanical failures a month after I take delivery.

As for my own decisions, I have access to a network of dealers, technicians, top-tier baristas and others that is not available to average consumers. Some of their advice begins, "Don't repeat this, but ...." I also have better hands-on access. I'm just looking for a way to get more of the same information into the hands of consumers, so they can make better-informed decisions.

And yes, I do believe a well-informed review by a journalist with integrity serves a useful purpose. One journalist friend read my original post and wrote to complain that I even referred to on-line reviews as "journalism."

luca wrote:Regarding clear winners at price points; surely it must at least be possible to create a short list, leaving a few machines that make espresso with different taste characteristics. For example, there is one prosumer machine on the Australian market that I wouldn't wish on my worst enemy at any price point - it has a ridiculously low shower screen, so you can't fit much coffee into the portafilter, it isn't put together so as to allow for easy servicing, it has a dual-walled steam wand that simply doesn't work - in fact it retains heat and, using fresh coffee and a large commercial grinder of some description, I was totally unable to get a shot that didn't start gushing alarmingly quickly as it finished. OK, so *maybe* that machine can be tweaked to make it better. Fine. That's the vendor's job, not the consumer's.

Unquestionably, although clear winners may not be easy to pick out, it's often very easy to pick out clear losers from a group. Technical inferiorities such as the ones that you point out also stand out to most competent reviewers--Such shortcomings would be immediately pointed out by any Home-Barista review. But what about this category that Marshall is referring to? The mystical "second machine" land of $1000-2000 (give or take), mostly HX, mostly E61 machines (with a number of exceptions, including the low-end double boiler machines)? E61 machines come in a seemingly infinite number of flavors. Price-wise, they seem more or less bracketed by the Expobar Lever ($1000 last I checked) and the Vibiemme Domobar Super (getting close to $2000 by now if it hasn't passed that). What is to separate these machines, tastewise? Well, surprisingly much. The Expobar has cheaper components, and it runs a lot hotter (from my reading), making its flushing routine much more irritating, and I'm sure that all of that causes less consistency for the machine. Granted. But I would say that the Expobar's shortcomings are well-known, and that it doesn't take a surreal amount of trouble to know that it's a machine that can make fine espresso, but with a lot more blood/sweat/tears than would be required from a VBM, or, for that matter, a double boiler machine. I believe that the reviews as they currently exist tend to find out these technical flaws and document very well the trouble you are likely to have making espresso with the machines in question.

I really think that quality control issues are, generally, well documented. Not for all machines, mind you. There are a lot of machines that are much less reviewed than others, and buyers take a more significant risk getting machines that don't have many or any professional reviews. Can you blame professionals, though? Heck, who wants to individually review ~200 E61 machines? What is that going to be but an incredible amount of work with the primary conclusion that 95% of them are just 'more of the same' with few if any distinguishing features for a few of them. The good, professional reviewers out here don't have the money for that. On a limited budget, professional reviewers look for machines that they think will be good, that already have somewhat of a following/interest, because they want their review to be able to steer users towards good machines, rather than away from crappy ones. But again, back to the real point: Espresso Review Journalism is not broken; we don't need to fix it; that said, it could probably be improved--with better funding to professional reviewers.

Finally, as a bit of a segue into the mixed bag that looking at results in the cup:

Marshall wrote:By the way, a stock Silvia in skilled hands makes pretty good espresso. The endless list of Silvia modifications and special techniques are the result of 1,000 obsessives (like us) focusing on it like a laser beam and putting our minds to ways to improve it. I still recommend it (with a PID) to beginners.

For a differing, professional opinion of the cup results, look at Jim Schulman's comment in Do you suffer from upgraditis?

another_jim wrote:More controversially, a recent stint with a PIDed pair of Silvias as a two group test machine left me much colder than I expected. The Silvia gives shots with the body of a commercial espresso machine, but with very poor taste clarity, worse than many other home machines. I would not recommend it as a starter machine for people primarily interested in coffee taste. I'd spend the extra few hundred and go straight to an Expobar, Bezzera, or Oscar as a starter machine.

Marshall wrote:One journalist friend read my original post and wrote to complain that I even referred to on-line reviews as "journalism."

I'm surprised more on-line reviewers weren't offended.

I have four points to make on this:

1. The espresso review community is us. There are some reviews in newspapers, but I think most people looking for machine reviews will end up at one or more of the forums.
2. We have improved tremendously over the last ten years. Back then, the top end was a Gaggia Classic and MDF, and espresso shots at cafes outside Seattle were uniformly undrinkable. In order to taste the range of good espresso, one had to make a pilgrimage to Seattle. Now many of us have commercial equipment, and access to some the best shot pullers on the planet. This has obviously raised the bar on our understanding of how good espresso can be.
3. One thing hasn't changed in ten years. Whenever a well known member of the community upgrades to more expensive equipment and asserts that this has improved his or her espresso experience, it sets off a storm of anger. "My machine is as good as yours," "I bought such and such based on your recommendation, now you tell me it isn't that great." Yes, this is disconcerting; but it's also humorously revealing. The complaints are not about the taste as such, instead, the complainers want to serve three star shots, not just solid home cooked ones, and are suddenly being confronted with the loss of a star.
4. But when it comes to 3 star espresso, the machine exists in the context of the person's skills, the coffees used, and the grinder. If I were using a medium dark roasted Brasil/Harar blend with a Rocky or MDF grinder, there wouldn't be much difference between a Gaggia Classic or an NS Aurelia (my current "as good as it gets" reference machine). If I'm using an Ecafe DP Yrg with a Robur, there is going to be a huge difference. In practical terms, I guarantee Marshall would have seen far less improvement if he were still using the Mini rather than the Max, or was pulling Monkey blend roasted to a rolling second in an off the shelf home roaster.

Our reviews are based on the pursuit of godshots and 3 star espresso. But the overall commitment and context this requires has forced me to reconsider what I would tell a non-hobbyist. My advice to people who don't want to be hardcore, but who do have some taste and some money to spend, is to get a good quality super auto and then use the very best coffees available. While they still don't achieve close to the mouth feel or flavor density of real espresso, they are beginning to do justice to the nuances of good coffees.

Unfortunately, none of the vendors has as yet sponsored a superauto shootout. I would have no problem doing such a test, since I think it would be the most honest and relevant piece of espresso journalism I could do for most of the buying public.

Marshall wrote:Ike, I don't see much of a parallel between buying wine and buying an espresso machine. I can sample lots of wines by the bottle or even by the glass before deciding which ones to buy by the case. When that case arrives, I can be reasonably certain it will taste like the bottle I sampled. I can also be reasonably certain that the wine will not develop chronic mechanical failures a month after I take delivery

IMO, the parallel is just that : A few years back I picked up a couple of cases of an excellent Barolo. The corks were crap and 1 in 3 went down the sink. Ditto '89 BV Latour Reserve. Great promise at release, but 50-50 after a few years. If wine were as reliable as my Vibiemme, I'd be happy as the proverbial pig.

another_jim wrote:One thing hasn't changed in ten years. Whenever a well known member of the community upgrades to more expensive equipment and asserts that this has improved his or her espresso experience, it sets off a storm of anger.

How true. When Mark Prince dumped his Silvia for a Livia, I felt so, so ... used!

another_jim wrote:One thing hasn't changed in ten years. Whenever a well known member of the community upgrades to more expensive equipment and asserts that this has improved his or her espresso experience, it sets off a storm of anger.

That's a very, very sad commentary.

Marshall wrote:1. Even expert consumers typically have meaningful experience with a very limited number of machines. By "meaningful" I mean long-term daily use.

I'd suggest that there is only a very, very small group that will develop a 'meaningful' experience as you describe it, with anything more than a handful of machine in their lifetime, and very few reviewers of any electromechanical kit develop a 'meaningful' relationship with the kit being reviewed/

2. Most amateur reviewers are in love with their own machines or need to justify the money they spent on them. At best they give their pride and joys the benefit of the doubt and at worst look at them through rose-colored glasses.

Uhm... yup...

3. Too many reviewers are strong on mechanics and engineering and weak on the palate, often because they have limited access to top-tier espresso bars. This leads to tech-heavy reviews with suspect conclusions (or no conclusions) about the cup..

Ehm, with the variations in beans, technique, water, humidity, elevation, roasts, the list goes on, it would be rather less than useful to have a reviewer tell me how this beans blended with that bean on that day post roast at this elevation with this grind and that humidity tasted with this much flush and that much pre-infusion with the variation in temperature because he's at 118V and you've got 112 at your house....

The technical aspects, features and performance of the machine are tangible. They won't vary much from place to place, from user to user. The taste of the cup is dependent on far too many other variables to suggest that the reviewers results are going to be your results.

5. Finally, too many of the most qualified reviewers, who also have the best access to multiple machines for long-term comparative reviews, are hamstrung by editorial policies that avoid offending advertisers or by personal ties that prevent offending friends and clients.

That's simply not journalism. That's advertising. Whole different (and far less honest) enterprise.

Psyd wrote:That's simply not journalism. That's advertising. Whole different (and far less honest) enterprise.

There's a difference

A trio of Mark Twain quotes on journalism :
"So I became a newspaperman. I hated to do it, but I couldn't find honest employment."

"I am personally acquainted with hundreds of journalists, and the opinion of the majority of them would not be worth tuppence in private, but when they speak in print it is the newspaper that is talking (the pygmy scribe is not visible) and then their utterances shake the community like the thunders of prophecy."

"If you don't read the newspaper you are uninformed. If you do read the newspaper you are misinformed."