So basically we have a dilemma. One the one hand, failure to do comparisons blind, using identical shot parameters, means that we run into a real danger of experimenter bias. Just trying to be evenhanded won't necessarily eliminate that bias, because many of the mechanisms involved operate at an unconscious level. For some interesting discussions on this point, see the "eco-friendly coffee thread" here: Want your coffee to taste better ? Call it Eco-friendly.

On the other, I think you guys are right that just dialing in a new grinder to be compared to identical parameters as one's current favorite grinder biases the test in favor of the "home team" grinder. Ideally, one wants to see how well each grinder performs when used to best effect, not when arbitarily trying to mimic another grinder's strengths.

So here is one solution: Have someone who loves each grinder dial in a particular coffee, in terms of grind, dose, and temperature, so that it tastes best on a particular machine. Then run two sets of blinded comparisons with that coffee. First, dial in the second grinder to match the flow rate of the first at the settings that work best for the first grinder. Then switch places, dialing in the second so that it can be compared blindly with the second grinder in the second grinder's comfort zone. If one grinder dominates on blind tasting in both comparisons, it is a clear winner. If each grinder edges the other out in its ideal zone, then we might say that one wins the comparison if it prevails by a larger margin in its comfort zone, but that is a less clear victory.

And of course, some grinders may be similar enough that the ideal shot parameters are very similar for both, making this sort of procedure pointless. I imagine that would be the case, for instance, with many of the titan conicals compared in the TGP thread, or if I were to do a head-to-head between my HG-One and my Pharos.

Aaron wrote:I would love to do more comparison testing with two dialed in grinders and an independent taster. I just need to find a local taster who wants to drink lots of excellent espresso. Any takers?

Maybe Scott (sweaner)? I wish I live closer for said test.

I would be game for that!

We will have to make it happen. Should be fun!

[creative nickname] wrote:So basically we have a dilemma. One the one hand, failure to do comparisons blind, using identical shot parameters, means that we run into a real danger of experimenter bias. Just trying to be evenhanded won't necessarily eliminate that bias, because many of the mechanisms involved operate at an unconscious level.

It's probably possible to design a valid, blind, "dialed in for best shot" test. But putting "blind" and "best shot" together involves significant technical hurdles. And, really, there's so much statistical noise involved in tests with so few panelists and so few samples, that even if you could blind without compromising absolute shot quality as well as shot quality equality from sample to sample, you're not going to come up with the sort of meaningful results which would allow you to offer an error bar and confidence level.

It's just an opinion, but I think you're better off leaving blinding alone, at least until you've got the parameters for "best shot" comparisons nailed.

"Best shot" aside, there are established rules for valid AB, ABX and "triangle" comparison testing; and we're not anywhere close. Without a lot of design (not to mention a two group machine), and a panel larger than two or three, the best we can do is "informed opinions from people with good taste." That's good enough for me. Even if you don't trust your ability to recognize and correct for bias, I do.

Rich