Perhaps we need more free-form, taste only competitions. Any coffee, roast, grinder and machine/modifications you want and tasters can compare side-by-side.

Testing without blinding allows results deformed by all sorts of bias. However, blind testing creates an enormous amount of "noise," which must be eliminated with additional methodology (such as control groups, AA testing, etc.), large panels, lots of samples, etc., to have any meaning. And by "meaning," I'm not talking about "statistical validity," I just mean good enough to draw a decent inference.

When you're talking about a few people testing a few grinders there's no "right" way to do it.

Furthermore... If the grinders are very different in the cup, you don't need to know much more than which one you like better -- for whatever reason, good or bad. Because... it's your taste and your money. While if the grinders are sufficiently similar in the cup to require blinding to get a reliable panel preference, then you don't need to know much more than which one you like better for other reasons.

That's not just true about grinders; and it's part of the reason that a critical element of blind comparison testing is establishing a consistently detectable difference between the subjects of comparison at all. I don't like blinding because I've a lot of education in statistics and some experience in sampling -- and despite recognizing that there's some utility of blinding in grinder testing, I can't overcome my prejudices about doing things "wrong." Bias on my part? Ironically, yes.

I do like the "best shot," method, even though bias can be a component of many conclusions, because it's compact, cheap, convenient, can be done by a panel of one, and doesn't promise more than it can deliver. Also, it allows the person doing the testing to adjust each grinder for whatever qualities are desired, beyond flow rate/dose. In other words, it's more like using a grinder as it would be used if you owned it.

That's not an argument for one over the other, both methods have enormous weaknesses; while neither -- at least not the way we're doing them -- exhibits many strengths. But one soldiers on.

The best piece of grinder testing I've ever read was Another Jim's "Beat the Robur" thread which -- yes -- included blinding as part of the methodology. It wasn't that, it was the detailed descriptions of what he was tasting. The man's got a palate and is very honest. Trust yourself.

Rich

peacecup wrote:There was a discussion about this a while ago in the knockbox forum - with some title I can't recall. Humans are very susceptible to perception bias, or something like that. If we buy something we love it's easy to believe it makes better espresso, for example.

Without some rigorous blind testing you can convince yourself, but I'm not 100% sure your results are truly objective. The human mind is very capable of influencing physical reactions. I like to cite Sir Peter Medewar's example of blushing, wherein an emotional encounter has the very obvious physical reaction of changing the color of one's skin.

If split second of personal encounter can cause one's face to turn red, imagine what weeks or months of reading and anticipation, not to mention a substantial financial outlay, can do to ones' tastebuds...

PC

I'm reminded of a taste test / study focusing on wine and the effects of perception. Inexpensive wine was put in expensive bottles. Not only did the participants say the preferred the more expensive looking wine, but brain scans proved it. Basically, they convinced themselves the wine was better because it was more expensive and that directly translated to them enjoying more than they (presumably) would otherwise.

boar_d_laze wrote:... Trust yourself.

Rich

Various studies have shown that we can't even trust ourselves in most cases. Link to the article referencing the wine study above http://www.theage.com.au/money/saving/i ... 2p8a9.html. I'm also reminded of high end audio claims, where people SWEAR they can hear a difference between a radio shack cable and a custom made cable that costs more than my car.

That doesn't mean we sincerely don't enjoy something more just because its new or more expensive (see article study above). However, I still wonder what results true blind tests would reveal. I'm guessing with most quality equipment it wouldn't be bad vs. good. More like I personally prefer this over that, but they are both pretty good.

Have a look at self-fulfilling prophecy (http://en.wikipedia.org/wiki/Self-fulfilling_prophecy). There are more theories regarding this phenomenon but to get the exact names I'll have to look at my Psychology books again. My study is way too long ago to remember all the names of the theories.

Another thing humans can't do is have subjective taste. When tasting two of the same espressos one after another the first one will always taste different then the second one just because we set our taste buds with the first one which affects the taste of the second espresso. Tasting them with an hour in between makes us dependend on memory and creates even more noise and bias. What we eat before tasting an espresso or in between gets out taste buds off. Even the size and shape of the cup creates a different spread of smell which will have an effect on taste. These are just a couple of examples of why humans aren't made to do this and the wine experiment from a while ago really made this clear.

Keeping all this in mind I started to just enjoy espresso and rate them as bad, good, very good or exceptional. After doing this I started to appreciate my espressos more with less fuss.

Good thoughts Erik. I was also going to ask how tasting espressos back to back would influence the overall testing.

Perhaps cloning someone directly before the tasting would alleviate the problem?

You guys are raising valid questions about the biases which often creep into non-blinded testing, just as I acknowledged them in my own post. However, you're neither acknowledging nor answering the questions I raised about noise.

What are your thoughts?

Rich

Can you formulate a couple of specific questions. I've read through the whole thread again but the questions about noise aren't clear to me.

What I was trying to say is that humans aren't build to do these types of testing, neither blind nor non-blind. The anatomy of humans just gives us too much variation between tests and then we aren't even considering biases which is an even larger downfall of humans.

For me I like the taste of big conicals better than big flat burrs but the main reason for me is the great consistency of large conical burrs that gives them a great advantage over other types of burrs.

erik82 wrote:Can you formulate a couple of specific questions. I've read through the whole thread again but the questions about noise aren't clear to me.

Me?

Mostly I'm referring to various forms of sampling error; but some non-sampling, residual errors too. Examples of sampling error included whether the "panel" can reliably distinguish any difference at all between grinders in AB testing; the effect of allowing filled baskets to sit around long enough to affect taste; clustering illusion; etc.

What I was trying to say is that humans aren't build to do these types of testing, neither blind nor non-blind. The anatomy of humans just gives us too much variation between tests and then we aren't even considering biases which is an even larger downfall of humans.

You raise an excellent point. Some things aren't testable; and some tests are meaningless.

For instance, I don't really care whether 7 out of 10 people prefer Pepsi to Coke. I know what I like and am happy with my choice even if it's not the one I'd make blindfolded.

For me I like the taste of big conicals better than big flat burrs but the main reason for me is the great consistency of large conical burrs that gives them a great advantage over other types of burrs.

Yes to big conicals for me too -- but with a different sequence of importance. First comes the wider and more defined spectrum of fruits and florals; mouthfeel is second; consistency, third.

Rich

First comes the wider and more defined spectrum of fruits and florals; mouthfeel is second;

If these are characteristics that you can reliably taste then it ought to be easy to document them using simple blind test comparisons. The among-shot variation due to all the factors other than burr size (i.e., the "noise") should not be large enough to impair your ability to differentiate between the grinders.

Not that anyone need do these tests - as stated, it ought to be enough for any given user to decide for themselves.

My biggest hang-up on all this has been that HBs have been a bit too fast to advise newbies that they need a large conical burr grinder to consistently make great espresso. World-class espresso, maybe. But a limited budget ought not to discourage people from getting into the world of espresso.