Espresso equipment test methodology

[cafeIKE]

Split from Titan Grinder Project by moderator...

Perhaps when this project is finished, or at the end of it, a real, statistically valid blind tasting can be done on at least a couple of grinder pairs being studied here. It may be that if the differences in the shots are really so obvious, that the tester should be asked not only to pick a preference on the shots but also to try to identify which grinder produced it.

These kinds of studies are a huge PITA to design, and even worse to execute.

I am not volunteering . . . . .

ken

To be valid, you'd need several of each grinder model to ensure differences were not summations of manufacturing tolerances.

Assuming a particular grinder type is prefered, there is problem of association:
Grinder A on Machine B with Coffee C at Temperature D vs Grinder B on Machine C with Coffee D at Temperature E.

[Ken Fox]

To be valid, you'd need several of each grinder model to ensure differences were not summations of manufacturing tolerances.

Assuming a particular grinder type is prefered, there is problem of association:
Grinder A on Machine B with Coffee C at Temperature D vs Grinder B on Machine C with Coffee D at Temperature E.

Ian,

You give a seemingly conclusive argument for never doing any kind of controlled coffee research, because you can never possibly know all the variables at play and hence never possibly control for them. The result is that you would have to do an undoable study, so complex and involved, so time-consuming and expensive, that you just throw up your hands and give up without trying.

I find this a sort of rubbish argument, one that says that doing research of any sort isn't worth the effort because you can't possibly account for all the variables. The beauty of the scientific method, when used properly, is that the researcher lays out all the known variables, how it was attempted to control them, and the methodology used to execute the study. Regardless of the results, the onus is then on the reader to design another study (hopefully based on a knowledge of the issues and why a particular result may have occurred) that will show why the first study reached the conclusions that it reached and why those conclusions were either in error or misleading.

Science does not generally advance by leaps and bounds, rather it advances in small steps. If we make no effort to go beyond the descriptive, e.g. "I just bought this huge honkin' grinder and lookit the coffee it makes! Sooooooo much better than that old cheaper grinder I used to own!," then we in fact learn next to nothing from experience. If instead an attempt is made to use the scientific method to study things, we must accept the limitations of that method but at the same time acknowledge that those results are better than anything else available to us.

I invite you to design and execute your own studies and to publish them here or in any other readily accessible venue. It is however much easier to sit around on the periphery and throw stones, than it is to design and execute a study. I guess that is because, as you point out, you really can't ever "prove" anything.

ken

[cafeIKE]

Ken,

Fine rifles are test fired at completion of manufacturer. The very best are set aside and sold as "One of One Hundred" or "One of One Thousand" on larger production. These "One of" units are coveted by marksmen and competition shooters. The run of the mill units are very accurate, but unlikely match winners or better than competing brands.

In high quality optics, Leitz, Zeiss, Swarovski, Nikon, Canon et al., considerable unit to unit variations exist, readily apparent to a skilled observer. All units pass rigorous manufacturing standards but if one selects from a large sample there is usually one that is head and shoulders better than its brethren. The difference is close that between TV and HiDef. When someone tries the missus' binocs and acclaims their clarity and brilliance we are scrupulous to inform they are "One of..."

When we are talking bimodal distribution of particles with size variations in microns and typical manufacturing tolerances maybe large enough to radically shift the nodes, it behoves the testers to determine the entire class, not just a specific unit, is worthy of consideration.

And since I'm stowing thrones, shipping the grinders all over hell's ½ acre gives me pause...

Here's a granite pebble for your burrs: Tester A grinds 20 pounds on new burrs, then ships to Tester B who grinds 30 pounds, then ships to Tester C. Tester C finds after 40 pounds that his results are diametrically opposed to Testers A and B. Revelation or Rubbish?

When one knows the variables and does not account for them, one may have rubbish results. First, Test the Test.

I'm all for 'testing', more correctly 'sampling', provided the results are

relevant to the audience at large.
not just laboratory knob-dicking.
dismissable for lack of rigor.

Anything that will someday enable us to have consecutive, not even necessarily identical, gShots... 40 years and counting.

[Ken Fox]

Ken,

Fine rifles are test fired at completion of manufacturer. The very best are set aside and sold as "One of One Hundred" or "One of One Thousand" on larger production. These "One of" units are coveted by marksmen and competition shooters. The run of the mill units are very accurate, but unlikely match winners or better than competing brands.

In high quality optics, Leitz, Zeiss, Swarovski, Nikon, Canon et al., considerable unit to unit variations exist, readily apparent to a skilled observer. All units pass rigorous manufacturing standards but if one selects from a large sample there is usually one that is head and shoulders better than its brethren. The difference is close that between TV and HiDef. When someone tries the missus' binocs and acclaims their clarity and brilliance we are scrupulous to inform they are "One of..."

When we are talking bimodal distribution of particles with size variations in microns and typical manufacturing tolerances maybe large enough to radically shift the nodes, it behoves the testers to determine the entire class, not just a specific unit, is worthy of consideration.

And since I'm stowing thrones, shipping the grinders all over hell's ½ acre gives me pause...

Here's a granite pebble for your burrs: Tester A grinds 20 pounds on new burrs, then ships to Tester B who grinds 30 pounds, then ships to Tester C. Tester C finds after 40 pounds that his results are diametrically opposed to Testers A and B. Revelation or Rubbish?

When one knows the variables and does not account for them, one may have rubbish results. First, Test the Test.

I'm all for 'testing', more correctly 'sampling', provided the results are

relevant to the audience at large.
not just laboratory knob-dicking.
dismissable for lack of rigor.

Anything that will someday enable us to have consecutive, not even necessarily identical, gShots... 40 years and counting.

Design your own studies and DO them. Point out the flaws in what I and others are doing, and correct them with your own work. Until then, I will use the best information available to me.

There is so little unbiased and objective work in coffee, that to read a post like yours is . . . . . . . painful.

ken

[JonR10]

And since I'm stowing thrones, shipping the grinders all over hell's ½ acre gives me pause...

Here's a granite pebble for your burrs: Tester A grinds 20 pounds on new burrs, then ships to Tester B who grinds 30 pounds, then ships to Tester C. Tester C finds after 40 pounds that his results are diametrically opposed to Testers A and B. Revelation or Rubbish?

Interesting - but probably moot. These monster grinders are most likely built to process large amounts of coffee between burr changes, so as long as user 1 and user 2 and user 3 only grind "normal for home use" amounts then there should be no signifigant wear in the burrs, yes?

Your point is valid, but I agree with ken in that this should not get in the way of the fun and informative experiment.








Now then, how do I wrangle an invitation to someone's house who is pulling shots from a Mazzer Robur?

[RapidCoffee]

Now then, how do I wrangle an invitation to someone's house who is pulling shots from a Mazzer Robur?

Any time, my friend - as long as it's in the next week or so. After that you'll have to put up with the Super Jolly.

[Walter]

Perhaps when this project is finished, or at the end of it, a real, statistically valid blind tasting can be done on at least a couple of grinder pairs being studied here.

To read "real, statistically valid" and "blind tasting" in one sentence is mildly amusing. For statistics to be valid you needed a huge number of probes and probants. Done with a few probants and probes the results may still be interesting, but "statistically" they are entirely irrelevant and might just tell us exactly nothing...

Ian,

You give a seemingly conclusive argument for never doing any kind of controlled coffee research, because you can never possibly know all the variables at play and hence never possibly control for them. The result is that you would have to do an undoable study, so complex and involved, so time-consuming and expensive, that you just throw up your hands and give up without trying.

I beg to disagree here too. Ian's points are, IMHO, very valid and what he does is not more than to point out that we are dealing with very complex matter here. A fact which is ever so often entirely ignored in "pseudo-scientifical" experiments. After all we're not doing "science" here, the extractions are not 100% reproduceable, we're not running e.g. GC/MS examinations of the extracted coffee and comparing the results and so forth. No, we're preparing shots and we're tasting. That is nothing remotely reminiscent of real science.

What I see here is an interesting comparison of good grinders with some really interesting facts and a scientifical touch, like the SEM images or particle size distribution measurements. Nothing more, nothing less. But to consider this "science" is a tad too ambitious, IMO...

[Walter]

Interesting - but probably moot. These monster grinders are most likely built to process large amounts of coffee between burr changes, so as long as user 1 and user 2 and user 3 only grind "normal for home use" amounts then there should be no signifigant wear in the burrs, yes?

Your point is valid, but I agree with ken in that this should not get in the way of the fun and informative experiment.

Yes, but these grinders also need a few kilos of beans during the "breaking in" period until they are "stable". I do not seem to recall having read anything whether or not this has be done before the tests started...

But I entirely agree with you about the "...fun and informative experiment"

[Ken Fox]

Perhaps when this project is finished, or at the end of it, a real, statistically valid blind tasting can be done on at least a couple of grinder pairs being studied here.

To read "real, statistically valid" and "blind tasting" in one sentence is mildly amusing. For statistics to be valid you needed a huge number of probes and probants. Done with a few probants and probes the results may still be interesting, but "statistically" they are entirely irrelevant and might just tell us exactly nothing...


I beg to disagree here too. Ian's points are, IMHO, very valid and what he does is not more than to point out that we are dealing with very complex matter here. A fact which is ever so often entirely ignored in "pseudo-scientifical" experiments. After all we're not doing "science" here, the extractions are not 100% reproduceable, we're not running e.g. GC/MS examinations of the extracted coffee and comparing the results and so forth. No, we're preparing shots and we're tasting. That is nothing remotely reminiscent of real science.

What I see here is an interesting comparison of good grinders with some really interesting facts and a scientifical touch, like the SEM images or particle size distribution measurements. Nothing more, nothing less. But to consider this "science" is a tad too ambitious, IMO...

Nothing has been posted in the grinder thread, So Far that would qualify as an experiment using the scientific method. As you state, some scientific tools have been used, however no real scientific experimentation has been done.

On the other hand, there is nothing special about blind tasting as an experimental technique that requires that it be treated differently than any other test methodology. All that statistics will do is to look at the probability that an observation could have occurred on the basis of chance alone, and to assign a numerical probability of that. If a properly designed experiment (and YES, you can do a properly designed experiment utilizing blind tasting) shows that an observed result could have occurred only 5% of the time on the basis of chance alone (5% confidence interval) then YES, ONE out of TWENTY times you might get that result by chance alone. If the confidence interval is 1%, then only 1% of the time you could expect that result by chance. The more the data points, the more the observed difference, then the more you can be confident that your results mean something.

There are also things to be learned from the more descriptive portions of the aforementioned article, even if they do not rise to the level of being scientific research. For one thing, they are additional pieces of information obtained by people generally having no ax to grind. You can't say that about the other information currently out there on these grinders.

Some people here and in other online coffee venues make an attempt to study and explain things, and they go to the trouble of writing these things up for the benefit of all readers. When those writings have flaws they should be pointed out constructively, giving the original author an opportunity to respond. This would constitute constructive criticism. What I can learn absolutely nothing from is people who criticize these efforts of others while offering absolutely nothing in return. If the "research" you see here does not meet your standard, then do some yourself that does, and enlighten us with the results. Isolated criticism with no other contribution is of no value at all, not to me nor to anyone else here.

ken

[cafeIKE]

Some people here and in other online coffee venues make an attempt to study and explain things, and they go to the trouble of writing these things up for the benefit of all readers. When those writings have flaws they should be pointed out constructively, giving the original author an opportunity to respond. This would constitute constructive criticism. What I can learn absolutely nothing from is people who criticize these efforts of others while offering absolutely nothing in return. If the "research" you see here does not meet your standard, then do some yourself that does, and enlighten us with the results. Isolated criticism with no other contribution is of no value at all, not to me nor to anyone else here.

ken

Earlier, in the original thread,

-I think this is valuable, although too limited in scope to "prove" anything
-Any measurements taken would be valueless in the absence of any observed differences in shot quality.
-one can't talk generically about "conicals" and "planar grinders" but rather about SPECIFIC grinders.

Methinks thou dost protest too much.

Lord knows, we don't need another generation of knick-knack paddy-whackers dumping their perfectly decent grinder and then bemoaning they still can't pull 1 in a row.

Let me reiterate. I applaud those who undertake projects and make great efforts to communicate their findings. Your work on HX PID, seminal in my Vibiemme decision. Dan's work on grinders last year, the trigger on the MC4. Jim's work on extraction, a large step up the ladder in consistency. Ad infinitum...

Apologies for my 'painful' ineloquent posts and if they appear baselessly critical. My intent is but to ensure the readers are aware there are 'issues' with the 'test' and 'results' are not annointed.

[old442]

I haven't looked at the TGP as a comprehensive scientific study but rather an in-depth review. I don't believe that a major scientific study is of value for existing products like these, especially when the question of who would pay for them is asked. If we are talking R&D purposes for a grinder manufacturer then the answer may be different.

It is indeed nice to see things like SEM pictures of the ground coffee, which may inspire others to find correlations that fuels further research. However, I don't recall any of the authors claiming this was a scientific study because of the pictures. It's just another tool, which few have access to, used to enlighten.

I was, and still am, excited about the TGP because this is different than most product reviews in that there is some side by side comparison AND multiple people doing the reviews. What makes this most interesting to me is that the reviews are being done by competent Home Baristas (Dan, back me up on this!). This should yield better buying guides than what is out there now (virtually nothing) for this level of grinder.

Reviews are subjective, as is taste. If each of the reviewers states that grinder A brings out brighter notes and that grinder B brings out deeper notes that helps in a decision making process when buying. Likewise, as they point out usability issues YOU are able to decide what is important to YOU when YOU make a purchase.

Take the TGP for what it is and use the information presented. If it's not what you're looking for then do your own evaluation and share that with others.

[old442]

Fom the first post in the TGP thread (Dan Kehn)

...this site's most extensive equipment review to-date...

I don't see the word "test", I see the word "review".

[another_jim]

I haven't looked at the TGP as a comprehensive scientific study but rather an in-depth review. I don't believe that a major scientific study is of value for existing products like these, especially when the question of who would pay for them is asked. If we are talking R&D purposes for a grinder manufacturer then the answer may be different.

The opportunity is too good not to do some testing.

People tend to regurgitate the bs they learned in the psychology department's "method of science 101" classes; mainly to assure themselves that it's way too complicated for them to do anything. Alternatively, people who do tests feel pressured to add all sorts of controls and other stuff. The main thing to know about all knowledge is that you can test until you're blue in the face, and not learn anything. For instance, if you start your car 10,000 times and record how long it takes to fire, you'll have nice bell shaped curve, but you still won't know a damn thing about the ignition system. This is exactly what psychologists do, since it's going to be a while before they can peek under the hood and come up with any hypotheses actually worth testing.

The situation is rather different with the TGP. We have lots of grinders. We have the opportunity to do some high quality physical measurements of ground coffee, we can measure how the grinders' pucks extract, and we can do taste comparisons. We already have several good hypotheses that we'll be testing. These predict how the shape of the particles, the ratio of fines to coarse, and the dispersion of coarse particles affects the mechanics of extraction and thereby the shot's taste.

If the hypotheses check out, we'll be able to do to things. Assess and predict the taste quality, that is the clarity, distinctness, etc of a grinder's shot by looking at the physical data. Compensate for variations in taste profile, that is the brightness, body, sweetness of the shots, by changing dose and grind.

In short, save discussions of "proof" and "certainty" for church or the philosophy forum. If you want to talk about testing espresso equipment, lets hear about solids yields, fines/coarse ratios, coarse grinding times, etc.