malachi wrote:I'm sure that Greg has that data. I know others do. Suggest you mention your desire for it to be shared in the appropriate thread in The Bench. I would guess people would be perfectly willing to share it with you.

Personally, I understand that people obsess about this sort of thing - I am just not one of them. I've see results from a reasonable bracket (197.9F, 200.0F and 203.0F) and confirmed the results myself as well as in collaboration with other testers. That's enough for me.

Actually, Chris, I don't think Greg has that sort of data. If he does, I don't think he has shared it with anyone I've talked with and I've discussed this issue with a number of people we all know.

I am going to come up with a test protocol to test this issue, not specifically on this machine but on any machine used in the home by a home user, who has access to a Scace Device and datalogger. I have had a few emails with Andy S. in the last day about exactly this issue. I suggested that the two of us come up with a protocol testing this, that we could publish before anyone actually tests their machine so there would be no chance for the test to be designed based upon how a user might expect his machine to perform.

It doesn't have to be too complicated, just a list of shot sequences and time intervals, with all shots graphed on top of the other shots made at that temperature and some simple statistical tests, done at maybe 3 or 4 selected temperatures to examine the desired temperature range.

My point, which I'm obviously not making very well, is that now that we have the capacity to do accurate and repeatable thermography with the Scace device, we are not using it in a way that measures what matters most to the home barista, which is how likely he is to get a shot at a range of desired temperatures when he walks up to his machine to pull a random shot at a random time interval.

Best,

ken
(who is still not convinced that this stuff matters, either, just who thinks it needs to be tested both with blind taste testing and with thermography designed for low volume usage)

The way I validated the results I received was simply to run the WBC protocol at the temps indicated above. Don't see why some other protocol would need to be invented.

malachi wrote:The way I validated the results I received was simply to run the WBC protocol at the temps indicated above. Don't see why some other protocol would need to be invented.

I do.

The WBC protocol measures what I would call multiple continuous shot temperature stability. It proceeds to disregard the data produced from the first several (is it 3?) shots, then concentrates on the rest, to a total of 14 shots, the last 6 of which are made at a shotmaking pace exceeding the skill level of almost every home barista out there. The fact that a machine might be "temperature stable" in a series of 6 shots pulled with 10 second intervals other than quick flushes and shot pulls, with 15 seconds to prepare the shot, does not even remotely mimic the work spread and production of a home barista. The data produced therefore means nothing to someone like myself and 99.8% of the home baristas reading this. I think that perhaps your work experience has skewed your impression of how typical home users would use a machine such as this, in the (generally) low volume setting of a home. I do again point out that this review is posted on "Home-Barista.com," which I would assume is directed at home baristas and not at professional baristas or owners of commercial establishments.

A typical home barista who cares about the quality of his shots, and who wants to avoid channeling and other flaws that are now more than evident with bottomless PFs, is unlikely to work at a pace greater than 1 double shot every 90 seconds, and one double every 2 minutes is probably more typical. The home barista pulls shots in runs of ones, occasional twos, and infrequent 3s (or more). One constantly sees posts from avid home baristas whose spouses or SOs don't drink much espresso, so the emphasis on "onesies," i.e. individual single shots pulled at varying idle intervals, does not seem misplaced in my opinion.

Instead of testing temperature stability over a 14 shot run that de-emphasizes what comes out in the first shot or two, and that extends far beyond what a home barista is apt to ever do, we need a test that looks at random first, second, and rare third shots in the way a home user would actually use this machine and all the others that claim to be more temperature stable than those machines made with "older" technology. This means we need a lot of isolated shots made after varying idle periods, to show what the home user is going to get in a use pattern that mimics how he or she would actually use the machine in the home, rather than some test that mimics how a professional barista in a barista competition would use it.

I'm almost 100% convinced that a test protocol mimicking typical home use will produce completely different "results" than would the WBC protocol, for every single machine out there except perhaps a tiny single boiler machine like a Silvia which probably would not do very well on either.

My opinion only.

ken

Interesting points...

I've split this to a more active forum as I think it's relevant to all home users and not specific to the GS3 review.

Would love to hear Greg's thoughts on this...

I'm inclined to agree with Ken and would be very interested in seeing the results on the "alternative" method not only on the GS3 but also on other machines more geared (and priced) towards an average enthusiast.

Br,
Teme

Ken Fox wrote:A typical home barista who cares about the quality of his shots, and who wants to avoid channeling and other flaws that are now more than evident with bottomless PFs, is unlikely to work at a pace greater than 1 double shot every 90 seconds, and one double every 2 minutes is probably more typical. The home barista pulls shots in runs of ones, occasional twos, and infrequent 3s (or more). One constantly sees posts from avid home baristas whose spouses or SOs don't drink much espresso, so the emphasis on "onesies," i.e. individual single shots pulled at varying idle intervals, does not seem misplaced in my opinion.

Thanks for making this point so clearly, Ken. Your scenario is an excellent description of what goes on in my kitchen. I'd also be interested in seeing a new protocol that reflects home espresso use a bit more closely.

- John

I completely agree as well. My typical usage pattern is to make 2 cappas in the morning, and several other drinks at pretty random intervals throughout the day - influenced only by my or my wife's mood. Garbage shots are failures, not part of a flushing routine. The most drinks ever made in quick succession is 5 - and this was a true rarity.

I have tremendous respect for the creators of the WBC protocols, but agree that they are not representative of the general community of "home baristas." I'd love to see testing and reviews that were more focused on the way that the majority of home baristas were likely to use the machines, and routines developed under conditions that were also more representative of this community. I think we'd find that there are more than ample resources here to work on that.

Just my .02.

EDIT: After rereading, I want to add that the fact that WBC protocol is not representative of home users is not a disparaging comment in any way - this is not the scenario it was intended to represent. I merely think it's time that we all agree that there are significant differences in the commercial and home settings, and that these difference may have significant impact on temperature stability. Frankly, I don't think anyone here would disagree with that. It's easily follows, then, that a testing protocol designed to mimic one scenario would prove inadequate to mimic the other.

malachi wrote:Interesting points...

I've split this to a more active forum as I think it's relevant to all home users and not specific to the GS3 review.

Would love to hear Greg's thoughts on this...

I would like to form a small committee to write this protocol, which should not be so time consuming or difficult to accomplish that it will dissuade people from completing it. Since it is my idea I would like to name the committee

My nominations for membership in this committee are Jim Schulman, Andy Schecter, and myself. I think we could write this up with an exchange of several emails or through posts in this forum. If anyone else wants to either join this astute committee or to make suggestions on how the protocol should be written, please indicate so in this thread.

One potential benefit of this process is that we could have a bunch of us home barista types posting shot temperature results and graphs, with details given on how the results were obtained. Perhaps this will motivate people to explore various approaches to obtaining the most stable and reproducible results on particular machines. This could help other owners of the same equipment to get more consistent results out of their equipment. As an example of such novel approaches, I have found through preliminary testing that using my PID'd Cimbali Junior at a very low boiler temperature with a miniscule flush produces very repeatable results, per this graph:



There is no point in keeping this sort of information private; it should be shared as everyone potentially benefits from disseminating knowledge like this. In addition, it will help to show exactly how much is to be gained by equipment upgrades, which may be less than initially thought once we come to realize how much performance can actually be gotten out of a given piece of equipment.

ken

I don't have the equipment to be part of testing the proposed protocol but think it should include atleast 6 back to back shots starting from various idle times, and also include back to back shot, to steaming for a cap, to shot, to steaming for say making 4 (or more) double shot caps steaming separately for each scenario. These are some the main actual reasons I upgraded from Silvia! Maybe also quantifying needed machine heatup times?

miKe mcKoffee wrote:I don't have the equipment to be part of testing the proposed protocol but think it should include atleast 6 back to back shots starting from various idle times, and also include back to back shot, to steaming for a cap, to shot, to steaming for say making 4 (or more) double shot caps steaming separately for each scenario. These are some the main actual reasons I upgraded from Silvia! Maybe also quantifying needed machine heatup times?

I don't have any problem with doing 6 or more back to back shots but I don't think this is how home users typically use their machines and will not be representative. I have made as many as 9 back to back shots testing my new lower PID'd boiler temps with small volume flushes, and found good shot stability, but I don't think that most home users would want to do this nor would it be representative of normal usage patterns and hence if one were to adjust their machine for this sort of usage presumably results with more normal, small or 1 shot runs, would suffer.

As to steaming, I think this needs to be avoided for two reasons: (1) different people are going to froth different ways with different and similar pieces of equipment and results will be all over the map and probably meaningless; (2) one can hide all sorts of sins in milk, and what really matters if one buys into tight temperature control as necessary, is to be able to pull straight shots which will be drunk straight under reproducible and tight temperature control.

I have some ideas myself, which I have not yet posted but will a little later.

ken