Long answer: over the years I have read many comments that effectively claim, "don't clean your coffee making equipment", "rinse it but never use soap", etc. etc. I have always found these statements to be extremely odd (e.g., see this post from over five years ago). Brewing coffee is cooking. With the obvious exception of cast iron/aluminum pots and pans*, what other kitchen implements would you recommend not cleaning thoroughly?
Some people claim that they can taste soap residue after washing and rinsing. If this is true, you have my sympathy. I always wash my glassware and dishes with soap, and rinse it throughly. Coffee equipment is no exception. (If soap residue is really an issue, try diluting your dish detergent 3-4X with water, rather than applying it full strength to the cookware. This makes it easier to rinse off.)
Other people state that cleaning espresso gear is a waste of time. I disagree. I accept input from all my senses, not just taste. If something smells, or if there is visible evidence of dirt, it needs to be cleaned. When you find dried-on food on your utensils in a restaurant, don't you ask for a clean fork? I do.
For espresso gear, this is not a subtle effect. Here is a picture of residual grinds from the grouphead of my Spaz after a slightly-updosed pour (16g). On the left is a quick shower screen brush and rinse. On the right is the subsequent PF wiggle. I don't think anything more needs to be said about the left side. Is that what you want to use for brewing the next shot? Yuck. Even the right side clearly shows spent grinds and discolored liquid, and I don't want that baking onto the grouphead and shower screen.

My cleaning regimen is similar to the OP's. After every shot I give a quick (1oz) grouphead rinse. After every series of shots I brush and rinse the shower screen, followed by PF wiggle and water backflush. The Spaz has a double shower screen to trap grinds, which I remove and scrub every few days. Once a week I do a detergent backflush. Anything less, and I believe the espresso quality degrades.
* which require seasoning to maintain nonstick properties



