I wipe it with a microfiber towel after each use. Once I noticed a faint brownish discoloration on the "mat" in my tamper rest (which happened many years and tampers ago), I realized that a tiny bit of coffee and/or oil was still on the tamper even though it looked clean. One wipe proved me correct when I saw the discoloration on the towel, and I've been doing it ever since.
I keep separate toweling for the PF and baskets, the tamper, the steam wand, and general cleaning up so that I don't cross-contaminate with milk, stale coffee, or debris. I buy microfiber towels in bundles (like these from Amazon
at $11.77 for a pack of 5 each in 4 different colors) and do my best to reserve a different color for each task. Sadly, my wife and our housekeeper do their best to thwart my best. When I discover a coffee towel in the pile of cleaning rags (or anywhere else that suggests it's been used for evil), I throw it out and add a new one from my stock. I may break down and have them embroidered with a skull & crossbones over the legend "NEVER USE THIS TOWEL!"
Maybe I expect too much - I can see how confusing a statement and policy like "the blue ones are yours to use as you wish" can be.