I think this is highly dependent on grinder and use case.
I have never been able to tell the difference in flavor after cleaning my Vario or Encore, I just do it as a part of regular maintenance. Haven't had my Niche for long enough to clean it, but I doubt i'll be cleaning it frequently. I don't think you need to frequently clean grinders that you single dose with, and especially grinders designed for single dosing. I can also see cleaning being much more important with a grinder like an SJ, especially with a doser. And obviously grinding darker, oily, roasts will necessitate more cleaning.
Like a lot of claims of taste here on HB, i'd like to see people identify dirty grinders in a blind taste test. And then I'd like to see just how dirty it needs to be for someone to identify it in the same test. I'd be willing to bet the vast majority of users on here keep their grinder plenty clean enough. FWIW I think a lot of people would be very surprised at how infrequently grinders in high end coffee shops get cleaned, much less pepper grinders in restaurants.