That's a good question Gary, which I've had to think about - and in so doing, I decided I was a bit of a sheep for cleaning my grinder because everybody else does it. After thinking about it for a bit though, I can think of two reasons.
First, the grinds that get stuck deep in the grooves of the long teeth of the grinder presumably have some effect on the quality of the grind. If they didn't the burrs wouldn't be made with such deep grooves. It would presumably be easier and cheaper to make less defined teeth, which would solve the problem of coffee getting stuck and not change the grind quality in any way.
Second, the gradual build up of stuck grinds may not present much of a problem, but when a old bit of compressed grounds breaks off and finds its way into your drink it could, I suppose, be noticeable.