Without knowing the "flavor defect" you're getting, it's hard to know if you're not fine enough so you're just getting fast, bulk flow, or too fine, so you're punching holes in the puck.

Another thing to consider is that lighter roasts can take longer than "espresso" roasts to hit their prime. At a week after roast, some of the fast extraction might be helped by letting the coffee rest a bit longer next time. Some of the Nordic roasts I won't open or freeze until three or four weeks after roast.