This is probably going to be a long response.
In addition, a bunch of the stuff I'm going to say is going to contradict what many (smart) people say and do.
Finally, in addition to (I hope) answering your question I'm going to give a lot of entirely unsolicited and perhaps undesired advice.
For all... I apologize in advance.
Dose:
The "optimal" dose is going to depend upon a number of attributes. First - the coffee. There is no magic dose that works for all coffees. Some coffees respond in certain, perhaps desirable, ways to certain dosing strategies. Others to others. Second - your personal taste. I have a friend who likes his espresso incredibly (to my taste) overextracted. His favorite shots tend to be about 0.75oz from a double basket in about 40 seconds. But he loves the taste. As a result, his optimal dose for any specific coffee is different from mine. Third - your machine. Variances in basket size can be significant as can the affects of different brew pressure profiles. Other factors like gasket sizing and dispersion patterns can even, in theory, play a part.
Yeah... I can hear you right now. "Wow. That's not helpful at all."
So... what do you do?
You
explore and you experiment. You know what you like and you have your machine. So, for each coffee you experiment with different doses to see what the results are. When you find results that you like for a specific coffee - then you can stick with that.
Of course... coffees change harvest to harvest and blends change batch formulation to batch formulation but that just means you need to be on your toes and be willing to re-explore.
Dosing:
I am a well-known opponent of grinding and dosing by weight.
In my opinion, there are numerous problems with this approach.
Coffee is an agricultural product and is fundamentally unpredictable, changing and wildly inconsistent. Density, moisture content... it's all a crap shoot.
Grinders tend to trap coffee and introduce uncontrolled variance and loss. Weighing the coffee creates an illusory sense of control while in reality, IMHO, decreasing your real control.
Focusing on weight removes your brain from the action - and your brain is a powerful tool in your favor. The more you can actually pay attention to what you're doing rather than trying to offload that attention to tools dumber than you... the better.
Instead of grinding and dosing by weight I'd strongly suggest doing so by volume.
You will almost certainly find that you are going to get better results in the cup. You will waste some coffee - yes. But it's not a significant loss, and the improvement in your espresso (and the decrease in your frustration) is well worth it.
With practice, you will find that you can be incredibly consistent dosing by volume (good baristas can dose 10 times in a row by volume with a variance of less than 0.3grams).
I spent a lot of time early on working on dosing by volume and then correlating the various volumes to approximate weights. I feel like this has paid off when it comes to communicating results to people - but has been nearly worthless when it comes to the espresso I drink.
Dosing Technique:
As a general rule it is not advisable to shake the portafilter side to side while dosing. This creates distribution problems in the basket - resulting in a bed that is of uneven density. Your extraction, as a result, will be uneven. The results through a naked portafilter can be very illuminating.
If you're trying to deal with the clumping of your grinder, it is far more effective to do so manually as part of your distribution. This will be as effective when it comes to 'de-clumping' and will not introduce extraction problems.
If you're trying to settle the coffee (to assist in updosing) then the better option is to tap the portafilter on the fork or the counter. Tap straight down and don't be overly aggressive. This will enable you to create and preserve even distribution.
Distribution:
I'm like a broken record on this topic... I know that. But honestly, distribution is one of the most important and least appreciated barista skills.
The idea is that you're trying to create a bed of coffee that is of even density within the basket and then preserve that bed.
There are a couple of very good techniques for doing this. While I, personally, believe that the so-called Stockfleth's Move (and its many variants) is the best option, I've also seen (and had) good results from the Schomer "NSEW" distribution technique.
Two things that both have in common that are incredibly important:
1 - you use your finger to distribute the coffee, and
2 - you
distribute the coffee (rather than merely leveling off the top surface).
I would predict that paying some attention to this area of technique is going to result in really noticeable changes to the quality of your espresso. There is a ton of data about distribution now. The naked portafilter really opened people's eyes to the importance of the techniques here and seems to have focused a lot of people on these two successful methodologies.
Extraction:
Again, while two of the factors behind your extraction parameters are your personal taste and your machine - the third is the coffee. I would suggest letting go of the "always pull 1-1.5oz in 25-27 seconds" and allowing yourself to taste the coffees and determine what extractions work the best for what coffees given your desired flavour profile.
At the end of the day, you need to be happy with your espresso.
If you taste it - and it tastes good to you - then it
is good. If you get there by breaking "rules"... if you're doing things "wrong" according to the "experts and pundits"... well more power to you.
Trust your taste.
I hope this was helpful and not overwhelming, annoying, abrasive or insulting.