Sounds like a tiny steam leak. Being rushed today, I'll quote myself from
condensation in my boiler pressure gauge:
"Don't laugh, but an easy way to find the smallest of steam leaks is by sticking one end of a short flexible tube into your ear and probe around the interior of the machine with the other end. Check carefully around fittings and joints. A turkey baster with the bulb end removed also works in a pinch."
As you said, the vacuum breaker is the most likely suspect, but it could also be a fitting has developed a small leak.
Reminder: Don't tighten fittings of a hot espresso machine in the attempt to seal a small leak, the fitting could snap off, releasing (superheated) boiling water!