Second that Tower is one of your best ways to help in trading. I usually build it turn 2 against a fast-leveling hero, and avoid bringing out a hero of my own until I’m using a spell or I have a solid tech 1 unit line.
Spectral Hound is often a good way to trade up against early hero aggression, just not against the black starter (deteriorate is killer).
Otherwise, I tend to rely on Scribe to help stall to tech 2.
But you’re absolutely not the first or last person to struggle playing P2 Blue against early aggro (I don’t have a great handle on it yet, and I’ve played like 20 games of P1 Black vs P2 Blue): it isn’t very well equipped to handle heroes like Garth / Vandy / Cala / Grave / etc…