We double-fenced the garden.
As a youngster Hooligan could leap tall buildings hedges in a single bound, so extreme measure were necessary.
There was already a wooden 'Hag' fence separating our property from the farm behind us, but no fencing on the other boundaries. The wooden fence wouldn't keep anything but a very dim cow in/out, so we used it as a support, running green wire diamond fencing along it. The green diamond wire disappears, from a distance all anyone really sees is the typical rural wooden fence.
We then built the same along the other sides of the property. At the front we built a more presentable closely-spaced wooden picket fence with lockable gate.
Then, because Hooligan came by her name honestly, we needed reinforcements. So along the fence we planted Brombeeren trellises and blueberry bushes on the sides with enough sun (in a small garden every centimeter needs to be used!), Thuja and climbing ivy on the sides where we needed a sight barrier.
And for good measure added a second fence, typical 'English garden' wooden picket hardscaping, in front of the border fence and hedge. With two fences sandwiching plantings, Hooligan couldn't get up enough of a head of steam to jump even if so tempted.
Then along came Heffalump. He is not an escape artist, but seriously clumsy and not too bright. One side of the property slopes, and he one day managed to get himself caught in between the fences and in his panic fell under a wire section of the fence after a rain eroded a bit of the slope. So back to the drawing board.
We used long metal U-shaped 'staples' from B/H to anchor the bottom of the green wire fence to the ground. Then added a stone border to prevent similar erosion. With your small pup, the greater danger is going under a fence, so this is where you should pay special attention.
So - Fence, hedge, fence. Because the plantings grow through the green wire you can't see it, keeping the neighbors happy.
Our fence posts are not installed in concrete. We are not allowed anything permanent, nor may we have solid fencing. But our 'impermanent' green diamond wire on a wooden fence does the job just as well, as the posts are set extra deep, while complying with building regs. OH is a good ol' Hoosier farm boy, he spent his youth fencing Grandpa's pastures, so building our fences was a doodle for him.
The borders have been secure for many years and many dogs. But now we have Robin Goodfellow, a mischievous sprite - or mayhap the devil incarnate. I'm waiting to see what surprises he might bring.
ETA:
Are your kittens and puppy new additions? How exciting! Here's to fun times ahead...
ETA:
The fencing materials (wooden posts, metal post holders, rolls of green diamond wire, hardware, etc.) were all bought from either Coop B/H, Hornbach, or Landi. Hornbach was the only place carrying old fashioned post-holers, which OH finds much easier to use.
Were it me, I'd add fencing either in front of or behind - or both - your existing hedges. No matter how thick the hedge, a determined pup could get through.