Just throwing out more ideas:
Is there a CPU spike on the VPS?
Anything weird about Wireguard on either end? Using kernel mode WG everywhere and not a user mode version, right?
As a test I would be inclined to try a very small mtu to see if it makes a difference. 1280 is a failsafe that I use when on unknown networks and trying to wg out.
Maybe try with a smaller packet size, like 1KB which I think is -l 1K
To be fair it wasn't a random banana.
It was this specific banana.