[TS3] Voice lag on local server after a spike in PC load?

Hello! I’m regularly hosting a TS3 server on my PC for gaming with friends. They will connect to it by my external IP, and I’ll use localhost, plus my wife participates often, from next room on her PC, so she uses our physical LAN IP to connect. Recently we’ve noticed this weird thing: after there’s been a spike in my internet network or CPU/GPU/RAM load, my voice will start lagging behind. Sometimes it’s barely noticeable, but after we started playing Monster Hunter World, after spike load increases such as loading and departing on a quest, my words would start reaching other people 5+ seconds later than they should. Even my wife would hear me with a huge delay. I’ve tried setting my sound settings in TS3 client to Windows Audio Session, DirectSound and Auto, it doesn’t make any difference. How do I solve that (other than hosting/paying for an external virtual server, obviously)? Is there perhaps a setting in the server files that might help? I understand the delay happening during the spike in system load, but it persists until I disconnect from the server and reconnect back. Then it goes back to normal until the next time.