Actually what happens is that I have packed drops/high latency when transfers over VPN are getting very slow, not fast. Then VPN server can easily reach half of the speed of my download DSL link (i.e 300Mbit/2=150Mbit) and then everything is OK. There are no issues when VPN is not used at all either. Problem is when the remote end behind VPN (=torrent sources) isn't that fast and download speed drops to say 10Mbit. Then torrent transfers are causing high latency/high packet drop on my link.
This is very similar case to this one (unresolved issue): https://forum.netgate.com/topic/125639/lots-of-packet-loss-and-high-ping-when-torrenting-through-pia-vpn
But it's not PIA VPN that I'm using (it's NordVPN).

What is surprising to me that, as said before, I had no such issues when Asus RT-AC68U was my router.