Ok, have a look into the DOCSIS Telemetry.

I was hell if my ISP rollout the OFDMA to the upstream some years ago. And your problem looks similar.
Idle was nice, but if you use the bandwidth, the error rat grows and grows and with it the retransmission and the latency explode.
It takes month and 2-3 construction sites to get a nice stable connection back.
Have a look into it fist.

Here is my transfer performance using Wireguard

DOWNLOADING FROM SERVER (Server upload performance)
fa6458705745c2fe12cf2ee4b989de6b[1].png

UPLOADING TO SERVER (Server download performance)
cbd266b143cfdf96762c54a44e8b5656[1].png
I'm very happy with these results.

UPD: the same issue as described at the beginning of my post is happening when connecting switch to pfSense and RouterA and RouterB to that switch thus hanging two routers on one pfSense port. Seems to be not an issue with virtual switch on pfSense as in this scenario using only one port.
Once separated Port5 and Port6 on pfSense to different private subnets and attaching RouterA and RouterB independently to pfSense box (+NAT with public VIPs) issue is gone. It appeared when both routers are connected to the same bridge or external switch they can't work reliably together. But I would still appreciate if someone can point me to the right direction how to investigate that further and perhaps with some Layer-2 debugging.