Yeah what confused me the most is that router->internet was testing at 750mbps+ with speedtest-cli, and desktop to router (through the same switch I'm normally connected through) was testing at 400+mbps (when downloading a dummy 1GB zip file I put in the web dir on the router... and it probably could have ramped up faster if it hadn't yanked the entire file in ~20 seconds).
That's what got me looking at the router configuration itself, because obviously the router->internet connection was fine, and the PC->router connection was fine -- so it had to be something on the router itself that was bottlenecking things.
UPD: the same issue as described at the beginning of my post is happening when connecting switch to pfSense and RouterA and RouterB to that switch thus hanging two routers on one pfSense port. Seems to be not an issue with virtual switch on pfSense as in this scenario using only one port.
Once separated Port5 and Port6 on pfSense to different private subnets and attaching RouterA and RouterB independently to pfSense box (+NAT with public VIPs) issue is gone. It appeared when both routers are connected to the same bridge or external switch they can't work reliably together. But I would still appreciate if someone can point me to the right direction how to investigate that further and perhaps with some Layer-2 debugging.
@akuma1x Yes it's easy enough to buy some secondhand/commodity hardware.
Anything you can find with enough network ports and an Atom C3XXX, or Intel i3/i5/i7 processors, or even some of the more recent fast Celeron and Xeon processors. Those are all good for a pfsense box. Try to stay away from the laptop-grade mobile processors, and the older Celeron J1900 stuff. Those are going to show their age and weaknesses quicker than the other ones.
HP and Dell made/make some good small form factor stuff. Just make sure you can add at least 1 multi-port INTEL network card in there and you'll be all set with a nice pfsense firewall box.