Slow speeds for internal static routes
-
I have a two pfSense firewalls. One physical and one virtual. I have multiple subnets behind the virtual one. I have setup static routes to point to each subnet to 192.168.8.5. It does work and transfers data, but it's very slow. SSH connections are fine, however trying to SFTP a file over to the VM's will copy 32k of data every 12-15 seconds. Watching the SFTP logs go by, it looks like it transfers a bit, and the connection is reset, then it reconnects and transfers another 32kb. File transfers over SMB or RDP are very similar. Sftp over the internet to the the wan work fine, as do other HTTP transfers.
If I configure a static route in Windows, the file transfer immediate speeds up to expected speed with other network gear in place (1Gbps'ish). I was previously running a Ubituiti USG without any problems with static routes, but much prefer pfSense.
What I have tried:
I have Bypass firewall rules for traffic on the same interface checked.
Disabling all hardware offloading options with mixed success. It seems to work better, but not great.
Upgraded hardware to a box with Intel Ethernet Controller I225-V cards and a 4 core Intel N3700 processor.
Disabling packet filtering to validate that isn't causing any issues.
Disabling pfBlockerNG (The only package installed)2.6.0-RELEASE (amd64)
built on Mon Jan 31 19:57:53 UTC 2022
FreeBSD 12.3-STABLE
-
@dannyh What is the OS of your hypervisor?
-
@nightlyshark
VMware vSphere 7.0 -
@dannyh Do you have:
- a whole network adapter passed-through to pfsense ? If not ->
- activated all hardware acceleration options for network adapters in vSphere ? If yes ->
- verified that the vmtools BSD kernel packages run successfully at boot in pfsense?
-
I'll check all of these items, I know for sure #1 is not the case. However it works great as long as I'm not using the static routes from the physical pfSense box.
I.e. From 192.168.8.8 to 192.168.13.5, transfers are slow if I just use my default gateway of 192.168.8.1 (physical pfSense box). if I set a static route through my Windows command line (route add 192.168.74.0 mask 255.255.255.0 192.168.8.15 metric 1) to just use the virtual pfSense box, it works great.
That, and 3 years of use without the physical pfSense box it was running full speed.
-
@dannyh Maybe with the addition of the physical pfsense box an MTU issue is created? Do you see any dropped or fragmented packets if you run wireshark?
-
@nightlyshark I'll check on that too and will get back.
-
@dannyh that sort of setup is going to be problematic for sure.
You have a host on your transit network, so you going to run into asymmetrical routing.. Unless for example you create routes on the 8.8 box which I think you did when you say this.
If I configure a static route in Windows
You should an actual transit network, ie no hosts on the network. When you connect routers, you should not have hosts on this network - especially if any communication between that host on the transit and devices on the downstream networks is going to happen. Unless you downstream router is also natting.
I just went over this in another thread.
https://forum.netgate.com/post/1086715
-
@johnpoz so, I could probably get away with using another interface as a separate path to that router. I.e. change the interface from lan to opt, and set the routes to go to 192.168.9.5 and leave my workstation in 192.168.8.0/24.
-
@dannyh yeah another interface works, or even just a vlan on the same physical lan interface would work too. You just want to get hosts off your transit network so you can't have asymmetrical traffic flow. Especially with a stateful firewall doing the routing you can see all kinds of issues with states as well and syn,acks being denied if the firewall/router never saw the syn to create the state, etc.
-
The new interface and routing through there resolved the slowness. Thanks for the help!
