Slow routing speeds

A Former User

Hello All,

I have recently made a change to my network so that my home and lab networks are separate entities. Thus I can break things in my lab without affecting anything in the home network and causing major headaches from the family.

Please see a diagram of how things are setup currently:

I have static routes in the UDM Pro pointing to the WAN IP of the pf device, so I can access lab networks without requiring direct connectivity inside the lab or a VPN.

However I noticed a drop in transfer speeds with this setup. When my NAS was directly connected to my UDM Pro I was happily seeing gigabit speeds, but now I am getting ~250Mbps.

To confirm this wasn't just an issue with SMB or NFS I decided to run some iperf tests at different points in the chain to work out where the issue was, here are the results:

Current Setup: ~250Mbps
Direct connection to switch, same subnet as NAS: ~950Mbps
Direct connection to switch, different subnet as NAS: ~300Mbps

This suggested to me that the pf device isn't able to route at full gigabit, however looking at the System Activity page when running the tests usage never went very high (screenshot below while first test was running) to suggest an issue.

pf specs:
AMD G-T40N Processor
4GB Memory
2.5.2-RELEASE

Any suggestions to resolve this would be helpful, thanks in advance!

johnpoz

@hngaminguk said in Slow routing speeds:

Direct connection to switch, different subnet as NAS: ~300Mbps

Well you are doing router on a stick, so your hairpinning traffic between vlans that use the same physical connection. While this could prevent full wirespeed, 1/3 does seem a bit low..

What other traffic is flowing over that same physical interface while you do your test? From your setup I could see a problem with asymmetrical traffic flow. Since you haven't called out the network being used to connect the udm to pfsense, is this a different network than what your PC is in for example.

In your test connected to the switch - where was it routed to the other vlan at pfsense or the udm.

I would do a test by placing your PC on the pfsense wan network.. Then from a client on network behind pfsense, do say an IP test between devices. This takes the udm out of the equation for routing. Also make sure while your doing this test, there is no other traffic flowing over this physical uplink from your switch to pfsense.

If you want to connect pfsense as a downstream router from udm, the connection from the udm to pfsense should be a transit network (no hosts on it) so that you don't have asymmetrical traffic flow.

A Former User

@johnpoz said in Slow routing speeds:

@hngaminguk said in Slow routing speeds:

Direct connection to switch, different subnet as NAS: ~300Mbps

Well you are doing router on a stick, so your hairpinning traffic between vlans that use the same physical connection. While this could prevent full wirespeed, 1/3 does seem a bit low..

What other traffic is flowing over that same physical interface while you do your test? From your setup I could see a problem with asymmetrical traffic flow. Since you haven't called out the network being used to connect the udm to pfsense, is this a different network than what your PC is in for example.

In your test connected to the switch - where was it routed to the other vlan at pfsense or the udm.

I would do a test by placing your PC on the pfsense wan network.. Then from a client on network behind pfsense, do say an IP test between devices. This takes the udm out of the equation for routing. Also make sure while your doing this test, there is no other traffic flowing over this physical uplink from your switch to pfsense.

If you want to connect pfsense as a downstream router from udm, the connection from the udm to pfsense should be a transit network (no hosts on it) so that you don't have asymmetrical traffic flow.

Thanks for your detailed response!

In relation to the what else is flowing over the same physical link during the test, nothing. My lab currently just has one device (the NAS)

In relation to how things are connected and what networks are being used hope the below helps:
PC connected to port 4 of UDM and on 192.168.20.0/24 network
pf connected to port 7 of UDM and on 192.168.0.0/24 network
NAS connected to port 2 of switch and on 10.0.10.0/24 network

The tests connected to switch when routing to another vlan were at the pf level. (10.0.0.0/24 to 10.0.10.0/24)

As seen above the network the pf is connected to is separate, the only devices on that network are management level (such as my Unifi AP)

Hope that helps, whilst I await a response I will complete the test mentioned in your fourth paragraph.

johnpoz

@hngaminguk said in Slow routing speeds:

pf connected to port 7 of UDM and on 192.168.0.0/24 network

Oh that is good, so you are using a transit network then.. There is nothing on this 192.168.0 network other than udm and pfsense.

Yeah I would put test device on this 192.168.0 and test something on lan network of pfsense one of your I assume multiple 10 networks. What speeds do you get then - simple iperf in both directions would be good test.

A Former User

@johnpoz said in Slow routing speeds:

@hngaminguk said in Slow routing speeds:

pf connected to port 7 of UDM and on 192.168.0.0/24 network

Oh that is good, so you are using a transit network then.. There is nothing on this 192.168.0 network other than udm and pfsense.

Yeah I would put test device on this 192.168.0 and test something on lan network of pfsense one of your I assume multiple 10 networks. What speeds do you get then - simple iperf in both directions would be good test.

So I changed the assigned network for my PC to the 192.168.0.0/24 network. Tested using iperf again to my NAS in the 10.0.10.0/24 network and again still getting ~250Mbps.

To confirm I am doing iperf3 -c NAS-IP and then iperf3 -c NAS-IP -R to complete a test in reverse.

If there are any other details you would like to know let me know and I can try to provide them.

Thanks again for the assistance!

johnpoz

@hngaminguk said in Slow routing speeds:

So I changed the assigned network for my PC to the 192.168.0.0/24 network

So when you say that I take it you also physically moved its connection or changed the port it was connected to be in this other vlan/network

Also your not natting - so how did this pc know how to get to your nas IP? What was it using for its gateway - did you change that to be the 192.168.0 IP of pfsense?

A Former User

@johnpoz said in Slow routing speeds:

@hngaminguk said in Slow routing speeds:

So I changed the assigned network for my PC to the 192.168.0.0/24 network

So when you say that I take it you also physically moved its connection or changed the port it was connected to be in this other vlan/network

Also your not natting - so how did this pc know how to get to your nas IP? What was it using for its gateway - did you change that to be the 192.168.0 IP of pfsense?

I changed the port to the 192.168.0.0/24 network.

The UDM has static route set so to go to 10.0.10.0/24 network it sends it to the WAN IP of the pf device. This is something I overlooked when I did the test... apologies

I did another test, this time manually setting the IP & Gateway so that the IP was on the 192.168.0 network and the gateway was the pf device. Completed another iperf test but the same result ~250Mbps.

Hope that helps.

johnpoz

@hngaminguk prob need someone like @stephenw10 to chime in here.. He is the hardware guru and good with this sort of stuff to figure out if say a hardware issue not able to do more than this.

I have no issues with with my sg4860 doing full speed routing between interfaces.

So this pfsense box - it has more than 1 physical interface.. You have 1 for wan and one for lan, and only on this lan interface do you have multiple vlans?

When I test between a pc on 1 network 192.168.2/24 to my nas on 192.168.9.10 I see pretty freaking close to full wire speed.

in both directions..

$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110
Connecting to host 192.168.9.10, port 5201
[  5] local 192.168.2.110 port 1048 connected to 192.168.9.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   102 MBytes   856 Mbits/sec
[  5]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  5]   2.00-3.00   sec   113 MBytes   947 Mbits/sec
[  5]   3.00-4.00   sec   113 MBytes   946 Mbits/sec
[  5]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[  5]   5.00-6.00   sec   113 MBytes   948 Mbits/sec
[  5]   6.00-7.00   sec   113 MBytes   945 Mbits/sec
[  5]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  5]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  sender
[  5]   0.00-10.02  sec  1.09 GBytes   936 Mbits/sec                  receiver

iperf Done.

$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 -R
Connecting to host 192.168.9.10, port 5201
Reverse mode, remote host 192.168.9.10 is sending
[  5] local 192.168.2.110 port 1070 connected to 192.168.9.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   105 MBytes   884 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   935 Mbits/sec
[  5]   2.00-3.00   sec   108 MBytes   903 Mbits/sec
[  5]   3.00-4.00   sec   108 MBytes   908 Mbits/sec
[  5]   4.00-5.00   sec   113 MBytes   947 Mbits/sec
[  5]   5.00-6.00   sec   106 MBytes   892 Mbits/sec
[  5]   6.00-7.00   sec   113 MBytes   945 Mbits/sec
[  5]   7.00-8.00   sec   111 MBytes   932 Mbits/sec
[  5]   8.00-9.00   sec   109 MBytes   912 Mbits/sec
[  5]   9.00-10.00  sec   106 MBytes   891 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   917 Mbits/sec  320             sender
[  5]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  receiver

iperf Done.

A Former User

@johnpoz said in Slow routing speeds:

@hngaminguk prob need someone like @stephenw10 to chime in here.. He is the hardware guru and good with this sort of stuff to figure out if say a hardware issue not able to do more than this.

I have no issues with with my sg4860 doing full speed routing between interfaces.

So this pfsense box - it has more than 1 physical interface.. You have 1 for wan and one for lan, and only on this lan interface do you have multiple vlans?

When I test between a pc on 1 network 192.168.2/24 to my nas on 192.168.9.10 I see pretty freaking close to full wire speed.

in both directions..

$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110
Connecting to host 192.168.9.10, port 5201
[  5] local 192.168.2.110 port 1048 connected to 192.168.9.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   102 MBytes   856 Mbits/sec
[  5]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  5]   2.00-3.00   sec   113 MBytes   947 Mbits/sec
[  5]   3.00-4.00   sec   113 MBytes   946 Mbits/sec
[  5]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[  5]   5.00-6.00   sec   113 MBytes   948 Mbits/sec
[  5]   6.00-7.00   sec   113 MBytes   945 Mbits/sec
[  5]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  5]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  sender
[  5]   0.00-10.02  sec  1.09 GBytes   936 Mbits/sec                  receiver

iperf Done.

$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 -R
Connecting to host 192.168.9.10, port 5201
Reverse mode, remote host 192.168.9.10 is sending
[  5] local 192.168.2.110 port 1070 connected to 192.168.9.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   105 MBytes   884 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   935 Mbits/sec
[  5]   2.00-3.00   sec   108 MBytes   903 Mbits/sec
[  5]   3.00-4.00   sec   108 MBytes   908 Mbits/sec
[  5]   4.00-5.00   sec   113 MBytes   947 Mbits/sec
[  5]   5.00-6.00   sec   106 MBytes   892 Mbits/sec
[  5]   6.00-7.00   sec   113 MBytes   945 Mbits/sec
[  5]   7.00-8.00   sec   111 MBytes   932 Mbits/sec
[  5]   8.00-9.00   sec   109 MBytes   912 Mbits/sec
[  5]   9.00-10.00  sec   106 MBytes   891 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   917 Mbits/sec  320             sender
[  5]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  receiver

iperf Done.

Thanks for the diagnostic help so far! Yes I have 2 physical ports on an intel based network card. One is WAN one is LAN and 3 networks currently using the LAN side, 10.0.0.0/24 direct on LAN, 10.0.10.0/24 on LAN.10 and 10.0.11.0/24 on LAN.11

As mentioned previously the only actual device right now on the lab side is my NAS so the 10.0.11.0/24 network has no devices at all.

As mentioned in my original post when just doing switching I can get ~950Mbps (at that speed I'm calling it gigabit)

johnpoz

@hngaminguk Im not doing switching there - that is routed through my sg4860

@stephenw10 is the hardware guru around here..

You didn't setup any sort of limiting or shaping did you..

When I just switch I see a bit higher, I see a small slight hit routing it through pfsense.. But its minor and I really have no idea what else my nas is doing at the moment.. Someone could be streaming a movie off my plex ;)

While routing and firewalling normally will see a small hit, it sure shouldn't 250 from gig..

A Former User

@johnpoz said in Slow routing speeds:

@hngaminguk Im not doing switching there - that is routed through my sg4860

@stephenw10 is the hardware guru around here..

You didn't setup any sort of limiting or shaping did you..

When I just switch I see a bit higher, I see a small slight hit routing it through pfsense.. But its minor and I really have no idea what else my nas is doing at the moment.. Someone could be streaming a movie off my plex ;)

While routing and firewalling normally will see a small hit, it sure shouldn't 250 from gig..

Okay thanks for the confirm, hopefully @stephenw10 will have some ideas.

Traffic shaping is completely off currently, I made doubly sure as it was one thing suggested on posts on reddit and here.

I agree it makes no sense as to why I am seeing that much of a performance degradation, other than the low spec of my pf device I am not sure as to another cause.

stephenw10

The original APU, that we sold as the VK-T40E, has a G-T40E CPU and Realtek NICs. That is good for ~350Mbps between two ports, or was last time I tested it.
The G-T40N appears to differ from this only in the intergrated GPU so I would expect it to perform almost identically.
Is this using a Realtek NIC? Is it single queue? (check the boot logs)

If so ~300Mbps is probably not much lower than expected with a router on a stick config.

Steve

johnpoz

@hngaminguk said in Slow routing speeds:

hopefully @stephenw10 will have some ideas.

Told you he would ;)

A Former User

@stephenw10 said in Slow routing speeds:

The original APU, that we sold as the VK-T40E, has a G-T40E CPU and Realtek NICs. That is good for ~350Mbps between two ports, or was last time I tested it.
The G-T40N appears to differ from this only in the intergrated GPU so I would expect it to perform almost identically.
Is this using a Realtek NIC? Is it single queue? (check the boot logs)

If so ~300Mbps is probably not much lower than expected with a router on a stick config.

Steve

Thanks for the information Steve, when I ordered this device (eBay) it claimed to have an Intel card (I never opened the unit to confirm). But I have just done so now and I can confirm it has a PCI card (NC7170).

Looking at the card, the processor is an Intel one and looking on google it also confirms it's an Intel based card. So not sure if there is a different limitation in play here?

In relation to the boot logs I looked in the "OS Boot" section of the logs but not sure if that's the location you are referring to? Secondly I don't know what exactly I need to look for if that is the correct log section.

stephenw10

What are the NICs assigned as? em0, em1, igb0?

A Former User

@stephenw10 said in Slow routing speeds:

What are the NICs assigned as? em0, em1, igb0?

WAN is em0
LAN is em1

Hope that helps.

stephenw10

It does yes. OK so the router-on-a-stick part here is that you're testing between two VLANs both on the LAN NIC?

em is also single queue though so, whilst a lot better than re, you will still be limited.
The later APU2 devices had quad core CPUs and igb NICs and with that and some heavy tweaking it was possible to get close to 1Gbps. Here you are using effectively quarter of that in any one direction. It is interesting looking at your top output that almost all the use is on one of the queues, probably the receive queue.

That NIC looks like is actually PCI-X. I haven't seen one of those is a very long time! If you can use a igb based NIC there you probably would see more throughput as it can use both CPU cores for transmit and receive. Not sure there are any PCI/PCI-X igb NICs though.

Ultimately you're unlikely to see more tan 500Mbps with that CPU.

Steve

johnpoz

@stephenw10 said in Slow routing speeds:

you're testing between two VLANs both on the LAN NIC?

I think he was testing through wan and lan (router on a stick on the lan).. So if I understand what he was testing he did end up testing through the 2 different nics, or alteast different ports on the same nic. If that makes any difference?

So what your saying in a nutshell - is not some setting he turn off to see a boost, and if he wants to see full wirespeed he needs better hardware.

A Former User

@stephenw10 said in Slow routing speeds:

It does yes. OK so the router-on-a-stick part here is that you're testing between two VLANs both on the LAN NIC?

em is also single queue though so, whilst a lot better than re, you will still be limited.
The later APU2 devices had quad core CPUs and igb NICs and with that and some heavy tweaking it was possible to get close to 1Gbps. Here you are using effectively quarter of that in any one direction. It is interesting looking at your top output that almost all the use is on one of the queues, probably the receive queue.

That NIC looks like is actually PCI-X. I haven't seen one of those is a very long time! If you can use a igb based NIC there you probably would see more throughput as it can use both CPU cores for transmit and receive. Not sure there are any PCI/PCI-X igb NICs though.

Ultimately you're unlikely to see more tan 500Mbps with that CPU.

Steve

Originally I was testing throughput from the UDM/home system to the pf/lab system which was also only getting ~250 - 300Mbps.

So even with an igb NIC (if I can get one) I am unlikely to see over 500Mbps routing through the WAN side and out the LAN as my original network diagram shows?

If that is the case to get full 1Gbps (or more) routing what CPU spec would be the minimum? Also out of the current netgate devices which one would be the best option? I may in the future want to do 10Gbps but if it costs more than £500 I would rather go for a lower option and upgrade down the line when 10Gbps is cheaper.

stephenw10

Mmm, I would expect to see at least 350Mbps there as the APU could do that even with Realtek NICs.
IIRC there were some determined users who fitted mPCIe igb NICs to it and got closer to 500Mbps. You will never see Gigabit 'wire speed' through that though.

I used to have a test box with a Core2Duo E4500 and that could just do gigabit with em NICs. If you look at a synthetic benchmark you can see why the G-T40 won't:
https://www.cpubenchmark.net/compare/AMD-G-T40E-vs-Intel-Core2-Duo-E4500-vs-Intel-Atom-C3558/264vs936vs3129
That doesn't give the full story but as a basic guide it's useful.

The C3558 is what we have in the 5100/6100/7100/.

Steve