Slow routing speeds
-
@hngaminguk said in Slow routing speeds:
Direct connection to switch, different subnet as NAS: ~300Mbps
Well you are doing router on a stick, so your hairpinning traffic between vlans that use the same physical connection. While this could prevent full wirespeed, 1/3 does seem a bit low..
What other traffic is flowing over that same physical interface while you do your test? From your setup I could see a problem with asymmetrical traffic flow. Since you haven't called out the network being used to connect the udm to pfsense, is this a different network than what your PC is in for example.
In your test connected to the switch - where was it routed to the other vlan at pfsense or the udm.
I would do a test by placing your PC on the pfsense wan network.. Then from a client on network behind pfsense, do say an IP test between devices. This takes the udm out of the equation for routing. Also make sure while your doing this test, there is no other traffic flowing over this physical uplink from your switch to pfsense.
If you want to connect pfsense as a downstream router from udm, the connection from the udm to pfsense should be a transit network (no hosts on it) so that you don't have asymmetrical traffic flow.
-
@johnpoz said in Slow routing speeds:
@hngaminguk said in Slow routing speeds:
Direct connection to switch, different subnet as NAS: ~300Mbps
Well you are doing router on a stick, so your hairpinning traffic between vlans that use the same physical connection. While this could prevent full wirespeed, 1/3 does seem a bit low..
What other traffic is flowing over that same physical interface while you do your test? From your setup I could see a problem with asymmetrical traffic flow. Since you haven't called out the network being used to connect the udm to pfsense, is this a different network than what your PC is in for example.
In your test connected to the switch - where was it routed to the other vlan at pfsense or the udm.
I would do a test by placing your PC on the pfsense wan network.. Then from a client on network behind pfsense, do say an IP test between devices. This takes the udm out of the equation for routing. Also make sure while your doing this test, there is no other traffic flowing over this physical uplink from your switch to pfsense.
If you want to connect pfsense as a downstream router from udm, the connection from the udm to pfsense should be a transit network (no hosts on it) so that you don't have asymmetrical traffic flow.
Thanks for your detailed response!
In relation to the what else is flowing over the same physical link during the test, nothing. My lab currently just has one device (the NAS)
In relation to how things are connected and what networks are being used hope the below helps:
PC connected to port 4 of UDM and on 192.168.20.0/24 network
pf connected to port 7 of UDM and on 192.168.0.0/24 network
NAS connected to port 2 of switch and on 10.0.10.0/24 networkThe tests connected to switch when routing to another vlan were at the pf level. (10.0.0.0/24 to 10.0.10.0/24)
As seen above the network the pf is connected to is separate, the only devices on that network are management level (such as my Unifi AP)
Hope that helps, whilst I await a response I will complete the test mentioned in your fourth paragraph.
-
@hngaminguk said in Slow routing speeds:
pf connected to port 7 of UDM and on 192.168.0.0/24 network
Oh that is good, so you are using a transit network then.. There is nothing on this 192.168.0 network other than udm and pfsense.
Yeah I would put test device on this 192.168.0 and test something on lan network of pfsense one of your I assume multiple 10 networks. What speeds do you get then - simple iperf in both directions would be good test.
-
@johnpoz said in Slow routing speeds:
@hngaminguk said in Slow routing speeds:
pf connected to port 7 of UDM and on 192.168.0.0/24 network
Oh that is good, so you are using a transit network then.. There is nothing on this 192.168.0 network other than udm and pfsense.
Yeah I would put test device on this 192.168.0 and test something on lan network of pfsense one of your I assume multiple 10 networks. What speeds do you get then - simple iperf in both directions would be good test.
So I changed the assigned network for my PC to the 192.168.0.0/24 network. Tested using iperf again to my NAS in the 10.0.10.0/24 network and again still getting ~250Mbps.
To confirm I am doing
iperf3 -c NAS-IPand theniperf3 -c NAS-IP -Rto complete a test in reverse.If there are any other details you would like to know let me know and I can try to provide them.
Thanks again for the assistance!
-
@hngaminguk said in Slow routing speeds:
So I changed the assigned network for my PC to the 192.168.0.0/24 network
So when you say that I take it you also physically moved its connection or changed the port it was connected to be in this other vlan/network
Also your not natting - so how did this pc know how to get to your nas IP? What was it using for its gateway - did you change that to be the 192.168.0 IP of pfsense?
-
@johnpoz said in Slow routing speeds:
@hngaminguk said in Slow routing speeds:
So I changed the assigned network for my PC to the 192.168.0.0/24 network
So when you say that I take it you also physically moved its connection or changed the port it was connected to be in this other vlan/network
Also your not natting - so how did this pc know how to get to your nas IP? What was it using for its gateway - did you change that to be the 192.168.0 IP of pfsense?
I changed the port to the 192.168.0.0/24 network.
The UDM has static route set so to go to 10.0.10.0/24 network it sends it to the WAN IP of the pf device. This is something I overlooked when I did the test... apologies
I did another test, this time manually setting the IP & Gateway so that the IP was on the 192.168.0 network and the gateway was the pf device. Completed another iperf test but the same result ~250Mbps.
Hope that helps.
-
@hngaminguk prob need someone like @stephenw10 to chime in here.. He is the hardware guru and good with this sort of stuff to figure out if say a hardware issue not able to do more than this.
I have no issues with with my sg4860 doing full speed routing between interfaces.
So this pfsense box - it has more than 1 physical interface.. You have 1 for wan and one for lan, and only on this lan interface do you have multiple vlans?
When I test between a pc on 1 network 192.168.2/24 to my nas on 192.168.9.10 I see pretty freaking close to full wire speed.
in both directions..
$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 Connecting to host 192.168.9.10, port 5201 [ 5] local 192.168.2.110 port 1048 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 102 MBytes 856 Mbits/sec [ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec [ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec [ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 5] 5.00-6.00 sec 113 MBytes 948 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec [ 5] 8.00-9.00 sec 113 MBytes 949 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec sender [ 5] 0.00-10.02 sec 1.09 GBytes 936 Mbits/sec receiver iperf Done. $ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 -R Connecting to host 192.168.9.10, port 5201 Reverse mode, remote host 192.168.9.10 is sending [ 5] local 192.168.2.110 port 1070 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 105 MBytes 884 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 935 Mbits/sec [ 5] 2.00-3.00 sec 108 MBytes 903 Mbits/sec [ 5] 3.00-4.00 sec 108 MBytes 908 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 947 Mbits/sec [ 5] 5.00-6.00 sec 106 MBytes 892 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 932 Mbits/sec [ 5] 8.00-9.00 sec 109 MBytes 912 Mbits/sec [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.07 GBytes 917 Mbits/sec 320 sender [ 5] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver iperf Done. -
@johnpoz said in Slow routing speeds:
@hngaminguk prob need someone like @stephenw10 to chime in here.. He is the hardware guru and good with this sort of stuff to figure out if say a hardware issue not able to do more than this.
I have no issues with with my sg4860 doing full speed routing between interfaces.
So this pfsense box - it has more than 1 physical interface.. You have 1 for wan and one for lan, and only on this lan interface do you have multiple vlans?
When I test between a pc on 1 network 192.168.2/24 to my nas on 192.168.9.10 I see pretty freaking close to full wire speed.
in both directions..
$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 Connecting to host 192.168.9.10, port 5201 [ 5] local 192.168.2.110 port 1048 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 102 MBytes 856 Mbits/sec [ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec [ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec [ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 5] 5.00-6.00 sec 113 MBytes 948 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec [ 5] 8.00-9.00 sec 113 MBytes 949 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec sender [ 5] 0.00-10.02 sec 1.09 GBytes 936 Mbits/sec receiver iperf Done. $ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 -R Connecting to host 192.168.9.10, port 5201 Reverse mode, remote host 192.168.9.10 is sending [ 5] local 192.168.2.110 port 1070 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 105 MBytes 884 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 935 Mbits/sec [ 5] 2.00-3.00 sec 108 MBytes 903 Mbits/sec [ 5] 3.00-4.00 sec 108 MBytes 908 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 947 Mbits/sec [ 5] 5.00-6.00 sec 106 MBytes 892 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 932 Mbits/sec [ 5] 8.00-9.00 sec 109 MBytes 912 Mbits/sec [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.07 GBytes 917 Mbits/sec 320 sender [ 5] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver iperf Done.Thanks for the diagnostic help so far! Yes I have 2 physical ports on an intel based network card. One is WAN one is LAN and 3 networks currently using the LAN side, 10.0.0.0/24 direct on LAN, 10.0.10.0/24 on LAN.10 and 10.0.11.0/24 on LAN.11
As mentioned previously the only actual device right now on the lab side is my NAS so the 10.0.11.0/24 network has no devices at all.
As mentioned in my original post when just doing switching I can get ~950Mbps (at that speed I'm calling it gigabit)
-
@hngaminguk Im not doing switching there - that is routed through my sg4860
@stephenw10 is the hardware guru around here..
You didn't setup any sort of limiting or shaping did you..
When I just switch I see a bit higher, I see a small slight hit routing it through pfsense.. But its minor and I really have no idea what else my nas is doing at the moment.. Someone could be streaming a movie off my plex ;)
While routing and firewalling normally will see a small hit, it sure shouldn't 250 from gig..
-
@johnpoz said in Slow routing speeds:
@hngaminguk Im not doing switching there - that is routed through my sg4860
@stephenw10 is the hardware guru around here..
You didn't setup any sort of limiting or shaping did you..
When I just switch I see a bit higher, I see a small slight hit routing it through pfsense.. But its minor and I really have no idea what else my nas is doing at the moment.. Someone could be streaming a movie off my plex ;)
While routing and firewalling normally will see a small hit, it sure shouldn't 250 from gig..
Okay thanks for the confirm, hopefully @stephenw10 will have some ideas.
Traffic shaping is completely off currently, I made doubly sure as it was one thing suggested on posts on reddit and here.
I agree it makes no sense as to why I am seeing that much of a performance degradation, other than the low spec of my pf device I am not sure as to another cause.
-
The original APU, that we sold as the VK-T40E, has a G-T40E CPU and Realtek NICs. That is good for ~350Mbps between two ports, or was last time I tested it.
The G-T40N appears to differ from this only in the intergrated GPU so I would expect it to perform almost identically.
Is this using a Realtek NIC? Is it single queue? (check the boot logs)If so ~300Mbps is probably not much lower than expected with a router on a stick config.
Steve
-
@hngaminguk said in Slow routing speeds:
hopefully @stephenw10 will have some ideas.
Told you he would ;)
-
@stephenw10 said in Slow routing speeds:
The original APU, that we sold as the VK-T40E, has a G-T40E CPU and Realtek NICs. That is good for ~350Mbps between two ports, or was last time I tested it.
The G-T40N appears to differ from this only in the intergrated GPU so I would expect it to perform almost identically.
Is this using a Realtek NIC? Is it single queue? (check the boot logs)If so ~300Mbps is probably not much lower than expected with a router on a stick config.
Steve
Thanks for the information Steve, when I ordered this device (eBay) it claimed to have an Intel card (I never opened the unit to confirm). But I have just done so now and I can confirm it has a PCI card (NC7170).
Looking at the card, the processor is an Intel one and looking on google it also confirms it's an Intel based card. So not sure if there is a different limitation in play here?
In relation to the boot logs I looked in the "OS Boot" section of the logs but not sure if that's the location you are referring to? Secondly I don't know what exactly I need to look for if that is the correct log section.
-
What are the NICs assigned as? em0, em1, igb0?
-
@stephenw10 said in Slow routing speeds:
What are the NICs assigned as? em0, em1, igb0?
WAN is em0
LAN is em1Hope that helps.
-
It does yes. OK so the router-on-a-stick part here is that you're testing between two VLANs both on the LAN NIC?
em is also single queue though so, whilst a lot better than re, you will still be limited.
The later APU2 devices had quad core CPUs and igb NICs and with that and some heavy tweaking it was possible to get close to 1Gbps. Here you are using effectively quarter of that in any one direction. It is interesting looking at your top output that almost all the use is on one of the queues, probably the receive queue.That NIC looks like is actually PCI-X. I haven't seen one of those is a very long time! If you can use a igb based NIC there you probably would see more throughput as it can use both CPU cores for transmit and receive. Not sure there are any PCI/PCI-X igb NICs though.
Ultimately you're unlikely to see more tan 500Mbps with that CPU.
Steve
-
@stephenw10 said in Slow routing speeds:
you're testing between two VLANs both on the LAN NIC?
I think he was testing through wan and lan (router on a stick on the lan).. So if I understand what he was testing he did end up testing through the 2 different nics, or alteast different ports on the same nic. If that makes any difference?
So what your saying in a nutshell - is not some setting he turn off to see a boost, and if he wants to see full wirespeed he needs better hardware.
-
@stephenw10 said in Slow routing speeds:
It does yes. OK so the router-on-a-stick part here is that you're testing between two VLANs both on the LAN NIC?
em is also single queue though so, whilst a lot better than re, you will still be limited.
The later APU2 devices had quad core CPUs and igb NICs and with that and some heavy tweaking it was possible to get close to 1Gbps. Here you are using effectively quarter of that in any one direction. It is interesting looking at your top output that almost all the use is on one of the queues, probably the receive queue.That NIC looks like is actually PCI-X. I haven't seen one of those is a very long time! If you can use a igb based NIC there you probably would see more throughput as it can use both CPU cores for transmit and receive. Not sure there are any PCI/PCI-X igb NICs though.
Ultimately you're unlikely to see more tan 500Mbps with that CPU.
Steve
Originally I was testing throughput from the UDM/home system to the pf/lab system which was also only getting ~250 - 300Mbps.
So even with an igb NIC (if I can get one) I am unlikely to see over 500Mbps routing through the WAN side and out the LAN as my original network diagram shows?
If that is the case to get full 1Gbps (or more) routing what CPU spec would be the minimum? Also out of the current netgate devices which one would be the best option? I may in the future want to do 10Gbps but if it costs more than £500 I would rather go for a lower option and upgrade down the line when 10Gbps is cheaper.
-
Mmm, I would expect to see at least 350Mbps there as the APU could do that even with Realtek NICs.
IIRC there were some determined users who fitted mPCIe igb NICs to it and got closer to 500Mbps. You will never see Gigabit 'wire speed' through that though.I used to have a test box with a Core2Duo E4500 and that could just do gigabit with em NICs. If you look at a synthetic benchmark you can see why the G-T40 won't:
https://www.cpubenchmark.net/compare/AMD-G-T40E-vs-Intel-Core2-Duo-E4500-vs-Intel-Atom-C3558/264vs936vs3129
That doesn't give the full story but as a basic guide it's useful.The C3558 is what we have in the 5100/6100/7100/.
Steve
-
@stephenw10 said in Slow routing speeds:
Mmm, I would expect to see at least 350Mbps there as the APU could do that even with Realtek NICs.
IIRC there were some determined users who fitted mPCIe igb NICs to it and got closer to 500Mbps. You will never see Gigabit 'wire speed' through that though.I used to have a test box with a Core2Duo E4500 and that could just do gigabit with em NICs. If you look at a synthetic benchmark you can see why the G-T40 won't:
https://www.cpubenchmark.net/compare/AMD-G-T40E-vs-Intel-Core2-Duo-E4500-vs-Intel-Atom-C3558/264vs936vs3129
That doesn't give the full story but as a basic guide it's useful.The C3558 is what we have in the 5100/6100/7100/.
Steve
Okay great thanks for the confirmation and the quick replies!
Looking at the CPU Mark scores alone I am not surprised it is only able to do ~300Mbps.
I will look into upgrading my router at some point then.
I will mark this as solved! Thanks @johnpoz & @stephenw10
