Slow routing speeds
-
@hngaminguk prob need someone like @stephenw10 to chime in here.. He is the hardware guru and good with this sort of stuff to figure out if say a hardware issue not able to do more than this.
I have no issues with with my sg4860 doing full speed routing between interfaces.
So this pfsense box - it has more than 1 physical interface.. You have 1 for wan and one for lan, and only on this lan interface do you have multiple vlans?
When I test between a pc on 1 network 192.168.2/24 to my nas on 192.168.9.10 I see pretty freaking close to full wire speed.
in both directions..
$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 Connecting to host 192.168.9.10, port 5201 [ 5] local 192.168.2.110 port 1048 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 102 MBytes 856 Mbits/sec [ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec [ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec [ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 5] 5.00-6.00 sec 113 MBytes 948 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec [ 5] 8.00-9.00 sec 113 MBytes 949 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec sender [ 5] 0.00-10.02 sec 1.09 GBytes 936 Mbits/sec receiver iperf Done. $ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 -R Connecting to host 192.168.9.10, port 5201 Reverse mode, remote host 192.168.9.10 is sending [ 5] local 192.168.2.110 port 1070 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 105 MBytes 884 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 935 Mbits/sec [ 5] 2.00-3.00 sec 108 MBytes 903 Mbits/sec [ 5] 3.00-4.00 sec 108 MBytes 908 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 947 Mbits/sec [ 5] 5.00-6.00 sec 106 MBytes 892 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 932 Mbits/sec [ 5] 8.00-9.00 sec 109 MBytes 912 Mbits/sec [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.07 GBytes 917 Mbits/sec 320 sender [ 5] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver iperf Done. -
@johnpoz said in Slow routing speeds:
@hngaminguk prob need someone like @stephenw10 to chime in here.. He is the hardware guru and good with this sort of stuff to figure out if say a hardware issue not able to do more than this.
I have no issues with with my sg4860 doing full speed routing between interfaces.
So this pfsense box - it has more than 1 physical interface.. You have 1 for wan and one for lan, and only on this lan interface do you have multiple vlans?
When I test between a pc on 1 network 192.168.2/24 to my nas on 192.168.9.10 I see pretty freaking close to full wire speed.
in both directions..
$ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 Connecting to host 192.168.9.10, port 5201 [ 5] local 192.168.2.110 port 1048 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 102 MBytes 856 Mbits/sec [ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec [ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec [ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 5] 5.00-6.00 sec 113 MBytes 948 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec [ 5] 8.00-9.00 sec 113 MBytes 949 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec sender [ 5] 0.00-10.02 sec 1.09 GBytes 936 Mbits/sec receiver iperf Done. $ iperf3.exe -c 192.168.9.10 -B 192.168.2.110 -R Connecting to host 192.168.9.10, port 5201 Reverse mode, remote host 192.168.9.10 is sending [ 5] local 192.168.2.110 port 1070 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 105 MBytes 884 Mbits/sec [ 5] 1.00-2.00 sec 111 MBytes 935 Mbits/sec [ 5] 2.00-3.00 sec 108 MBytes 903 Mbits/sec [ 5] 3.00-4.00 sec 108 MBytes 908 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 947 Mbits/sec [ 5] 5.00-6.00 sec 106 MBytes 892 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 932 Mbits/sec [ 5] 8.00-9.00 sec 109 MBytes 912 Mbits/sec [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.07 GBytes 917 Mbits/sec 320 sender [ 5] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver iperf Done.Thanks for the diagnostic help so far! Yes I have 2 physical ports on an intel based network card. One is WAN one is LAN and 3 networks currently using the LAN side, 10.0.0.0/24 direct on LAN, 10.0.10.0/24 on LAN.10 and 10.0.11.0/24 on LAN.11
As mentioned previously the only actual device right now on the lab side is my NAS so the 10.0.11.0/24 network has no devices at all.
As mentioned in my original post when just doing switching I can get ~950Mbps (at that speed I'm calling it gigabit)
-
@hngaminguk Im not doing switching there - that is routed through my sg4860
@stephenw10 is the hardware guru around here..
You didn't setup any sort of limiting or shaping did you..
When I just switch I see a bit higher, I see a small slight hit routing it through pfsense.. But its minor and I really have no idea what else my nas is doing at the moment.. Someone could be streaming a movie off my plex ;)
While routing and firewalling normally will see a small hit, it sure shouldn't 250 from gig..
-
@johnpoz said in Slow routing speeds:
@hngaminguk Im not doing switching there - that is routed through my sg4860
@stephenw10 is the hardware guru around here..
You didn't setup any sort of limiting or shaping did you..
When I just switch I see a bit higher, I see a small slight hit routing it through pfsense.. But its minor and I really have no idea what else my nas is doing at the moment.. Someone could be streaming a movie off my plex ;)
While routing and firewalling normally will see a small hit, it sure shouldn't 250 from gig..
Okay thanks for the confirm, hopefully @stephenw10 will have some ideas.
Traffic shaping is completely off currently, I made doubly sure as it was one thing suggested on posts on reddit and here.
I agree it makes no sense as to why I am seeing that much of a performance degradation, other than the low spec of my pf device I am not sure as to another cause.
-
The original APU, that we sold as the VK-T40E, has a G-T40E CPU and Realtek NICs. That is good for ~350Mbps between two ports, or was last time I tested it.
The G-T40N appears to differ from this only in the intergrated GPU so I would expect it to perform almost identically.
Is this using a Realtek NIC? Is it single queue? (check the boot logs)If so ~300Mbps is probably not much lower than expected with a router on a stick config.
Steve
-
@hngaminguk said in Slow routing speeds:
hopefully @stephenw10 will have some ideas.
Told you he would ;)
-
@stephenw10 said in Slow routing speeds:
The original APU, that we sold as the VK-T40E, has a G-T40E CPU and Realtek NICs. That is good for ~350Mbps between two ports, or was last time I tested it.
The G-T40N appears to differ from this only in the intergrated GPU so I would expect it to perform almost identically.
Is this using a Realtek NIC? Is it single queue? (check the boot logs)If so ~300Mbps is probably not much lower than expected with a router on a stick config.
Steve
Thanks for the information Steve, when I ordered this device (eBay) it claimed to have an Intel card (I never opened the unit to confirm). But I have just done so now and I can confirm it has a PCI card (NC7170).
Looking at the card, the processor is an Intel one and looking on google it also confirms it's an Intel based card. So not sure if there is a different limitation in play here?
In relation to the boot logs I looked in the "OS Boot" section of the logs but not sure if that's the location you are referring to? Secondly I don't know what exactly I need to look for if that is the correct log section.
-
What are the NICs assigned as? em0, em1, igb0?
-
@stephenw10 said in Slow routing speeds:
What are the NICs assigned as? em0, em1, igb0?
WAN is em0
LAN is em1Hope that helps.
-
It does yes. OK so the router-on-a-stick part here is that you're testing between two VLANs both on the LAN NIC?
em is also single queue though so, whilst a lot better than re, you will still be limited.
The later APU2 devices had quad core CPUs and igb NICs and with that and some heavy tweaking it was possible to get close to 1Gbps. Here you are using effectively quarter of that in any one direction. It is interesting looking at your top output that almost all the use is on one of the queues, probably the receive queue.That NIC looks like is actually PCI-X. I haven't seen one of those is a very long time! If you can use a igb based NIC there you probably would see more throughput as it can use both CPU cores for transmit and receive. Not sure there are any PCI/PCI-X igb NICs though.
Ultimately you're unlikely to see more tan 500Mbps with that CPU.
Steve
-
@stephenw10 said in Slow routing speeds:
you're testing between two VLANs both on the LAN NIC?
I think he was testing through wan and lan (router on a stick on the lan).. So if I understand what he was testing he did end up testing through the 2 different nics, or alteast different ports on the same nic. If that makes any difference?
So what your saying in a nutshell - is not some setting he turn off to see a boost, and if he wants to see full wirespeed he needs better hardware.
-
@stephenw10 said in Slow routing speeds:
It does yes. OK so the router-on-a-stick part here is that you're testing between two VLANs both on the LAN NIC?
em is also single queue though so, whilst a lot better than re, you will still be limited.
The later APU2 devices had quad core CPUs and igb NICs and with that and some heavy tweaking it was possible to get close to 1Gbps. Here you are using effectively quarter of that in any one direction. It is interesting looking at your top output that almost all the use is on one of the queues, probably the receive queue.That NIC looks like is actually PCI-X. I haven't seen one of those is a very long time! If you can use a igb based NIC there you probably would see more throughput as it can use both CPU cores for transmit and receive. Not sure there are any PCI/PCI-X igb NICs though.
Ultimately you're unlikely to see more tan 500Mbps with that CPU.
Steve
Originally I was testing throughput from the UDM/home system to the pf/lab system which was also only getting ~250 - 300Mbps.
So even with an igb NIC (if I can get one) I am unlikely to see over 500Mbps routing through the WAN side and out the LAN as my original network diagram shows?
If that is the case to get full 1Gbps (or more) routing what CPU spec would be the minimum? Also out of the current netgate devices which one would be the best option? I may in the future want to do 10Gbps but if it costs more than £500 I would rather go for a lower option and upgrade down the line when 10Gbps is cheaper.
-
Mmm, I would expect to see at least 350Mbps there as the APU could do that even with Realtek NICs.
IIRC there were some determined users who fitted mPCIe igb NICs to it and got closer to 500Mbps. You will never see Gigabit 'wire speed' through that though.I used to have a test box with a Core2Duo E4500 and that could just do gigabit with em NICs. If you look at a synthetic benchmark you can see why the G-T40 won't:
https://www.cpubenchmark.net/compare/AMD-G-T40E-vs-Intel-Core2-Duo-E4500-vs-Intel-Atom-C3558/264vs936vs3129
That doesn't give the full story but as a basic guide it's useful.The C3558 is what we have in the 5100/6100/7100/.
Steve
-
@stephenw10 said in Slow routing speeds:
Mmm, I would expect to see at least 350Mbps there as the APU could do that even with Realtek NICs.
IIRC there were some determined users who fitted mPCIe igb NICs to it and got closer to 500Mbps. You will never see Gigabit 'wire speed' through that though.I used to have a test box with a Core2Duo E4500 and that could just do gigabit with em NICs. If you look at a synthetic benchmark you can see why the G-T40 won't:
https://www.cpubenchmark.net/compare/AMD-G-T40E-vs-Intel-Core2-Duo-E4500-vs-Intel-Atom-C3558/264vs936vs3129
That doesn't give the full story but as a basic guide it's useful.The C3558 is what we have in the 5100/6100/7100/.
Steve
Okay great thanks for the confirmation and the quick replies!
Looking at the CPU Mark scores alone I am not surprised it is only able to do ~300Mbps.
I will look into upgrading my router at some point then.
I will mark this as solved! Thanks @johnpoz & @stephenw10
-
Marking this as solved, basically the processor in use is not enough processing power to handle full gigabit routing. My solution options are either a) Upgrade the network card to an igb NIC and complete some tweaking to maybe get 500Mbps or b) Upgrade the router as a whole to a better CPU.
Myself I am likely going for option 2, just need to decide if I am going for a netgate device or custom built.
-
@hngaminguk I may be a bit biased, but I love my sg4860.. And when its time to retire it I will most certainly be sticking with netgate box.
If mine blew up today - I would prob go with the 6100 base.. It would allow for 10 in the future and I could leverage the 2.5ge ports today.. I am not a fan of switch ports on a router..
And like my 4860 its overkill for my current internet speeds.. But never no what tomorrow might bring, etc.
-
@johnpoz said in Slow routing speeds:
@hngaminguk I may be a bit biased, but I love my sg4860.. And when its time to retire it I will most certainly be sticking with netgate box.
If mine blew up today - I would prob go with the 6100 base.. It would allow for 10 in the future and I could leverage the 2.5ge ports today.. I am not a fan of switch ports on a router..
And like my 4860 its overkill for my current internet speeds.. But never no what tomorrow might bring, etc.
Thanks for your input
With a max budget of £500 ($650 approx) I think the highest option I could go with is the 2100 since the 3100 is EOS.
Only annoyance being that the 2100 states a max of 881Mbps for Firewall (10k ACLs) I am not well versed into knowing how many ACLs I am using but I assume my setup currently has less? So I could likely hit 1Gbps?
-
@hngaminguk well to be honest, there was some threads recently about actual speeds not meeting those numbers via testing.. I don't recall exactly what model that was - might of been related to 10g, not sure..
-
@hngaminguk said in Slow routing speeds:
Only annoyance being that the 2100 states a max of 881Mbps for Firewall (10k ACLs) I am not well versed into knowing how many ACLs I am using but I assume my setup currently has less? So I could likely hit 1Gbps?
No, using fewer firewall rules will not allow it to hit 1Gbps between two subnets. Enabling pf to set any number of rules will introduce that overhead.
Steve
-
@stephenw10 said in Slow routing speeds:
@hngaminguk said in Slow routing speeds:
Only annoyance being that the 2100 states a max of 881Mbps for Firewall (10k ACLs) I am not well versed into knowing how many ACLs I am using but I assume my setup currently has less? So I could likely hit 1Gbps?
No, using fewer firewall rules will not allow it to hit 1Gbps between two subnets. Enabling pf to set any number of rules will introduce that overhead.
Steve
Okay thanks for the confirmation, in that case I will have to go for a 3rd party option such as https://www.ebay.co.uk/itm/Intel-Atom-E3845-4-LAN-3G-4G-4G-RAM-64G-SSD-Fanless-pfSense-Firewall-AES-NI-/114644549859?mkcid=16&mkevt=1&_trksid=p2349624.m46890.l49286&mkrid=710-127635-2958-0
