Slow throughput on SG-4860 (~600mbs on 1gbs line)



  • Hey folks - I recently upgraded to a SG-4860 to take advantage of a speed bump in my FiOS connection to their 1 gig service.

    After the upgrade I am maxing out at about 500-700 mbs …although I rarely get about 600.

    I was so sure it was the line or Verizon gateway that I scheduled a service visit before checking anything else. When we connected directly to the Verizon gateway (which is in bridge mode, BTW) it pulled a sold 950mbs. I say we, because I was able to replicate with a laptop directly connected.

    I've run speed tests using curl and iperf from the PF box with nothing else connected - so I'm reasonably sure it's not other devices sipping data while I'm testing.

    I have 3 site-to-site VPNs running too (the test doesn't go through any of them, FWIW). But I assume they create some processor overhead, even if there isn't much/any traffic flowing. My LAN has 4 vLANS which may also create some overhead...?

    On a related note - I have hardware cypto enabled on both ends, and the tunnels max out well below the line speed of either endpoint.

    I've also read it might be a single vs multi thread issue. I've tried running speed tests on multiple machines at the same time and the result is a sum total of about 600 mbps.

    It could be my cabling - I've swapped the WAN cable already. And I don't think it's the switches on the LAN side since I've connected directly to the SG-4860's LAN port and run the tests from the PF box itself.

    Anyone have any more tips for troubleshooting?


  • Galactic Empire

    Are these results from speedtest trough the tunnel? Sorry, it's not that clear to me.



  • @ivor:

    Are these results from speedtest trough the tunnel? Sorry, it's not that clear to me.

    Totally on me! I wasn't clear!

    ~600mbs is from the LAN side of the firewall and LAN clients straight to 'the internet'…
    I did the ole curl -o spedtest , Fast.com and several public iperf servers

    (through OVPN tunnel I max out at 50 mbs...again, another thread I need to start once I resolve this :) )


  • Galactic Empire

    Check MTU's on your ports, modem and switch. 4860 can do 1Gbps without any problems so it's MTU or cables perhaps.



  • @ivor:

    Check MTU's on your ports, modem and switch. 4860 can do 1Gbps without any problems so it's MTU or cables perhaps.

    played around with different MTUs tonight: 1450, 1400, 1350, 1300 and for good measure 9000 … no appreciable change in speed.

    I also swapped cables with some from work which were in a setup pulling 1gpbs symmetrically. So it's not the cables.

    I watched the processors during the tests and at least one pegs each time. I'm thinking perhaps the 4860 can do 1gpbs total, but not in a single thread?


  • Galactic Empire

    @SpaceBass:

    @ivor:

    Check MTU's on your ports, modem and switch. 4860 can do 1Gbps without any problems so it's MTU or cables perhaps.

    played around with different MTUs tonight: 1450, 1400, 1350, 1300 and for good measure 9000 … no appreciable change in speed.

    I also swapped cables with some from work which were in a setup pulling 1gpbs symmetrically. So it's not the cables.

    I watched the processors during the tests and at least one pegs each time. I'm thinking perhaps the 4860 can do 1gpbs total, but not in a single thread?

    When was the unit purchased? If it's recent, please submit a ticket to https://customercare.netgate.com/ and reference this thread. We'll figure out what's going on a lot faster.

    Thanks!


  • Rebel Alliance Netgate Administrator

    If you upgraded your config from an older machine to a newer one you might have to enable powerd to get your speeds.

    System -> Advanced -> Miscellaneous, enable PowerD, HiAdaptive



  • Are you having the SG-4860 terminate a PPPoE connection?  If so, that's your problem.  I get about the same performance on mine because of a FreeBSD bug.  There's more info at https://redmine.pfsense.org/issues/4821.



  • @chrismacmahon:

    If you upgraded your config from an older machine to a newer one you might have to enable powerd to get your speeds.

    System -> Advanced -> Miscellaneous, enable PowerD, HiAdaptive

    Thanks - I've tried both ways and unfortunately, PowerD seems to –if anything --slow things down further.



  • Are you using Snort?  if so turn it off and try again as Snort is single threaded.  Your performance with one snort instance is actually above average considering you have an Atom cpu in there.



  • @hescominsoon:

    Are you using Snort?  if so turn it off and try again as Snort is single threaded.  Your performance with one snort instance is actually above average considering you have an Atom cpu in there.

    no snort here :)



  • What happens when you plug the SG-4860 WAN interface directly into the ONT - do you still experience throughput issues?  From what I can gather form your first post, the pfSense box is actually downstream of the Verizon gateway - is there any particular reason why you decided to set it  up that way?

    Hope this helps.



  • @tman222:

    What happens when you plug the SG-4860 WAN interface directly into the ONT - do you still experience throughput issues?  From what I can gather form your first post, the pfSense box is actually downstream of the Verizon gateway - is there any particular reason why you decided to set it  up that way?

    Hope this helps.

    You might be on to something. Admittedly it’s years since I could claim even a working understanding of how VZ was implementing FiOS. Back then I’d get a huge ONT on the side of my house.

    In this install they ran a really super slim single strand fiber from the pole straight through my exterior wall and into a Verizon router. That router has an AD/DC supply zip tied to it. Otherwise there’s no ONT that I can see.

    Should I get a fiber to ether exchanger for my PF box? I have a jar full of  gbics I haven’t used in years ;)



  • I think there might be some confusion as to what we are calling the Verizon Gateway and ONT.  The ONT includes the device (looks like a modem) that takes fiber as input and provides ethernet/coax outputs.  The Verizon Gateway (at least to me) is the router/firewall Verizon provides - with gigabit I assume they would have given you the black G1100 (Fios Quantum Gateway) unless you decided to keep the (red colored) Actiontec.  Do you have your pfSense box currently plugged into the ONT directly or into the Verizon Gateway (router/firewall)?

    Hope this helps.



  • @SpaceBass:

    @ivor:

    Are these results from speedtest trough the tunnel? Sorry, it's not that clear to me.

    Totally on me! I wasn't clear!

    ~600mbs is from the LAN side of the firewall and LAN clients straight to 'the internet'…
    I did the ole curl -o spedtest , Fast.com and several public iperf servers

    (through OVPN tunnel I max out at 50 mbs...again, another thread I need to start once I resolve this :) )

    Keep in mind FIOS is rated UP TO @940 megabits even though they advertise gigabit.  It may not be anything with your gear you might jsut be hitting a bandwidth limitation in your aarea.  I ahve a freind that subs to VZ Gb and he rarely sees above 800 himself.



  • 940 Mbps happens to be about what most people will see with a single 1 Gbps interface, after taking into account protocol overhead, so they're still pushing 1 Gbps through the line.

    Of course, the caveat is that they're providing "Up to 940 Mbps"… if you're in-depth on the technology behind FiOS, it is actually shared connectivity between you and possibly up to 31 other neighbors, so if a couple of your neighbors also have gigabit service and have major downloads going on, that could actually prevent you from getting full speed.



  • The poster did mention that he was able to achieve a higher speed when connected directly to the Verizon gateway (back in the first post), so I'm not convinced it's an issue with the connection.

    I'm still curious though as to how exactly this is setup:  Is the pfSense box hooked into the gateway and gateway into the ONT, or is the pfSense box hooked directly into the ONT?  I would recommend connecting the pfSense box into the ONT and then using the Verizon gateway downstream of the pfSense box (if it's required to be in use, e.g. to provide internet connectivity to STB's).

    Hope this helps.



  • Having the exact same issue. When my computer is plugged straight into the ONT, I get 940 Mbps each way every time. When downstream of the SG-4860 running a 2.4-RC snapshot (Oct 4), download tops out at 400-500 Mbps. Upload can usually hit 700 Mbps, strangely.

    This is a fresh install, so no old config slowing it down.

    Let me know how I can help triage this. Thanks.



  • Fixed it- was necessary to enable PowerD, and set to Hiadaptive or Maximum (slightly faster). Hey, maybe this thing should be enabled by default, huh?



  • @tman222:

    I'm still curious though as to how exactly this is setup:  Is the pfSense box hooked into the gateway and gateway into the ONT, or is the pfSense box hooked directly into the ONT?  I would recommend connecting the pfSense box into the ONT and then using the Verizon gateway downstream of the pfSense box (if it's required to be in use, e.g. to provide internet connectivity to STB's).

    Hope this helps.

    VZ box (where fiber terminates) –cat6-->PF Wan port

    I know it sounds confusing because I've seen different FiOS set ups, but in this install, fiber runs from the pole, through a hole in my wall all the way to the Vz box (let's just call it the VZ Box). The VZ box has a locked port where the fiber terminates. And it has a switch built in. One switch port is connected to my PF box.

    Wifi and routing are disabled on the VZ box (it's in bridge mode).

    We don't have phone or TV through Verizon - internet only. No STBs in the mix.



  • Are you sure the computer on the lan is capable of 1gb speeds?

    Its far from a given.



  • @kejianshi:

    Are you sure the computer on the lan is capable of 1gb speeds?

    Its far from a given.

    That's a valid concern - I've tried a few machines including a current (ha!!!) MacPro, a 2017 MacBook Pro (15" with USB-C GigE adaptor), a 2015 iMac and my main workstation, an i7 4770k with 32gb of ram and dual gig NICs running Ubuntu 17.04 … thank goodness for super long ethernet cables! (which I've also tested by using shorter ones and newer ones).

    But here's the real proof - those machines get ~900mbs when wired directly to the VZ Box.



  • I'd say take your Mac and plug it into the LAN port of your pfsense directly with a very short cable and run the test again.

    If its still bad, save the config, install the previous version and restore the config.  Test again.

    If its not fixed, its a nic problem or isp problem.  Or maybe a Mac problem.

    Otherwise, a pfsense problem if a rollback fixes it.



  • @SpaceBass:

    @tman222:

    I'm still curious though as to how exactly this is setup:  Is the pfSense box hooked into the gateway and gateway into the ONT, or is the pfSense box hooked directly into the ONT?  I would recommend connecting the pfSense box into the ONT and then using the Verizon gateway downstream of the pfSense box (if it's required to be in use, e.g. to provide internet connectivity to STB's).

    Hope this helps.

    VZ box (where fiber terminates) –cat6-->PF Wan port

    I know it sounds confusing because I've seen different FiOS set ups, but in this install, fiber runs from the pole, through a hole in my wall all the way to the Vz box (let's just call it the VZ Box). The VZ box has a locked port where the fiber terminates. And it has a switch built in. One switch port is connected to my PF box.

    Wifi and routing are disabled on the VZ box (it's in bridge mode).

    We don't have phone or TV through Verizon - internet only. No STBs in the mix.

    This is an interesting configuration that I have not come across before.  Are you saying that the fiber terminates into a box/device that does Firewalling/WiFi/NAT routing as well?  Usually what I've seen the fiber tends to terminate into the ONT where the outputs are ethernet, coax, and phone jack(s) and the routing is done by the Verizon supplied router/gateway (which connects via coax/ethernet back to the ONT).    Are you implying that in your case the ONT and router/gateway are all just one box/device where the fiber connects to?

    Thanks in advance for the clarification.



  • I think he is describing fiber terminating at an ONT with ethernet to pfsense.  Thats the best way to do it.



  • @kejianshi:

    I think he is describing fiber terminating at an ONT with ethernet to pfsense.  Thats the best way to do it.

    Gotcha - I guess the part about putting the Vz Box in bridge mode is what threw me off since the ONT usually does not do any routing/wifi.



  • If he is putting a VZ box in bridge mode then he is most likely not connected directly to the ONT.  Rather he is connected to the VZ router which has a MOCA connection back to the ONT.

    My bad.  Still, direct connect to the ONT is best.

    Not really sure what verizon was thinking when they decided to go with that architecture.



  • @kejianshi:

    I think he is describing fiber terminating at an ONT with ethernet to pfsense.  Thats the best way to do it.

    Looks like what I have (and all I have) from VZ is the current ONT. Sorry if I've confused everyone with terms or lack there of :)




  • Ah ok, so it looks like the ONT is connected directly to the WAN port on pfSense (i.e. no Verizon router/gateway inbetween).

    In trying to isolate the problem I assume you have already tried to do am iperf/iperf3 test between your laptop and the pfSense box to make sure that the LAN interface(s) on the pfSense boxes are able to push out 1 gbit of traffic to downstream devices?  Also, have you looked at your network statistics?  Are you seeing any dropped packets and/or errors?

    One other thing I might recommend is trying to tune your network a little bit since you have a very high speed connection.

    https://forum.pfsense.org/index.php?topic=113496.0
    https://calomel.org/freebsd_network_tuning.html

    Hope this helps.



  • @tman222:

    assume you have already tried to do am iperf/iperf3 test between your laptop and the pfSense box to make sure that the LAN interface(s) on the pfSense boxes are able to push out 1 gbit of traffic to downstream devices?
    Hope this helps.

    you know…. for all the iperf tests I've done (VPNs, server to server, host to server, etc) I dont think I've done one from a workstation to pfSense... what a simple thing and silly thing to over look! Thank you!

    (now I just have to wait :) lost my SG-4860 to a [potential] HW failure. As soon as I have one back in commission, I'll test that).

    Thank you!!



  • In a connection where you have A linked to the WAN of B and the lan of B connected to the wan of C,

    an iperf test from A to C will tell you the most about the health of B, which is hopefully the pfsense.

    Nice picture.  You couldn't take a bigger one?  (kidding)



  • With my SG-4860 back and running, i wanted to revisit this thread.

    Between a wired workstation (over a switch) to my PF box, i get a little north of 1gbps in each direction. So the bottleneck isn't there.

    I've also learned some things about IPerf and traffic and threading (which is to say I learned very little but knew next to nothing when I started).

    If I run a multithreaded test with a larger transfer size I get marginally better performance. But still not 1gbps

    iperf -c iperf.he.net -w 1MB -P 5
    ------------------------------------------------------------
    Client connecting to iperf.he.net, TCP port 5001
    TCP window size: 1.00 MByte (WARNING: requested 1.00 MByte)
    ------------------------------------------------------------
    [  9] local 10.15.1.115 port 52292 connected with 216.218.227.10 port 5001
    [  8] local 10.15.1.115 port 52291 connected with 216.218.227.10 port 5001
    [  5] local 10.15.1.115 port 52288 connected with 216.218.227.10 port 5001
    [  7] local 10.15.1.115 port 52289 connected with 216.218.227.10 port 5001
    [  6] local 10.15.1.115 port 52290 connected with 216.218.227.10 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  9]  0.0-10.0 sec   150 MBytes   126 Mbits/sec
    [  8]  0.0-10.0 sec   151 MBytes   127 Mbits/sec
    [  6]  0.0-10.0 sec   150 MBytes   126 Mbits/sec
    [  5]  0.0-10.0 sec   150 MBytes   125 Mbits/sec
    [  7]  0.0-10.0 sec  79.1 MBytes  66.3 Mbits/sec
    [SUM]  0.0-10.0 sec   680 MBytes   570 Mbits/sec
    


  • It might be able that more the one issue will be invited in that case.

    First of all PPPoE usage of the Internet account, this is then more single CPU core threaded and 
    nothing can help you out with it either another more powerful or strong CPU.

    The gateway device from the ISP if it will be not a real or pure modem, but more a router will be also
    doing NAT at the WAN port, and then you will be also able to ask you ISP what to do or how to solve
    this problem. Perhaps he will send you another "box" that is doing not NAT or is acting as a pure modem.

    The SG-4860 unit is doing for someone called @gonzopancho at a 1Gbit/s symmetric Internet connection
    nearly or around ~900 MBit/s and together with the TCP/IP overhead and passing firewall rules through it,
    it might be the nearly 1 GBit/s real throughput.

    If you now gets only 500 - 600 MBit/s I personally would do a fresh and full install without any additional packets
    and have a look what is going on then. The ADI 2.4.1 serial Image is able to download and pretty new, so it could be
    if you are using VLANs at the WAN port that the version 2.4.0 will be a better option at this time for you.

    Can you set this up for us and your self? Before doing so please do a configuration backup but don´t play it back
    on the fresh installation to prevent a typo or earlier set up false in the test. Normally it will take 30 minutes for you.

    And then you will be able to see what is going on in real. Would be my way in that case.


Log in to reply