No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..



  • So actually I have this same issue, EXCEPT that I'm on a full 1Gig connection (symmetrical) in my Data Center.

    No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..

    If I remove pfsense from the equation I get a solid 982Mb up/down. (connecting laptop directly to the ethernet line provided by the Data Center.

    Originally I was thinking this was due to Suricata/Snort so pulled them from the mix (and while it did improve the speed, it was only by 1-2Mb difference (which makes me happy actually that it was not causing that much of a hit). So I rolled everything back to basics (reverted all tunables, uninstalled all packages.. and no matter what, my download is still roughly half of my upload via pfsense.. Related to this I would like to find out why I'm maxing out at 600Mb (using the same processor as netgate's XG-7100 1U server with 8GB of RAM... CPU is hit hard when doing my tests (hits about 40-60% CPU load).


  • Netgate Administrator

    You have any traffic shaping in play?

    Any packages installed?

    Check Status > Interfaces are they all synced at 1Gb? Any errors showing?

    40-60% cpu load could be two cores at 100%. Try running top -aSH at the CLI to see how that usage is broken down.

    Steve



  • @stephenw10 Nope.. I eliminated all the packages and removed all traffic shaping.. So its pretty much a basic pfsense box.

    All the interfaces are synced at 1Gb (actually Fixed Speed/Duplex since auto-negotiate sometimes causes issues).

    This is what things look like when idle (sending about 1-2Mb of data from some residual streams):

    last pid: 70132;  load averages:  0.11,  0.09,  0.08                                                                                                    up 0+11:42:20  13:24:17
    200 processes: 5 running, 156 sleeping, 39 waiting
    CPU:  0.1% user,  0.0% nice,  0.7% system,  1.0% interrupt, 98.2% idle
    Mem: 17M Active, 113M Inact, 338M Wired, 27M Buf, 7408M Free
    Swap: 16G Total, 16G Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
       11 root     155 ki31     0K    64K CPU2    2 690:14  99.02% [idle{idle: cpu2}]
       11 root     155 ki31     0K    64K RUN     3 688:31  98.55% [idle{idle: cpu3}]
       11 root     155 ki31     0K    64K CPU1    1 684:06  97.76% [idle{idle: cpu1}]
       11 root     155 ki31     0K    64K CPU0    0 673:14  95.98% [idle{idle: cpu0}]
       12 root     -92    -     0K   624K WAIT    0  18:11   2.99% [intr{irq261: igb1:que 0}]
       12 root     -92    -     0K   624K WAIT    1   6:43   1.11% [intr{irq257: igb0:que 1}]
    29579 root      20    0 10196K  5648K select  2   6:51   0.71% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
    48075 root      20    0 10196K  5620K select  0   4:06   0.64% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
       12 root     -60    -     0K   624K WAIT    1   2:49   0.39% [intr{swi4: clock (0)}]
       12 root     -92    -     0K   624K WAIT    0   2:11   0.28% [intr{irq256: igb0:que 0}]
    73588 root      52  -10  9588K  5404K select  2   1:02   0.13% /usr/local/sbin/tincd --config=/usr/local/etc/tinc
    64525 root      20    0  7812K  4204K CPU3    3   0:00   0.13% top -aSH
    48682 root      20    0  6600K  2380K bpf     3   0:59   0.12% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
       20 root     -16    -     0K    16K -       3   0:33   0.09% [rand_harvestq]
    63955 root      20    0  6404K  2560K select  2   0:33   0.07% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
       12 root     -92    -     0K   624K WAIT    2   1:08   0.07% [intr{irq263: igb1:que 2}]
    80585 root      20    0  6900K  2456K nanslp  1   0:07   0.05% [dpinger{dpinger}]
    81868 root      20    0  6900K  2456K nanslp  3   0:07   0.05% [dpinger{dpinger}]
       12 root     -92    -     0K   624K WAIT    1   0:24   0.03% [intr{irq262: igb1:que 1}]
       19 root     -16    -     0K    16K pftm    0   0:53   0.03% [pf purge]
    81494 root      20    0  6900K  2456K nanslp  1   0:07   0.03% [dpinger{dpinger}]
    81182 root      20    0  6900K  2456K nanslp  3   0:07   0.03% [dpinger{dpinger}]
    23476 root      20    0 12396K 12500K select  2   0:11   0.02% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}
    56889 unbound   20    0 52768K 25640K kqread  3   0:04   0.02% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
       12 root     -92    -     0K   624K WAIT    3   0:47   0.02% [intr{irq259: igb0:que 3}]
       15 root     -68    -     0K    80K -       1   0:02   0.01% [usb{usbus0}]
    36417 root      20    0 12904K  8148K select  1   0:00   0.01% sshd: admin@pts/0 (sshd)
       12 root     -92    -     0K   624K WAIT    2   0:35   0.01% [intr{irq258: igb0:que 2}]
       12 root     -92    -     0K   624K WAIT    3   0:24   0.01% [intr{irq264: igb1:que 3}]
       12 root     -88    -     0K   624K WAIT    0   0:01   0.01% [intr{irq23: ehci0}]
       15 root     -68    -     0K    80K -       0   0:02   0.01% [usb{usbus0}]
       12 root     -72    -     0K   624K WAIT    3   0:02   0.01% [intr{swi1: netisr 0}]
    81868 root      20    0  6900K  2456K sbwait  1   0:02   0.01% [dpinger{dpinger}]
    80585 root      20    0  6900K  2456K sbwait  3   0:02   0.00% [dpinger{dpinger}]
    81182 root      20    0  6900K  2456K sbwait  3   0:01   0.00% [dpinger{dpinger}]
    81494 root      20    0  6900K  2456K sbwait  3   0:02   0.00% [dpinger{dpinger}]
    81868 root      20    0  6900K  2456K nanslp  3   0:02   0.00% [dpinger{dpinger}]
    81494 root      20    0  6900K  2456K nanslp  0   0:02   0.00% [dpinger{dpinger}]
       21 root     -16    -     0K    48K psleep  3   0:02   0.00% [pagedaemon{dom0}]
    56889 unbound   20    0 52768K 25640K kqread  1   0:03   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
       25 root      20    -     0K    32K sdflus  2   0:02   0.00% [bufdaemon{/ worker}]
    81182 root      20    0  6900K  2456K nanslp  2   0:02   0.00% [dpinger{dpinger}]
    80585 root      20    0  6900K  2456K nanslp  3   0:02   0.00% [dpinger{dpinger}]
        0 root     -16    -     0K   624K swapin  0   1:09   0.00% [kernel{swapper}]
    

    And this is how things look when I'm doing a speed small test (from 1-2% usage to 20-30%. When I do more data tests from multiple clients, then this goes up. but this is based on a single client doing a speed test):

    last pid: 31858;  load averages:  0.22,  0.24,  0.16                                                                                                    up 0+11:48:39  13:30:36
    200 processes: 5 running, 156 sleeping, 39 waiting
    CPU:  1.7% user,  0.0% nice,  1.7% system,  9.1% interrupt, 87.5% idle
    Mem: 19M Active, 113M Inact, 338M Wired, 27M Buf, 7407M Free
    Swap: 16G Total, 16G Free
    
    Message from syslogd@MCEFW at Nov  7 13:26:57 ... TIME    WCPU COMMAND
    MCEFW php-fpm[340]: /index.php: SuccesCPU3    3 694:27  89.60min' from: 10.10.30.253 (Local Database)
       11 root     155 ki31     0K    64K RUN     0 678:56  87.13% [idle{idle: cpu0}]
       11 root     155 ki31     0K    64K CPU2    2 696:16  86.87% [idle{idle: cpu2}]
       11 root     155 ki31     0K    64K CPU1    1 689:57  84.43% [idle{idle: cpu1}]
       12 root     -92    -     0K   624K WAIT    2   1:14  10.67% [intr{irq263: igb1:que 2}]
       12 root     -92    -     0K   624K WAIT    1   0:35  10.32% [intr{irq262: igb1:que 1}]
       12 root     -92    -     0K   624K WAIT    0  18:28   12.28% [intr{irq261: igb1:que 0}]
       12 root     -92    -     0K   624K WAIT    3   0:31   5.05% [intr{irq264: igb1:que 3}]
       12 root     -92    -     0K   624K WAIT    3   0:53   2.70% [intr{irq259: igb0:que 3}]
       12 root     -92    -     0K   624K WAIT    1   6:51   2.14% [intr{irq257: igb0:que 1}]
      340 root      20    0 89064K 35676K accept  2   0:11   1.48% php-fpm: pool nginx (php-fpm)
       12 root     -92    -     0K   624K WAIT    0   2:23   1.37% [intr{irq256: igb0:que 0}]
      341 root      52    0 88936K 34364K accept  3   0:07   1.25% php-fpm: pool nginx (php-fpm)
    29579 root      20    0 10196K  5648K select  3   6:54   1.16% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
    11255 root      52    0 88936K 33928K accept  0   0:10   0.75% php-fpm: pool nginx (php-fpm)
    48075 root      20    0 10196K  5620K select  3   4:09   0.48% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
       12 root     -60    -     0K   624K WAIT    2   2:50   0.36% [intr{swi4: clock (0)}]
       20 root     -16    -     0K    16K -       1   0:33   0.22% [rand_harvestq]
    48682 root      20    0  6600K  2380K bpf     2   1:00   0.21% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
    73588 root      52  -10  9588K  5404K select  1   1:03   0.17% /usr/local/sbin/tincd --config=/usr/local/etc/tinc
    63955 root      20    0  6404K  2560K select  3   0:33   0.16% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
       12 root     -92    -     0K   624K WAIT    2   0:37   0.14% [intr{irq258: igb0:que 2}]
    64525 root      20    0  7812K  4220K CPU0    0   0:01   0.13% top -aSH
    21883 root      20    0 23592K  9080K kqread  0   0:00   0.12% nginx: worker process (nginx)
    

    Still very puzzled as to why it's so slow.



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    Nope.. I eliminated all the packages and removed all traffic shaping.. So its pretty much a basic pfsense box.

    Is this a box that you can physically factory reset? You say above, you had those packages installed, but now they are removed. I believe I have heard it mentioned here several times that some bits and pieces and settings and preferences from some of these packages sometimes are still remaining in the system.

    So, if you can, I would factory reset, then start throwing the tests back at it and see what you get.

    Jeff



  • @akuma1x Well, I realized that not everything was cleared out (example my firewall rules still had references to the Traffic shaping queues) so I factory reset it as well.. (after a backup and then restored it) to make sure everything was back sans the data.

    And still having the same issue.

    Download is pretty much half of the upload (even though its a symmetrical pipe) and neither is anywhere close to my SLA Speed of 1G. and if I bypass pfsense and just my Mac or a Windows laptop, I get full line speed.



  • @MrSassinak

    No, I mean completely factory reset it. Don't install a backup, just run it plain vanilla from scratch.

    Do the basic first-run setup wizard screens, then do your testing.

    Jeff



  • @akuma1x I reset everything back to factory (fresh software load.. all my data gone.. sniff) and still have the same problem.


  • Netgate Administrator

    @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    All the interfaces are synced at 1Gb (actually Fixed Speed/Duplex since auto-negotiate sometimes causes issues).

    You should never have to set Gigabit Ethernet at fixed speed/duplex. Are you still doing that after the reset?
    Try putting a switch in between the modem and firewall and leave the interfaces at autoselect.

    Steve


  • LAYER 8 Global Moderator

    Yeah if your hard coding gig, your doing it wrong!! And you have something wrong that should be looked at..



  • @stephenw10 I don't HAVE to set the speed/duplex.. I just do it to eliminate it as a variable. Its been running autonegotiation forever (which it finds the speed correctly). But when there are speed issues, the first thing is always to reduce/eliminate as many variables as possible for the cause. (its why we dumbed the installation down to pretty much factory spec).


  • LAYER 8 Global Moderator

    Setting speed and duplex on copper gig would not a variable you should ever mess with.. It should always be left at auto.



  • @johnpoz While I do agree in principal, historically, I've had issues sometimes with Cisco and Broadcom core switches (not with THIS system mind you, but with others and eliminating autoneg got rid of those problems).

    But as I'm at the end of my rope here, I'm willing to try just about anything.. (this problem has bene going on for a while, but as our data needs are ramping up, its now becoming more of an issue since we can't take advantage our SLA speed.



  • What happens many times when you try to force speed or duplex in Gigabit circuits is the other end of the conversation gets confused because Gigabit links want (just about demand, actually) to be set to auto-negotiate. If one end gets confused about speed or duplex (especially duplex), speeds will suffer tremendously.

    I know where you are coming from by using prior bad experiences with auto-negotiate and thus wanting to hard-code, but that is strongly discouraged in Gigabit land. You can sometimes get away with it in 10/100 setups, but even there these days most everything expects auto-negotiate on copper.


  • LAYER 8 Global Moderator

    Its not principle... its part of the standard..

    Clause 40 (1000BASE-T) makes special use of Auto-Negotiation and requires additional MII registers. This use is summarized below. Details are provided in 40.5.

    Auto-Negotiation is mandatory for 1000BASE-T (see 40.5.1).
    1000BASE-T requires an ordered exchange of Next Page messages (see 40.5.1.2), or optionally an exchange of an Extended Next Page message
    1000BASE-T parameters are configured based on information provided by the exchange of Next Page messages.
    1000BASE-T uses MASTER and SLAVE to define PHY operations and to facilitate the timing of transmit and receive operations. Auto-Negotiation is used to provide information used to configure MASTER-SLAVE status (see 40.5.2).
    1000BASE-T transmits and receives Next Pages for exchange of information related to MASTER-SLAVE operation. The information is specified in MII registers 9 and 10 (see 32.5.2 and 40.5.1.1), which are required in addition to registers 0-8 as defined in 28.2.4.
    1000BASE-T adds new message codes to be transmitted during Auto-Negotiation (see 40.5.1.3).
    1000BASE-T adds 1000BASE-T full duplex and half duplex capabilities to the priority resolution table (see 28B.3) and MII Extended Status Register (see 22.2.2.4).
    1000BASE-T is defined as a valid value for “x” in 28.3.1 (e.g., link_status_1GigT.) 1GigT represents that the 1000BASE-T PMA is the signal source.
    

    If your hard setting it - this for sure could cause you issues!

    If you have issues with devices auto doing gig - then you need to figure out why that is to get gig.. Not hard set it.



  • Folks, while I appreciate this is a hot topic, I think we are moving away from the central issue. autoneg or not, the problem of speed still exists (setting it fixed or autoneg, has zero impact on the issue).

    As I mentioned earlier, I have had autoneg on for quite a long time but as our data needs are ramping up, its now becoming more of an issue since we can't take advantage our SLA speed hence in the interest of eliminating this as a variable (I don't control the data center upstream switch but I have confirmed with them channel speed, and optimum MTU size).



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    Folks, while I appreciate this is a hot topic, I think we are moving away from the central issue. autoneg or not, the problem of speed still exists (setting it fixed or autoneg, has zero impact on the issue).

    As I mentioned earlier, I have had autoneg on for quite a long time but as our data needs are ramping up, its now becoming more of an issue since we can't take advantage our SLA speed hence in the interest of eliminating this as a variable (I don't control the data center upstream switch but I have confirmed with them channel speed, and optimum MTU size).

    You said in an earlier post you had it hard-coded. We assumed you still did.

    Actually, reading through again, you said two different things. You said in one post you had it set to hard-coded. Then later you said it has been set to auto-negotiation forever. Which is it?


  • LAYER 8 Global Moderator

    You can not actually troubleshoot a speed issue if your hard coding gig.. You can not - because now you have thrown in a known problem that could be the cause of the problem..

    You need to forget the old days.. The only reason you would hard code a gig interface, is your forcing to use a lower speed than gig.



  • Also, have you tried swapping NICs if that is possible? Or swapping which NIC port is in use. Not out of the realm of possibilty that you have a physical hardware issue with a NIC or its port. Don't forget the obvious such as the cable on the pfSense WAN and LAN connection. I once had a speed/connectivity issue caused by a slightly bent gold finger contact inside an RJ45 port on a Dell motherboard. I was the second-level network support guy and my field premises tech was at the site. We had fiddled with the Cisco switch port, tried a different port, pulled out the RJ45 jack from the wall and re-punched the terminations, swapped the patch cables both in the wiring closet and at the PC. Nada. Then my field guy just happened to be peering at the right spot on the rear of the PC and saw the bent pin inside the RJ45 port on the motherboard. Swapped the motherboard and problem solved.



  • @bmeeks Historically it has been set to autonegotiation. Since the speed has become a problem, in my effort to uncover what is the root cause, I've made several changes/modifications/tests (in sequence):

    • Disabling traffic shaping
    • Uninstalling all packages
    • Namely, rolling back all tunables
    • Setting Speed-Duplex to be fixed (this once it showed it had no impact, it was set back to be autospeed duplex. I think this is the part that confused people.. it was not left on, but it was set, tested, then reset back to auto)
    • And then reinstalling pfsense and then rolling back to previous configuration.

    Then per akuma1x, I reset it back to factory (ie: out of the box with ZERO changes other than what's stock from the installation).

    So right now, I have a clean zero configuration pfsense 2.4.4p3 install running on a Intel C3558 CPU with 8GB of RAM and 128GSSD. (nics are igb), and I am still stuck with 200Mb down and 400Mb up on a 1Gb Connection.



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    @bmeeks Historically it has been set to autonegotiation. Since the speed has become a problem, in my effort to uncover what is the root cause, I've made several changes/modifications/tests (in sequence):

    • Disabling traffic shaping
    • Uninstalling all packages
    • Namely, rolling back all tunables
    • Setting Speed-Duplex to be fixed (this once it showed it had no impact, it was set back to be autospeed duplex. I think this is the part that confused people.. it was not left on, but it was set, tested, then reset back to auto)
    • And then reinstalling pfsense and then rolling back to previous configuration.

    Then per akuma1x, I reset it back to factory (ie: out of the box with ZERO changes other than what's stock from the installation).

    So right now, I have a clean zero configuration pfsense 2.4.4p3 install running on a Intel C3558 CPU with 8GB of RAM and 128GSSD. (nics are igb), and I am still stuck with 200Mb down and 400Mb up on a 1Gb Connection.

    Look over my post immediately above this one. Don't discount that you may have a hardware issue somewhere.



  • @bmeeks I did try that as well.. I can't swap NICS themselves (part of the MB itself), but I did move from igb0 to igb1 (basically swapped the default LAN and WAN interfaces I used) just as a test (before the factory reset), as well as get the DC guys to test out their line to me..

    I'm pretty sure I'm going to need to bring some peace offerings of the liquid or edible sort next time, since I've been asking them to double check all their connections, speed, configuration, isolation, and even run a new line to me.



  • Do you have a way to run something like an iperf client/server setup so you can test each pfSense interface to a local host? That would let you see if the issue is within the pfSense box or outside.

    For example, test from a laptop or machine directly connected to the LAN to an iperf instance on pfSense. Then do the same on the WAN port (or even better, though pfSense itself from LAN to WAN or vice-versa).

    pfSense itself can most definitely do Gigabit without a sweat. So if iperf indicates the problem is within the pfSense setup, my bet is hardware because the software is known to be capable of gigabit transfers with ease.



  • @bmeeks Let me also add one other historical note..

    Previously before moving to this hardware set, we were running a D525 1.8Ghz Atoms with 4GB Ram and 128SSD (I know.. way underpowered) but we were getting symmetrical 400Mb. (they used the intel em driver, not the igb driver). What prompted the change was ramping up the speed (and since we were rolling out VPN, needed AES-NI for more efficient connections and speeds).

    At the time, I assumed the reason we could never crack above 400Mb was the system (D525's are fine for minor home use, but a heavily used 1Gb connection with TINC, IPsec, and OpenVPN connections and several outbound streaming services...that was just not going to cut it as a pfsense box). But to date, I have yet to crack the 400Mb barrier with pfsense.

    I have not yet tried another system (mostly because I have a preference for pfsense (been using it commercially for a long time (since the 1.x days) and before that m0n0wall) so I'm mostly familiar with it.. and I find it to be faster than most linux derived systems (otherwise untangle would be on my radar). Love linux as a server (we have a lot), but for performance, I find BSD based systems to be better. I know my Cisco router can do the full 1G, as well as Ubiquiti Edge Router as well as Vyatta. (I tested with all of them before in this DC before moving our kit here..so I know the speed itself is good.. just have to figure out why my beloved pfsense is not doing its job.)



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    All the interfaces are synced at 1Gb (actually Fixed Speed/Duplex since auto-negotiate sometimes causes issues).

    If you do that, you have to do it at both ends, otherwise you will likely have problem. Whether fixed 1 Gb or auto, you must have the same configuration at both ends of the cable. Generally speaking, you should use auto, unless you have some need to use fixed. One example would be if you're using fibre, where the SFP tends to be fixed.



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    @bmeeks Let me also add one other historical note..

    Previously before moving to this hardware set, we were running a D525 1.8Ghz Atoms with 4GB Ram and 128SSD (I know.. way underpowered) but we were getting symmetrical 400Mb. (they used the intel em driver, not the igb driver). What prompted the change was ramping up the speed (and since we were rolling out VPN, needed AES-NI for more efficient connections and speeds).

    At the time, I assumed the reason we could never crack above 400Mb was the system (D525's are fine for minor home use, but a heavily used 1Gb connection with TINC, IPsec, and OpenVPN connections and several outbound streaming services...that was just not going to cut it as a pfsense box). But to date, I have yet to crack the 400Mb barrier with pfsense.

    I have not yet tried another system (mostly because I have a preference for pfsense (been using it commercially for a long time (since the 1.x days) and before that m0n0wall) so I'm mostly familiar with it.. and I find it to be faster than most linux derived systems (otherwise untangle would be on my radar). Love linux as a server (we have a lot), but for performance, I find BSD based systems to be better. I know my Cisco router can do the full 1G, as well as Ubiquiti Edge Router as well as Vyatta. (I tested with all of them before in this DC before moving our kit here..so I know the speed itself is good.. just have to figure out why my beloved pfsense is not doing its job.)

    Try a test using iperf to check out the interfaces. There are many folks using pfSense for Gigabit symmetrical connections. In fact, during testing of Snort with Inline IPS mode on pfSense-2.5 the Netgate tester recorded 1.8 Gigabits/sec of sustained throughput. Without Snort running, that rose to 3.2 Gigabits/sec. So the software is certainly capable. That was three interfaces running on an SG-5100.

    Of course there could be an issue with a particular driver on the FreeBSD side. And since pfSense is based on FreeBSD any driver problems would show up. Don't know for sure about any identified igb issues, but then I don't keep up with that area.



  • @bmeeks I completely agree.. I did a lot of research before jumping in the forums and these days and 1G internet connection is passé. Its why I'm very puzzled.. (and why we tried a number of system tunes and other changes because I know the software is capable (based on many accounts) and I believe the hardware should be fine (since its basically the same hardware Netgate is selling for 10gb connections)

    If I use a public iperf server (from internal client through pfsense to public iperf server (thankfully HE has one in the same DC I'm in), I get:


    Client connecting to iperf.he.net, TCP port 5201
    TCP window size: 325 KByte (default)

    [ 3] local 10.10.10.160 port 55814 connected with 216.218.227.10 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0- 0.0 sec 109 KBytes 287 Mbits/sec

    But if I do an internal iperf (from internal client to pfsense on the LAN side):

    Client connecting to 10.10.10.254, TCP port 5201
    TCP window size: 85.0 KByte (default)

    [ 3] local 10.10.10.160 port 48020 connected with 10.10.10.254 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0-10.0 sec 735 MBytes 617 Mbits/sec
    [ 4] 0.00-8.03 sec 886 MBytes 926 Mbits/sec



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    @bmeeks I completely agree.. I did a lot of research before jumping in the forums and these days and 1G internet connection is passé. Its why I'm very puzzled.. (and why we tried a number of system tunes and other changes because I know the software is capable (based on many accounts) and I believe the hardware should be fine (since its basically the same hardware Netgate is selling for 10gb connections)

    If I use a public iperf server (from internal client through pfsense to public iperf server (thankfully HE has one in the same DC I'm in), I get:


    Client connecting to iperf.he.net, TCP port 5201
    TCP window size: 325 KByte (default)

    [ 3] local 10.10.10.160 port 55814 connected with 216.218.227.10 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0- 0.0 sec 109 KBytes 287 Mbits/sec

    But if I do an internal iperf (from internal client to pfsense on the LAN side):

    Client connecting to 10.10.10.254, TCP port 5201
    TCP window size: 85.0 KByte (default)

    [ 3] local 10.10.10.160 port 48020 connected with 10.10.10.254 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0-10.0 sec 735 MBytes 617 Mbits/sec
    [ 4] 0.00-8.03 sec 886 MBytes 926 Mbits/sec

    Is there a way for you to put something on the WAN side (another machine perhaps connected to the same switch) and do an iperf test through pfSense from LAN to WAN? The key is to be able to reliably eliminate any dependency on an external host. To really test the pfSense hardware and software you need an iperf endpoint directly on the WAN network and the other endpoint directly on the LAN network.

    With that HE test, I suspect there are other hosts, routers or connections between you and them even if in the same DC. Just trying to make sure you test only pfSense itself. That's the only way to narrow down the problem.

    Do you perhaps have another Intel server NIC that uses say the em driver that you could substitute for testing? It's certainly not impossible for there to be a software issue in the NIC driver for igb. However, if true and widespread I would expect to see a lot of posts here complaining. The igb driver is used by some of the NICs in Netgate's SG-5100.



  • Don't know if anything in this thread directly applies to you, and the hardware is likely some different, but here is a post from last year about igb throughput problems: https://forum.netgate.com/topic/133704/poor-performance-on-igb-driver/43.

    And are the NIC ports on your board genuine Intel chips or are they a clone from another vendor? If not genuine Intel, that may be part of the equation to consider. I found some instances on a Google search where the HP clone of say the Intel i350 didn't perform as well with the igb driver as the native Intel card did in the same box.


  • LAYER 8 Global Moderator

    Yeah you really need to just test pfsense here, and take the internet out of the equation completely..

    Test 1
    boxA --- switch --- boxB

    Can they do gig without pfsense between..

    example

    $ iperf3.exe -c 192.168.9.10
    warning: Ignoring nonsense TCP MSS 0
    Connecting to host 192.168.9.10, port 5201
    [  5] local 192.168.9.101 port 62734 connected to 192.168.9.10 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec   110 MBytes   922 Mbits/sec
    [  5]   1.00-2.00   sec   113 MBytes   952 Mbits/sec
    [  5]   2.00-3.00   sec   113 MBytes   949 Mbits/sec
    [  5]   3.00-4.00   sec   115 MBytes   966 Mbits/sec
    [  5]   4.00-5.00   sec   113 MBytes   950 Mbits/sec
    [  5]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
    [  5]   6.00-7.00   sec   113 MBytes   950 Mbits/sec
    [  5]   7.00-8.00   sec   113 MBytes   950 Mbits/sec
    [  5]   8.00-9.00   sec   113 MBytes   948 Mbits/sec
    [  5]   9.00-10.00  sec   113 MBytes   950 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  1.10 GBytes   949 Mbits/sec                  sender
    [  5]   0.00-10.03  sec  1.10 GBytes   945 Mbits/sec                  receiver
    
    iperf Done.
    

    Now do test with pfsense

    boxA --- wan pfsense lan --- boxB

    What do you get?



  • @johnpoz Yup.. I can do full Gig without pfsense.

    root@coreplex:~# iperf3 -p 5201 -c 10.10.10.141
    Connecting to host 10.10.10.141, port 5201
    [ 4] local 10.10.10.160 port 44144 connected to 10.10.10.141 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-1.00 sec 112 MBytes 936 Mbits/sec
    [ 4] 1.00-2.00 sec 111 MBytes 935 Mbits/sec
    [ 4] 2.00-3.00 sec 111 MBytes 935 Mbits/sec

    And on the Lan side from pfsense to the client, I can do full Gig as well

    Connecting to host 10.10.10.141, port 5201
    [ 5] local 10.10.10.254 port 27396 connected to 10.10.10.141 port 5201
    [ ID] Interval Transfer Bitrate
    [ 5] 0.00-1.00 sec 111 MBytes 927 Mbits/sec
    [ 5] 1.00-2.00 sec 109 MBytes 911 Mbits/sec
    [ 5] 3.00-4.00 sec 111 MBytes 935 Mbits/sec
    [ 5] 4.00-5.00 sec 111 MBytes 930 Mbits/sec
    [ 5] 5.00-6.00 sec 111 MBytes 931 Mbits/sec
    [ 5] 6.00-7.00 sec 110 MBytes 924 Mbits/sec

    Folks, I have run through a lot of these tests before..I know the box is capable of full Gig (ran these tests with just linux on the box prior to installing pfsense on it). And I know the switch (on the lan side) is capable of full 10Gb (yes, 10Gb, not a typo) and I know the connection itself is capable of doing full 1Gb (if I remove pfsense and run the line directly to a Mac or Windows Laptop, we can get full 1Gb). So the main culprit is pfsense.



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    @johnpoz Yup.. I can do full Gig without pfsense.

    root@coreplex:~# iperf3 -p 5201 -c 10.10.10.141
    Connecting to host 10.10.10.141, port 5201
    [ 4] local 10.10.10.160 port 44144 connected to 10.10.10.141 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-1.00 sec 112 MBytes 936 Mbits/sec
    [ 4] 1.00-2.00 sec 111 MBytes 935 Mbits/sec
    [ 4] 2.00-3.00 sec 111 MBytes 935 Mbits/sec

    And on the Lan side from pfsense to the client, I can do full Gig as well

    Connecting to host 10.10.10.141, port 5201
    [ 5] local 10.10.10.254 port 27396 connected to 10.10.10.141 port 5201
    [ ID] Interval Transfer Bitrate
    [ 5] 0.00-1.00 sec 111 MBytes 927 Mbits/sec
    [ 5] 1.00-2.00 sec 109 MBytes 911 Mbits/sec
    [ 5] 3.00-4.00 sec 111 MBytes 935 Mbits/sec
    [ 5] 4.00-5.00 sec 111 MBytes 930 Mbits/sec
    [ 5] 5.00-6.00 sec 111 MBytes 931 Mbits/sec
    [ 5] 6.00-7.00 sec 110 MBytes 924 Mbits/sec

    Folks, I have run through a lot of these tests before..I know the box is capable of full Gig (ran these tests with just linux on the box prior to installing pfsense on it). And I know the switch (on the lan side) is capable of full 10Gb (yes, 10Gb, not a typo) and I know the connection itself is capable of doing full 1Gb (if I remove pfsense and run the line directly to a Mac or Windows Laptop, we can get full 1Gb). So the main culprit is pfsense.

    So help us understand the path a bit better. Which IP is what device in the about output. I take it your second test was this way --

    BoxA --> pfSenseLAN port

    and you had an iperf agent on pfSense and BoxA. Is that correct?

    If the above is true, can you do the same thing on the pfSense WAN port. This will nail down if the issue is something on say the WAN port, or if there truly is a routing bottleneck within the pfSense/FreeBSD network routing code.

    I don't see how we've checked the actual WAN port in the box yet based on the data posted above -- unless I'm missing some detail. You tested the LAN port and it looks fully capable. But what about the WAN port? Can you simply unplug the WAN cable from the firewall and put your laptop directly connected to the WAN port with a crossover cable? Give your laptop an IP in the WAN subnet and then run iperf again.

    And just for grins, try disabling the pf firewall engine. You can do that under the SYSTEM menus. That will turn pfSense into essentially a dumb router by taking all of the firewall code out of the path. That setting is under SYSTEM > ADVANCED and then the Firewall & NAT tab. It's a checkbox.


  • LAYER 8 Global Moderator

    @bmeeks said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    with a crossover cable?

    Don't need a crossover, gig interfaces all support auto mdi-x

    Just do the test please all the way through pfsense, doing nat and firewall..



  • @bmeeks said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    crossover cable?

    Crossover cables are not required with Gb NICs. With 10 & 100 Mb, two pairs were used, one in each direction. With Gb, all 4 pairs are used for both directions.



  • @johnpoz If you look at my earlier comment you will see the result to that.

    From internal client through pfsense (ie: through NAT and firewall) I get this.
    And iperf.he.net is a resource within the same data center as me (on a 10Gb connection)

    Client connecting to iperf.he.net, TCP port 5201
    TCP window size: 325 KByte (default)
    [ 3] local 10.10.10.160 port 55814 connected with 216.218.227.10 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0- 0.0 sec 109 KBytes 287 Mbits/sec

    10.10.10.160 is an internal client (physical box, trying to avoid vm's for these tests connected to 10Gb Switch)

    .160 ---> 10GB Switch ---> pfSense Lan Port <PFSENSE BOX> pfSense WAN port ---> Data Center CORE Switch -> iperf.he.net



  • @johnpoz Don't need a crossover cable.. nor should one need. One can reach native speeds with all the requisite hardware and without pfSense as the FW.



  • @bmeeks I can do the last test when I get back to the DC.. but I think I mentioned earlier, that if I plug in directly to the DC line (no pfsense, just give my laptop a public IP, and start testing, I get native speeds.. NEVER have a slow down.. Its only when traffic is going through pfsense it gives me slow speeds.

    I also before even installing pfsense, I ran similar tests with linux on the same hardware (had to make sure its sound before installing what will be OUR "core" Router/FW/VPN Server.) So I KNOW the hardware is good (at least under linux, Ubuntu Server 14.04, no tweaks, just patched up to what was current then and then installed iperf (and iperf3) and then started to test as a burn in and performance validator)



  • @MrSassinak said in No matter what I do, through pfSense I'm getting between 190-200Mb down, and between 400-600Mb up..:

    @bmeeks I can do the last test when I get back to the DC.. but I think I mentioned earlier, that if I plug in directly to the DC line (no pfsense, just give my laptop a public IP, and start testing, I get native speeds.. NEVER have a slow down.. Its only when traffic is going through pfsense it gives me slow speeds.

    I also before even installing pfsense, I ran similar tests with linux on the same hardware (had to make sure its sound before installing what will be OUR "core" Router/FW/VPN Server.) So I KNOW the hardware is good (at least under linux, Ubuntu Server 14.04, no tweaks, just patched up to what was current then and then installed iperf (and iperf3) and then started to test as a burn in and performance validator)

    The only thing I've not seen proved here thus far to my satisfaction is whether or not the actual physical WAN port on the pfSense box has been verified to pass gigabit traffic flow. I don't doubt that you are not getting gigabit through the box, but we are trying to figure out why. Is it because pfSense itself just can't do it (unlikely, but never impossible)? Or is it because there is some weird issue with just the physical WAN port? I think you said earlier you had swapped physical ports around, but depending on how you are testing through pfSense you might still have had that "bad" port in the mix. We can't see your entire configuration so we have to make assumptions about some things. For instance, if you have just two ports on the box and you swapped LAN and WAN around that won't help as that would just swap the throughput problem from one port to the other. On the other hand, if you have 4 ports in the box and are only using say two of them and you swap LAN and/or WAN to the other ports that is a more valid test. Of course there could still be some backplane issue that ports shared. Wouldn't know that without digging into the details of the motherboard.

    What we are all trying to say is that if you can connect two laptops or two other physical boxes directly to the pfSense NIC ports (one on the LAN and the other on the WAN) and then run an 'iperf' between those two connected machines that will test just pfSense with no other variables. If that gives the same poor throughput, I would next try with the pf firewall disabled. The last desperate test would be to boot the firewall box on a Linux distro live CD and do the iperf test that way. If suddenly Linux has poor throughput as well, then something has gone weird in the hardware. On the other hand, if Linux works like a champ in the same connection scenario, then that definitely points the finger at pfSense or FreeBSD (more likely FreeBSD as the core network code is not altered all that much in pfSense).

    And lastly, I am assuming we are still using a plain-vanilla pfSense install with an empty firewall rule set and no imported previous configuration info. Just install pfSense, select interfaces, set IP addresses and then test.



  • @bmeeks Maybe I'm not explaining things clearly.. so let me try it this way.. my old friends, succinct bullet points and diagrams.

    Current Hardware:
    Supermicro RS-A2SDI-4C4L-FIO Intel Atom C3558 Networking Front I/O 1U Rackmount w/ 4X GbE LAN
    8GB RAM
    128GB SSD

    Services Normally Running on pfSense:
    OpenVPN Server (4 Site-to-Site Client Connections to Japan, Taiwan, HK, and California) - Normally chews up about 2Mb on the pipe (will spike depending on what actions are taking place)
    TINC (Mesh Network) for connecting VPC's (10 of these) - Normally chews up 1Mb on the pipe (will spike depending on what actions are taking place)
    DNS Services - uses about 30Kb average
    NTP (final time source for company) - uses about 5Kb average.

    I have a separate dedicated pfsense instance that handles client level VPN (running in a VM in our ESX cluster. the performance there is terrible as well, but 200Mb is adequate for client communication)

    Previous tuning was based on the 1Gb tuning from here: https://calomel.org/freebsd_network_tuning.html. (had no impact on performance or stability).

    This is the "typical" configuration that I have running all the time:

    Servers <---> 10GB Cisco LAN SWITCH <---> igb0 [pfSense box] igb1 <---> Data Center Core Switch

    Variants/Tests that I have tried (after each test, then reverted back to the above configuration):

    [pfSense box but running linux] igb1 <---> Data Center Core Switch <---> Test Server (in the DC but not internet connected)
    Purpose: To confirm hardware is solid. (did 24 hour burn-in test with this first before installing pfsense)

    Servers <---> 10GB Cisco LAN SWITCH <---> igb1 [pfSense box] igb0 <---> Data Center Core Switch
    Purpose: to see if slow performance moves from WAN to LAN, which it did not. LAN was still solid at 1Gb speeds, and WAN still had the same performance (roughly 200Mb down and 400Mb up)

    Servers <---> 1GB Dell LAN SWITCH <---> igb1 [pfSense box] igb0 <---> Data Center Core Switch
    Purpose: to see if slow performance moves from WAN to LAN, which it did not. LAN was still solid at 1Gb speeds, and WAN still had the same performance (roughly 200Mb down and 400Mb up)

    Single Server (Xeon E5) <---> igb1 [pfSense box] igb0 (fixed speed and duplex) <---> Data Center Core Switch (same fixed speed and duplex)
    Purpose: to rule out autoneg as a variable.

    Single Server (Xeon E5) <---> igb1 [pfSense box] igb0 <---> Data Center Core Switch
    Purpose: To streamline the connection and rule out switch communication as a variable.

    Servers <---> 10GB Cisco LAN SWITCH <---> igb2 [pfSense box] igb3 <---> Data Center Core Switch
    Purpose: To confirm ports are a variable. LAN was still solid at 1Gb speeds, and WAN still had the same performance (roughly 200Mb down and 400Mb up)

    Servers <---> igb0 [pfSense box] igb1 <---> Data Center Core Switch
    Purpose: To remove the switch from the equation to confirm no misconfiguration there.

    MacBook <---> 10GB Cisco LAN SWITCH <---> Data Center Core Switch
    Purpose: To to remove pfSense to confirm 1Gb Speeds possible

    MacBook <---> Data Center Core Switch
    Purpose: To to remove pfSense to confirm 1Gb Speeds possible

    Windows Laptop <---> 10GB Cisco LAN SWITCH <---> Data Center Core Switch
    Purpose: To to remove pfSense to confirm 1Gb Speeds possible (confirming OS/Hardware was not a fluke)

    Servers <---> 10GB Cisco LAN SWITCH <---> igb0 [pfSense box, all packages and features rolled back to baseline] igb1 <---> Data Center Core Switch
    Purpose: To to confirm its not a misconfiguration of pfsense.

    Servers <---> 10GB Cisco LAN SWITCH <---> igb0 [pfSense box, reset to factory (backup restored)] igb1 <---> Data Center Core Switch
    Purpose: To to confirm its not a misconfiguration of pfsense.

    Servers <---> 10GB Cisco LAN SWITCH <---> igb0 [pfSense box, reset to factory] igb1 <---> Data Center Core Switch
    Purpose: To to confirm its not a misconfiguration of pfsense.

    Servers <---> igb0 [pfSense box, reset to factory] igb1 <---> Data Center Core Switch
    Purpose: To remove the switch from the equation to confirm no misconfiguration there and confirm ports are still not a problem.

    Servers <---> igb0 [backup pfSense box, reset to factory] igb1 <---> Data Center Core Switch
    Purpose: To remove the switch from the equation to confirm no misconfiguration there and confirm ports are still not a problem.

    Single Server (Xeon E5) <---> igb1 [pfSense box, reset to factory] igb0 <---> Data Center Core Switch
    Purpose: To streamline the connection and rule out switch communication as a variable.

    The only test I have not done is the firewall disabled using pfsense.. I will do that test this weekend (at a client site right now) but will see if that yields some results.

    I hope this clears up things.


  • Netgate Administrator

    I would try putting some simple unmanaged switch between the pfSense WAN and the datacentre core switch.

    Whilst you were able to see 1Gbps from the WAN to the Coreswitch with Linux running on the hardware the FreeBSD igb driver might be doing something odd, or at least different. We do see that sort of thing occasionally, though usually with crappy SOHO routers.

    Steve



  • @MrSassinak
    Thanks for the details. I will need to read through them carefully and digest what is provided. About to leave for an extended weekend at a college football game, so won't have a chance to get back on this until probably Monday.


Log in to reply