Outbound traffic stalling



  • I have ran many tests with many theories of the issue. I will try and be succinct.

    Issue is outbound traffic only (SCP, SMTP) stalls when trying to copy/upload anything substantial (1MB+).

    Layout:

    ADSL2+ modem (Internode) in Bridge mode
    pfsense 2.0BETA5 box connecting to the modem on WAN (rl0) in PPPoE mode and 16 port Gbit switch on LAN (dc0).

    FACTS

    • If modem is in PPPoE modem and pfsense is in DHCP client, then there is no issue.
    • If I connect my laptop (Linux Mint) to the modem and have it act as PPPoE client and make a connection, then there is no issue.
    • I changed the rl0 NIC with an SMC NIC, which ended up being rl0 anyway. No resolve.
    • I swapped the assignment of the NICs so the dc0 is on WAN. No resolve.
    • I changed the Ethernet cable on WAN going to the modem. No resolve.
    • I tried lowering the MTU and MSS values for the WAN device. Down to 1412. No resolve.

    So I can run with pfsense WAN in DHCP etc, but I would really prefer to control as much as I can on 1 box (pfsense box).
    I am thinking this is a bug in 2.0BETA5.



  • Sounds like you need (your ISP requires) a lower MTU on WAN. 1412 may not be low enough



  • This is the angle I took at one stage of my troubleshooting.

    MTU or MSS?

    Would this only effect outbound traffic and not inbound, such as my situation?

    And why does Pfsense struggle with this (in PPPoE) but the modem does not when it is running PPPoE auth? (the modem is not from the ISP, its $40 off the shelf)

    Thanks for responding.



  • IT seems they are saying the standard 1492 is what it should need.

    http://www.internode.on.net/support/faq/broadband_adsl/using_internode_adsl/#What_is_this_MTU_size_problem_th

    Also, my laptop acting as a PPPoE client had no issues too. I did not adjust the MTU value there.



  • @DaveQB:

    MTU or MSS?

    both.



  • hmm

    I have played with both. I have had MTU down to as low as 1412 and MTU to 760. Is there a combination that is needed here, like a safe  :) ?



  • If you set both to 1300 that should definitely be safe, that's the lowest I've heard of needing. Try that and if you're still seeing stalls, go to Diagnostics>Packet capture, pick WAN, enter 0 for packet count, click start. Start transferring some stuff until it stalls and let it stall for about 30 seconds, then click Stop on the capture. Download the pcap and email it to me (cmb at pfsense dot org) with a link to this thread.



  • Fantastic! Thanks cmp. Will do that tonight!

    Thank you very much for the offer of support on this, much appreciate it.
    One can donate to the pfsense project right?

    Ahh found it for anyone interested:
    http://www.pfsense.org/donate.html



  • From looking at the pcaps, it's not what I would normally expect with DSL, losing large frames. There is a ton of packet loss, but it's most all on frame sizes under 500 bytes, and a lot of it on frames under 100 bytes. This is traffic going from you to the ISP, it's leaving your WAN and somehow disappearing. In virtually all such cases it's something upstream of the firewall, like the modem in DSL cases. Where it's the firewall related you usually aren't going to see the lost packets show up on WAN. Granted we are still looking at it from the PPPoE perspective, and it's hard to see lower level issues from the firewall itself - the one that comes to mind is the occasional Realtek NIC that has broken hardware checksum offloading and sends packets out on occasion with wrong or missing checksums. Try disabling hardware checksum offloading under System>Advanced, does that make any difference?



  • Thanks cmb.

    That is odd the traffic is not smooth. Do you know from your experience what the cause could be or is it many and varied?
    It is a new housing estate, in a new house, with new cables and all. I have the same ISP at my previous home. Internode is known for their premium quality (at a slight premium in cost).
    I am using a TP-link 8840 modem. Only $40, could that be causing this?
    Having said that though, my testing has proving the modem in bridge mode works in other configurations (laptop or modem itself authenticating etc all work fine.)

    I did have the WAN on a rl0 NIC, but I since switched it with the LAN which was dc0 NIC (not sure which chipset that is).
    So now WAN is dc0 and LAN is rl0. LAN is working as expected still.

    I tried turning off checksum offloading as well as some other settings as jimp suggested in this thread.

    http://forum.pfsense.org/index.php/topic,32725.0.html

    Thanks cmb.


  • Rebel Alliance Developer Netgate

    @DaveQB:

    I did have the WAN on a rl0 NIC, but I since switched it with the LAN which was dc0 NIC (not sure which chipset that is).
    So now WAN is dc0 and LAN is rl0. LAN is working as expected still.

    FYI- The quality of dc(4) NICs is not all that great either, but it really depends on the actual chip on the card. Personally, I've had just as much bad luck with them as with rl(4) NICs over the years. But it may just be my luck, of course.



  • Thanks jimp.

    So could these cards cause the packet loss cmp noted in my pcaps? Or could it be the modem?

    Maybe I should do a packet capture in the current "working" configuration I have setup.

    On another note, perhaps unrelated, I noticed when I started an scp upload the Internet connection is totally unusable. I can't browse the web (DNS fails), ssh to other external hosts fails. I had to put a limit on the scp (which is the only way I could complete an upload when pfsense is in PPPoE mode(the mode I am having issues with)). Is this just pfsense giving the upload total bandwidth and I need to be setting traffic shaping myself? Previously, uploads would lag my connection, but not totally destroy it.

    Thanks guys.



  • Could be the NICs, the modem, the combination of the NICs and modem not playing nicely together, general issues on your Internet connection, or something completely unrelated though that's probably unlikely. Rather than just swapping the assignments can you replace the cards entirely with less questionable ones like Intels?



  • Thanks cmp.

    I did swap the rl0 with an SMC NIC. But it came up as rl0. So must be a realtek chipset underneath.

    I am pretty sure I don't have any Intel nics. I do have a motherboard (with dual nics) that will is about to be repurposed but I can use it as a test unit. Install pfsense and import my config.
    That might not happen until the weekend though.

    2 things that I can do before then that may help is do a packet capture in the current "working" state and also plug my laptop directly into the modem, by-passing the pfsense box and do some uploading and packet capture on their with wireshark.

    Do these sound like worthwhile tests?

    Thanks all.



  • Is the "Intel PWLA8391 GT 10/100/1000" a good NIC for pfsense?



  • Adding info pertaining to the NICs.

    WAN (wan)                -> dc0        -> 192.168.1.2
      LAN (lan)                -> rl0        -> 10.3.3.1

    A scp on the LAN interface.

    [2.0-RC1][admin@pfsense.rizal]/root(1): scp david@david:30mb .
    david@david.rizal's password:
    30mb                                                                                                  100%  30MB  2.7MB/s  00:11
    [2.0-RC1][admin@pfsense.rizal]/root(2):

    Also, I am getting better pings from google than my ISP..

    [2.0-RC1][admin@pfsense.rizal]/root(4): ping google.com
    PING google.com (66.102.11.104): 56 data bytes
    64 bytes from 66.102.11.104: icmp_seq=0 ttl=56 time=23.838 ms
    64 bytes from 66.102.11.104: icmp_seq=1 ttl=56 time=22.917 ms
    64 bytes from 66.102.11.104: icmp_seq=2 ttl=56 time=23.882 ms
    64 bytes from 66.102.11.104: icmp_seq=3 ttl=56 time=22.892 ms
    64 bytes from 66.102.11.104: icmp_seq=4 ttl=56 time=22.886 ms
    64 bytes from 66.102.11.104: icmp_seq=5 ttl=56 time=22.631 ms
    64 bytes from 66.102.11.104: icmp_seq=6 ttl=56 time=26.551 ms
    64 bytes from 66.102.11.104: icmp_seq=7 ttl=56 time=25.804 ms
    64 bytes from 66.102.11.104: icmp_seq=8 ttl=56 time=24.782 ms
    ^C
    –- google.com ping statistics ---
    9 packets transmitted, 9 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 22.631/24.020/26.551/1.330 ms
    [2.0-RC1][admin@pfsense.rizal]/root(5): ping internode.on.net
    PING internode.on.net (150.101.140.197): 56 data bytes
    64 bytes from 150.101.140.197: icmp_seq=0 ttl=57 time=44.259 ms
    64 bytes from 150.101.140.197: icmp_seq=1 ttl=57 time=44.363 ms
    64 bytes from 150.101.140.197: icmp_seq=2 ttl=57 time=48.035 ms
    64 bytes from 150.101.140.197: icmp_seq=3 ttl=57 time=46.951 ms
    64 bytes from 150.101.140.197: icmp_seq=4 ttl=57 time=46.958 ms
    ^C
    –- internode.on.net ping statistics ---
    5 packets transmitted, 5 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 44.259/46.113/48.035/1.524 ms
    [2.0-RC1][admin@pfsense.rizal]/root(6):



  • @DaveQB:

    On another note, perhaps unrelated, I noticed when I started an scp upload the Internet connection is totally unusable. I can't browse the web (DNS fails), ssh to other external hosts fails. I had to put a limit on the scp (which is the only way I could complete an upload when pfsense is in PPPoE mode(the mode I am having issues with)). Is this just pfsense giving the upload total bandwidth and I need to be setting traffic shaping myself? Previously, uploads would lag my connection, but not totally destroy it.

    Thanks guys.

    I have never experienced this entire uselessness of my Internet connection while uploading.
    Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?

    PS just did a test and if I am downloading full throttle I can still get some connectivity (ssh in, issue commands etc) but if I am uploading it is all but useless.
    I think this is related to my issue. I am thinking about talking to my ISP. I ordered the VoIP add-on package and I know this ISP does QoS for their VoIP users. Could it mis-configured  or simply causing issues?



  • I've seen misconfigured QoS on provider networks (MPLS private WANs) cause serious and strange issues before.

    Generally it's impossible to make your connection completely unusable under normal circumstances because of load (aside from something intentional like a DDoS attack), TCP is likely to be most all your traffic, and it has congestion avoidance that keeps any TCP connection from completely killing your connection. Slow and drastically higher latency, yes, especially where you peg your upload, but not completely unusable.

    @DaveQB:

    Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?

    No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.



  • I am seeing similar stalling when uploading large files.  I have switched the NICs out (from Broadcom to Intel).  No real difference.  If anything, the symptoms seem slightly worse.  Trying to upload a 3GB over the web fails every time.  Trying a different approach now.

    FYI: No packet loss here.



  • @cmb:

    I've seen misconfigured QoS on provider networks (MPLS private WANs) cause serious and strange issues before.

    Generally it's impossible to make your connection completely unusable under normal circumstances because of load (aside from something intentional like a DDoS attack), TCP is likely to be most all your traffic, and it has congestion avoidance that keeps any TCP connection from completely killing your connection. Slow and drastically higher latency, yes, especially where you peg your upload, but not completely unusable.

    Cool. Backs up my theory but my next post may dispute that theory though.
    When I say unusable, I can get a web page to load but it takes several failed page loads due to DNS failure and then content download failure. To an extent I have never seens before unless under an unusual circumstances like you said (DDoS etc)
    It is practically unusable.

    @cmb:

    No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.

    Cool thanks for clarifying.



  • Did some more thorough testing.

    Account1 = the account I have been testing with the whole time in this thread.
    Account2 = another account with same ISP and I have been using this account for about 4-5 years in this house without an issue (with a NetGear Modem)
    ADSL1 modem = an old Alcatel Speed Touch modem that does not even have PPPoE build in used. About 10 years old.
    ADSL2 modem = a TP-Link TD-8840

    At my new house I have been doing the testing above with Account1. I then did another test over the weekend where I used this ADSL1 modem. It is was in bridge mode and the pfsense box WAN in PPPoE.
    Same issue occoured. Uploads hit about 3MB and then stalled. (have packet captures.)

    Did some in another house with a different account (same ISP).

    Summary of tests and results from me taking my firewall and 2 different modems to this other house.

    • ADSL1 modem used with Account1 and my pfsense firewall in PPPoE. Uploads stalled.
    • ADSL1 modem used with Account2 and my pfsense firewall in PPPoE. Uploads stalled.
    • ADSL2 modem used with Account1 and my pfsense firewall in PPPoE. Uploads stalled.
    • ADSL2 modem used with Account2 and my pfsense firewall in PPPoE. Uploads stalled.
    • ADSL2 modem used with Account1 in PPPoE mode so pfsense firewall in STATIC IPv4 on the WAN resulted in uploads working but the same unusable state for the connection. DNS queries failed.
    • ADSL2 modem used with Account2 in PPPoE mode so pfsense firewall in STATIC IPv4 on the WAN resulted in uploads working but the same unusable state for the connection. DNS queries failed.

    So it seems the stalling issue only has the pfsense box as its common item across all the testing.
    And it also seems it is not a QoS issue on the ISP side as I used a different account (that does not have a VoIP plan) on a different phone line/residence. So this issue could be separate and simply a modem issue??

    I have packet captures for some of these tests.



  • @cmb:

    @DaveQB:

    Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?

    No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.

    I guess what I meant was more do these commercial routers/modems have some baseline shaping rules that pfsense does not as it allows you to totally control the traffic flow?



  • I went through the wizard for setting up Traffic Shaping and just used PRIQ as it is a simple setup and I am new to pfSense.
    This solved the issue with uploads destroying my net connection and making all but useless.
    (A side note, I found I had to set the UP and DOWN bandwidth to less than what the modem was precisely reporting as the speed it had connected to the DSLAM at)

    So maybe this same traffic shaping will allow this install to be in PPPoE mode with this modem.

    I might give it one more go this weekend.



  • Has there been any progress on this issue? We seem to be affected by the same bug.

    Our setup differs from the one described by DaveQB, though. We have pfSense configured with three interfaces (all bce). It is also the default gateway for all machines in the LAN. Routes are configured on the box for traffic to other branch offices. All checksum offloading has been disabled, but it didn't help. pfSense is configured for bypassing traffic on the same interface.

    When I try to transfer a file using scp from office A to office B using pfSense as default gateway the transfer will stall leaving an incomplete file on the receiving end which is exactly 48k. Interrupting the stalled transfer and immediately trying again will succeed.

    Circumventing pfSense by adding a route on office A's server to office B's LAN using the VPN router directly (which is also in the LAN) works without any problem.

    I have been tinkering with kernel MTU parameters, but to no avail.

    We are running 2.0-RC1 (amd64) built on Sat Feb 26 18:07:23 EST 2011. This is an in place upgrade from 1.2.3-RELEASE built on Sun Dec 6 23:21:36 EST 2009.



  • @helpdeskadam:

    Has there been any progress on this issue? We seem to be affected by the same bug.

    Our setup differs from the one described by DaveQB, though. We have pfSense configured with three interfaces (all bce). It is also the default gateway for all machines in the LAN. Routes are configured on the box for traffic to other branch offices. All checksum offloading has been disabled, but it didn't help. pfSense is configured for bypassing traffic on the same interface.

    When I try to transfer a file using scp from office A to office B using pfSense as default gateway the transfer will stall leaving an incomplete file on the receiving end which is exactly 48k. Interrupting the stalled transfer and immediately trying again will succeed.

    Circumventing pfSense by adding a route on office A's server to office B's LAN using the VPN router directly (which is also in the LAN) works without any problem.

    I have been tinkering with kernel MTU parameters, but to no avail.

    We are running 2.0-RC1 (amd64) built on Sat Feb 26 18:07:23 EST 2011. This is an in place upgrade from 1.2.3-RELEASE built on Sun Dec 6 23:21:36 EST 2009.

    I haven't done any more. Mine is working great if I have pfsense in DHCP client mode (from the modem) rather than PPPoE mode (and modem in bridge mode) AND basic shaping setup. If either of these 2 things are not true, then my connection is hosed.

    2.0-RC1 (i386)
    built on Mon Feb 28 18:12:00 EST 2011


Locked