Outbound traffic stalling
-
From looking at the pcaps, it's not what I would normally expect with DSL, losing large frames. There is a ton of packet loss, but it's most all on frame sizes under 500 bytes, and a lot of it on frames under 100 bytes. This is traffic going from you to the ISP, it's leaving your WAN and somehow disappearing. In virtually all such cases it's something upstream of the firewall, like the modem in DSL cases. Where it's the firewall related you usually aren't going to see the lost packets show up on WAN. Granted we are still looking at it from the PPPoE perspective, and it's hard to see lower level issues from the firewall itself - the one that comes to mind is the occasional Realtek NIC that has broken hardware checksum offloading and sends packets out on occasion with wrong or missing checksums. Try disabling hardware checksum offloading under System>Advanced, does that make any difference?
-
Thanks cmb.
That is odd the traffic is not smooth. Do you know from your experience what the cause could be or is it many and varied?
It is a new housing estate, in a new house, with new cables and all. I have the same ISP at my previous home. Internode is known for their premium quality (at a slight premium in cost).
I am using a TP-link 8840 modem. Only $40, could that be causing this?
Having said that though, my testing has proving the modem in bridge mode works in other configurations (laptop or modem itself authenticating etc all work fine.)I did have the WAN on a rl0 NIC, but I since switched it with the LAN which was dc0 NIC (not sure which chipset that is).
So now WAN is dc0 and LAN is rl0. LAN is working as expected still.I tried turning off checksum offloading as well as some other settings as jimp suggested in this thread.
http://forum.pfsense.org/index.php/topic,32725.0.html
Thanks cmb.
-
I did have the WAN on a rl0 NIC, but I since switched it with the LAN which was dc0 NIC (not sure which chipset that is).
So now WAN is dc0 and LAN is rl0. LAN is working as expected still.FYI- The quality of dc(4) NICs is not all that great either, but it really depends on the actual chip on the card. Personally, I've had just as much bad luck with them as with rl(4) NICs over the years. But it may just be my luck, of course.
-
Thanks jimp.
So could these cards cause the packet loss cmp noted in my pcaps? Or could it be the modem?
Maybe I should do a packet capture in the current "working" configuration I have setup.
On another note, perhaps unrelated, I noticed when I started an scp upload the Internet connection is totally unusable. I can't browse the web (DNS fails), ssh to other external hosts fails. I had to put a limit on the scp (which is the only way I could complete an upload when pfsense is in PPPoE mode(the mode I am having issues with)). Is this just pfsense giving the upload total bandwidth and I need to be setting traffic shaping myself? Previously, uploads would lag my connection, but not totally destroy it.
Thanks guys.
-
Could be the NICs, the modem, the combination of the NICs and modem not playing nicely together, general issues on your Internet connection, or something completely unrelated though that's probably unlikely. Rather than just swapping the assignments can you replace the cards entirely with less questionable ones like Intels?
-
Thanks cmp.
I did swap the rl0 with an SMC NIC. But it came up as rl0. So must be a realtek chipset underneath.
I am pretty sure I don't have any Intel nics. I do have a motherboard (with dual nics) that will is about to be repurposed but I can use it as a test unit. Install pfsense and import my config.
That might not happen until the weekend though.2 things that I can do before then that may help is do a packet capture in the current "working" state and also plug my laptop directly into the modem, by-passing the pfsense box and do some uploading and packet capture on their with wireshark.
Do these sound like worthwhile tests?
Thanks all.
-
Is the "Intel PWLA8391 GT 10/100/1000" a good NIC for pfsense?
-
Adding info pertaining to the NICs.
WAN (wan) -> dc0 -> 192.168.1.2
LAN (lan) -> rl0 -> 10.3.3.1A scp on the LAN interface.
[2.0-RC1][admin@pfsense.rizal]/root(1): scp david@david:30mb .
david@david.rizal's password:
30mb 100% 30MB 2.7MB/s 00:11
[2.0-RC1][admin@pfsense.rizal]/root(2):Also, I am getting better pings from google than my ISP..
[2.0-RC1][admin@pfsense.rizal]/root(4): ping google.com
PING google.com (66.102.11.104): 56 data bytes
64 bytes from 66.102.11.104: icmp_seq=0 ttl=56 time=23.838 ms
64 bytes from 66.102.11.104: icmp_seq=1 ttl=56 time=22.917 ms
64 bytes from 66.102.11.104: icmp_seq=2 ttl=56 time=23.882 ms
64 bytes from 66.102.11.104: icmp_seq=3 ttl=56 time=22.892 ms
64 bytes from 66.102.11.104: icmp_seq=4 ttl=56 time=22.886 ms
64 bytes from 66.102.11.104: icmp_seq=5 ttl=56 time=22.631 ms
64 bytes from 66.102.11.104: icmp_seq=6 ttl=56 time=26.551 ms
64 bytes from 66.102.11.104: icmp_seq=7 ttl=56 time=25.804 ms
64 bytes from 66.102.11.104: icmp_seq=8 ttl=56 time=24.782 ms
^C
–- google.com ping statistics ---
9 packets transmitted, 9 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.631/24.020/26.551/1.330 ms
[2.0-RC1][admin@pfsense.rizal]/root(5): ping internode.on.net
PING internode.on.net (150.101.140.197): 56 data bytes
64 bytes from 150.101.140.197: icmp_seq=0 ttl=57 time=44.259 ms
64 bytes from 150.101.140.197: icmp_seq=1 ttl=57 time=44.363 ms
64 bytes from 150.101.140.197: icmp_seq=2 ttl=57 time=48.035 ms
64 bytes from 150.101.140.197: icmp_seq=3 ttl=57 time=46.951 ms
64 bytes from 150.101.140.197: icmp_seq=4 ttl=57 time=46.958 ms
^C
–- internode.on.net ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 44.259/46.113/48.035/1.524 ms
[2.0-RC1][admin@pfsense.rizal]/root(6): -
On another note, perhaps unrelated, I noticed when I started an scp upload the Internet connection is totally unusable. I can't browse the web (DNS fails), ssh to other external hosts fails. I had to put a limit on the scp (which is the only way I could complete an upload when pfsense is in PPPoE mode(the mode I am having issues with)). Is this just pfsense giving the upload total bandwidth and I need to be setting traffic shaping myself? Previously, uploads would lag my connection, but not totally destroy it.
Thanks guys.
I have never experienced this entire uselessness of my Internet connection while uploading.
Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?PS just did a test and if I am downloading full throttle I can still get some connectivity (ssh in, issue commands etc) but if I am uploading it is all but useless.
I think this is related to my issue. I am thinking about talking to my ISP. I ordered the VoIP add-on package and I know this ISP does QoS for their VoIP users. Could it mis-configured or simply causing issues? -
I've seen misconfigured QoS on provider networks (MPLS private WANs) cause serious and strange issues before.
Generally it's impossible to make your connection completely unusable under normal circumstances because of load (aside from something intentional like a DDoS attack), TCP is likely to be most all your traffic, and it has congestion avoidance that keeps any TCP connection from completely killing your connection. Slow and drastically higher latency, yes, especially where you peg your upload, but not completely unusable.
Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?
No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.
-
I am seeing similar stalling when uploading large files. I have switched the NICs out (from Broadcom to Intel). No real difference. If anything, the symptoms seem slightly worse. Trying to upload a 3GB over the web fails every time. Trying a different approach now.
FYI: No packet loss here.
-
@cmb:
I've seen misconfigured QoS on provider networks (MPLS private WANs) cause serious and strange issues before.
Generally it's impossible to make your connection completely unusable under normal circumstances because of load (aside from something intentional like a DDoS attack), TCP is likely to be most all your traffic, and it has congestion avoidance that keeps any TCP connection from completely killing your connection. Slow and drastically higher latency, yes, especially where you peg your upload, but not completely unusable.
Cool. Backs up my theory but my next post may dispute that theory though.
When I say unusable, I can get a web page to load but it takes several failed page loads due to DNS failure and then content download failure. To an extent I have never seens before unless under an unusual circumstances like you said (DDoS etc)
It is practically unusable.@cmb:
No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.
Cool thanks for clarifying.
-
Did some more thorough testing.
Account1 = the account I have been testing with the whole time in this thread.
Account2 = another account with same ISP and I have been using this account for about 4-5 years in this house without an issue (with a NetGear Modem)
ADSL1 modem = an old Alcatel Speed Touch modem that does not even have PPPoE build in used. About 10 years old.
ADSL2 modem = a TP-Link TD-8840At my new house I have been doing the testing above with Account1. I then did another test over the weekend where I used this ADSL1 modem. It is was in bridge mode and the pfsense box WAN in PPPoE.
Same issue occoured. Uploads hit about 3MB and then stalled. (have packet captures.)Did some in another house with a different account (same ISP).
Summary of tests and results from me taking my firewall and 2 different modems to this other house.
- ADSL1 modem used with Account1 and my pfsense firewall in PPPoE. Uploads stalled.
- ADSL1 modem used with Account2 and my pfsense firewall in PPPoE. Uploads stalled.
- ADSL2 modem used with Account1 and my pfsense firewall in PPPoE. Uploads stalled.
- ADSL2 modem used with Account2 and my pfsense firewall in PPPoE. Uploads stalled.
- ADSL2 modem used with Account1 in PPPoE mode so pfsense firewall in STATIC IPv4 on the WAN resulted in uploads working but the same unusable state for the connection. DNS queries failed.
- ADSL2 modem used with Account2 in PPPoE mode so pfsense firewall in STATIC IPv4 on the WAN resulted in uploads working but the same unusable state for the connection. DNS queries failed.
So it seems the stalling issue only has the pfsense box as its common item across all the testing.
And it also seems it is not a QoS issue on the ISP side as I used a different account (that does not have a VoIP plan) on a different phone line/residence. So this issue could be separate and simply a modem issue??I have packet captures for some of these tests.
-
@cmb:
Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?
No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.
I guess what I meant was more do these commercial routers/modems have some baseline shaping rules that pfsense does not as it allows you to totally control the traffic flow?
-
I went through the wizard for setting up Traffic Shaping and just used PRIQ as it is a simple setup and I am new to pfSense.
This solved the issue with uploads destroying my net connection and making all but useless.
(A side note, I found I had to set the UP and DOWN bandwidth to less than what the modem was precisely reporting as the speed it had connected to the DSLAM at)So maybe this same traffic shaping will allow this install to be in PPPoE mode with this modem.
I might give it one more go this weekend.
-
Has there been any progress on this issue? We seem to be affected by the same bug.
Our setup differs from the one described by DaveQB, though. We have pfSense configured with three interfaces (all bce). It is also the default gateway for all machines in the LAN. Routes are configured on the box for traffic to other branch offices. All checksum offloading has been disabled, but it didn't help. pfSense is configured for bypassing traffic on the same interface.
When I try to transfer a file using scp from office A to office B using pfSense as default gateway the transfer will stall leaving an incomplete file on the receiving end which is exactly 48k. Interrupting the stalled transfer and immediately trying again will succeed.
Circumventing pfSense by adding a route on office A's server to office B's LAN using the VPN router directly (which is also in the LAN) works without any problem.
I have been tinkering with kernel MTU parameters, but to no avail.
We are running 2.0-RC1 (amd64) built on Sat Feb 26 18:07:23 EST 2011. This is an in place upgrade from 1.2.3-RELEASE built on Sun Dec 6 23:21:36 EST 2009.
-
Has there been any progress on this issue? We seem to be affected by the same bug.
Our setup differs from the one described by DaveQB, though. We have pfSense configured with three interfaces (all bce). It is also the default gateway for all machines in the LAN. Routes are configured on the box for traffic to other branch offices. All checksum offloading has been disabled, but it didn't help. pfSense is configured for bypassing traffic on the same interface.
When I try to transfer a file using scp from office A to office B using pfSense as default gateway the transfer will stall leaving an incomplete file on the receiving end which is exactly 48k. Interrupting the stalled transfer and immediately trying again will succeed.
Circumventing pfSense by adding a route on office A's server to office B's LAN using the VPN router directly (which is also in the LAN) works without any problem.
I have been tinkering with kernel MTU parameters, but to no avail.
We are running 2.0-RC1 (amd64) built on Sat Feb 26 18:07:23 EST 2011. This is an in place upgrade from 1.2.3-RELEASE built on Sun Dec 6 23:21:36 EST 2009.
I haven't done any more. Mine is working great if I have pfsense in DHCP client mode (from the modem) rather than PPPoE mode (and modem in bridge mode) AND basic shaping setup. If either of these 2 things are not true, then my connection is hosed.
2.0-RC1 (i386)
built on Mon Feb 28 18:12:00 EST 2011
