Outbound traffic stalling
-
I have ran many tests with many theories of the issue. I will try and be succinct.
Issue is outbound traffic only (SCP, SMTP) stalls when trying to copy/upload anything substantial (1MB+).
Layout:
ADSL2+ modem (Internode) in Bridge mode
pfsense 2.0BETA5 box connecting to the modem on WAN (rl0) in PPPoE mode and 16 port Gbit switch on LAN (dc0).FACTS
- If modem is in PPPoE modem and pfsense is in DHCP client, then there is no issue.
- If I connect my laptop (Linux Mint) to the modem and have it act as PPPoE client and make a connection, then there is no issue.
- I changed the rl0 NIC with an SMC NIC, which ended up being rl0 anyway. No resolve.
- I swapped the assignment of the NICs so the dc0 is on WAN. No resolve.
- I changed the Ethernet cable on WAN going to the modem. No resolve.
- I tried lowering the MTU and MSS values for the WAN device. Down to 1412. No resolve.
So I can run with pfsense WAN in DHCP etc, but I would really prefer to control as much as I can on 1 box (pfsense box).
I am thinking this is a bug in 2.0BETA5. -
Sounds like you need (your ISP requires) a lower MTU on WAN. 1412 may not be low enough
-
This is the angle I took at one stage of my troubleshooting.
MTU or MSS?
Would this only effect outbound traffic and not inbound, such as my situation?
And why does Pfsense struggle with this (in PPPoE) but the modem does not when it is running PPPoE auth? (the modem is not from the ISP, its $40 off the shelf)
Thanks for responding.
-
IT seems they are saying the standard 1492 is what it should need.
http://www.internode.on.net/support/faq/broadband_adsl/using_internode_adsl/#What_is_this_MTU_size_problem_th
Also, my laptop acting as a PPPoE client had no issues too. I did not adjust the MTU value there.
-
-
hmm
I have played with both. I have had MTU down to as low as 1412 and MTU to 760. Is there a combination that is needed here, like a safe :) ?
-
If you set both to 1300 that should definitely be safe, that's the lowest I've heard of needing. Try that and if you're still seeing stalls, go to Diagnostics>Packet capture, pick WAN, enter 0 for packet count, click start. Start transferring some stuff until it stalls and let it stall for about 30 seconds, then click Stop on the capture. Download the pcap and email it to me (cmb at pfsense dot org) with a link to this thread.
-
Fantastic! Thanks cmp. Will do that tonight!
Thank you very much for the offer of support on this, much appreciate it.
One can donate to the pfsense project right?Ahh found it for anyone interested:
http://www.pfsense.org/donate.html -
From looking at the pcaps, it's not what I would normally expect with DSL, losing large frames. There is a ton of packet loss, but it's most all on frame sizes under 500 bytes, and a lot of it on frames under 100 bytes. This is traffic going from you to the ISP, it's leaving your WAN and somehow disappearing. In virtually all such cases it's something upstream of the firewall, like the modem in DSL cases. Where it's the firewall related you usually aren't going to see the lost packets show up on WAN. Granted we are still looking at it from the PPPoE perspective, and it's hard to see lower level issues from the firewall itself - the one that comes to mind is the occasional Realtek NIC that has broken hardware checksum offloading and sends packets out on occasion with wrong or missing checksums. Try disabling hardware checksum offloading under System>Advanced, does that make any difference?
-
Thanks cmb.
That is odd the traffic is not smooth. Do you know from your experience what the cause could be or is it many and varied?
It is a new housing estate, in a new house, with new cables and all. I have the same ISP at my previous home. Internode is known for their premium quality (at a slight premium in cost).
I am using a TP-link 8840 modem. Only $40, could that be causing this?
Having said that though, my testing has proving the modem in bridge mode works in other configurations (laptop or modem itself authenticating etc all work fine.)I did have the WAN on a rl0 NIC, but I since switched it with the LAN which was dc0 NIC (not sure which chipset that is).
So now WAN is dc0 and LAN is rl0. LAN is working as expected still.I tried turning off checksum offloading as well as some other settings as jimp suggested in this thread.
http://forum.pfsense.org/index.php/topic,32725.0.html
Thanks cmb.
-
I did have the WAN on a rl0 NIC, but I since switched it with the LAN which was dc0 NIC (not sure which chipset that is).
So now WAN is dc0 and LAN is rl0. LAN is working as expected still.FYI- The quality of dc(4) NICs is not all that great either, but it really depends on the actual chip on the card. Personally, I've had just as much bad luck with them as with rl(4) NICs over the years. But it may just be my luck, of course.
-
Thanks jimp.
So could these cards cause the packet loss cmp noted in my pcaps? Or could it be the modem?
Maybe I should do a packet capture in the current "working" configuration I have setup.
On another note, perhaps unrelated, I noticed when I started an scp upload the Internet connection is totally unusable. I can't browse the web (DNS fails), ssh to other external hosts fails. I had to put a limit on the scp (which is the only way I could complete an upload when pfsense is in PPPoE mode(the mode I am having issues with)). Is this just pfsense giving the upload total bandwidth and I need to be setting traffic shaping myself? Previously, uploads would lag my connection, but not totally destroy it.
Thanks guys.
-
Could be the NICs, the modem, the combination of the NICs and modem not playing nicely together, general issues on your Internet connection, or something completely unrelated though that's probably unlikely. Rather than just swapping the assignments can you replace the cards entirely with less questionable ones like Intels?
-
Thanks cmp.
I did swap the rl0 with an SMC NIC. But it came up as rl0. So must be a realtek chipset underneath.
I am pretty sure I don't have any Intel nics. I do have a motherboard (with dual nics) that will is about to be repurposed but I can use it as a test unit. Install pfsense and import my config.
That might not happen until the weekend though.2 things that I can do before then that may help is do a packet capture in the current "working" state and also plug my laptop directly into the modem, by-passing the pfsense box and do some uploading and packet capture on their with wireshark.
Do these sound like worthwhile tests?
Thanks all.
-
Is the "Intel PWLA8391 GT 10/100/1000" a good NIC for pfsense?
-
Adding info pertaining to the NICs.
WAN (wan) -> dc0 -> 192.168.1.2
LAN (lan) -> rl0 -> 10.3.3.1A scp on the LAN interface.
[2.0-RC1][admin@pfsense.rizal]/root(1): scp david@david:30mb .
david@david.rizal's password:
30mb 100% 30MB 2.7MB/s 00:11
[2.0-RC1][admin@pfsense.rizal]/root(2):Also, I am getting better pings from google than my ISP..
[2.0-RC1][admin@pfsense.rizal]/root(4): ping google.com
PING google.com (66.102.11.104): 56 data bytes
64 bytes from 66.102.11.104: icmp_seq=0 ttl=56 time=23.838 ms
64 bytes from 66.102.11.104: icmp_seq=1 ttl=56 time=22.917 ms
64 bytes from 66.102.11.104: icmp_seq=2 ttl=56 time=23.882 ms
64 bytes from 66.102.11.104: icmp_seq=3 ttl=56 time=22.892 ms
64 bytes from 66.102.11.104: icmp_seq=4 ttl=56 time=22.886 ms
64 bytes from 66.102.11.104: icmp_seq=5 ttl=56 time=22.631 ms
64 bytes from 66.102.11.104: icmp_seq=6 ttl=56 time=26.551 ms
64 bytes from 66.102.11.104: icmp_seq=7 ttl=56 time=25.804 ms
64 bytes from 66.102.11.104: icmp_seq=8 ttl=56 time=24.782 ms
^C
–- google.com ping statistics ---
9 packets transmitted, 9 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.631/24.020/26.551/1.330 ms
[2.0-RC1][admin@pfsense.rizal]/root(5): ping internode.on.net
PING internode.on.net (150.101.140.197): 56 data bytes
64 bytes from 150.101.140.197: icmp_seq=0 ttl=57 time=44.259 ms
64 bytes from 150.101.140.197: icmp_seq=1 ttl=57 time=44.363 ms
64 bytes from 150.101.140.197: icmp_seq=2 ttl=57 time=48.035 ms
64 bytes from 150.101.140.197: icmp_seq=3 ttl=57 time=46.951 ms
64 bytes from 150.101.140.197: icmp_seq=4 ttl=57 time=46.958 ms
^C
–- internode.on.net ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 44.259/46.113/48.035/1.524 ms
[2.0-RC1][admin@pfsense.rizal]/root(6): -
On another note, perhaps unrelated, I noticed when I started an scp upload the Internet connection is totally unusable. I can't browse the web (DNS fails), ssh to other external hosts fails. I had to put a limit on the scp (which is the only way I could complete an upload when pfsense is in PPPoE mode(the mode I am having issues with)). Is this just pfsense giving the upload total bandwidth and I need to be setting traffic shaping myself? Previously, uploads would lag my connection, but not totally destroy it.
Thanks guys.
I have never experienced this entire uselessness of my Internet connection while uploading.
Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?PS just did a test and if I am downloading full throttle I can still get some connectivity (ssh in, issue commands etc) but if I am uploading it is all but useless.
I think this is related to my issue. I am thinking about talking to my ISP. I ordered the VoIP add-on package and I know this ISP does QoS for their VoIP users. Could it mis-configured or simply causing issues? -
I've seen misconfigured QoS on provider networks (MPLS private WANs) cause serious and strange issues before.
Generally it's impossible to make your connection completely unusable under normal circumstances because of load (aside from something intentional like a DDoS attack), TCP is likely to be most all your traffic, and it has congestion avoidance that keeps any TCP connection from completely killing your connection. Slow and drastically higher latency, yes, especially where you peg your upload, but not completely unusable.
Curious, is this a case where hardware firewalls/routers do some slight trafic shaping under the covers while pfsense truly does what you tell it and nothing without you setting it up and this there needs to be a baseline traffic shaping setup to mitigate this?
No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.
-
I am seeing similar stalling when uploading large files. I have switched the NICs out (from Broadcom to Intel). No real difference. If anything, the symptoms seem slightly worse. Trying to upload a 3GB over the web fails every time. Trying a different approach now.
FYI: No packet loss here.
-
@cmb:
I've seen misconfigured QoS on provider networks (MPLS private WANs) cause serious and strange issues before.
Generally it's impossible to make your connection completely unusable under normal circumstances because of load (aside from something intentional like a DDoS attack), TCP is likely to be most all your traffic, and it has congestion avoidance that keeps any TCP connection from completely killing your connection. Slow and drastically higher latency, yes, especially where you peg your upload, but not completely unusable.
Cool. Backs up my theory but my next post may dispute that theory though.
When I say unusable, I can get a web page to load but it takes several failed page loads due to DNS failure and then content download failure. To an extent I have never seens before unless under an unusual circumstances like you said (DDoS etc)
It is practically unusable.@cmb:
No. The "hardware firewalls/routers" are actually no different, they're no more "hardware" than what you're running now, they're a piece of hardware running an OS that does all the work, effectively identical, just different software and hardware. Without shaping, the limits and queuing on your connection are entirely upstream, with any router or firewall.
Cool thanks for clarifying.
