Packet Loss and Excessive Bandwidth Usage
-
I have pfSense connected to a 35/35 DIA Fiber connection as my primary firewall/router. In the last week, it has developed a few issues. I haven't changed anything that I specifically recall off of the top of my head
The first issue is that there was been an inordinate amount of packet loss for in/outbound connections through pfSense.Example Outbound MTR (from pfSense to Remote Computer) HOST: pfSense.DOMAIN.com Loss% Snt Last Avg Best Wrst StDev 1.|-- rrcs-FIBER-ROUTER-HERE-1.midsout 0.0% 10 1.1 1.2 1.1 1.3 0.1 2.|-- rrcs-24-172-50-29.midsout 10.0% 10 5.3 12.4 5.1 35.6 11.0 3.|-- 24.27.255.250 30.0% 10 4.8 7.3 4.8 21.0 6.0 4.|-- ten3-0-0.chrlncsa-pe-rtr0 30.0% 10 7.6 10.3 7.6 25.8 6.8 5.|-- ten3-0-0.chrlncsa-p-rtr01 20.0% 10 7.6 8.4 7.5 14.0 2.3 6.|-- ge-2-1-0.chrlncpop-rtr1.s 50.0% 10 11.0 9.8 8.3 11.0 1.2 7.|-- bu-ether44.atlngamq46w-bc 30.0% 10 13.3 13.7 12.7 15.3 0.9 8.|-- 66.109.1.44 0.0% 10 30.6 32.5 30.6 33.9 1.0 9.|-- ae-0-0.cr0.dfw10.tbone.rr 30.0% 10 33.5 33.0 30.7 33.7 1.1 10.|-- ae1.pr1.dfw10.tbone.rr.co 10.0% 10 30.5 31.3 30.4 38.0 2.5 11.|-- 10ge5-4.core1.dal1.he.net 10.0% 10 42.3 34.0 30.4 42.3 4.6 12.|-- southern-light-rail-inc.1 20.0% 10 30.6 30.5 30.4 30.6 0.1 13.|-- he-atl-dallas-nlr.sox.net 0.0% 10 40.5 40.8 40.5 41.0 0.1 14.|-- bcdcgw1-hurricane.sox.net 20.0% 10 51.5 51.5 51.2 51.7 0.2 15.|-- campus1-rtr.gatech.edu 10.0% 10 51.7 52.2 51.6 53.2 0.6 16.|-- resnet-rtr.gatech.edu 20.0% 10 51.6 52.1 51.6 52.4 0.3 17.|-- res404s-REMOTE-COMPUTER-IP-196.re 10.0% 10 51.7 52.0 51.5 52.3 0.3Example Inbound MTR (from remote computer) Rosss-iMac.local (0.0.0.0) Fri Feb 21 13:27:20 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1\. www.asusnetwork.net 0.0% 8 0.5 0.4 0.3 0.5 0.1 2\. 128.61.104.1 0.0% 8 0.6 1.0 0.6 3.0 0.8 3\. campus1-rtr-130-207-254-137.gatech.edu 0.0% 8 0.7 1.6 0.6 7.9 2.6 4\. 130.207.254.153 0.0% 8 0.9 1.1 0.7 3.0 0.8 5\. tr-bcdcgw1.sox.net 0.0% 8 1.1 1.1 0.9 1.2 0.1 6\. xe-7-3-0.189.asbn0.tr-cps.internet2.edu 0.0% 8 15.4 15.3 15.1 15.7 0.2 7\. 107.14.16.81 0.0% 8 43.0 42.5 42.3 43.0 0.2 8\. ae-2-0.cr0.dca10.tbone.rr.com 0.0% 8 44.1 44.5 43.5 45.3 0.7 9\. 107.14.19.21 0.0% 8 51.9 52.1 51.5 53.1 0.5 10\. ten1-3.rlghncrdc-p-rtr01.southeast.rr.com 0.0% 8 48.9 49.0 48.8 49.3 0.2 11\. ten3-0-0.fyvlncr-pe-rtr01.southeast.rr.com 0.0% 8 47.7 47.8 47.7 48.1 0.2 12\. ten3-0-0.fyvlncr-p-rtr01.southeast.rr.com 0.0% 8 47.9 47.8 47.5 48.2 0.2 13\. ten4-0-0.chrlncsa-pe-rtr01.southeast.rr.com 0.0% 8 47.6 47.7 47.4 47.8 0.1 14\. ten2-0-0.clmascmhe-p-rtr1.southeast.rr.com 0.0% 8 47.5 47.8 47.4 48.2 0.3 15\. lag3.twcc.clmascmhe-a2701.sc.rr.com 0.0% 7 48.8 49.0 48.1 51.7 1.2 16\. rrcs-24-172-50-30.midsouth.biz.rr.com 14.3% 7 52.6 52.7 52.4 52.9 0.2 17\. rrcs-PFSENSE-PUBLIC-IP-3.midsouth.biz.rr.com 42.9% 7 52.4 52.6 52.2 53.0 0.4Example Outbound MTR (from server behind pfSense to remote computer) pbx.local (0.0.0.0) Fri Feb 21 13:30:29 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1\. 192.168.40.1 0.0% 15 0.4 0.4 0.2 0.5 0.1 2\. rrcs-FIBER-ROUTER-HERE-1.midsouth.biz.rr.com 0.0% 15 1.6 1.6 1.3 1.9 0.1 3\. rrcs-24-172-50-29.midsouth.biz.rr.com 26.7% 15 7.4 8.9 5.6 16.3 4.0 4\. 24.27.255.250 26.7% 15 5.2 8.1 5.1 36.8 9.5 5\. ten3-0-0.chrlncsa-pe-rtr01.southeast.rr.com 26.7% 15 8.0 8.1 7.8 9.2 0.4 6\. ten3-0-0.chrlncsa-p-rtr01.southeast.rr.com 20.0% 15 7.8 13.4 7.8 42.7 12.7 7\. ge-2-1-0.chrlncpop-rtr1.southeast.rr.com 26.7% 15 9.4 9.4 8.0 11.9 1.3 8\. bu-ether34.atlngamq46w-bcr00.tbone.rr.com 13.3% 15 15.6 14.6 13.3 15.8 0.8 9\. 66.109.6.36 33.3% 15 33.5 32.9 31.2 34.9 1.2 10\. ae-0-0.cr0.dfw10.tbone.rr.com 0.0% 15 33.0 32.9 31.5 34.8 1.1 11\. ae1.pr1.dfw10.tbone.rr.com 13.3% 15 30.8 33.2 30.6 61.6 8.5 12\. 10ge5-4.core1.dal1.he.net 26.7% 15 34.7 33.2 30.7 37.1 2.7 13\. southern-light-rail-inc.10gigabitethernet4-2.core1.dal1.he.net 13.3% 15 30.6 30.8 30.6 31.0 0.1 14\. he-atl-dallas-nlr.sox.net 26.7% 15 41.1 41.0 40.9 41.2 0.1 15\. bcdcgw1-hurricane.sox.net 33.3% 15 52.1 52.2 51.6 54.3 0.8 16\. campus1-rtr.gatech.edu 33.3% 15 52.0 52.6 52.0 53.9 0.5 17\. resnet-rtr.gatech.edu 6.7% 15 52.1 52.6 51.9 56.0 1.0 18\. res404s-REMOTE-COMPUTER-IP-196.res.gatech.edu 21.4% 14 52.5 52.3 52.0 52.9 0.3Example Inbound MTR (from remote computer to server *not* behind pfSense, but still on DIA fiber) Rosss-iMac.local (0.0.0.0) Fri Feb 21 13:32:16 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1\. www.asusnetwork.net 0.0% 6 0.5 0.5 0.3 0.5 0.1 2\. 128.61.104.1 0.0% 6 1.3 1.0 0.7 1.3 0.2 3\. campus1-rtr-130-207-254-137.gatech.edu 0.0% 6 0.9 0.8 0.6 0.9 0.1 4\. 130.207.254.153 0.0% 6 0.8 5.1 0.8 25.9 10.2 5\. tr-bcdcgw1.sox.net 0.0% 6 1.2 1.1 1.0 1.2 0.1 6\. xe-7-3-0.189.asbn0.tr-cps.internet2.edu 0.0% 6 15.5 15.9 15.0 18.7 1.4 7\. 107.14.16.81 0.0% 6 42.4 42.4 42.0 42.8 0.2 8\. 107.14.19.132 0.0% 6 45.7 44.6 43.6 45.7 0.8 9\. 107.14.19.43 0.0% 6 53.1 52.0 50.7 53.1 0.9 10\. ten1-3.rlghncrdc-p-rtr01.southeast.rr.com 0.0% 6 49.0 49.0 48.9 49.1 0.1 11\. ten3-0-0.fyvlncr-pe-rtr01.southeast.rr.com 0.0% 6 48.0 47.9 47.7 48.3 0.2 12\. ten3-0-0.fyvlncr-p-rtr01.southeast.rr.com 0.0% 6 47.7 47.8 47.6 48.1 0.2 13\. ten4-0-0.chrlncsa-pe-rtr01.southeast.rr.com 0.0% 6 47.7 47.8 47.6 48.1 0.2 14\. ten2-0-0.clmascmhe-p-rtr1.southeast.rr.com 0.0% 6 84.4 55.5 47.6 84.4 14.7 15\. lag3.twcc.clmascmhe-a2701.sc.rr.com 0.0% 6 55.9 49.7 48.2 55.9 3.1 16\. rrcs-24-172-50-30.midsouth.biz.rr.com 0.0% 5 52.4 52.2 52.2 52.4 0.1 17\. rrcs-SERVER-NOTBEHIND-PFSENSE-125.midsouth.biz.rr.com 0.0% 5 52.0 52.4 52.0 52.8 0.3The MTRs show the packet loss in both directions. The last one is a server that is directly attached to the fiber without going through pfSense. It works fine, albeit very slowly due to the next issue.
Here's a rough network diagram.[Internet]----[TWC Demarc (Fiber to Ethernet)]----[HP ProCurve 2610]----[Devices on Public Network and pfSense LAN] | [pfSense WAN and LAN] Ethernet from Fiber goes into port 3 of 2610 to untagged VLAN 5; Port 5 (untagged VLAN5) goes into pfSense WAN port pfSense LAN goes into port 25 with all applicable VLANs (1 untagged, 10,20,30,40,50 tagged) configured appropriately Devices that need direct public access are connected to ports on the 2610 untagged VLAN 5 General LAN devices are connected to one of the other (10,20,30,40,50) VLANs depending on their purpose. VLAN1 is for management only.2. pfSense seems to be maxing out my 35M bandwidth. On my ProCurve, I can see bandwidth usage per port. It has been saying (I don't know for how long) for at least the past few days that ports 3 and 5 are using about 35%-37% of the 100mbps available to them. Additionally, if I connect my laptop directly to the circuit (bypassing pfSense, but leaving it connected), everything is very slow. I looked at the traffic graphs on pfSense, and they show little to no traffic, which is in deed correct. I have disconnected the LAN completely from pfSense and all traffic flowed normally, and the bandwidth issues cleared.
What would cause this? I can provide more details/info if needed.
Thanks!