Lost Packets and slow traffic
-
Having a few network problems and I think I have it narrowed down to a pfSense box.
A little background on the network:
-3 sites
-Each site has its own pfSense box
-The sites are connected via a private network provided by internet provider.
-The sites also have IPsec tunnels setup that can be enabled if the private network has problems
-Each pfSense box is a Dell PowerEdge 2950 with on board Broadcom nicsFrom:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_CardsApplied in the /boot/loader.conf.local file:
kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0And for packet loss System tunables changed:
net.isr.direct_force=1
net.isr.direct=1We are working on a project to upgrade servers at each site. We upgraded 1 site and noticed no issues. We upgraded the 2nd site and started to notice packet loss. We saw the packet loss from site 2 to site 3 that has not been touched yet. Also it seemed a bit slow.
We tried enabling the IPsec tunnel and still saw packet loss.
So I started doing pings from site 3 to site 2 watching the interfaces. Default windows ping size worked. Ping size of 20972 and up fails. I could see it going from site 3 going into the LAN port and going out the internet provider private network port in site 3. It showed up in site 2 on the internet provider private network port and it went out the LAN port. I could see it come back into the LAN port in site 2 but I could never see it go out any interface as it tried to go back to site 3. I attached JPG of ping path.
If we pinged from Site 3 to 1 it worked with the larger ping. Also tried multiple computers in each location to make sure it was not just the server we just upgraded.
First we tried uninstalling packages. We uninstalled squid, squidguard, and pfblocker. Still no success for the larger pings. Next step we tried was to upgrade from 2.0.3 to 2.1, with no success. Only packages still installed are ntop and OpenVPN client export utility.
Also tried moving the LAN to another port/network card and it didn't help.
Getting the No buffer space available error when trying to ping while SSH'ed in.
What would be other troubleshooting steps to take besides reinstalling? With the new version of pfSense do we still need all the changes for the Broadcom cards?
![Ping path.JPG](/public/imported_attachments/1/Ping path.JPG)
![Ping path.JPG_thumb](/public/imported_attachments/1/Ping path.JPG_thumb) -
Can you elaborate on your hardware?
-
So does your site upgrade involve changing the pfSense box or just the servers behind it? If just the servers these new boxes are capable of pushing more traffic that the old models?
Steve
-
Just upgrading the server software. Servers came with Windows Server 2008 32 bit. We have been upgrading them to Windows Server 2008 R2. Once all the software is back in the servers, the servers seem more responsive.
We were waiting on upgrading the pfSense boxes from 2.0.3 to the newer version until later. We upgraded the suspected pfSense box to 2.1 but it still did not fix the issue.
At this point we planned additional down time for the boxes. For some reason the box would not take a clean install off of the live cd for version 2.1. So we did a clean install of 2.0.3 and then upgraded to 2.1 before anything was installed. All the configuration was put back in slowly watching to make sure only the configuration that was needed got added back in. This fixed the Internet provider private network for losing packets.
We are still losing packets on the IPsec VPN tunnel but with the speed of the tunnel and the equipment having the ability to re-request the packets, the traffic is getting through at a reasonable rate. So we are going to hold troubleshooting this part of the link until the upgrading of the servers is complete.
pfSense boxes were re-tasked Dell PowerEdge 2950s( a little old-purchased in 2006-but still should be enough power for this)
Dual CPUs - Intel Xeon Processor 5050
Intel(R) Xeon(TM) CPU 3.00GHz
8 CPUs: 2 package(s) x 2 core(s) x 2 HTT threads
2 gigs of ram
on-board Broadcom NICs
additonal 4 port StaTech PCI express gigabit Ethernet network adapter card