Packet loss on new firewalls

alanon

Hello, we are currently running 8 pfsense boxes in our environment. We have several hundred servers that are data collectors so we load balance them both for performance and to have different apparent IP's. Our heaviest ones average 25Mb at all times. Just simple browsing is not possible, some sites load, others never load. There are no additional rules in place. The only changes I've made are the advanced state tables settings (increased and set algorithm to conservative, which is the same across all boxes)

I recently added two more (to bring us to 8) and now on both boxes I am seeing 6-8% packet loss. Here's what I've tried (all of these are built on Dell PE1750's)

Flipping nic's (two embedded) making left WAN and right LAN, and switching them. Same 6-8% packet loss to various public IP's. Internal pings have no packet loss either way.
Bought a new NIC and used that for the WAN, same issue.
Different builds of PFsense, same 6-8% packet loss

I am at a loss and don't know what else to try? I have spoke with the networking person at the DC and he says there are no issues with their router, nor is he seeing anything strange on his end.

Any suggestions would be greatly appreciated.

Thanks.

mhab12

We use a Dell PE 1750 and I have seen issues with the internal NICs before. We're now using a dual port Intel server NIC.

Through the various pfSense versions we've also encountered IRQ conflicts. You might go into the BIOS and disable com ports (if you don't need them) and USB (same). I know the 1750 BIOS lets you manually assign IRQ per resource too, so at a minimum make sure the two built in NICs aren't sharing one IRQ.

alanon

Thanks, I'll check the bios settings tomorrow.

It seems strange that adding the new PCI NIC that the problem still exists, and with the same % of packet loss.

wallabybob

How have you determined "6-8% packet loss"?

What NICs are you using?

Does the CPU utilisation go to 100% when packet loss occurs?

What are the interrupt rates (shown by # vmstat -i)?

What is packet size distribution - lots of small packet? mostly large packets?

alanon

We are using the integrated NIC's. I checked the BIOS settings and evrything is OK there.

What I am doing is just running a ping to an external IP, from a PC using that firewall as it's gateway. I am using 4.2.2.2, since it is one of the larger DNS serevrs. CPU usage is minimal. We are ordering Intel NIC's to use isntead of the integrated NIC's.

I decided to try the pre release version of 1.2.3, when I run that, off the CD, I do not lose any packets. I did a ping for about 2 hours. I am wondering if there are newer drivers in this version, or if it's related to running it off the CD and maye the issue is when it's installed….

Thanks.

wallabybob

pfSense 1.2.3 is based on FreeBSD 7.1 so (generally) has newer drivers than pfSense 1.2.2 which is based on FreeBSD 7.0. Earlier versions of pfSense are based on 6 series FreeBSD.

The developers say pfSense 1.2.3 should be considered "production ready" - see http://blog.pfsense.org/?p=377.