Bad performance across high-end Pfsense box

[Kh3ops]

Hi everyone,

We've been using pfsense for years now and we've got very good performance results 'till last month…
Our trafic has been growing much since one year and lately, users reported slow transfert rates and packet loss.

Stats :

• Version : 1.2.3-RELEASE
• Effective "going-through-firewall" trafic : 400 Mpbs (out) / 50 Mbps (in)
• Active sessions : 250.000
• Packets : 40.000 (in) / 50.000 (out)
• Hardware : Intel(R) Xeon(R) CPU X3460 @ 2.80GHz  / Dedicated network interface card (Gigabit ethernet) / 4 GB RAM
• CPU usage : 25%
• Memory usage : 10%

Actually, when running some tests, we get very poor results :

• Pinging WAN ip address from outside gives almost 0% packet loss and stable latency (25 - 30ms)
• Pinging LAN host from outside gives about 10% packet loss and variable latency (stable around 30ms with lots of peaks to 60/80ms)
• Transfert on same LAN hosts : 600 Mbps
• Transfert between hosts on different LAN networks (going through pfsense) : between 4 and 5 MBps. With lots of "stalled" state during transfert. Transfert hangs then go on, etc...

Actually, I have no idea where to look out. I disabled hardware checksuming without any result.

Any help or idea would be much appreciated!

Thanks,

Gaëtan

[Supermule]

If you are running the same realease, then error is elsewhere….

Nothing changed in the 1.2.3 release.

[GruensFroeschli]

The described symptoms sound to me like a failing NIC.

Do you get anything in the systemlog?

[Supermule]

Bad cable??

[Kh3ops]

We have two redundant pfsense boxes.

Moving Carp VIP to the other box gives same "poor" results.

[jahonix]

So your pfSense boxes did not change in any way, right?
Look at what's in front or behind them. Any kind of router/modem ahead, even a GBIC module to connect your line to the ISP may be failing.
The LAN switch behind those two pfSense boxes, …

When hardware and software of your redundant pfSense boxes didn't change then look at the surrounding.

[Kh3ops]

No, boxes haven't changed in any way.

ISP is all right. I'm running tests from a bout before firewall (on WAN network and it gets very higher results).
Both firewalls are connected on two different switches. And switches reports 0 error/CRC on connected ports.

When I run a "fetch" on the "backup" box on a sample 1G file, I get very high performance (> 30 MBps). As soon as VIP is moved to this backup box, trafic falls down immediatly.

[ccb056]

Looks like your CPU is being maxed out.  The X3460 is a quad core, and you've pegged it at 25%, right?  Something I've noticed with pfsense and multicore systems is all inbound traffic gets tied to a core, all outbound traffic gets tied to a core.  For instance, on a quad core system the highest CPU you will see is 75% (if you're running snort, 50% without snort).  It looks like the max pps you can send in any particular direction with this CPU is 50k.

[dreamslacker]

By any chance do you have or implemented traffic shaping recently?  The rules might be catching inter-LAN traffic hence, limiting the transfers to your upload cap (~50mbit/s).