PFsense can't do above 90k pps? Is there anything to improve pps performance?

shukko

My Pfsense setup:

Transparent bridge mode installed exactly using trendchiller.com document..
3 Ethernet cards 1x wan 1x lan 1x management
100mbits of connection to internet on a 1 Gbps port

Hardware details:

Intel brand 1U server
S5000 series mainboard
quad core cpu Intel Xeon E5310 @ 1.60GHz - SMP enabled
2 O/B Intel ethernet cards + 2 more connected to PCI-X port

em0: Intel(R) PRO/1000 Network Connection Version - 6.2.9
em1: Intel(R) PRO/1000 Network Connection Version - 6.2.9
em2: Intel(R) PRO/1000 Network Connection Version - 6.2.9
em3: Intel(R) PRO/1000 Network Connection Version - 6.2.9

73 GB SAS HDD + 2 GB ECC DDR2 Buffered Ram
–---------------------

This server setup exactly CAN NOT DO ABOVE 90k pps under a dos attack!
90k pps is the limit!

I can see on my switch that I am getting around 80Mbps traffic at 120k pps

But On my PFsense directly connected to this switch I see only 42Mbps and 90k pps at most.

Smp is enabled on server.And with full block rules added to firewall Cpu load is at most %60
memory usage never goes beyond %40 with 1.000.000 state limit.

When under a DOS attack there are not any kind of slowness or lag on server.GUI works as usual - very fast.
It just does not count any packets beyond 90k.

Is this normal?

Is 90k pps for a server like that a physical limit?

I read all the documents I can find and can't find anything about max pps limit of pfsense.
Somebody said on this forum m0n0wall can do much better than that.
I know that physical limit of 100Mbit ethernet is around 144k pps
But all my connections are GBIT connections.
I can easily do 100Mbits full routing over my firewall under normal conditions.
Thousands of connections. Very very steady. %10 Cpu at most.

but when under a DOS attack. Although I see that my switch sends 120K packets in 80 Mbps to PFsense
PFsense shows only 42 Mbps 90k pps?
Is there something I am doing wrong ?

OpenBSD PF can do around 300K pps under testing connections with 4 bridges google says.
there is only 1 bridge in my situation. And just 80Mbps of traffic?
Only the pps rate is somewhat high.I am really loosing my trust to Pfsense..

Please someone explains me what is going wrong with my setup…

I will add some graphics below.
the first three is what pfsense shows.
The last one is PRTG graphic from SNMP port of the switch.

shukko

here are some graphics under a dos attack:

packets.png_thumb

system.png_thumb

traffic.png_thumb

attack8.jpg_thumb

eri--

Well the first you should start with is the icmp tweaking.
After that start with increasing with ip input queue tweaking. This are all sysctl that help you get higher pps.
I would recommend even trying a 1.2.1 install to get things better though i am talkin in air here since it would be nice to see some detailed stats before.

shukko

@ermal:

Well the first you should start with is the icmp tweaking.
After that start with increasing with ip input queue tweaking. This are all sysctl that help you get higher pps.
I would recommend even trying a 1.2.1 install to get things better though i am talkin in air here since it would be nice to see some detailed stats before.

How?
At least point me to something for reference please.
for example will

kern.ipc.nmbclusters=32768

will do good for me ?

shukko

I tried 1.2.1RC1 Latest snapshot today.
And whatever I did my transparent bridge did not work as expected.
After everything was setup my bridge allows traffic from inside to outside but whatever firewall rule I tried I can't make it work from outside to inside.
I guess this is some bug with the snapshot image I used.
Will try 1.2.1 when it's released later.

Now I have to stick with 1.2 release and 90k max pps problem is still here.
I need a little word from the developers please. Please somebody say me 90k is what you can get most.
Then I'll try something else. even will buy a hardware firewall..

dotdash

Not that I doubt what you are seeing, but I'd like to see some controlled tests. In theory (which I know is not real world), 1.2 should be able to do much better. See http://www.tancsa.com/blast.html The older versions of m0n0 use 4.x, which is still a very fast stack. I expect 7.x (1.2.1/1.3) will come close to the 4.x levels.
Were you seeing any problems on the network when this was happening?

shukko

Network setup is very basic

ISP ==> SWITCH1 ==> PFSENSE ==> ROUTER ==>SWITCH2 ==>SERVERS

I read traffic from switch1 using prtg on a private port.
When I see 100Mbps incoming with 120k pps on my switch1

I read 42Mbps 90k pps on my pfsense.

That's the problem.

Pfsense machine does not lag or lock down. Cpu %60 at most. ram %40. GUI works very fast and normal.
Any Idea?

shukko

I also want to add that I already tried m0n0wall.
But because of hardware problems I can't use it.
My hardware is quite new. Mono does not work with it.

dotdash

Double check your net.inet.ip.fastforwarding. IIRC, this gets turned off under certain configurations, like when you are running IPSec…

shukko

sysctl -a | grep net.inet.ip.forwarding

net.inet.ip.forwarding: 1

dotdash

@shukko:

sysctl -a | grep net.inet.ip.forwarding

net.inet.ip.forwarding: 1

Yeah, you're forwarding packets, but what does
sysctl -a | grep net.inet.ip.fastforwarding
show you?

shukko

it is also the same

sysctl -a | grep net.inet.ip.fastforwarding

net.inet.ip.fastforwarding: 1