Pfsense freeze at DDoS attack - Tuning?
-
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.
-
With every ISP in the world trying to do real-time DPI on every connection, you would think that while they are analyzing packets you don't want them to look at, they could analyze packets you do want them to notice and stop something like this from ever hitting your firewall. I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
-
@cmb:
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.
I got my state tables filled up with only 40mbit traffic. Instantly. 20sec "attack" only but pfsense was freezed first sec.
I almost daily get ddos'ed (my customer do) Some customers are running gameservers, and scriptkiddies like to knock them out.
I even got DDoS mitigation by providor, using arbor network (best ddos protection ever).Before i couldnt take any attack/flood over 800mbit. Now i dont get knocked off straight away.My drop is on 1Gbit.
E3-2.8ghz
8gb ram
ssd disk 120gbi've attached some rrd graphs - high spike shows the attack…
-
And…. Is the traffic in question on that port? 1433?
-
Oooops, i did mix some things together. Ignore the "OVH method" and MS SQL.
But when i did a capture i saw this… many many many of this from different IPs:
84 38.733142 119.246.140.4 76.28.11.29 TCP 60 46288→80 [SYN, ECN, CWR] Seq=0 Win=0 Len=0
88 38.733361 76.28.11.29 119.246.140.4 TCP 58 80→46288 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 -
I gave my second box (virtual) 96GB memory hoping that the table state would survive ;D
But it knocked it out anyway, and only with 20mbit SYN flood. :-XThe first post about 400mbit SSDP attack is understandable, but how come a 20mbit SYN flood just "crash" the box? Shall we suddenly just block a whole protocol?… not even that would do it, i would say....
-
I'm maybe mislead here, but my understanding is that a syn flood can be blocked unless the address is spoofed…
If the address is spoofed with a syn flood attack, you would need to have a situation where something inside your network is compromised.
A server or a LAN client or someone inside a VPN with knowledge of your network?
So, malicious client A inside your net pretending to be client B inside your net forging IPs and sending SYN to computers that never ACK
So, if this is the case, who is inside your network?
-
I'm maybe mislead here, but my understanding is that a syn flood can be blocked unless the address is spoofed…
If the address is spoofed with a syn flood attack, you would need to have a situation where something inside your network is compromised.
A server or a LAN client or someone inside a VPN with knowledge of your network?
So, malicious client A inside your net pretending to be client B inside your net forging IPs and sending SYN to computers that never ACK
So, if this is the case, who is inside your network?
The attack has nothing to do with my LAN clients, other than they get attacked by time. I made a research and found some stressers online to simulate some attacks. I tried many and pfsense survived most of the times, but one specific attack type was quite efficient to pfsense box's - it crash the box in no time with minimal bandwidth. I have other people who participate in my "firewall test", They "survived" (not using pfsene) - pfsense is way ahead on many features, troughput etc but handling this is a challenging job for it. I may try it on 2.2 when the critical bugs has been fixed….
-
There is an option in PFSense that has a linear reduction in state live time after a threshhold. I'm not at home to look at it, but I think it's under advanced or something in the general settings.
In my case, I have it set to 3mil state, with a 4mil hard cap. So after 3mil states are created, the live time of those states will keep reducing at more states get added. By the time 4mil states exist, if a state isn't refreshed almost immediately, the state will get killed as being "old".
Another idea that popped into my head, I have no practice in these kinds of issues, but it seems as if the client machine is ACKing all of those SYNs, as it should. Would rate limiting new connections per client be an acceptable trade-off, assuming it can be done at a per client level.
Another possibility could be rate limiting via traffic shaping, how much bandwidth SYN packets get for that customer.
Just throwing around ideas.
-
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Lowprofile has tested against your setup and it takes your setup offline at once.
Only one provider here in Denmark is running as nothing has happened and they are using Juniper gear.
-
I know you have my IP in Denmark Supermule… Please spare me (-;
-
I will mate :D
This is just testing and you can bring down just about anything out there. Its quite scary.
The state table fills up instantly but everything is responsive and nothing in the logs. Your drop pipeline is just filled and nothing to do about it since pfSense doesnt seem to be able to handle this specific kind of SYN traffic.
-
I just checked the servers running there on public IPs - Nothing hit yet.
Hope it stays that way. -
Dont worry mate. It wont come from any of us here.
But it can be ordered online and its very easy to take down pfsense.org and the forum if we want to (we dont).
The first seconds of the attack, pfsense dies on the WAN side while states are flooding but only on 1 specific type of SYN. Nothing in the logs and nothing reported by SNort as well.
-
I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
A SYN is a SYN is a SYN generally, which is why null routing the IP being attacked is the typical response. Can't tell the difference between legit users and the DDoS traffic in most cases.
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Again, what test is this? PM me if you don't want to post publicly.
-
Hi Chris
We use something like this
https://www.vdos-s.com
You can order any kind of attack using whatever length of time you want.
You can write to Lowprofile and give him a testing address to see how this is seen by pfSense and the results.
When using port 80 and for my own testing purposes, I have pfblockerNG running and it takes a lot of traffic away from the webserver itself but pfSense dies fast anyway. I see around 20mbit of traffic on the server and its responsive from the inside and you dont see anything wrong with it.
Its just offline.
-
@cmb:
I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
A SYN is a SYN is a SYN generally, which is why null routing the IP being attacked is the typical response. Can't tell the difference between legit users and the DDoS traffic in most cases.
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Again, what test is this? PM me if you don't want to post publicly.
PM sent! - regarding null route, i am running a layer 2 setup and a null route trough my providor cost me $200 each time :) despite i have prof. dos protection.
-
There is an option in PFSense that has a linear reduction in state live time after a threshhold. I'm not at home to look at it, but I think it's under advanced or something in the general settings.
In my case, I have it set to 3mil state, with a 4mil hard cap. So after 3mil states are created, the live time of those states will keep reducing at more states get added. By the time 4mil states exist, if a state isn't refreshed almost immediately, the state will get killed as being "old".
Another idea that popped into my head, I have no practice in these kinds of issues, but it seems as if the client machine is ACKing all of those SYNs, as it should. Would rate limiting new connections per client be an acceptable trade-off, assuming it can be done at a per client level.
Another possibility could be rate limiting via traffic shaping, how much bandwidth SYN packets get for that customer.
Just throwing around ideas.
I forgot to reply you! :-\ thanks for the tips! :)
I will try playing with those states, if others have better ideas let us know :) -
Its not to be found under system -> advanced…
-
It's where ever you set max states, I think. I forgot to check when I got home yesterday….. Definitely in the menu at the top left.