Pfsense freeze at DDoS attack - Tuning?
-
- Careful with SYN proxy, it breaks the window sizing, which means your max window is 64KB. High latency links will be crazy slow.
Yes don't use SYN proxy, that's a solution to a 1990s problem. Unless you have a device opened that's susceptible to 1990s problems of course (things with shitty IP stacks that don't have modern SYN flood mitigation built-in, talking '90s Linux versions, Windows NT 4 and earlier, etc).
- Someone needs to test loading PFSense? I got a 50Mb connection, soon to be 100Mb.
We've done and continue to do load testing. Things in 2.2 are faster than ever. Not as fast as we'd like, as we'd like to see 10 Gbps wire speed at 64 byte frames and will be doing work towards that goal, but there isn't a firewall in existence today that'll do that.
I got a 50Mb connection, soon to be 100Mb.
That's all? :) We have two gigabit drops at our primary datacenter (and redundant 10G fiber run from there to our office next door), and max that out with ease. I have 300 Mb service at home, which an APU can max out with room to spare, much less anything faster. With any normal usage, multi-Gbps is achievable.
Problem is that enterprise boxes should be able to handle that amount of traffic.
A 400 Mb flood of 64 byte packets all opening new connections? That's hell on any firewall if you're passing the traffic. The lowest limit of any firewall is in its ability to handle new connections. 400 Mbps of SYN flood traffic is over 800,000 new connections/sec. Not sure about OP's specific combination of hardware but generally speaking that's probably close to double the most you're going to get through pf. The most expensive Cisco ASA maxes out at 350,000 new connections/sec, and costs $150K-300K+ USD per box depending on which features you license.
So OP's effectively dealing with an attack that's more than twice as big as the biggest Cisco ASA that costs as much as a house could handle. Cisco's as "enterprise" as it gets. The fact pf can't deal with it either shouldn't surprise you.
Firewalls in general are the wrong answer for DDoS. No matter what firewall it is, it's going to have a hell of a hard time dealing with huge floods of new connections.
Block the flooded traffic and you'll be in as good of shape as you can be if it's hitting your firewall. Better when you're under an attack that big to have your upstream null route the IP that's being attacked. Or if you absolutely have to keep it up, send that through a DDoS mitigation provider first.
-
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.
-
With every ISP in the world trying to do real-time DPI on every connection, you would think that while they are analyzing packets you don't want them to look at, they could analyze packets you do want them to notice and stop something like this from ever hitting your firewall. I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
-
@cmb:
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.
I got my state tables filled up with only 40mbit traffic. Instantly. 20sec "attack" only but pfsense was freezed first sec.
I almost daily get ddos'ed (my customer do) Some customers are running gameservers, and scriptkiddies like to knock them out.
I even got DDoS mitigation by providor, using arbor network (best ddos protection ever).Before i couldnt take any attack/flood over 800mbit. Now i dont get knocked off straight away.My drop is on 1Gbit.
E3-2.8ghz
8gb ram
ssd disk 120gbi've attached some rrd graphs - high spike shows the attack…
-
And…. Is the traffic in question on that port? 1433?
-
Oooops, i did mix some things together. Ignore the "OVH method" and MS SQL.
But when i did a capture i saw this… many many many of this from different IPs:
84 38.733142 119.246.140.4 76.28.11.29 TCP 60 46288→80 [SYN, ECN, CWR] Seq=0 Win=0 Len=0
88 38.733361 76.28.11.29 119.246.140.4 TCP 58 80→46288 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460 -
I gave my second box (virtual) 96GB memory hoping that the table state would survive ;D
But it knocked it out anyway, and only with 20mbit SYN flood. :-XThe first post about 400mbit SSDP attack is understandable, but how come a 20mbit SYN flood just "crash" the box? Shall we suddenly just block a whole protocol?… not even that would do it, i would say....
-
I'm maybe mislead here, but my understanding is that a syn flood can be blocked unless the address is spoofed…
If the address is spoofed with a syn flood attack, you would need to have a situation where something inside your network is compromised.
A server or a LAN client or someone inside a VPN with knowledge of your network?
So, malicious client A inside your net pretending to be client B inside your net forging IPs and sending SYN to computers that never ACK
So, if this is the case, who is inside your network?
-
I'm maybe mislead here, but my understanding is that a syn flood can be blocked unless the address is spoofed…
If the address is spoofed with a syn flood attack, you would need to have a situation where something inside your network is compromised.
A server or a LAN client or someone inside a VPN with knowledge of your network?
So, malicious client A inside your net pretending to be client B inside your net forging IPs and sending SYN to computers that never ACK
So, if this is the case, who is inside your network?
The attack has nothing to do with my LAN clients, other than they get attacked by time. I made a research and found some stressers online to simulate some attacks. I tried many and pfsense survived most of the times, but one specific attack type was quite efficient to pfsense box's - it crash the box in no time with minimal bandwidth. I have other people who participate in my "firewall test", They "survived" (not using pfsene) - pfsense is way ahead on many features, troughput etc but handling this is a challenging job for it. I may try it on 2.2 when the critical bugs has been fixed….
-
There is an option in PFSense that has a linear reduction in state live time after a threshhold. I'm not at home to look at it, but I think it's under advanced or something in the general settings.
In my case, I have it set to 3mil state, with a 4mil hard cap. So after 3mil states are created, the live time of those states will keep reducing at more states get added. By the time 4mil states exist, if a state isn't refreshed almost immediately, the state will get killed as being "old".
Another idea that popped into my head, I have no practice in these kinds of issues, but it seems as if the client machine is ACKing all of those SYNs, as it should. Would rate limiting new connections per client be an acceptable trade-off, assuming it can be done at a per client level.
Another possibility could be rate limiting via traffic shaping, how much bandwidth SYN packets get for that customer.
Just throwing around ideas.
-
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Lowprofile has tested against your setup and it takes your setup offline at once.
Only one provider here in Denmark is running as nothing has happened and they are using Juniper gear.
-
I know you have my IP in Denmark Supermule… Please spare me (-;
-
I will mate :D
This is just testing and you can bring down just about anything out there. Its quite scary.
The state table fills up instantly but everything is responsive and nothing in the logs. Your drop pipeline is just filled and nothing to do about it since pfSense doesnt seem to be able to handle this specific kind of SYN traffic.
-
I just checked the servers running there on public IPs - Nothing hit yet.
Hope it stays that way. -
Dont worry mate. It wont come from any of us here.
But it can be ordered online and its very easy to take down pfsense.org and the forum if we want to (we dont).
The first seconds of the attack, pfsense dies on the WAN side while states are flooding but only on 1 specific type of SYN. Nothing in the logs and nothing reported by SNort as well.
-
I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
A SYN is a SYN is a SYN generally, which is why null routing the IP being attacked is the typical response. Can't tell the difference between legit users and the DDoS traffic in most cases.
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Again, what test is this? PM me if you don't want to post publicly.
-
Hi Chris
We use something like this
https://www.vdos-s.com
You can order any kind of attack using whatever length of time you want.
You can write to Lowprofile and give him a testing address to see how this is seen by pfSense and the results.
When using port 80 and for my own testing purposes, I have pfblockerNG running and it takes a lot of traffic away from the webserver itself but pfSense dies fast anyway. I see around 20mbit of traffic on the server and its responsive from the inside and you dont see anything wrong with it.
Its just offline.
-
@cmb:
I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
A SYN is a SYN is a SYN generally, which is why null routing the IP being attacked is the typical response. Can't tell the difference between legit users and the DDoS traffic in most cases.
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Again, what test is this? PM me if you don't want to post publicly.
PM sent! - regarding null route, i am running a layer 2 setup and a null route trough my providor cost me $200 each time :) despite i have prof. dos protection.
-
There is an option in PFSense that has a linear reduction in state live time after a threshhold. I'm not at home to look at it, but I think it's under advanced or something in the general settings.
In my case, I have it set to 3mil state, with a 4mil hard cap. So after 3mil states are created, the live time of those states will keep reducing at more states get added. By the time 4mil states exist, if a state isn't refreshed almost immediately, the state will get killed as being "old".
Another idea that popped into my head, I have no practice in these kinds of issues, but it seems as if the client machine is ACKing all of those SYNs, as it should. Would rate limiting new connections per client be an acceptable trade-off, assuming it can be done at a per client level.
Another possibility could be rate limiting via traffic shaping, how much bandwidth SYN packets get for that customer.
Just throwing around ideas.
I forgot to reply you! :-\ thanks for the tips! :)
I will try playing with those states, if others have better ideas let us know :) -
Its not to be found under system -> advanced…