Pfsense freeze at DDoS attack - Tuning?
-
Hi
When i got hit by various ddos attack types, e.x SSDP at 400mbit (have 1gbit connection) the firewall looses packets.
A constant ping shows every 5th ping is getting timeout. Only for a second, hence packetloss. I am having ddos service from providor but they only guaranteed 100mbit clean traffic.Why does pfsense freezes/doesn't respond for an attack which is half the bandwidth? It is struggling at 4-500mbit incoming traffic. I think i need to tune somewhere…
Some good hints and advice? I would appreciate it so much!
My box:
Supermicro
i3, 2.8ghz 2/4 core.
16GB ram
120 SSd disk
1Gbit NIC (quad)
1Gbit uplinkthanks in advance!
-
Skift udbyder Anders og det har ikke noget med tuning at gøre ;)
-
I have new information now…
It is now confirmed that the bottleneck is PFsense in this case. It seems like 1 special attack does freeze the box. I made a packet capture and got this:
84 38.733142 119.246.140.4 76.28.11.29 TCP 60 46288→80 [SYN, ECN, CWR] Seq=0 Win=0 Len=0
88 38.733361 76.28.11.29 119.246.140.4 TCP 58 80→46288 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460What kind of attack is this? and how to stop it? I smell some SYN attack? It seems like pfsense tries to reply the "request", hence the freeze. (many of these lines)
My "ip": 76.28.11.29 (faked)
-
What are your WAN firewall rules? By default, PFSense should be dropping incoming data, not responding to it.
-
WAN rules are mostly open with all proto and "any" "any" to several servers. Some customers are running "unmanaged" so everything is opened on their IP. They got hit on port 80, webservers or my dns servers on port 53 (no open resolvers)
I will try to play with synproxy, i have to get it fixed since i do have anti-ddos by providor but pfsense can't even handle small syn flood. I will try this guide. http://pfsensesetup.com/syn-flood-prevention-in-pfsense/
-
When tested yesterday on a 100mbit homelink the load looked like this on pfsense.
It dropped packages (ping) but remained online and did not hang at any moment.
In production, I dont have a netgear CG3000 infront and I believe some of the loss could be accounted for by that.
I dont get why you run any/any inbound since it opens up for a wide range of ways to hammer the backend servers.
-
You are running with your firewall wide open on the WAN? Why would you do such a thing?
-
You are running with your firewall wide open on the WAN? Why would you do such a thing?
Honeypot? Works well, apparently. :D :D :D
-
You are running with your firewall wide open on the WAN? Why would you do such a thing?
It is not wide open. It is open according to the need from customers. I have some virtual servers running (hosting)
Open or not. It should not be the reason :-) You can give me any ip (send me PM) where pfsense is in front and i would be able to make it freeze by sending 1 special attack. If anyone would like to try, send me pm.
-
Sure - Try pfsense.org
-
-
Well - If its truly an issue, I'm sure they will be the ones who want to know, so seems like a logical choice.
-
Supermule will confirm…. when he is online.... He also wanted to test :)
-
I need mine up - So I'd rather not be the test subject.
-
I need mine up - So I'd rather not be the test subject.
Trust me. We all want. :) Though i only did it for some seconds. i will try different scenarios and return
-
Have you looked at Calomel.org for tuning tips?
https://calomel.org/freebsd_network_tuning.html
Sysctl.conf (so most option can be set by adding them to the system tunables menu option)
# General Security and DoS mitigation #net.bpf.optimize_writers=0 # bpf are write-only unless program explicitly specifies the read filter (default 0) #net.bpf.zerocopy_enable=0 # zero-copy BPF buffers, breaks dhcpd ! (default 0) net.inet.ip.check_interface=1 # verify packet arrives on correct interface (default 0) #net.inet.ip.portrange.randomized=1 # randomize outgoing upper ports (default 1) net.inet.ip.process_options=0 # ignore IP options in the incoming packets (default 1) #net.inet.ip.random_id=1 # assign a random IP_ID to each packet leaving the system (default 0) net.inet.ip.redirect=0 # do not send IP redirects (default 1) #net.inet.ip.accept_sourceroute=0 # drop source routed packets since they can not be trusted (default 0) #net.inet.ip.sourceroute=0 # if source routed packets are accepted the route data is ignored (default 0) net.inet.ip.stealth=1 # do not reduce the TTL by one(1) when a packets goes through the firewall (default 0) #net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent to IP broadcast addresses (default 0) #net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address Mask Request packets (default 0) #net.inet.icmp.maskrepl=0 # replies are not sent for ICMP address mask requests (default 0) #net.inet.icmp.log_redirect=0 # do not log redirected ICMP packet attempts (default 0) net.inet.icmp.drop_redirect=1 # no redirected ICMP packets (default 0) #net.inet.icmp.icmplim=500 # number of ICMP/TCP RST packets/sec, increase for bittorrent or many clients. (default 200) #net.inet.icmp.icmplim_output=1 # show "Limiting open port RST response" messages (default 1) #net.inet.tcp.always_keepalive=0 # tcp keep alive detection for dead peers, can be spoofed (default 1) net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on initial connection (default 0) #net.inet.tcp.ecn.enable=1 # explicit congestion notification (ecn) warning: some ISP routers abuse ECN (default 0) net.inet.tcp.fast_finwait2_recycle=1 # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0) net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid spoofed icmp/udp floods (default 1) #net.inet.tcp.maxtcptw=15000 # max number of tcp time_wait states for closing connections (default 5120) net.inet.tcp.msl=5000 # 5s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000) net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1) #net.inet.tcp.rfc3042=1 # on packet loss trigger the fast retransmit algorithm instead of tcp timeout (default 1) net.inet.udp.blackhole=1 # drop udp packets destined for closed sockets (default 0) net.inet.tcp.blackhole=2 # drop tcp packets destined for closed ports (default 0) #net.route.netisr_maxqlen=2048 # route queue length (rtsock using "netstat -Q") (default 256) security.bsd.see_other_uids=0 # users only see their own processes. root can see all (default 1)
-
Problem is that enterprise boxes should be able to handle that amount of traffic.
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
-
-
Careful with SYN proxy, it breaks the window sizing, which means your max window is 64KB. High latency links will be crazy slow.
-
Someone needs to test loading PFSense? I got a 50Mb connection, soon to be 100Mb.
-
-
- Careful with SYN proxy, it breaks the window sizing, which means your max window is 64KB. High latency links will be crazy slow.
Yes don't use SYN proxy, that's a solution to a 1990s problem. Unless you have a device opened that's susceptible to 1990s problems of course (things with shitty IP stacks that don't have modern SYN flood mitigation built-in, talking '90s Linux versions, Windows NT 4 and earlier, etc).
- Someone needs to test loading PFSense? I got a 50Mb connection, soon to be 100Mb.
We've done and continue to do load testing. Things in 2.2 are faster than ever. Not as fast as we'd like, as we'd like to see 10 Gbps wire speed at 64 byte frames and will be doing work towards that goal, but there isn't a firewall in existence today that'll do that.
I got a 50Mb connection, soon to be 100Mb.
That's all? :) We have two gigabit drops at our primary datacenter (and redundant 10G fiber run from there to our office next door), and max that out with ease. I have 300 Mb service at home, which an APU can max out with room to spare, much less anything faster. With any normal usage, multi-Gbps is achievable.
Problem is that enterprise boxes should be able to handle that amount of traffic.
A 400 Mb flood of 64 byte packets all opening new connections? That's hell on any firewall if you're passing the traffic. The lowest limit of any firewall is in its ability to handle new connections. 400 Mbps of SYN flood traffic is over 800,000 new connections/sec. Not sure about OP's specific combination of hardware but generally speaking that's probably close to double the most you're going to get through pf. The most expensive Cisco ASA maxes out at 350,000 new connections/sec, and costs $150K-300K+ USD per box depending on which features you license.
So OP's effectively dealing with an attack that's more than twice as big as the biggest Cisco ASA that costs as much as a house could handle. Cisco's as "enterprise" as it gets. The fact pf can't deal with it either shouldn't surprise you.
Firewalls in general are the wrong answer for DDoS. No matter what firewall it is, it's going to have a hell of a hard time dealing with huge floods of new connections.
Block the flooded traffic and you'll be in as good of shape as you can be if it's hitting your firewall. Better when you're under an attack that big to have your upstream null route the IP that's being attacked. Or if you absolutely have to keep it up, send that through a DDoS mitigation provider first.
-
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.
-
With every ISP in the world trying to do real-time DPI on every connection, you would think that while they are analyzing packets you don't want them to look at, they could analyze packets you do want them to notice and stop something like this from ever hitting your firewall. I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
-
@cmb:
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.
I got my state tables filled up with only 40mbit traffic. Instantly. 20sec "attack" only but pfsense was freezed first sec.
I almost daily get ddos'ed (my customer do) Some customers are running gameservers, and scriptkiddies like to knock them out.
I even got DDoS mitigation by providor, using arbor network (best ddos protection ever).Before i couldnt take any attack/flood over 800mbit. Now i dont get knocked off straight away.My drop is on 1Gbit.
E3-2.8ghz
8gb ram
ssd disk 120gbi've attached some rrd graphs - high spike shows the attack…
-
And…. Is the traffic in question on that port? 1433?
-
Oooops, i did mix some things together. Ignore the "OVH method" and MS SQL.
But when i did a capture i saw this… many many many of this from different IPs:
84 38.733142 119.246.140.4 76.28.11.29 TCP 60 46288→80 [SYN, ECN, CWR] Seq=0 Win=0 Len=0
88 38.733361 76.28.11.29 119.246.140.4 TCP 58 80→46288 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460
-
I gave my second box (virtual) 96GB memory hoping that the table state would survive ;D
But it knocked it out anyway, and only with 20mbit SYN flood. :-XThe first post about 400mbit SSDP attack is understandable, but how come a 20mbit SYN flood just "crash" the box? Shall we suddenly just block a whole protocol?… not even that would do it, i would say....
-
I'm maybe mislead here, but my understanding is that a syn flood can be blocked unless the address is spoofed…
If the address is spoofed with a syn flood attack, you would need to have a situation where something inside your network is compromised.
A server or a LAN client or someone inside a VPN with knowledge of your network?
So, malicious client A inside your net pretending to be client B inside your net forging IPs and sending SYN to computers that never ACK
So, if this is the case, who is inside your network?
-
I'm maybe mislead here, but my understanding is that a syn flood can be blocked unless the address is spoofed…
If the address is spoofed with a syn flood attack, you would need to have a situation where something inside your network is compromised.
A server or a LAN client or someone inside a VPN with knowledge of your network?
So, malicious client A inside your net pretending to be client B inside your net forging IPs and sending SYN to computers that never ACK
So, if this is the case, who is inside your network?
The attack has nothing to do with my LAN clients, other than they get attacked by time. I made a research and found some stressers online to simulate some attacks. I tried many and pfsense survived most of the times, but one specific attack type was quite efficient to pfsense box's - it crash the box in no time with minimal bandwidth. I have other people who participate in my "firewall test", They "survived" (not using pfsene) - pfsense is way ahead on many features, troughput etc but handling this is a challenging job for it. I may try it on 2.2 when the critical bugs has been fixed….
-
There is an option in PFSense that has a linear reduction in state live time after a threshhold. I'm not at home to look at it, but I think it's under advanced or something in the general settings.
In my case, I have it set to 3mil state, with a 4mil hard cap. So after 3mil states are created, the live time of those states will keep reducing at more states get added. By the time 4mil states exist, if a state isn't refreshed almost immediately, the state will get killed as being "old".
Another idea that popped into my head, I have no practice in these kinds of issues, but it seems as if the client machine is ACKing all of those SYNs, as it should. Would rate limiting new connections per client be an acceptable trade-off, assuming it can be done at a per client level.
Another possibility could be rate limiting via traffic shaping, how much bandwidth SYN packets get for that customer.
Just throwing around ideas.
-
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Lowprofile has tested against your setup and it takes your setup offline at once.
Only one provider here in Denmark is running as nothing has happened and they are using Juniper gear.
-
I know you have my IP in Denmark Supermule… Please spare me (-;
-
I will mate :D
This is just testing and you can bring down just about anything out there. Its quite scary.
The state table fills up instantly but everything is responsive and nothing in the logs. Your drop pipeline is just filled and nothing to do about it since pfSense doesnt seem to be able to handle this specific kind of SYN traffic.
-
I just checked the servers running there on public IPs - Nothing hit yet.
Hope it stays that way.
-
Dont worry mate. It wont come from any of us here.
But it can be ordered online and its very easy to take down pfsense.org and the forum if we want to (we dont).
The first seconds of the attack, pfsense dies on the WAN side while states are flooding but only on 1 specific type of SYN. Nothing in the logs and nothing reported by SNort as well.
-
I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
A SYN is a SYN is a SYN generally, which is why null routing the IP being attacked is the typical response. Can't tell the difference between legit users and the DDoS traffic in most cases.
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Again, what test is this? PM me if you don't want to post publicly.
-
Hi Chris
We use something like this
You can order any kind of attack using whatever length of time you want.
You can write to Lowprofile and give him a testing address to see how this is seen by pfSense and the results.
When using port 80 and for my own testing purposes, I have pfblockerNG running and it takes a lot of traffic away from the webserver itself but pfSense dies fast anyway. I see around 20mbit of traffic on the server and its responsive from the inside and you dont see anything wrong with it.
Its just offline.
-
@cmb:
I mean surely if they can methodically drop traffic you want they can methodically drop traffic you don't want?
A SYN is a SYN is a SYN generally, which is why null routing the IP being attacked is the typical response. Can't tell the difference between legit users and the DDoS traffic in most cases.
Chris…This is my home setup that we did testing on.
It doesnt use all ressources but the connection (100mbit) goes offline at once.
Again, what test is this? PM me if you don't want to post publicly.
PM sent! - regarding null route, i am running a layer 2 setup and a null route trough my providor cost me $200 each time :) despite i have prof. dos protection.
-
There is an option in PFSense that has a linear reduction in state live time after a threshhold. I'm not at home to look at it, but I think it's under advanced or something in the general settings.
In my case, I have it set to 3mil state, with a 4mil hard cap. So after 3mil states are created, the live time of those states will keep reducing at more states get added. By the time 4mil states exist, if a state isn't refreshed almost immediately, the state will get killed as being "old".
Another idea that popped into my head, I have no practice in these kinds of issues, but it seems as if the client machine is ACKing all of those SYNs, as it should. Would rate limiting new connections per client be an acceptable trade-off, assuming it can be done at a per client level.
Another possibility could be rate limiting via traffic shaping, how much bandwidth SYN packets get for that customer.
Just throwing around ideas.
I forgot to reply you! :-\ thanks for the tips! :)
I will try playing with those states, if others have better ideas let us know :)
-
Its not to be found under system -> advanced…
-
It's where ever you set max states, I think. I forgot to check when I got home yesterday….. Definitely in the menu at the top left.
-
FOUND IT!
Thanks mate! ;)