Pfsense freeze at DDoS attack - Tuning?
-
Hi
When i got hit by various ddos attack types, e.x SSDP at 400mbit (have 1gbit connection) the firewall looses packets.
A constant ping shows every 5th ping is getting timeout. Only for a second, hence packetloss. I am having ddos service from providor but they only guaranteed 100mbit clean traffic.Why does pfsense freezes/doesn't respond for an attack which is half the bandwidth? It is struggling at 4-500mbit incoming traffic. I think i need to tune somewhere…
Some good hints and advice? I would appreciate it so much!
My box:
Supermicro
i3, 2.8ghz 2/4 core.
16GB ram
120 SSd disk
1Gbit NIC (quad)
1Gbit uplinkthanks in advance!
-
Skift udbyder Anders og det har ikke noget med tuning at gøre ;)
-
I have new information now…
It is now confirmed that the bottleneck is PFsense in this case. It seems like 1 special attack does freeze the box. I made a packet capture and got this:
84 38.733142 119.246.140.4 76.28.11.29 TCP 60 46288→80 [SYN, ECN, CWR] Seq=0 Win=0 Len=0
88 38.733361 76.28.11.29 119.246.140.4 TCP 58 80→46288 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460What kind of attack is this? and how to stop it? I smell some SYN attack? It seems like pfsense tries to reply the "request", hence the freeze. (many of these lines)
My "ip": 76.28.11.29 (faked)
-
What are your WAN firewall rules? By default, PFSense should be dropping incoming data, not responding to it.
-
WAN rules are mostly open with all proto and "any" "any" to several servers. Some customers are running "unmanaged" so everything is opened on their IP. They got hit on port 80, webservers or my dns servers on port 53 (no open resolvers)
I will try to play with synproxy, i have to get it fixed since i do have anti-ddos by providor but pfsense can't even handle small syn flood. I will try this guide. http://pfsensesetup.com/syn-flood-prevention-in-pfsense/ -
When tested yesterday on a 100mbit homelink the load looked like this on pfsense.
It dropped packages (ping) but remained online and did not hang at any moment.
In production, I dont have a netgear CG3000 infront and I believe some of the loss could be accounted for by that.
I dont get why you run any/any inbound since it opens up for a wide range of ways to hammer the backend servers.
-
You are running with your firewall wide open on the WAN? Why would you do such a thing?
-
You are running with your firewall wide open on the WAN? Why would you do such a thing?
Honeypot? Works well, apparently. :D :D :D
-
You are running with your firewall wide open on the WAN? Why would you do such a thing?
It is not wide open. It is open according to the need from customers. I have some virtual servers running (hosting)
Open or not. It should not be the reason :-) You can give me any ip (send me PM) where pfsense is in front and i would be able to make it freeze by sending 1 special attack. If anyone would like to try, send me pm. -
Sure - Try pfsense.org
-
-
Well - If its truly an issue, I'm sure they will be the ones who want to know, so seems like a logical choice.
-
Supermule will confirm…. when he is online.... He also wanted to test :)
-
I need mine up - So I'd rather not be the test subject.
-
I need mine up - So I'd rather not be the test subject.
Trust me. We all want. :) Though i only did it for some seconds. i will try different scenarios and return
-
Have you looked at Calomel.org for tuning tips?
https://calomel.org/freebsd_network_tuning.html
Sysctl.conf (so most option can be set by adding them to the system tunables menu option)
# General Security and DoS mitigation #net.bpf.optimize_writers=0 # bpf are write-only unless program explicitly specifies the read filter (default 0) #net.bpf.zerocopy_enable=0 # zero-copy BPF buffers, breaks dhcpd ! (default 0) net.inet.ip.check_interface=1 # verify packet arrives on correct interface (default 0) #net.inet.ip.portrange.randomized=1 # randomize outgoing upper ports (default 1) net.inet.ip.process_options=0 # ignore IP options in the incoming packets (default 1) #net.inet.ip.random_id=1 # assign a random IP_ID to each packet leaving the system (default 0) net.inet.ip.redirect=0 # do not send IP redirects (default 1) #net.inet.ip.accept_sourceroute=0 # drop source routed packets since they can not be trusted (default 0) #net.inet.ip.sourceroute=0 # if source routed packets are accepted the route data is ignored (default 0) net.inet.ip.stealth=1 # do not reduce the TTL by one(1) when a packets goes through the firewall (default 0) #net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent to IP broadcast addresses (default 0) #net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address Mask Request packets (default 0) #net.inet.icmp.maskrepl=0 # replies are not sent for ICMP address mask requests (default 0) #net.inet.icmp.log_redirect=0 # do not log redirected ICMP packet attempts (default 0) net.inet.icmp.drop_redirect=1 # no redirected ICMP packets (default 0) #net.inet.icmp.icmplim=500 # number of ICMP/TCP RST packets/sec, increase for bittorrent or many clients. (default 200) #net.inet.icmp.icmplim_output=1 # show "Limiting open port RST response" messages (default 1) #net.inet.tcp.always_keepalive=0 # tcp keep alive detection for dead peers, can be spoofed (default 1) net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on initial connection (default 0) #net.inet.tcp.ecn.enable=1 # explicit congestion notification (ecn) warning: some ISP routers abuse ECN (default 0) net.inet.tcp.fast_finwait2_recycle=1 # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0) net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid spoofed icmp/udp floods (default 1) #net.inet.tcp.maxtcptw=15000 # max number of tcp time_wait states for closing connections (default 5120) net.inet.tcp.msl=5000 # 5s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000) net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1) #net.inet.tcp.rfc3042=1 # on packet loss trigger the fast retransmit algorithm instead of tcp timeout (default 1) net.inet.udp.blackhole=1 # drop udp packets destined for closed sockets (default 0) net.inet.tcp.blackhole=2 # drop tcp packets destined for closed ports (default 0) #net.route.netisr_maxqlen=2048 # route queue length (rtsock using "netstat -Q") (default 256) security.bsd.see_other_uids=0 # users only see their own processes. root can see all (default 1) -
Problem is that enterprise boxes should be able to handle that amount of traffic.
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
-
-
Careful with SYN proxy, it breaks the window sizing, which means your max window is 64KB. High latency links will be crazy slow.
-
Someone needs to test loading PFSense? I got a 50Mb connection, soon to be 100Mb.
-
-
- Careful with SYN proxy, it breaks the window sizing, which means your max window is 64KB. High latency links will be crazy slow.
Yes don't use SYN proxy, that's a solution to a 1990s problem. Unless you have a device opened that's susceptible to 1990s problems of course (things with shitty IP stacks that don't have modern SYN flood mitigation built-in, talking '90s Linux versions, Windows NT 4 and earlier, etc).
- Someone needs to test loading PFSense? I got a 50Mb connection, soon to be 100Mb.
We've done and continue to do load testing. Things in 2.2 are faster than ever. Not as fast as we'd like, as we'd like to see 10 Gbps wire speed at 64 byte frames and will be doing work towards that goal, but there isn't a firewall in existence today that'll do that.
I got a 50Mb connection, soon to be 100Mb.
That's all? :) We have two gigabit drops at our primary datacenter (and redundant 10G fiber run from there to our office next door), and max that out with ease. I have 300 Mb service at home, which an APU can max out with room to spare, much less anything faster. With any normal usage, multi-Gbps is achievable.
Problem is that enterprise boxes should be able to handle that amount of traffic.
A 400 Mb flood of 64 byte packets all opening new connections? That's hell on any firewall if you're passing the traffic. The lowest limit of any firewall is in its ability to handle new connections. 400 Mbps of SYN flood traffic is over 800,000 new connections/sec. Not sure about OP's specific combination of hardware but generally speaking that's probably close to double the most you're going to get through pf. The most expensive Cisco ASA maxes out at 350,000 new connections/sec, and costs $150K-300K+ USD per box depending on which features you license.
So OP's effectively dealing with an attack that's more than twice as big as the biggest Cisco ASA that costs as much as a house could handle. Cisco's as "enterprise" as it gets. The fact pf can't deal with it either shouldn't surprise you.
Firewalls in general are the wrong answer for DDoS. No matter what firewall it is, it's going to have a hell of a hard time dealing with huge floods of new connections.
Block the flooded traffic and you'll be in as good of shape as you can be if it's hitting your firewall. Better when you're under an attack that big to have your upstream null route the IP that's being attacked. Or if you absolutely have to keep it up, send that through a DDoS mitigation provider first.
-
It times out at 75mbit using af specific sort of DDoS packages despite tuning.
On what hardware? What specifically are you running to throw traffic at it?
75 Mbps of 64 byte frames is over 150,000 pps (which if it weren't all new connections wouldn't be a big deal with fast enough hardware), and 150,000 new connections/sec of traffic you're passing with any firewall requires some serious horsepower.