What is the biggest attack in GBPS you stopped
-
-
Enabled Snort to see if it made things worse.
The answer to that is yes and no… initial phase was really good, since it took out the initial spike in CPU and made the load more even (slower boost). CPU had a shorter MAX load interval than on previous attacks.
It activated a more even CPU usage on the 8 cores and recovery was really good.
Traffic graphs didnt fare as well as with Cron only running.
-
-
Enabling Apinger did not do any good to the box as expected.
More CPU on top and a minute longer to recover. For what its worth, the impact didnt seem as big as the first test but still a trend (big).
Traffic graphs very unresponsive as the rest of the GUI. It dies on the first CPU spike in the GUI.
-
So maybe some services are more resource hungry and/or less refined in the scheme of things.
I wonder if there would be any benefit in having two firewalls in series, where the forward facing/1st was stripped of all unnecessary tasks/services, then the 2nd inline firewall had the unnecessary services/tasks running on it?
It certainly seems like there might not be any one property, service/task which is at fault, but maybe a combination of things which can affect how well the system stays up.
Have you seen the links I pm'ed almabes regarding setting up pfsense to reduce/avoid syn floods? If so did you give them ago and how did they perform?
-
-
I have come a BIG step closer to locating the culprit.
Look at the graphs when NTPD is enabled.
It destroys the GUI completely and takes the interfaces offline in the GUI. No response from them. The graphs is a 3 minute attack and only maybe 10 seconds are showing.
Whats really interesting is the VmWare graph. When it spikes for the last time, the GUI comes back and the CPU graph in the GUI starts working again.
Wonder if NTPD and Apinger together could make something?
-
-
-
-
Running stateless with the box not having any limits to states pr sec. and other things pr. rule settings…
Box ran fine. Responsive. A little fallout on the traffic graphs but not so bad as seen before.
CPU load is a LOT less and only 5 dropped packets to Google via ping.
Instead of beeing crippled to a halt, it actually routed traffic to the server behind.
Whats a little odd, is that the traffic doubled in bandwith from around 4-5mbit/s to around 8-10mbit/s running stateless compared to SynProxy state.
-
-
Last one for today…
Enabling NTPD was also something that crippled the box.
Whats really interesting in the graphs in VmWare. The last 3 is the following:
1: Stateless NOT running Apinger and NTPD. No CPU hits 100% and the box is responsive and routes traffic.
2: Stateless Runing Apinger but NOT NTPD. 1 CPU (nr. 4) is 100% and the box stops routing and loses packets.
3: Stateless NOT running Apinger but running NTPD. 1 CPU is 100% (nr. 3) and the box stops responding and loses packets.1st graph doesnt have the small "bump" at the end of the attack and is responsive all along. When enabling Apinger OR NTPD OR both, then the box dies and recovery time is long (minutes). Recovery time is longer when running Apinger than with NTPD running.
When running SynProxy state the same pattern can be seen when attacked. Some CPU runs 100% and the box is dead.
Last image is a better view of the cpu usage.
1st one maxes out and packet loss occurs. 2nd does not and routes everything fine. 3rd is initially fine, but as soon as 1 cpu hits 100%, then the box is gone. (about halfway into the attack).
-
What I noticed was that pfSense would not let go of states for several minutes (5+). So when I was hit with 4.8M states, I'd still see 2.9M several minutes later. It wasn't until I rebooted the box several hours later that it returned to 3,500 states. IMHO boxes that are unresponsive after attacks still haven't released their states. Mine for whatever reason, recovered almost immediately after the attacks ceased.
Here is my hardware:
Intel Core i3-2100 Sandy Bridge dual core
Intel BOXDQ77MK LGA 1155 Intel Q77
4GB RAM
320 GB 7200RM HD
2 x Intel EXPI9301CTBLK 10/ 100/ 1000Mbps PCI-Express Network AdapterAlso note, that the initial SYN flood significantly burdened the UI but not the console, and when I increased states, the box was fine. The interface that was being attacked was disabled, but the box and the other three interfaces were working perfectly.
-
My setup:
Cisco Comcast business CPE -> Cisco SG300 switch -> WAN side of two firewalls, one HW one VM
VM firewall is running on a Dell precision 7500 with dual Xeon 5650 processors and 48 GB RAM
ESXi 6.0
Official ESF pfSense 2.2.2 OVA (2GB ram and 2 cores)
WAN goes to a broadcom add in NIC
LAN is coming out the onboard NIC and plugs into a catalyst 2948 switchThe Hardware firewall is a VK40-TE. It runs the nanoBSD version of pfSense 2.2.2
Wan side plugs into the SG300
Lan into the 2948I have 5 IPs, so the two firewalls have different WAN interface IP addresses. Additionally, on the VM firewall, I 1:1 natted a windows box and opened RDP.
I set up two laptops on the LAN, one wireless, because I wanted to sit on my couch. The other was plugged into the 2948. The wired laptop was configured to use the VM as it's gateway. I fired of ping -t www.google.com, ping -t <other firewall="" ip="">, and ping -t <comcast public="" cpe="" ip="">.
The results were unexpected.
During the attack, I could ping the WAN interface of the "opposite" firewall and get a 2ms response, as if nothing was happening. After a few seconds, www.google.com failed to reply, or came back with 2100+ ms replies, through BOTH firewalls. Even the un attacked one. The Comcastic gateway device went from sub 10 ms pings to 300-400ms.
Even after the attack stopped, the Comcastic gateway failed to route traffic. I had to power cycle it to get back on the grid.</comcast></other>
-
Did one more test, because I noticed a configuration error in my test setup.
After I reconfigured, I RDPd over to the test Win2k12 box's public IP, fired up wireshark and had supermule attack it.
I set up a ping to www.google.com through the un atacked firewall. It immediately started timing out.So again, my comcast gateway quickly crapped on itself, but pfSense didn't break a sweat. I was able to watch the packet capture over the RDP connection through the instance of pfSense under attack.
So, does the Cisco DPC3939B suck rocks, or was it "protecting" me by taking the brunt of the attack?
My states never got above 40k.
CPU hit about 30%, once.
RDP session never blinked.As soon as the attack stopped, www.google.com was pingable again.
-
Its your GW Cisco box that crapped itself and took the heat of pfsense.
-
Its your GW Cisco box that crapped itself and took the heat of pfsense.
I agree. My next step is to hit up a friend of mine that is the owner and chief packet plumber of an ISP. I'll haul the VM box over there and we'll test.
-
After deleting the Vmware Tools then I disabled Apinger and NTPD.
The graphs on ESXi looked the same as "normal". No jitter from Apinger and NTPD afterwards.
Recovery was instant. Traffic graphs didnt respond well.
Might be relevant for tuning apinger.
https://forum.pfsense.org/index.php?topic=37740.msg194857#msg194857WRT the gui going unresponsive during an attack, you guys wont be logged into the gui in real life anyway, it will probably occur when you are at home cooking on the bbq, so the fact the gui goes unresponsive is perhaps best looked at as a side show distraction not really relevant in the scheme of things.
How are those affected handling the log data? Are you just writing to pfsense's own log or are you syslogging it out to another machine?
WRT to NTPD, I noticed back over Xmas some of the NTP pools were being dominated by certain countries. I didnt like this so I moved away from the default NTP pools used by pfsense and chose some of my own to put in pfsense, although building our own NTP server might be an option http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html but I should also point out, it might be possible to hijack/affect the GPS signals based on a hack I read about a while back (which has quite major significance when you think about it) but cant find the link to it atm.
-
GUI is constantly monitored here by employees.
So yes it is.
-
Ok, but I'd suggest that would be a minority but could be wrong as I know I'm not constantly logged in monitoring things.
Do you still log to syslog though so you have historical data in which to look back over data and spot any patterns?
-
FYI PFSense defaults to an established TCP state to last 24 hours, even with zero traffic, as long as no FIN packet happens.
-
GUI is constantly monitored here by employees.
So yes it is.
I have to agree here. One of the things i do for several clients is proactively monitor their firewalls.
-
Ok, but I'd suggest that would be a minority but could be wrong as I know I'm not constantly logged in monitoring things.
Do you still log to syslog though so you have historical data in which to look back over data and spot any patterns?
I actually use the UI but also monitory with OpenNMS. I pulled my system logs after the attack and preserved some Wireshark pcaps. I have Security Onion running to do pcaps on the same mirrored interface, but haven't set up syslog yet. That shouldn't be too tough to do in the scheme of things.
-
I have come a BIG step closer to locating the culprit.
Look at the graphs when NTPD is enabled.
It destroys the GUI completely and takes the interfaces offline in the GUI. No response from them. The graphs is a 3 minute attack and only maybe 10 seconds are showing.
Whats really interesting is the VmWare graph. When it spikes for the last time, the GUI comes back and the CPU graph in the GUI starts working again.
Wonder if NTPD and Apinger together could make something?
This makes sense because the kind of attack you hit me with is classified as an NTP attack.
![Screen Shot 2015-05-24 at 10.54.49 PM.png](/public/imported_attachments/1/Screen Shot 2015-05-24 at 10.54.49 PM.png)
![Screen Shot 2015-05-24 at 10.54.49 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-05-24 at 10.54.49 PM.png_thumb)
![Screen Shot 2015-05-24 at 10.55.16 PM.png](/public/imported_attachments/1/Screen Shot 2015-05-24 at 10.55.16 PM.png)
![Screen Shot 2015-05-24 at 10.55.16 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-05-24 at 10.55.16 PM.png_thumb) -
https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks/
https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/
https://www.acunetix.com/blog/articles/ntp-reflection-ddos-attacks/
https://forum.pfsense.org/index.php?topic=71396.0
"If you have appropriate WAN rules to stop the Internet from reaching your firewall's NTP server, then good news, you have nothing to do. However, if you have opened your NTP service up on purpose or if you have overly permissive rules (e.g. "allow all on WAN") and you don't want to change them, you can apply the following fix to change the behavior of the NTP daemon so it will no longer respond to the monlist command:"https://redmine.pfsense.org/issues/3496
https://redmine.pfsense.org/issues/3384 -
I dont have any NTP ports open on WAN. Only open port is 80.
-
-
-
CONCLUSION:
As soon as you have a working port forward (NOT DISABLED) to a server behind and pfSense needs to route it, youre dead.
Even at limited SYN Proxy state enabled and only 3mbit traffic passing through and states never reaches above 10%. Disabling every service on the firewall and still dead.
If you dont have a portforward, then its fine but that misses the main reason for having this setup….. IMHO.
Its definately FreeBSD/PF related and I dont believe its in the ESF code unless they alter the Packet Filter code.
I would be glad if anyone would run a base OpenBSD/FreeBSD with PF enabled as a frontend so we can continue testing.
-
@Supermule, tell me, is the script posted in this thread the one causing the problem you see?
If so, does it also cause the problem you describe here? https://forum.pfsense.org/index.php?topic=87571.msg492268#msg492268
It seems like we are going around in circles at this stage this is why I ask.
Its also like CMB says here https://forum.pfsense.org/index.php?topic=87571.msg493401#msg493401
"DDoS is hell on stateful firewalls is the basic summary of this thread. It's not specific to anything in any particular firewall."The very nature of any stateful firewall will cause an increase in resource use like you are seeing.
With this is mind, what can you do to limit your exposure [edit]
ofto [/edit] the weakness of a stateful firewall?Lots of suggestions here
http://www.cisco.com/web/about/security/intelligence/guide_ddos_defense.htmlSome users who might be hosting a website that is providing services/products only to punters in their own country, can limit access to the ip addresses to those assigned. If its something being offered further afield like to a continent, then rinse and repeat the above but with all the continents ip address blocks.
If its something flogged globally, then consider a website sat behind the TLD specific to that country, ie if in the UK then a website assigned to a .co.uk could help, but then you'd need something to redirect the originating ip address to the correct country domain. This approach can lesson but not eradicate 100% a DDOS attack of sorts.
Perhaps having something that temporarily disables the port forward in realtime when the CPU activity reaches a threshold might be a way around the problem to avoid taking out the firewall if the other tuning options like increasing the default states et al doesnt work.
Either way, theres lots of ways to skin the cat!
-
@Supermule, tell me, is the script posted in this thread the one causing the problem you see?
It can be altered to do so.
If so, does it also cause the problem you describe here? https://forum.pfsense.org/index.php?topic=87571.msg492268#msg492268
Yes. It takes more bandwith to do it. Limiting bandwith will make pfsense survive with no ports open.
It seems like we are going around in circles at this stage this is why I ask.
Its also like CMB says here https://forum.pfsense.org/index.php?topic=87571.msg493401#msg493401
"DDoS is hell on stateful firewalls is the basic summary of this thread. It's not specific to anything in any particular firewall."The very nature of any stateful firewall will cause an increase in resource use like you are seeing.
With this is mind, what can you do to limit your exposure [edit]
ofto [/edit] the weakness of a stateful firewall?This takes down a stateless setup as well with 15mbit of traffic. So its not related to the states.
Some users who might be hosting a website that is providing services/products only to punters in their own country, can limit access to the ip addresses to those assigned. If its something being offered further afield like to a continent, then rinse and repeat the above but with all the continents ip address blocks.
If its something flogged globally, then consider a website sat behind the TLD specific to that country, ie if in the UK then a website assigned to a .co.uk could help, but then you'd need something to redirect the originating ip address to the correct country domain. This approach can lesson but not eradicate 100% a DDOS attack of sorts.
Perhaps having something that temporarily disables the port forward in realtime when the CPU activity reaches a threshold might be a way around the problem to avoid taking out the firewall if the other tuning options like increasing the default states et al doesnt work.
It doesnt matter. The states itself is not the issue. Increase it and you wont hit the limit. As soon as you port forward, its dead. But then you cant host anything behind it.
Either way, theres lots of ways to skin the cat!
IMHO there is no cat to skin currently. As stated, if you port forward, youre dead.
-
It doesnt matter. The states itself is not the issue. Increase it and you wont hit the limit. As soon as you port forward, its dead. But then you cant host anything behind it.
This behaviour which takes down stateless & stateful fw's is seen with that script posted in the msg link below?
https://forum.pfsense.org/index.php?topic=91856.msg523649#msg523649
-
https://forum.pfsense.org/index.php?topic=91856.msg523921#msg523921
-
@Supermule, tell me, is the script posted in this thread the one causing the problem you see?
If so, does it also cause the problem you describe here? https://forum.pfsense.org/index.php?topic=87571.msg492268#msg492268
It seems like we are going around in circles at this stage this is why I ask.
Its also like CMB says here https://forum.pfsense.org/index.php?topic=87571.msg493401#msg493401
"DDoS is hell on stateful firewalls is the basic summary of this thread. It's not specific to anything in any particular firewall."The very nature of any stateful firewall will cause an increase in resource use like you are seeing.
With this is mind, what can you do to limit your exposure [edit]
ofto [/edit] the weakness of a stateful firewall?Lots of suggestions here
http://www.cisco.com/web/about/security/intelligence/guide_ddos_defense.htmlSome users who might be hosting a website that is providing services/products only to punters in their own country, can limit access to the ip addresses to those assigned. If its something being offered further afield like to a continent, then rinse and repeat the above but with all the continents ip address blocks.
If its something flogged globally, then consider a website sat behind the TLD specific to that country, ie if in the UK then a website assigned to a .co.uk could help, but then you'd need something to redirect the originating ip address to the correct country domain. This approach can lesson but not eradicate 100% a DDOS attack of sorts.
Perhaps having something that temporarily disables the port forward in realtime when the CPU activity reaches a threshold might be a way around the problem to avoid taking out the firewall if the other tuning options like increasing the default states et al doesnt work.
Either way, theres lots of ways to skin the cat!
DDOS attacks can't be stopped because it's trying to shove 100Gb/s down a 1Gb pipe. Lots of different IPs does not a DDOS make. 3Mb of traffic hitting a 10Gb firewall is not what anyone in their right mind would call a DDOS. If a firewall dies, it's because of a slow path that is definitely not O(1).
Packets hitting the firewall should trigger an O(1) to see if the state exists, if it does, pass, if it does not, go to next check
Packets not passed should trigger an O(n) to compare the new state against firewall rules. If good, add and pass, else block
These two I can't see being an issue without some absurdly crazy firewall rules.NAT sits in there somewhere, not sure where, and possibly some other routing related stuff
Whatever is going on, if I take the 30Mb/s that I saw SuperMule take down my firewall with, and assume I only have one CPU, then it's taking over 100k clock cycles per packet. Since I have a quad core and all 4 cores were getting hosed, and the packets were actually quite large, it was more like 1mil clockcycles per packet. Since my system was not keeping up, that means it was worse than 1mil/packet.
I'm not sure what kind of slow path warrants 1mil cycles to decide what to do with a packet. 1mil cycles is a lot of work. You can encrypt 2.3mil bits with AES. Another way to put it. 1Gb/s over SMB was about 0.5% cpu on my old 2.67ghz cpu, which once translated down to 1mhz(1mil cycles per second), then into 1500 byte packets, is about 780 packets per second. In the time PFSense processes one of these packets, my Windows box could have transferred 780 packets via SMB.
The funny thing is it isn't the number of states. SuperMule did an attack that hit a forwarded port, which means the states were being created. It was up against the 4mil state limit, yes CPU was higher than normal, but the system was perfectly stable. A similar attack against blocked ports resulted in the same thing, everything was fine. PFSense does handle lots of blocked traffic just fine. Whatever is going on is triggering something other than just blocking traffic.
Lets make an analogy. If normal person stands on scale and it says they weigh 75 tons, you don't ask them to take off their shoes. That's about the same magnitude difference.
-
I was just thinking, when my 4mil states are full up and the firewall is trying to expire states and whatever is done, scanning 4mil states could be a bit of work. I wonder how the attack would do if the state table was made to something small, like 10k states. Expiring states may require an O(n) scan when lots of states are made about the same time.
Another thought is I have "Firewall Adaptive Timeouts"(System->Advanced->Firewall/NAT) set to 4mil states, which means by the time my state table gets full, states are being expired instantly. If expiration causes a full scan of at least triggers often, then creating states quickly and just as quickly expiring them may be the cause. I don't know. Just a thought.
That could explain why it takes a few tens of seconds before I started to feel the hurt of the attack, the issue didn't full trigger until the state table got full.
edit: more thoughts
I wonder if I set the target expiration to 4mil, but set the max to something larger. I'm not sure what PFSense/FreeBSD has to do when the table gets full, it may trigger a bad code path. Maybe I should have the table set to something like 5mil max states, but leave the adaptive timeout to 4mil.
edit2:
If the issue is an issue involving states, a good test to make an extreme could be to try a few combinations. 1mil max states with a target of 10k, should never get much past 10k, but shouldn't hit the max state limit. 10k max with 10k target, 10k max with no adaptive, etc.
-
@supermule So whats different with your setup then compared to what I have setup?
edit:
CPU Microcode perhaps? My VM is running on an AMD CPU which has some bugs affecting threading performance, all the others affected who have posted are running Intel CPU's iirc so maybe thats why I dont see the problem?edit2: This might also be a factor especially considering the vmware point which I've had to tweak as its possible to get the time to drift on vm's.
http://en.wikipedia.org/wiki/HPETAnyone know if HPET is built into the pfsense builds?
http://www.freebsd.org/cgi/man.cgi?query=hpet&apropos=0&sektion=0&manpath=FreeBSD+9-current&format=html -
I was just thinking, when my 4mil states are full up and the firewall is trying to expire states and whatever is done, scanning 4mil states could be a bit of work. I wonder how the attack would do if the state table was made to something small, like 10k states. Expiring states may require an O(n) scan when lots of states are made about the same time.
Another thought is I have "Firewall Adaptive Timeouts"(System->Advanced->Firewall/NAT) set to 4mil states, which means by the time my state table gets full, states are being expired instantly. If expiration causes a full scan of at least triggers often, then creating states quickly and just as quickly expiring them may be the cause. I don't know. Just a thought.
That could explain why it takes a few tens of seconds before I started to feel the hurt of the attack, the issue didn't full trigger until the state table got full.
edit: more thoughts
I wonder if I set the target expiration to 4mil, but set the max to something larger. I'm not sure what PFSense/FreeBSD has to do when the table gets full, it may trigger a bad code path. Maybe I should have the table set to something like 5mil max states, but leave the adaptive timeout to 4mil.
You get 1 packet come in which causes x-number-of-lines-of-code to run & some memory spaces to be filled up.
Consider the time it takes for 1 packet to be processed by the fw, then add the time for the state to expire and before long you could easily fill up the available/free ram and also swamp the cpu by getting it to run x-number-of-lines-of-code per incoming packet, by having some code assigned to 1 core dominating the CPU especially if some threaded code has a higher priority than other code.
What if you could throttle the packets coming in before the states were processed? Would that prevent the firewall from crashing/hanging?
-
What if you could throttle the packets coming in before the states were processed? Would that prevent the firewall from crashing/hanging?
Isn't that the core idea of "a DDoS shouldn't be dealt with by the firewall, but upstream?"
-
@supermule So whats different with your setup then compared to what I have setup?
edit:
CPU Microcode perhaps? My VM is running on an AMD CPU which has some bugs affecting threading performance, all the others affected who have posted are running Intel CPU's iirc so maybe thats why I dont see the problem?edit2: This might also be a factor especially considering the vmware point which I've had to tweak as its possible to get the time to drift on vm's.
http://en.wikipedia.org/wiki/HPETAnyone know if HPET is built into the pfsense builds?
http://www.freebsd.org/cgi/man.cgi?query=hpet&apropos=0&sektion=0&manpath=FreeBSD+9-current&format=htmlJust to note; I was running bare metal with the same predictable results, I posted my machine specs in this thread, but will append them to my signature.
jimp or cmb made a recommendation on this thread somewhere about tuning pfSense settings to better handle DDOS.
I know that with the default state limit of 394000, the UI and most every other service seems to lock up, but system utilization under top is minimal at best. The screen shot I posed of the console shows the system utilization with a 394K state table. Increasing the state table makes the box responsive, but the interface being attacked stops responding with 4Mbit of attack traffic.