What is the biggest attack in GBPS you stopped
-
Thanks Tim.
You are 100% correct.
-
Silly question - Is it possible to set a max cpu % that may be used by the packet filter? Keep some in reserve for other processes?
-
Someone needs to use DTrace and make a flame-graph of what methods are being called in the kernel.
-
Shall we test again Harvy??
-
I don't know how to do flame graphs, I've only seen them in Netflix presentations talking about optimizing FreeBSD.
-
Tested Fortigate Virtual Server and after enabling Flood Protection, it ran perfectly during all tests.
1CPU and 1GB RAM in a VmWare VM.
Had an email conversation with a guy named Dave Huffman and he was able to replicate the scenario but only using DoS and not DDoS. Not that important, but it seems pfSense is not able to handle legitimate traffic vs. offending IP's.
It chokes somewhere in the stack.
-
Do you need help figuring out how to enable profiling or some other debugging software?
-
Yes because I need to get to the bottom of this.
-
I would start here. http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html
You can do remote kernel debugging as an option. It's not for the faint of heart. Debugging never is.
-
Yes because I need to get to the bottom of this.
Try a USB nic on the wan, see how the data is handled differently.
You'll find these useful as well.
https://software.intel.com/sites/default/files/profiling_debugging_freebsd_kernel_321772.pdf
https://2008.asiabsdcon.org/papers/P8B-paper.pdfMutexs can catch some people out, but they are just locks to ensure the code doesnt deadlock in a multicore environment.
IMO profiling is better than retrospectively debugging crash dumps as you can make the crash dumps misleading in some situations masking the real root cause of the problem. I've found bugs in programming languages that have existed for over 15 years, thats how difficult some of these bugs are to find, even though it took me less than a week to find, theres a lot of exposed software out there.
Main difficultly are multi cores when it comes to debugging, you could be looking at the code running on one core whilst a bug in code running on another core creates the problem which crashes the code running on the core you are looking at. You can mask some problems by running on a single core but you will still have the problem, just less often as these things are just inglorious clockwork turkmachines.
-
Yes but doesnt crash as in crash….
It just goes to a standstill and is unresponsive. You dont see anything on the console and in the logs besides excessive traffic.
I dont see any queueing on the NIC's as well so its pretty odd.
-
Maybe something is getting out of order, perhaps a queue/buffer is being processed LIFO when it should be FIFO which would generally only show up under load considering the speeds of todays CPU's.
-
Could be.
The fact is that it cannot sort traffic in legitimate and non-legit traffic.
Its pretty weird that a lot of excess ressources is not used on the webserver to keep the GUI alive at least.
When the flood control setting is activated in the Fortinet VM appliance then it can handle everything we threw at it.
It can sort traffic at wirespeed, but pfsense cannot.
It sortof feels like pfsense is responding to packets that shouldnt get a response and holds everything else…
I dont know where to start.
A guy called Dave Huffmann contacted and has written a script that can replicate most of what we see. I dont know if he has come up with an answer yet on whats causing it to slow down.
I am yet to receive news from him.
-
Lookup flame graphs and Netflix. They're very useful for looking for offending code paths.
-
http://techblog.netflix.com/2014/11/nodejs-in-flames.html
-
Looks really interesting and is a good read.
How to impelement it in pfsense??
Can this be implemented somehow in a package?
http://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html
There is code in there…
-
This is how my logs look like…
May 14 14:27:51 php-fpm[1097]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:27:41 php-fpm[1097]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:27:29 php-fpm[71742]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:27:19 php-fpm[71742]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:27:07 php-fpm[71510]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:26:56 php-fpm[71510]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:26:45 php-fpm[53920]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:26:35 php-fpm[53920]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:26:23 php-fpm[1097]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:26:13 php-fpm[1097]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:26:01 php-fpm[71742]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:25:51 php-fpm[71742]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:25:39 php-fpm[71510]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:25:29 php-fpm[71510]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:25:27 check_reload_status: Reloading filter
May 14 14:25:27 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 14 14:25:27 check_reload_status: Restarting ipsec tunnels
May 14 14:25:27 check_reload_status: updating dyndns Yousee
May 14 14:25:23 check_reload_status: Reloading filter
May 14 14:25:23 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 14 14:25:23 check_reload_status: Restarting ipsec tunnels
May 14 14:25:23 check_reload_status: updating dyndns Yousee
May 14 14:25:17 php-fpm[53920]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 14 14:25:13 check_reload_status: Reloading filter
May 14 14:25:13 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 14 14:25:13 check_reload_status: Restarting ipsec tunnels
May 14 14:25:13 check_reload_status: updating dyndns Yousee
May 14 14:25:11 check_reload_status: Reloading filter
May 14 14:25:11 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 14 14:25:11 check_reload_status: Restarting ipsec tunnels
May 14 14:25:11 check_reload_status: updating dyndns Yousee
May 14 14:25:03 php-fpm[53920]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 14 14:24:55 check_reload_status: Reloading filter
May 14 14:24:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 14 14:24:55 check_reload_status: Restarting ipsec tunnels
May 14 14:24:55 check_reload_status: updating dyndns Yousee -
Looks really interesting and is a good read.
How to impelement it in pfsense??
Can this be implemented somehow in a package?
http://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html
There is code in there…
IMHO, I wouldn't put this into a package or integrate it into pfSense. It's a development debugging tool that ideally should be installed on dev or test machines.
-
Hmmm.
EDIT:
Thinking it would work as a great tool in debugging issues related to pfsense and give the IT-admins a better insight of whats going on in their environments.
-
Hi guys
Just want to share some info.
I am going away from pfsense. PFsense is a great firewall but all these buggy versions out there have cost me $$$$ - 2.2* has been so unstable and i've lost the trust to pfsense.
Regarding DDoS, pfsense cannot handle a simple flood SYN. In 2.2* it got more worse. You can spend many weeks in tuning, tweaking etc, but then you will then have a system which is unreliable at the end. Too much core changing.
I managed to get it somehow 80% resistent to SYN floods in 2.1.5, but it had its sideaffects. I now experienced unstability generel.
I've tried fortigate VM appliance with 1gb ram and 1core (trial) - i was surprised how stable it was with same hardware (virtual) You have a special option to block SYN/ICMP/FIN etc floods. Very simple option. See screenshot.
I used 10min to install it and further 10min to set it up. Activated the ddos policy. and bingo i had a stable setup. I know fortigate cost much more, but most of the appliances are built on linux or freebsd. I have concluded the pfsense does lack this crucial "feature" and protection.
No more packet drop, even the attack was on +100mbit SYN flood. Very stable, not a single drop in ping.
Sad to say, but this has proven the source to the issue = PFsense.I am now investing in a proper firewall. VM or box, doesnt matter, it just wont be PFsense. I liked pfsense untill i got these issues and some serious stability issues in newer versions. Time to move on. ;)
