What is the biggest attack in GBPS you stopped
-
Thanks for checking back in, lowprofile. Would definitely appreciate if you could just share some brief tips of your findings with people here. Enough others are interested that I think they'll run with it in doing more testing and putting together recommendations for specific scenarios. I'd like to put out a guide myself, just going to be a bit until I have enough time for that.
Too much custom work to make a how-to guide at this moment, but looking forward to see the new corrections in 2.2.2
Those new config options made 2.2.1 actually, no changes in that regard from 2.2.1 to 2.2.2. There weren't any "corrections" technically I guess, as nothing changed by default, we just exposed all those timer values for configuration since they're greatly helpful in some circumstances.
-
During part of the test, the incoming bandwidth was around 40Mb/s, and I was still getting packetloss to my Admin interface. The bandwidth DDOS was the only part of the DDOS where PFSense was responding correctly, the other parts of the DDOS that did not consume 100% of the bandwidth left it unstable.
You're using the traffic shaper, that's almost certainly what caused that.
Those messing with this and doing traffic shaping on the same box, all bets are off there. ALTQ is not very fast for the kind of scale abuse we're talking here, and queuing in general really complicates things. If you're looking to handle as big of a DDoS as possible, you don't want to be running traffic shaping.
-
I am on 2.1.5, and due to the Kernel panic error (CARP+Limiter) i haven't upgraded to 2.2.1, but it seems like it may got fixed in 2.2.2 - I will give it some days yet to hear from others.
I'll then upgrade to 2.2.2 and make a how-to-guide, since there is too much unnecessary tweaks/changes on my present setup, which also isn't proper documented as well. It isn't pretty with all those extra tuning from all over the net (freeBSD recommendation etc) which is implemented.I will rather start from beginning, and make a solid setup on the new 2.2.2 - So this will include proper test with DDoS. If anyone is interested to participate in this test and tuning, please let me know. I assume SuperMule will be a part of this test.
It requires 2.2.2, we can take a session trough skype. I am located in +1 GMT timezone. Expect some DDoS, not volume attacks, but SYN floods of maximum 60-80mbit. -
I dont use traffic shaper at all and are affected in the exact same way as others using it.
@cmb:
During part of the test, the incoming bandwidth was around 40Mb/s, and I was still getting packetloss to my Admin interface. The bandwidth DDOS was the only part of the DDOS where PFSense was responding correctly, the other parts of the DDOS that did not consume 100% of the bandwidth left it unstable.
You're using the traffic shaper, that's almost certainly what caused that.
Those messing with this and doing traffic shaping on the same box, all bets are off there. ALTQ is not very fast for the kind of scale abuse we're talking here, and queuing in general really complicates things. If you're looking to handle as big of a DDoS as possible, you don't want to be running traffic shaping.
-
If upgraded to 2.2.2 then we need 3-4 people willing to be tested.
If both bare metal and using VM's could be a mix, then it would be perfect.
Same setup with traffic shaper. Used and not used.
Volunteers can contact me on PM. Attacks will be restricted to 2-10 mins depending on wish from the tested party.
Different attack types (tcp/udp) will be used. Pipe should be 100 mbit+ preferably.
-
Nice thread but also some overkill statements from people :D
- I was one of the first to locate this issue, and since last i've been much more experienced and today having an almost bulletproof setup regarding SYN flood.
Too much custom work to make a how-to guide at this moment, but looking forward to see the new corrections in 2.2.2
Will make some test in upcoming weeks.
I'm under the impression that a similar issue can be triggered by UDP, not just TCP. I think SuperMule showed many out of state UDP packets from many IP+port combos can trigger issues without consuming all of the bandwidth.
- I was one of the first to locate this issue, and since last i've been much more experienced and today having an almost bulletproof setup regarding SYN flood.
-
Exactly.
Nice thread but also some overkill statements from people :D
- I was one of the first to locate this issue, and since last i've been much more experienced and today having an almost bulletproof setup regarding SYN flood.
Too much custom work to make a how-to guide at this moment, but looking forward to see the new corrections in 2.2.2
Will make some test in upcoming weeks.
I'm under the impression that a similar issue can be triggered by UDP, not just TCP. I think SuperMule showed many out of state UDP packets from many IP+port combos can trigger issues without consuming all of the bandwidth.
- I was one of the first to locate this issue, and since last i've been much more experienced and today having an almost bulletproof setup regarding SYN flood.
-
If UDP can cause it, I wonder if ICMP can also cause it, heck, even a custom protocol. what I'm getting at is I wonder if it's an issue with the firewall and IP, when lots of different IPs are getting blocked.
-
Reading this thread with interest.
I might be interested in participating in testing of pfsense. I've set up a pfsense which should replace our existing firewall (shorewall/iptables on linux).
We have the 2 firewalls (existing and a new pfsense) on the same pipe running 1 gbit which is also used for productions system (so a minimal test in regards to time span would be OK in the middle of the night… timezone gmt+2 since we have to announce this to our customers, probably your daytime :) ...)
A few questions:
From where does the simulated attack origin ?
Is it special crafted UDP traffic ? (low and slow attack ?)
Will the simulated attack influence our primary linux firewall ? (I guess not since it's not using full pipe)
What is your settings for timeouts etc. in pfsense ? (the things cmb pointed out, our linux firewall is tuned for this after some annoying floods, but I'm new to pfsense so good to know recommended settings... also it's not easy to migrate settings when things are named differently... but could read up on this... though it's faster to have recommended settings, but maybe there isn't any recommended settings yet ?)
The pfsense is on a hardware server with http://ark.intel.com/products/75779/Intel-Xeon-Processor-E5-1620-v2-10M-Cache-3_70-GHz and 32gb ram, so no VM.
-
From everywhere… using spoofed IP's.
No. It can be tailored to use special crafted packages.
Depending on the packet load of the pipe, it can be.
I have time tonight at 10PM CET.
Send me a port and IP to test. Make sure it responds to ICMP on WAN so I can monitor the response from here and test various setups regarding the attack.
It will take 2-10 mins depending on response from the ICMP. If no response at all on PING then its a quick test, if normal reply attack will change using different approach until pfsense doesnt respond.
-
I guess no test tonight….
-
Sorry, have kids who wouldn't sleep.
I'll send you IP on PM if you still have time…
-
Send a message through the system, but can't see a sent message… did you get it ?
- 12 days later
-
Just a short notice on the matter.
Even after the 2.2.2 upgrade the GUI itself stille becomes useless during a SYN flood.
http://youtu.be/Jji4lW8gW1c
It even records packet loss to the LAN side and response times gets 10 times longer. Traffic graphs doesnt update at all and the worst part is the amount of traffic coming in.
15mbps…. States running around 150K out of 8MM. No real load on the server itself, but pf is dead. If the attack gets a little bigger (around 40mbps) then it goes offline completely and doesnt handle traffic at all.
-
Testing again this time stateless.
http://youtu.be/CGDo9pAQDlo
It completely downs pfSense and render the GUI useless/unresponsive.
-
Have you tested FreeBSD and/or OpenBSD?
-
FreeBSD yes, OpenBSD no.
-
but pf is dead. … goes offline completely and doesnt handle traffic at all. ... It completely downs pfSense and render the GUI useless/unresponsive.
-
no good news on the horizon then :(
-
Do you have a fix for this? Any ideas Doktor??
-
No, I have no fix for your top secret instant DoS. You know what? Either do a proper full disclosure or go away. Tired of reading this useless "PM me and I'll DoS you" crap for months.
-
Do you actually think that by going public with a script that can down any pfsense installation with a bandwith usage of 40mbps would be a wise idea??
What the hell is wrong with you?
-
With me? This "PM to get DoS-ed" BS is not how you get things fixed… Either work with those concerned (that includes FreeBSD upstream), or just publish it. Seriously noone is interested in crappy Youtube videos of unresponsive GUI.
-
But we want to test things and actually have something to point out before we introduce the world.
We want to see if others are affected and sees the same as we do.
Then we did write to ESF and told them there was a huge problem.
Not much has come back…
We upgrade and harden the damn thing to get a clue of whats actually going on when it hits and WHY 15mbps downs the thing!
I would love to open a redmine ticket for this, but I havent got a clue of which direction to point people in...!
So cut the crap and help if you can. Otherwise STFU.
-
Who the hell is "we"? I for sure don't want to see any Youtube "tests". Reminds me of the endless crappy antivirus "reviews" done on Youtube in a VM. If all you wrote to someone was "Hey, there's a huge problem, PM me and I'll DoS you", there's no surprise not much came back. You need to provide a testcase to reproduce the thing. Not this nonsense.
-
Lowprofile is also in this test scenario.
He has the email conversation with ESF.
-
If responsable disclosure has been done and no suitable answer is given sometimes full disclosure is the way to go.
In any way the bad guys are probably allready aware of the details so it will not hurt so much and help the community to find solutions if there is one, and if not, then be aware seems better than beleiving we are safe.
Of course this is my way of seeing things. Not using pfsense on professional things i use it only for now on personal adsl lines . Also if this is a FreeBSD issue and not a pfsense only thing trying to reach the bds guys could be the solution.
-
Its in the OS. Hardware can easily handle it if you got some muscle.
I can take this site offline using a specific type of traffic that takes no more than 70-80Mbps bandwith.
When that traffic hits pfSense, its dead. Goes offline instantly. No matter how powerful the hardware is.
I run 8 Core, 16GB ram and SSD. Dead in a second if it hits.
Exploiting the multithreading capabilities perhaps?
-
Perhaps :)
-
@ghislain26:
hi,
i am hit by ddos (upd flood mostly) and looking for solutions, hopefully opensource ones. I wanted to know what was the biggest multi gigabits attack you successfully stopped with your pfsense setup in the field ( so not with nullrouting at ISP level) and what the hardware used was.
My actuel issue is on the 5 to 10 gbps DDOS udp flood attacks so i search to see if a 20gbps filtering firewall could work in the real world of April 2015 and help me mitigate 1-16gbps attacks. My problem is to filter myself not ask upstrream to help so i really speak of how i can filter this and if anyone here had setup playing at this level of gbps.
regards,
Ghislain.Some DDOS attacks can be nullified by simply changing the ip address(es) at the dns level.
Where a DNS lookup is taking place, you need to identify the rogue who is doing the dns lookup and send them off to 23.37.28.215 or 195.99.147.120 if you have a sense of humour which contrary to popular belief also includes these guys 77.87.229.22. ;D -
I can tell you this much….
Windows firewall doesnt get affected by any of these attacks. If you put the server out front and only have WF running and forwarding traffic to the server then it can handle it easily.
It seems to only affect UNIX/Linux/BSD distros.
But MS are no longer supporting ISA server or its later rebranded versions last time I looked, but there might still be a way of exploiting the windows core in similar circumstances.
-
Could be. And yes its not supported any more.
But we were testing…..
-
No….but maybe some updates to what they find or not find??
Maybe hints to what could be done to minimize impact by adding things to system -> tunables??
Have you considered that CMB is now under contract and cant disclose? This was something disclosed by Snowden, some individuals were forced/required to form a legal entity under guidance of the NSA.
http://www.tomsguide.com/us/nsa-tech-coercion,news-17517.html
-
HAHAHAHAHAHAHAHA :D
If thats the case, then pfSense is dead as of THIS moment :D
-
Some DDOS attacks can be nullified by simply changing the ip address(es) at the dns level.
Where a DNS lookup is taking place, you need to identify the rogue who is doing the dns lookup and send them off to 23.37.28.215 or 195.99.147.120 if you have a sense of humour which contrary to popular belief also includes these guys 77.87.229.22. ;Dthe issue is on a webserver with XX+ domains the udp attack do not show which one is targetted and also some domains are handled by the main branch of the customer of our customer's office in another country with days of business paperwork nonsense to finaly react and change the dns :)
this is why i started to look at beeffy machines with pfsense to help but first i try to gather information about people using it for this and it seems no one, that answer here, use pfsense in multi gigabit setup or has experienced multi gigabit attacks on a pfsense box. I am happy thet they do not get attacked but i would have loved they had been to have some feedback ;p Supermule is providing feedback on "small scale" attack that would take down a firewall like this so i am not closer to any solution right now (and still fight on DC side to get a POC setup) :)
-
Lowprofile and I both have 10bgit setups in the production scenario and they are affected the same way.
We dont need to run high bandwith attacks like DNS, NTP, SSDP or anything like that when the interesting stuff takes place when using small bandwith scripts that takes the firewall offline.
When using little bandwith, then the attacker multiplies in numbers since they dont need 1gbit or more to take you offline. They only need a 50mbit pipe to do so.
THATS the scary part!
-
Could be. And yes its not supported any more.
But we were testing…..
Its possible to make the Windows core hang in some circumstances from the net even desktops behind fw's, but havent tested Win8 or later. Seen it on ubuntu 14.04 as well.
@ghislain26:
Some DDOS attacks can be nullified by simply changing the ip address(es) at the dns level.
Where a DNS lookup is taking place, you need to identify the rogue who is doing the dns lookup and send them off to 23.37.28.215 or 195.99.147.120 if you have a sense of humour which contrary to popular belief also includes these guys 77.87.229.22. ;Dthe issue is on a webserver with XX+ domains the udp attack do not show which one is targetted and also some domains are handled by the main branch of the customer of our customer's office in another country with days of business paperwork nonsense to finaly react and change the dns :)
this is why i started to look at beeffy machines with pfsense to help but first i try to gather information about people using it for this and it seems no one, that answer here, use pfsense in multi gigabit setup or has experienced multi gigabit attacks on a pfsense box. I am happy thet they do not get attacked but i would have loved they had been to have some feedback ;p Supermule is providing feedback on "small scale" attack that would take down a firewall like this so i am not closer to any solution right now (and still fight on DC side to get a POC setup) :)
I wonder if the L2 cache is causing a problem. Can this exploit be tried on a non AMD64 instruction set cpu if such a chip/device exists which can run pfsense and handle the bandwith? Its not something I can test on my RPi's sadly. ;)
http://www.lshift.net/blog/2013/10/08/cpu-cache-collisions-in-the-context-of-performance/
Edit. A quick search suggests its not possible to switch off the L2 or any other cache now adays in the bios, but one way around this might be to limit it to a single core instead which will be doable on VM's like ESXI.
When I suggest a non AMD64 cpu, please include the x86 32bit cpu's as well.
I sadly can not participate in this little experiment due to only having 5MB download, but would be interested to see the data generated by the scripts none the less, so Supermule if you dont mind sharing the script via pm, I'd be curious to see what it generates to see what patterns are observed over a closed network between a couple of machines.
FWIW.
-
I run this in the test bench
http://ark.intel.com/products/33927/Intel-Xeon-Processor-E5420-12M-Cache-2_50-GHz-1333-MHz-FSB
12M L2 cache. I actually dont know how big the L1 cache is.
-
This is what I run in the datacenters pfsense clusters
http://ark.intel.com/products/47920/Intel-Xeon-Processor-X5670-12M-Cache-2_93-GHz-6_40-GTs-Intel-QPI
-
Full disclosure: I'm a hobbyist here, no multi-gigabit access.
You said this works on Linux too. Is there common networking code between the two?
I would surely like to see a patch before this goes public, but IMO full disclosure to official channels might be a good option. For example, https://www.freebsd.org/security/reporting.html if it's freebsd itself, or https://www.kb.cert.org/vuls/html/report-a-vulnerability/ if it's cross-platform. Both offer encryption keys to send confidential information.