What is the biggest attack in GBPS you stopped

[cmb]

Thanks for checking back in, lowprofile. Would definitely appreciate if you could just share some brief tips of your findings with people here. Enough others are interested that I think they'll run with it in doing more testing and putting together recommendations for specific scenarios. I'd like to put out a guide myself, just going to be a bit until I have enough time for that.

@lowprofile:

Too much custom work to make a how-to guide at this moment, but looking forward to see the new corrections in 2.2.2

Those new config options made 2.2.1 actually, no changes in that regard from 2.2.1 to 2.2.2. There weren't any "corrections" technically I guess, as nothing changed by default, we just exposed all those timer values for configuration since they're greatly helpful in some circumstances.

[cmb]

@Harvy66:

During part of the test, the incoming bandwidth was around 40Mb/s, and I was still getting packetloss to my Admin interface. The bandwidth DDOS was the only part of the DDOS where PFSense was responding correctly, the other parts of the DDOS that did not consume 100% of the bandwidth left it unstable.

You're using the traffic shaper, that's almost certainly what caused that.

Those messing with this and doing traffic shaping on the same box, all bets are off there. ALTQ is not very fast for the kind of scale abuse we're talking here, and queuing in general really complicates things. If you're looking to handle as big of a DDoS as possible, you don't want to be running traffic shaping.

[lowprofile]

I am on 2.1.5, and due to the Kernel panic error (CARP+Limiter) i haven't upgraded to 2.2.1, but it seems like it may got fixed in 2.2.2 - I will give it some days yet to hear from others.
I'll then upgrade to 2.2.2 and make a how-to-guide, since there is too much unnecessary tweaks/changes on my present setup, which also isn't proper documented as well. It isn't pretty with all those extra tuning from all over the net (freeBSD recommendation etc) which is implemented.

I will rather start from beginning, and make a solid setup on the new 2.2.2 - So this will include proper test with DDoS. If anyone is interested to participate in this test and tuning, please let me know. I assume SuperMule will be a part of this test.
It requires 2.2.2, we can take a session trough skype. I am located in +1 GMT timezone. Expect some DDoS, not volume attacks, but SYN floods of maximum 60-80mbit.

[Supermule]

I dont use traffic shaper at all and are affected in the exact same way as others using it.

@cmb:

@Harvy66:

During part of the test, the incoming bandwidth was around 40Mb/s, and I was still getting packetloss to my Admin interface. The bandwidth DDOS was the only part of the DDOS where PFSense was responding correctly, the other parts of the DDOS that did not consume 100% of the bandwidth left it unstable.

You're using the traffic shaper, that's almost certainly what caused that.

Those messing with this and doing traffic shaping on the same box, all bets are off there. ALTQ is not very fast for the kind of scale abuse we're talking here, and queuing in general really complicates things. If you're looking to handle as big of a DDoS as possible, you don't want to be running traffic shaping.

[Supermule]

If upgraded to 2.2.2 then we need 3-4 people willing to be tested.

If both bare metal and using VM's could be a mix, then it would be perfect.

Same setup with traffic shaper. Used and not used.

Volunteers can contact me on PM. Attacks will be restricted to 2-10 mins depending on wish from the tested party.

Different attack types (tcp/udp) will be used. Pipe should be 100 mbit+ preferably.

[Harvy66]

@lowprofile:

Nice thread but also some overkill statements from people  :D

• I was one of the first to locate this issue, and since last i've been much more experienced and today having an almost bulletproof setup regarding SYN flood.
  Too much custom work to make a how-to guide at this moment, but looking forward to see the new corrections in 2.2.2

Will make some test in upcoming weeks.

I'm under the impression that a similar issue can be triggered by UDP, not just TCP. I think SuperMule showed many out of state UDP packets from many IP+port combos can trigger issues without consuming all of the bandwidth.

[Supermule]

Exactly.

@Harvy66:

@lowprofile:

Nice thread but also some overkill statements from people  :D

• I was one of the first to locate this issue, and since last i've been much more experienced and today having an almost bulletproof setup regarding SYN flood.
  Too much custom work to make a how-to guide at this moment, but looking forward to see the new corrections in 2.2.2

Will make some test in upcoming weeks.

I'm under the impression that a similar issue can be triggered by UDP, not just TCP. I think SuperMule showed many out of state UDP packets from many IP+port combos can trigger issues without consuming all of the bandwidth.

[Harvy66]

If UDP can cause it, I wonder if ICMP can also cause it, heck, even a custom protocol. what I'm getting at is I wonder if it's an issue with the firewall and IP, when lots of different IPs are getting blocked.

[ledj]

Reading this thread with interest.

I might be interested in participating in testing of pfsense. I've set up a pfsense which should replace our existing firewall (shorewall/iptables on linux).

We have the 2 firewalls (existing and a new pfsense) on the same pipe running 1 gbit which is also used for productions system (so a minimal test in regards to time span would be OK in the middle of the night… timezone gmt+2 since we have to announce this to our customers, probably your daytime :) ...)

A few questions:

From where does the simulated attack origin ?

Is it special crafted UDP traffic ? (low and slow attack ?)

Will the simulated attack influence our primary linux firewall ? (I guess not since it's not using full pipe)

What is your settings for timeouts etc. in pfsense ? (the things cmb pointed out, our linux firewall is tuned for this after some annoying floods, but I'm new to pfsense so good to know recommended settings... also it's not easy to migrate settings when things are named differently... but could read up on this... though it's faster to have recommended settings, but maybe there isn't any recommended settings yet ?)

The pfsense is on a hardware server with http://ark.intel.com/products/75779/Intel-Xeon-Processor-E5-1620-v2-10M-Cache-3_70-GHz and 32gb ram, so no VM.

[Supermule]

From everywhere… using spoofed IP's.

No. It can be tailored to use special crafted packages.

Depending on the packet load of the pipe, it can be.

I have time tonight at 10PM CET.

Send me a port and IP to test. Make sure it responds to ICMP on WAN so I can monitor the response from here and test various setups regarding the attack.

It will take 2-10 mins depending on response from the ICMP. If no response at all on PING then its a quick test, if normal reply attack will change using different approach until pfsense doesnt respond.

[Supermule]

I guess no test tonight….

[ledj]

Sorry, have kids who wouldn't sleep.

I'll send you IP on PM if you still have time…

[ledj]

Send a message through the system, but can't see a sent message… did you get it ?

[Supermule]

Just a short notice on the matter.

Even after the 2.2.2 upgrade the GUI itself stille becomes useless during a SYN flood.

http://youtu.be/Jji4lW8gW1c

It even records packet loss to the LAN side and response times gets 10 times longer. Traffic graphs doesnt update at all and the worst part is the amount of traffic coming in.

15mbps…. States running around 150K out of 8MM. No real load on the server itself, but pf is dead. If the attack gets a little bigger (around 40mbps) then it goes offline completely and doesnt handle traffic at all.

[Supermule]

Testing again this time stateless.

http://youtu.be/CGDo9pAQDlo

It completely downs pfSense and render the GUI useless/unresponsive.

[Nullity]

Have you tested FreeBSD and/or OpenBSD?

[Supermule]

FreeBSD yes, OpenBSD no.

[doktornotor]

@Supermule:

but pf is dead. … goes offline completely and doesnt handle traffic at all. ... It completely downs pfSense and render the GUI useless/unresponsive.

[gadnet]

no good news on the horizon then :(

[Supermule]

Do you have a fix for this? Any ideas Doktor??