What is the biggest attack in GBPS you stopped

[torontob]

I do not appreciate SM's disclosure.  No developer would.  However, I am trying to respond to it in a way that I can contribute.
Users absolutely DO code.

Most of users do NOT code! We contribute by download, test, and report. Devs make their money indirectly from this. Don't act superior to other users. The fact that you did coding or not has NO VALUE. You did it for your own personal gain regardless. One doesn't need a business major to understand this. This project exists because of it's user-base (Small or big). You are not important any more than any other user here. Don't flatter yourself!

I do NOT understand why you make this about Supermule's character or method of disclosure while this is simply a really bad thing happening out there that has nothing to do with anything but the issue itself. Attack the problem, not the messenger. Are you pissed that he found it? I am certain he is not lying when he says Devs give him shit for reporting this. Once again, people like you are trying to change focus from what he is saying. And you are full of shit because if you saw no value in what Supermule found then you wouldn't be so angry here. If you don't believe this is serious then forget it and move on. If you do believe he is onto something and you still DO NOT APPRECIATE HIM then, …. (fill the dots yourself)

And what the heck is a white hat? Who decides who is a white hat - you? LOL - No one trusts you when you take the high moral ground to everyone else! STOP ATTACKING HIS CHARECTER - this is stupid

[Supermule]

Geesus guys!

Take it easy. Just got up on a sunday and all this bickering is not good for the community.

Lets get into solution mode and see if we can assist the dev's on this one since it seems to be pretty severe.

Go grab a coffee and a donut :)

I will read my PM's in the meantime with people wanting to help. Really appreciated guys!

[doktornotor]

[tim.mcmanus]

My apologies.  The Jameson made Mr. Hyde come out.

[Harvy66]

@Supermule:

Geesus guys!

Take it easy. Just got up on a sunday and all this bickering is not good for the community.

Lets get into solution mode and see if we can assist the dev's on this one since it seems to be pretty severe.

Go grab a coffee and a donut :)

I will read my PM's in the meantime with people wanting to help. Really appreciated guys!

Coffee sounds really good. Still waiting to get a coffee grinder and a french press. I don't trust coffee filters unless stainless steel, and those are hard to find outside of a french press. I don't like to use paper filters because it seems like a such a waste of paper and many are treated with a range of chemicals, and plastic filters, well, food grade nylon is only rated for 165f before it starts leeching chemicals.

I drink my tea every morning, but I really want free leaf and a tea ball, so I can cut down on waste materials. Took a lot of reviews, but I found a tea that uses compress air to whiten their bags, instead of chemicals, their bags don't use staples only tied so less waste, and chemical analysis shows very low levels of lead and other heavy metals compared to other brands and well below safety limits, and virtually no pesticide residue.

We may be living long compared to people decades ago, but I take the stance that if everything you eat is technically safe, you're just going to get death by a thousand cuts. I don't just want "safe" levels just below the maximum, I want the best that can be done, with in reason.

There's my random weekend off-topic rant.

Next on the agenda, replacing mercury filled florescent lightbulbs with LED lights, but ones that don't have a strong blue spectrum that causes eye strain.

[jimp]

Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?

Specifically: Is the traffic in question actually passed, or blocked? Is a service on the firewall running pfSense (such as the GUI) exposed to the test source or is the traffic being passed through to an internal host (port forward, routed, etc)? – This is important because a SYN flood to pfSense as a host is completely different than a flood through pfSense as a forwarder/firewall.

If the traffic is passing through the firewall, using rules to clamp down state limits, or going stateless properly (floating quick OUT rules to pass out with no state along with the pass in rules on the other tabs) may help.

If the traffic is targeting the firewall itself, then there are things that can be tweaked (syncache parameters, for example), but it's yet another reason the services on the firewall such as the GUI and SSH should not be exposed to the Internet in general. State limits can help there as well.

Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.

[almabes]

@jimp:

Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?

Yes.
Supermule, Tim.Mcmanus, and I (almabes) are going to work on this.  I will get together some documentation once we get the problem isolated.

To quote the late, great Joe Cocker
We could use a little help from our friends.

[torontob]

@jimp:

Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.

Hi Jimp,

In my tests with Supermule, he was able to take down my pfSense 2.2.2 factory fresh install with no ports open. By take down, I mean the GUI was not responding, ping to pfSense was met with no response, ping to google.com stopped responding, and I could not browse the internet. It tooks about 30-60 seconds for pfsense to become responsive once he stopped the attack. Supermule said he was attacking with 5-6mbit. Following is what I had from the console at one point during the attack.

I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHT

After attack stop, I saw this: 05-18-15 11:48:00 [ There were error(s) loading the rules: pfctl: DIOCXCOMMIT: Device busy - The line in question reads [0]: ]

[tim.mcmanus]

Can you post screen shots of your AllGraphs for the System tab instead if just the processor?

[jimp]

Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?

If so, it would be worth reporting upstream as well.

[tim.mcmanus]

@jimp:

Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?

If so, it would be worth reporting upstream as well.

I can probably do this, but it'll be about a week before I can present data and metrics.  I already have a clean FreeBSD 10.1 and Apache 4.2 server built for testing.  Enabling PF shouldn't be too difficult, I just need to plot out the architecture and possibly roll another 10.1 VM with two interfaces and a point to dead end traffic.

[cmb]

@Supermule:

OPNSense responds better (35 seconds) before its taken offline.

Make up your own mind…

You mean their Ubuntu VPS web server that isn't behind a firewall at all. Yeah, make up your own mind indeed.

[tim.mcmanus]

@torontob:

I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHT

Did your WAN IP address change during this event?  Do you have a static IP?

The DHCP service is exiting with an error 1, is this a persistent error in your logs?

[torontob]

@tim.mcmanus:

@torontob:

I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHT

Did your WAN IP address change during this event?  Do you have a static IP?

The DHCP service is exiting with an error 1, is this a persistent error in your logs?

This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.

[tim.mcmanus]

@torontob:

@tim.mcmanus:

@torontob:

I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHT

Did your WAN IP address change during this event?  Do you have a static IP?

The DHCP service is exiting with an error 1, is this a persistent error in your logs?

This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.

This was the first line that caught my attention:

May 18 15:48:22	php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.

From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.

Could this issue be related to the external DHCP leases granted by ISPs?  Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again?  Is this even possible with a pure SYN flood?

Has anyone tested this with a static IP?

[fsansfil]

Talking of DHCP, mine is going all crazy on WAN…

dhclient[21355]: Many bogus options seen in offers. Taking this offer in spite of bogus options - hope for the best!
dhclient[21355]: option nisplus-servers (100) larger than buffer.
dhclient[9876]: rejecting bogus offer.
dhclient[9876]: option option-97 (100) larger than buffer.
dhclient[90315]: rejecting bogus offer.
dhclient[90315]: option streettalk-directory-assistance-server (97) larger than buffer.
dhclient[2514]: rejecting bogus offer.
dhclient[2514]: option option-109 (111) larger than buffer.
dhclient[57118]: rejecting bogus offer.
dhclient[57118]: option nntp-server (105) larger than buffer.
dhclient[99613]: rejecting bogus offer.
dhclient[99613]: option routers (81) larger than buffer.
dhclient[35967]: rejecting bogus offer.
dhclient[35967]: option option-82 (97) larger than buffer.

None of those were solicitated/initiated by me.

Something is wrong.
"hope for the best!"

F.

[Supermule]

This is what I have in the logs during the initial phases of a SYN flood.

May 19 05:39:27 php-fpm[13762]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:11 php-fpm[13762]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns Yousee
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns Yousee

It keeps going on and on…

[torontob]

From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.

Could this issue be related to the external DHCP leases granted by ISPs?  Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again?  Is this even possible with a pure SYN flood?

Has anyone tested this with a static IP?

I am not sure if this was related as I was connecting DSL and disconnecting during test. I would say ignore DHCP logs safely…

[Supermule]

More

May 19 05:44:58 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:48 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:37 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:27 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:16 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:06 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:55 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:45 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:34 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:24 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:13 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:03 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:40:57 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:55 check_reload_status: Reloading filter
May 19 05:40:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:55 check_reload_status: Restarting ipsec tunnels
May 19 05:40:55 check_reload_status: updating dyndns Yousee
May 19 05:40:50 check_reload_status: Reloading filter
May 19 05:40:50 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:50 check_reload_status: Restarting ipsec tunnels
May 19 05:40:50 check_reload_status: updating dyndns Yousee
May 19 05:40:47 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:31 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:25 check_reload_status: Reloading filter
May 19 05:40:25 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:25 check_reload_status: Restarting ipsec tunnels
May 19 05:40:25 check_reload_status: updating dyndns Yousee
May 19 05:40:24 check_reload_status: Reloading filter
May 19 05:40:24 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:24 check_reload_status: Restarting ipsec tunnels
May 19 05:40:24 check_reload_status: updating dyndns Yousee
May 19 05:40:15 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:07 check_reload_status: Reloading filter
May 19 05:40:07 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:07 check_reload_status: Restarting ipsec tunnels
May 19 05:40:07 check_reload_status: updating dyndns Yousee
May 19 05:39:59 check_reload_status: Reloading filter
May 19 05:39:59 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:59 check_reload_status: Restarting ipsec tunnels
May 19 05:39:59 check_reload_status: updating dyndns Yousee
May 19 05:39:57 php-fpm[19082]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:44 php-fpm[19082]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500

[torontob]

Supermule 'Morning,

What procedure do you use to ensure only 2-3 or 5-6 mbps of bandwidth is used during attack? From your signature, it seems like you have a Google fiber or some other Gigabit internet connection.