What is the biggest attack in GBPS you stopped
-
My apologies. The Jameson made Mr. Hyde come out.
-
Geesus guys!
Take it easy. Just got up on a sunday and all this bickering is not good for the community.
Lets get into solution mode and see if we can assist the dev's on this one since it seems to be pretty severe.
Go grab a coffee and a donut :)
I will read my PM's in the meantime with people wanting to help. Really appreciated guys!
Coffee sounds really good. Still waiting to get a coffee grinder and a french press. I don't trust coffee filters unless stainless steel, and those are hard to find outside of a french press. I don't like to use paper filters because it seems like a such a waste of paper and many are treated with a range of chemicals, and plastic filters, well, food grade nylon is only rated for 165f before it starts leeching chemicals.
I drink my tea every morning, but I really want free leaf and a tea ball, so I can cut down on waste materials. Took a lot of reviews, but I found a tea that uses compress air to whiten their bags, instead of chemicals, their bags don't use staples only tied so less waste, and chemical analysis shows very low levels of lead and other heavy metals compared to other brands and well below safety limits, and virtually no pesticide residue.
We may be living long compared to people decades ago, but I take the stance that if everything you eat is technically safe, you're just going to get death by a thousand cuts. I don't just want "safe" levels just below the maximum, I want the best that can be done, with in reason.
There's my random weekend off-topic rant.
Next on the agenda, replacing mercury filled florescent lightbulbs with LED lights, but ones that don't have a strong blue spectrum that causes eye strain.
-
Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?
Specifically: Is the traffic in question actually passed, or blocked? Is a service on the firewall running pfSense (such as the GUI) exposed to the test source or is the traffic being passed through to an internal host (port forward, routed, etc)? – This is important because a SYN flood to pfSense as a host is completely different than a flood through pfSense as a forwarder/firewall.
If the traffic is passing through the firewall, using rules to clamp down state limits, or going stateless properly (floating quick OUT rules to pass out with no state along with the pass in rules on the other tabs) may help.
If the traffic is targeting the firewall itself, then there are things that can be tweaked (syncache parameters, for example), but it's yet another reason the services on the firewall such as the GUI and SSH should not be exposed to the Internet in general. State limits can help there as well.
Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.
-
Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?
Yes.
Supermule, Tim.Mcmanus, and I (almabes) are going to work on this. I will get together some documentation once we get the problem isolated.To quote the late, great Joe Cocker
We could use a little help from our friends. -
Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.
Hi Jimp,
In my tests with Supermule, he was able to take down my pfSense 2.2.2 factory fresh install with no ports open. By take down, I mean the GUI was not responding, ping to pfSense was met with no response, ping to google.com stopped responding, and I could not browse the internet. It tooks about 30-60 seconds for pfsense to become responsive once he stopped the attack. Supermule said he was attacking with 5-6mbit. Following is what I had from the console at one point during the attack.
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTAfter attack stop, I saw this: 05-18-15 11:48:00 [ There were error(s) loading the rules: pfctl: DIOCXCOMMIT: Device busy - The line in question reads [0]: ]
-
Can you post screen shots of your AllGraphs for the System tab instead if just the processor?
-
Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?
If so, it would be worth reporting upstream as well.
-
Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?
If so, it would be worth reporting upstream as well.
I can probably do this, but it'll be about a week before I can present data and metrics. I already have a clean FreeBSD 10.1 and Apache 4.2 server built for testing. Enabling PF shouldn't be too difficult, I just need to plot out the architecture and possibly roll another 10.1 VM with two interfaces and a point to dead end traffic.
-
OPNSense responds better (35 seconds) before its taken offline.
Make up your own mind…
You mean their Ubuntu VPS web server that isn't behind a firewall at all. Yeah, make up your own mind indeed.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
This was the first line that caught my attention:
May 18 15:48:22 php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
-
Talking of DHCP, mine is going all crazy on WAN…
dhclient[21355]: Many bogus options seen in offers. Taking this offer in spite of bogus options - hope for the best!
dhclient[21355]: option nisplus-servers (100) larger than buffer.
dhclient[9876]: rejecting bogus offer.
dhclient[9876]: option option-97 (100) larger than buffer.
dhclient[90315]: rejecting bogus offer.
dhclient[90315]: option streettalk-directory-assistance-server (97) larger than buffer.
dhclient[2514]: rejecting bogus offer.
dhclient[2514]: option option-109 (111) larger than buffer.
dhclient[57118]: rejecting bogus offer.
dhclient[57118]: option nntp-server (105) larger than buffer.
dhclient[99613]: rejecting bogus offer.
dhclient[99613]: option routers (81) larger than buffer.
dhclient[35967]: rejecting bogus offer.
dhclient[35967]: option option-82 (97) larger than buffer.None of those were solicitated/initiated by me.
Something is wrong.
"hope for the best!"F.
-
This is what I have in the logs during the initial phases of a SYN flood.
May 19 05:39:27 php-fpm[13762]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:11 php-fpm[13762]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns Yousee
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns YouseeIt keeps going on and on…
-
From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
I am not sure if this was related as I was connecting DSL and disconnecting during test. I would say ignore DHCP logs safely…
-
More
May 19 05:44:58 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:48 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:37 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:27 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:16 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:06 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:55 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:45 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:34 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:24 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:13 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:03 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:40:57 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:55 check_reload_status: Reloading filter
May 19 05:40:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:55 check_reload_status: Restarting ipsec tunnels
May 19 05:40:55 check_reload_status: updating dyndns Yousee
May 19 05:40:50 check_reload_status: Reloading filter
May 19 05:40:50 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:50 check_reload_status: Restarting ipsec tunnels
May 19 05:40:50 check_reload_status: updating dyndns Yousee
May 19 05:40:47 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:31 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:25 check_reload_status: Reloading filter
May 19 05:40:25 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:25 check_reload_status: Restarting ipsec tunnels
May 19 05:40:25 check_reload_status: updating dyndns Yousee
May 19 05:40:24 check_reload_status: Reloading filter
May 19 05:40:24 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:24 check_reload_status: Restarting ipsec tunnels
May 19 05:40:24 check_reload_status: updating dyndns Yousee
May 19 05:40:15 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:07 check_reload_status: Reloading filter
May 19 05:40:07 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:07 check_reload_status: Restarting ipsec tunnels
May 19 05:40:07 check_reload_status: updating dyndns Yousee
May 19 05:39:59 check_reload_status: Reloading filter
May 19 05:39:59 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:59 check_reload_status: Restarting ipsec tunnels
May 19 05:39:59 check_reload_status: updating dyndns Yousee
May 19 05:39:57 php-fpm[19082]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:44 php-fpm[19082]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500 -
Supermule 'Morning,
What procedure do you use to ensure only 2-3 or 5-6 mbps of bandwidth is used during attack? From your signature, it seems like you have a Google fiber or some other Gigabit internet connection.
-
I use my home connection and not my datacenter for this :)
My home upload is not that fast…
-
CMB accused me of beeing the guy behind the forum downtime yesterday.
That saddened me…
Just because I say and keep saying that there is a major flaw in the way pf handles packets, then I must be the guy taking the forum down. :(
He asked for pcaps and I told him they were available for download in this thread. Didnt hear from him again.
I need to get somebody with the right debugging tools involved in this.
https://forum.pfsense.org/index.php?topic=91856.msg517232#msg517232
I sadly can not participate in this little experiment due to only having 5MB download, but would be interested to see the data generated by the scripts none the less, so Supermule if you dont mind sharing the script via pm, I'd be curious to see what it generates to see what patterns are observed over a closed network between a couple of machines.
I might be able to help but no promises and the flamegraphs mentioned earlier in this thread probably wont help debug this problem either, but thats my opinion, this could even be a BIOS issue but until I get my hands on a script that can trigger this sort of thing, I cant say for sure I can only speculate theoretically speaking at this stage.
-
The kind Doktor apparently didnt notice that….
A lot of scripts are emerging right now that has these capabilites built in. That makes pfsense compeletely useless for any that hosts public services if a 3mbit DDoS can bring it down and take slave and 3rd links down as well.
What do you consider it to be on a scale of 1-10 if EVERYBODY can be taken offline instantly by scripts that is everybodys for the taking in a not so distant future.
What would happen to the forum if one is pissed on some of the devs or fort that matter somebody else?
Take it offline as long as they like…. the netgate store and everything else related to ESF can be gone in seconds.
OPNSense responds better (35 seconds) before its taken offline. Pfsense takes 1-2 seconds.
Make up your own mind...
PM me some links so I can inspect them please.
