What is the biggest attack in GBPS you stopped
-
Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?
Yes.
Supermule, Tim.Mcmanus, and I (almabes) are going to work on this. I will get together some documentation once we get the problem isolated.To quote the late, great Joe Cocker
We could use a little help from our friends. -
Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.
Hi Jimp,
In my tests with Supermule, he was able to take down my pfSense 2.2.2 factory fresh install with no ports open. By take down, I mean the GUI was not responding, ping to pfSense was met with no response, ping to google.com stopped responding, and I could not browse the internet. It tooks about 30-60 seconds for pfsense to become responsive once he stopped the attack. Supermule said he was attacking with 5-6mbit. Following is what I had from the console at one point during the attack.
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTAfter attack stop, I saw this: 05-18-15 11:48:00 [ There were error(s) loading the rules: pfctl: DIOCXCOMMIT: Device busy - The line in question reads [0]: ]
-
Can you post screen shots of your AllGraphs for the System tab instead if just the processor?
-
Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?
If so, it would be worth reporting upstream as well.
-
Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?
If so, it would be worth reporting upstream as well.
I can probably do this, but it'll be about a week before I can present data and metrics. I already have a clean FreeBSD 10.1 and Apache 4.2 server built for testing. Enabling PF shouldn't be too difficult, I just need to plot out the architecture and possibly roll another 10.1 VM with two interfaces and a point to dead end traffic.
-
OPNSense responds better (35 seconds) before its taken offline.
Make up your own mind…
You mean their Ubuntu VPS web server that isn't behind a firewall at all. Yeah, make up your own mind indeed.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
This was the first line that caught my attention:
May 18 15:48:22 php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
-
Talking of DHCP, mine is going all crazy on WAN…
dhclient[21355]: Many bogus options seen in offers. Taking this offer in spite of bogus options - hope for the best!
dhclient[21355]: option nisplus-servers (100) larger than buffer.
dhclient[9876]: rejecting bogus offer.
dhclient[9876]: option option-97 (100) larger than buffer.
dhclient[90315]: rejecting bogus offer.
dhclient[90315]: option streettalk-directory-assistance-server (97) larger than buffer.
dhclient[2514]: rejecting bogus offer.
dhclient[2514]: option option-109 (111) larger than buffer.
dhclient[57118]: rejecting bogus offer.
dhclient[57118]: option nntp-server (105) larger than buffer.
dhclient[99613]: rejecting bogus offer.
dhclient[99613]: option routers (81) larger than buffer.
dhclient[35967]: rejecting bogus offer.
dhclient[35967]: option option-82 (97) larger than buffer.None of those were solicitated/initiated by me.
Something is wrong.
"hope for the best!"F.
-
This is what I have in the logs during the initial phases of a SYN flood.
May 19 05:39:27 php-fpm[13762]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:11 php-fpm[13762]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns Yousee
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns YouseeIt keeps going on and on…
-
From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
I am not sure if this was related as I was connecting DSL and disconnecting during test. I would say ignore DHCP logs safely…
-
More
May 19 05:44:58 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:48 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:37 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:27 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:16 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:06 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:55 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:45 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:34 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:24 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:13 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:03 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:40:57 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:55 check_reload_status: Reloading filter
May 19 05:40:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:55 check_reload_status: Restarting ipsec tunnels
May 19 05:40:55 check_reload_status: updating dyndns Yousee
May 19 05:40:50 check_reload_status: Reloading filter
May 19 05:40:50 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:50 check_reload_status: Restarting ipsec tunnels
May 19 05:40:50 check_reload_status: updating dyndns Yousee
May 19 05:40:47 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:31 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:25 check_reload_status: Reloading filter
May 19 05:40:25 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:25 check_reload_status: Restarting ipsec tunnels
May 19 05:40:25 check_reload_status: updating dyndns Yousee
May 19 05:40:24 check_reload_status: Reloading filter
May 19 05:40:24 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:24 check_reload_status: Restarting ipsec tunnels
May 19 05:40:24 check_reload_status: updating dyndns Yousee
May 19 05:40:15 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:07 check_reload_status: Reloading filter
May 19 05:40:07 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:07 check_reload_status: Restarting ipsec tunnels
May 19 05:40:07 check_reload_status: updating dyndns Yousee
May 19 05:39:59 check_reload_status: Reloading filter
May 19 05:39:59 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:59 check_reload_status: Restarting ipsec tunnels
May 19 05:39:59 check_reload_status: updating dyndns Yousee
May 19 05:39:57 php-fpm[19082]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:44 php-fpm[19082]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500 -
Supermule 'Morning,
What procedure do you use to ensure only 2-3 or 5-6 mbps of bandwidth is used during attack? From your signature, it seems like you have a Google fiber or some other Gigabit internet connection.
-
I use my home connection and not my datacenter for this :)
My home upload is not that fast…
-
CMB accused me of beeing the guy behind the forum downtime yesterday.
That saddened me…
Just because I say and keep saying that there is a major flaw in the way pf handles packets, then I must be the guy taking the forum down. :(
He asked for pcaps and I told him they were available for download in this thread. Didnt hear from him again.
I need to get somebody with the right debugging tools involved in this.
https://forum.pfsense.org/index.php?topic=91856.msg517232#msg517232
I sadly can not participate in this little experiment due to only having 5MB download, but would be interested to see the data generated by the scripts none the less, so Supermule if you dont mind sharing the script via pm, I'd be curious to see what it generates to see what patterns are observed over a closed network between a couple of machines.
I might be able to help but no promises and the flamegraphs mentioned earlier in this thread probably wont help debug this problem either, but thats my opinion, this could even be a BIOS issue but until I get my hands on a script that can trigger this sort of thing, I cant say for sure I can only speculate theoretically speaking at this stage.
-
The kind Doktor apparently didnt notice that….
A lot of scripts are emerging right now that has these capabilites built in. That makes pfsense compeletely useless for any that hosts public services if a 3mbit DDoS can bring it down and take slave and 3rd links down as well.
What do you consider it to be on a scale of 1-10 if EVERYBODY can be taken offline instantly by scripts that is everybodys for the taking in a not so distant future.
What would happen to the forum if one is pissed on some of the devs or fort that matter somebody else?
Take it offline as long as they like…. the netgate store and everything else related to ESF can be gone in seconds.
OPNSense responds better (35 seconds) before its taken offline. Pfsense takes 1-2 seconds.
Make up your own mind...
PM me some links so I can inspect them please.
-
Its been 3+ months since this was forwarded to the devs by Lowprofile and me.
There has been NOTHING other than scattered emails and one PM to me about pcaps from CMB and he could get it since its in this thread.
I find that fact pretty scary since people buy commercial support and gold memberships to support the project.
But when its really needed, everything went quiet and almost no responde other than accusations of me bringing down the forum.
I would love to help out, but I am not a freebsd/linux guru and I cannot provide anything besides point out there is an issue, trying to enlighten people and show how it affects pfsense and other firewall distros.
Thats why I ask for testers and help from people like Doktornotor since I know he is a dev. and all I get is shit from him.
I have promised not to spread the script since as it is, it can take down any firewall running pfsense and other OS'.
Its like having a dirty bomb and people asking for copies, than all hell breaks lose.
What else can I do to draw attention to this and have a response to the issue?
You wont get this problem fixed quickly then.
BTW SYN floods if this is what is being alluded to have been around since 1996 according to Cisco. http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_9-4/syn_flooding_attacks.html
http://en.wikipedia.org/wiki/SYN_floodJust because someone says they are a dev doesnt make them a good dev either. Demonstrating one's knowledge is usually a good way to proceed and as Pfsense is opensource, in theory the argument for a dirty bomb is negated by virtue of the nature of opensource software.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
This was the first line that caught my attention:
May 18 15:48:22 php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
This thread from the beta testing might be relevant to your line of thought with regard to problems arising when the IP addresses are reset by the ISP's.
"still having to delete entries from the state table"
https://forum.pfsense.org/index.php?topic=86087.0 -
INteresting!
