What is the biggest attack in GBPS you stopped
-
The following is my opinion.
I don't really see this thread as being squarely aimed at root causing the issue as much as one of bringing to light the issue to make the community and pfSense team aware of the issue. Awareness has been accomplished.
Very disappointed in the response (or seemingly lack of response) by the pfSense team to either acknowledge or refute this issue. I would expect this from a fortune 500 company, but not from a grass roots originated, community contributed, effort like pfSense. :o
Really seems as though the pfSense team is not interested in engaging. This to me is an indication that there really is an issue and that the issue is beyond what they are capable of dealing with due to a technical limitation within the product. Or that there could be an unpalatable to the community agenda involved. Whatever the case or reason may be the response thus far does not instil confidence in the product or ESF. :(
Since I do not use this product in a business environment the issue itself is not of much concern to me. The pfSense team seemingly lack response and engagement though is very unpalatable.
If I were using pfSense in a business environment I would be very concerned though that with such a small amount of traffic pfSense could be taken off-line by a disgruntled employee, dissatisfied customer, or an unscrupulous competitor.
Regardless of one's philosophy of where such a low bandwidth "attack" should be mitigated. No modern firewall should be susceptible to being knocked off-line by as little as a few megabits of traffic. ::) This would be totally unacceptable to me in a business environment.
-
Its been 3+ months since this was forwarded to the devs by Lowprofile and me.
There has been NOTHING other than scattered emails and one PM to me about pcaps from CMB and he could get it since its in this thread.
I find that fact pretty scary since people buy commercial support and gold memberships to support the project.
But when its really needed, everything went quiet and almost no responde other than accusations of me bringing down the forum.
I would love to help out, but I am not a freebsd/linux guru and I cannot provide anything besides point out there is an issue, trying to enlighten people and show how it affects pfsense and other firewall distros.
Thats why I ask for testers and help from people like Doktornotor since I know he is a dev. and all I get is shit from him.
I have promised not to spread the script since as it is, it can take down any firewall running pfsense and other OS'.
Its like having a dirty bomb and people asking for copies, than all hell breaks lose.
What else can I do to draw attention to this and have a response to the issue?
The following is my opinion.
I don't really see this thread as being squarely aimed at root causing the issue as much as one of bringing to light the issue to make the community and pfSense team aware of the issue. Awareness has been accomplished.
Very disappointed in the response (or seemingly lack of response) by the pfSense team to either acknowledge or refute this issue. I would expect this from a fortune 500 company, but not from a grass roots originated, community contributed, effort like pfSense. :o
Really seems as though the pfSense team is not interested in engaging. This to me is an indication that there really is an issue and that the issue is beyond what they are capable of dealing with due to a technical limitation within the product. Or that there could be an unpalatable to the community agenda involved. Whatever the case or reason may be the response thus far does not instil confidence in the product or ESF. :(
Since I do not use this product in a business environment the issue itself is not of much concern to me. The pfSense team seemingly lack response and engagement though is very unpalatable.
If I were using pfSense in a business environment I would be very concerned though that with such a small amount of traffic pfSense could be taken off-line by a disgruntled employee, dissatisfied customer, or an unscrupulous competitor.
Regardless of one's philosophy of where such a low bandwidth "attack" should be mitigated. No modern firewall should be susceptible to being knocked off-line by as little as a few megabits of traffic. ::) This would be totally unacceptable to me in a business environment.
-
Its been 3+ months since this was forwarded to the devs by Lowprofile and me.
There has been NOTHING other than scattered emails and one PM to me about pcaps from CMB and he could get it since its in this thread.
I find that fact pretty scary since people buy commercial support and gold memberships to support the project.
But when its really needed, everything went quiet and almost no responde other than accusations of me bringing down the forum.
I would love to help out, but I am not a freebsd/linux guru and I cannot provide anything besides point out there is an issue, trying to enlighten people and show how it affects pfsense and other firewall distros.
Thats why I ask for testers and help from people like Doktornotor since I know he is a dev. and all I get is shit from him.
I have promised not to spread the script since as it is, it can take down any firewall running pfsense and other OS'.
Its like having a dirty bomb and people asking for copies, than all hell breaks lose.
What else can I do to draw attention to this and have a response to the issue?
Only time will tell what the devs agenda is on this. For all I know there is enough testing done here. It is up to us the users to decide whether we care or not now. The devs make their money different ways and have different priorities. The good doktor is trying to troll. He is useless so far himself. Supermule has produced results. What have you done? At least don't blame the guy. He can't spoon feed you many times over. If he releases this script to the forum, don't you think someone will pick the script and attack everyone? He already says he went to devs 3 months ago and scripts are available so I am not sure what your point is? Do you still not believe this happening or what? If you think this is fake then let us know how otherwise let the man get the attention needed. He said it repeatedly that devs are not responding and yet you go on ranting about how he should do more free work for you. STOP IT doktornotor - you are annoying and trolling. PM supermule if you have good intentions.
-
Supermule has produced results. What have you done? At least don't blame the guy. **With all due respect to Supermule, and I am actively working with him, he hasn't produced results. He's provided a use case. That use case triggers an adverse result. That's what he's produced. I will be working with him to narrow down if that is rooted in FreeBSD 10.1 or pfSense. It'll take a few weeks for me to make infrastructure changes and build servers, but I think it'll narrow it down.
I'm not here to defend Supermule, but I am absolutely here to work with him to put the effort into determining the root cause with the resources I have available to me.
Isn't that how open source communities work?
I understand Supermule's frustration because the devs on this forums are so cocky at times and jump at you the moment you talk about security or features. I have seen that first hand. I am just in the process of setting a test bed myself but if I see same results then I would know for sure that anyone who says otherwise is trolling or stalling what Supermule is trying to put out here. If he is a whistleblower then you should appreciate him regardless of whether he helps anymore or not.
We, the users, do NOT code! but we give the devs the power to see all the download and usage statistics so YES we are directly contributing to make this product better by simply using it.
Turn the thread into a progress report if you care - otherwise, I don't believe anything you say unless a new version comes that is not prone to the same attack.
Pfsense is AWESOME! but if they turn this into a personal war against Supermule then it's really stupid of devs even if it is for buying time. He has raised his voice and has all the right to do so in whatever way he wants. This is an opensource project so suck it, fix it, and move on. No one is giving handouts here - everyone contributes!**
-
I do not appreciate SM's disclosure. No developer would. However, I am trying to respond to it in a way that I can contribute.
Users absolutely DO code.Most of users do NOT code! We contribute by download, test, and report. Devs make their money indirectly from this. Don't act superior to other users. The fact that you did coding or not has NO VALUE. You did it for your own personal gain regardless. One doesn't need a business major to understand this. This project exists because of it's user-base (Small or big). You are not important any more than any other user here. Don't flatter yourself!
I do NOT understand why you make this about Supermule's character or method of disclosure while this is simply a really bad thing happening out there that has nothing to do with anything but the issue itself. Attack the problem, not the messenger. Are you pissed that he found it? I am certain he is not lying when he says Devs give him shit for reporting this. Once again, people like you are trying to change focus from what he is saying. And you are full of shit because if you saw no value in what Supermule found then you wouldn't be so angry here. If you don't believe this is serious then forget it and move on. If you do believe he is onto something and you still DO NOT APPRECIATE HIM then, …. (fill the dots yourself)
And what the heck is a white hat? Who decides who is a white hat - you? LOL - No one trusts you when you take the high moral ground to everyone else! STOP ATTACKING HIS CHARECTER - this is stupid
-
Geesus guys!
Take it easy. Just got up on a sunday and all this bickering is not good for the community.
Lets get into solution mode and see if we can assist the dev's on this one since it seems to be pretty severe.
Go grab a coffee and a donut :)
I will read my PM's in the meantime with people wanting to help. Really appreciated guys!
-
-
My apologies. The Jameson made Mr. Hyde come out.
-
Geesus guys!
Take it easy. Just got up on a sunday and all this bickering is not good for the community.
Lets get into solution mode and see if we can assist the dev's on this one since it seems to be pretty severe.
Go grab a coffee and a donut :)
I will read my PM's in the meantime with people wanting to help. Really appreciated guys!
Coffee sounds really good. Still waiting to get a coffee grinder and a french press. I don't trust coffee filters unless stainless steel, and those are hard to find outside of a french press. I don't like to use paper filters because it seems like a such a waste of paper and many are treated with a range of chemicals, and plastic filters, well, food grade nylon is only rated for 165f before it starts leeching chemicals.
I drink my tea every morning, but I really want free leaf and a tea ball, so I can cut down on waste materials. Took a lot of reviews, but I found a tea that uses compress air to whiten their bags, instead of chemicals, their bags don't use staples only tied so less waste, and chemical analysis shows very low levels of lead and other heavy metals compared to other brands and well below safety limits, and virtually no pesticide residue.
We may be living long compared to people decades ago, but I take the stance that if everything you eat is technically safe, you're just going to get death by a thousand cuts. I don't just want "safe" levels just below the maximum, I want the best that can be done, with in reason.
There's my random weekend off-topic rant.
Next on the agenda, replacing mercury filled florescent lightbulbs with LED lights, but ones that don't have a strong blue spectrum that causes eye strain.
-
Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?
Specifically: Is the traffic in question actually passed, or blocked? Is a service on the firewall running pfSense (such as the GUI) exposed to the test source or is the traffic being passed through to an internal host (port forward, routed, etc)? – This is important because a SYN flood to pfSense as a host is completely different than a flood through pfSense as a forwarder/firewall.
If the traffic is passing through the firewall, using rules to clamp down state limits, or going stateless properly (floating quick OUT rules to pass out with no state along with the pass in rules on the other tabs) may help.
If the traffic is targeting the firewall itself, then there are things that can be tweaked (syncache parameters, for example), but it's yet another reason the services on the firewall such as the GUI and SSH should not be exposed to the Internet in general. State limits can help there as well.
Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.
-
Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?
Yes.
Supermule, Tim.Mcmanus, and I (almabes) are going to work on this. I will get together some documentation once we get the problem isolated.To quote the late, great Joe Cocker
We could use a little help from our friends. -
Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.
Hi Jimp,
In my tests with Supermule, he was able to take down my pfSense 2.2.2 factory fresh install with no ports open. By take down, I mean the GUI was not responding, ping to pfSense was met with no response, ping to google.com stopped responding, and I could not browse the internet. It tooks about 30-60 seconds for pfsense to become responsive once he stopped the attack. Supermule said he was attacking with 5-6mbit. Following is what I had from the console at one point during the attack.
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTAfter attack stop, I saw this: 05-18-15 11:48:00 [ There were error(s) loading the rules: pfctl: DIOCXCOMMIT: Device busy - The line in question reads [0]: ]
-
Can you post screen shots of your AllGraphs for the System tab instead if just the processor?
-
Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?
If so, it would be worth reporting upstream as well.
-
Also, has anyone taken a stock FreeBSD 10.1 box with pf and a basic ruleset to see if the same happens there?
If so, it would be worth reporting upstream as well.
I can probably do this, but it'll be about a week before I can present data and metrics. I already have a clean FreeBSD 10.1 and Apache 4.2 server built for testing. Enabling PF shouldn't be too difficult, I just need to plot out the architecture and possibly roll another 10.1 VM with two interfaces and a point to dead end traffic.
-
OPNSense responds better (35 seconds) before its taken offline.
Make up your own mind…
You mean their Ubuntu VPS web server that isn't behind a firewall at all. Yeah, make up your own mind indeed.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
This was the first line that caught my attention:
May 18 15:48:22 php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
-
Talking of DHCP, mine is going all crazy on WAN…
dhclient[21355]: Many bogus options seen in offers. Taking this offer in spite of bogus options - hope for the best!
dhclient[21355]: option nisplus-servers (100) larger than buffer.
dhclient[9876]: rejecting bogus offer.
dhclient[9876]: option option-97 (100) larger than buffer.
dhclient[90315]: rejecting bogus offer.
dhclient[90315]: option streettalk-directory-assistance-server (97) larger than buffer.
dhclient[2514]: rejecting bogus offer.
dhclient[2514]: option option-109 (111) larger than buffer.
dhclient[57118]: rejecting bogus offer.
dhclient[57118]: option nntp-server (105) larger than buffer.
dhclient[99613]: rejecting bogus offer.
dhclient[99613]: option routers (81) larger than buffer.
dhclient[35967]: rejecting bogus offer.
dhclient[35967]: option option-82 (97) larger than buffer.None of those were solicitated/initiated by me.
Something is wrong.
"hope for the best!"F.
