Our Sites become unavailable randomly
-
Lol, a security appliance in front of what we already consider to be our security appliance? Isn't that what PFSense is already, a firewall?
All this discussion is good, don't get me wrong, but it is all distracting from the real issue I was having.
Seems to be a common tact. People offer other solutions rather then explain why pfSense doesn't measure up to the task.
There is another thread here where it's been demonstrated that pfSense can be knocked-out with by a SYN "flood" of only a few mbps. One of the typical responses was; DDoS should be mitigated upstream. LOL.
Apparently some people just can't face the fact that pfSense seems to have some serious issues, just because their usage model doesn't seem to be affected (at the moment).
If you think pfSense is a security appliance, you're very misguided.
Using a stateful packet inspection device to mitigate a DDOS SYN flood–or any DDOS for that matter--is, well, a gross misunderstanding of what a security appliance is as opposed to what pfSense is. DDOS, by design, is a capacity overflow attack. That capacity could be bandwidth, states, or anything else.
-
If you think pfSense is a security appliance, you're very misguided.
Don't tell me. I never claimed pfSense is a security appliance. Tell these people. https://www.pfsense.org/
"Open Source Security"
"We make network security easy."
"Providing comprehensive network security solutions for the enterprise, large business and SOHO, pfSense solutions bring together the most advanced technology available to make protecting your network easier than ever before. Our products are built on the most reliable platforms and are engineered to provide the highest levels of performance, stability and confidence."
Using a stateful packet inspection device to mitigate a DDOS SYN flood–or any DDOS for that matter--is, well, a gross misunderstanding of what a security appliance is as opposed to what pfSense is. DDOS, by design, is a capacity overflow attack. That capacity could be bandwidth, states, or anything else.
Regardless of ones philosophy on where and how DDOS attacks should be mitigated, as little as a few mbps of any traffic should not take down a modern day firewall. If it does then it is flawed. And to simply say it's not the proper place, device, etc. to mitigate is just making excuses.
-
Regardless of ones philosophy on where and how DDOS attacks should be mitigated, as little as a few mbps of any traffic should not take down a modern day firewall. If it does then it is flawed. And to simply say it's not the proper place, device, etc. to mitigate is just making excuses.
You cannot mitigate DDOS with a stateful packet inspection device, period. That's what PF is. It's the wrong device to do the job.
A stateless packet device acting as a firewall can. That's why there's a market for $300K Palo Alto devices. You get what you pay for.
-
Actually, I'll admit I'm wrong. pfSense is a security appliance. That was my mistake.
The other mistake I made was classifying a DDOS as a security issue, which it is not.
-
You cannot mitigate DDOS with a stateful packet inspection device, period. That's what PF is. It's the wrong device to do the job.
Which misses the point. A few mbps of any traffic, regardless of whether or not is a DDOS or otherwise, should not take down a modern firewall. If it does then it is flawed.
-
You can't bring down a system with 55 DNS requests. Maybe if you're running Snort with auto-blocking and haven't whitelisted your own IPs, and the traffic triggers a signature in such a way that it ends up adding your own IPs to the block list. That's another explanation where a reboot would fix things, since it clears Snort's blocking table. Do you have Snort installed?
Only 55 connections almost certainly isn't an open resolver. Those usually rack up hundreds of thousands or more on fast connections, completely filling your upload to the extent your connection will be nearly unusable (the fun of UDP floods). Your datacenter would have said something if you were using far more bandwidth than usual (at least if it's a worthwhile DC). I also have doubts those 55 connections even have a relation to the problem, though it's possible. Did you get packet captures of what they were doing?
You also can't bring down a system with a few Mbps DDoS IF it's sized and configured accordingly to handle that kind of resource exhaustion attack.
-
We are a small company with not that much traffic at all.
I assure you that it was indeed the 55 udp connection that where routed to our Webserver which also runs our DNS. As soon as my ISP blocked the IP on their end the problem has not returned.
I am not running many packages, just: mailreport, OpenVPN Client Export Utility, pfBlockerNG, RRD Summary and Service Watchdog.
I did not even get details about the actual attacker. My ISP said they were from Russia and I just assumed then that it if he saw immediately that it was Russian then the IP must show up on a russian country blocklist. I assume you that pfBlockerNG is enabled and Russia and China from the top 20 list are the only two selected.
-
And what was the IP or IPs? Go to diag, tables - and pick your table that you created under country.. I assume it was ipv4 and not ipv6, etc.
So attached is very small sample, and where was the firewall rule using this?
This data is gotten from "Geolite Data by Maxmind Inc. - ISO 3166" Does not mean it is 100% inclusive of every single netblock that is russia, or maybe it wasn't russia and that is just what your isp said, maybe it was the Ukraine or something like that. Without the IP in question and validation that it was in your table to block, and that the table was used correctly in a rule, etc..
As to 55 connections taking him down.. Would depend on many factors - what that query was exactly, what version dns is he actually running? A misconfiguration in say rate limiting could shoot you in the foot.. You go no details to work with here at all to why his site was down..
Lets see this firewall rule that is blocking russia and china, and lets see the IP of this so called 55 connection attacker, etc.
-
I have no special settings at all (not using rate limiting), just a basic setup.
I have not created a table… Pretty sure pfBlockerNG does this on it's own. Also the rules are automatically created by pfBlockerNG.
I already stated DNS is running from Windows 2012 R2.
-
Regardless of ones philosophy on where and how DDOS attacks should be mitigated, as little as a few mbps of any traffic should not take down a modern day firewall. If it does then it is flawed. And to simply say it's not the proper place, device, etc. to mitigate is just making excuses.
You cannot mitigate DDOS with a stateful packet inspection device, period. That's what PF is. It's the wrong device to do the job.
A stateless packet device acting as a firewall can. That's why there's a market for $300K Palo Alto devices. You get what you pay for.
Wrong. It depends on some factors. But if your firewall is not capable to stop a 10mbit SYN flood when others can, then there is something wrong. Period.
Solution? Replace your "firewall" with a real firewall. -
This is the same solution that keeps popping into my head…
-
What is the IP of your so called attacker and can look to see that its listed in pfbng as russia. Also post up your rules where pfbng is blocking.. Lets see your wan rules..
If your running dns on 2k12 to the public - it could stop responding if you looked at it funny ;)
" But if your firewall is not capable to stop a 10mbit SYN flood "
Good luck stopping that with a Million dollar firewall, If your pipe is only 8mbps…
-
I do not know the IP of the attacker as I didn't get those details from our ISP.
I am willing to accept that the address may simply not be included in pfBlockerNG's blocklist.
So that is not the real issue…
I would like to know why PFSense is blocking wan dns requests while this attacked was connecting. It was not a high bandwidth attack, it was 55 connections from a single IP.
-
Who said it was blocking anything.. Did you sniff on your dns server and validate that traffic was not getting there? Was pfsense logging that it was blocking? Without know what the traffic was you have no idea what it was asking your ns to do - its quite possible those queries were preventing it from doing anything else, etc.
You have no DETAILS yet you want to know why something happened?? So my car stopped running - what was the problem? Can't you tell me with that amount of information? BTW it was red and the tires where new - isn't that enough to tell my why it stopped running?
You don't know what your rules are, we don't know the setup of your network, we don't know the setup of your websites - we don't even know that they were down to be honest.. We have no details of anything to even take a guess to what your problem might have been. For all we know query was causing high load on your windows machine and that is why the websites didn't load, etc. etc..
-
No I did not do those things. However like I already stated, once I shut off our main PFSense box and then CARP kicks in the secondary box, the dns requests are instantly working again.
That proves it's not a problem with our DNS server.
-
No it doesn't it proves those queries are no longer going to your dns server..
Dude without any details there is NO WAY to even guess to what your actual problem is/was will be…
-
it proves those queries are no longer going to your dns server..
Yeah that's exactly what I'm trying to say. PFSense is crapping out and no longer sends through the port 53 udp traffic.
If this problem creeps up again some time in the future I will dig deeper. It just that our setup is so generic without any fancy packages or special settings.
-
Here's a quick advise: Stop running public DNS servers. Especially stupid idea on shitty lines.
-
"It just that our setup is so generic without any fancy packages"
Well then lets see it.. It can not be too generic since you have a carp setup. So what is in front of pfsense? How is your webserver/dns/whateverelse server connected to both of your pfsense boxes?
What just blows me away is how someone could be in charge of a network and having what I assume is a production down issue and not even get the IP that is supposedly causing your problem that you thought you were blocking, etc. etc. But your not sure since you don't even know what IPs are in the list your blocking nor what the IPs was.. You could of clearly just looked in pfsense to see the number of connections to your dns server from the outside, etc.
Why would you shut off your main pfsense without some basic details of what was going on, etc.
Just blows my freaking mind!!!
Also does one configure such setup, so where is your other ns for these domains. Also behind pfsense pointing to the same 2k12 box? So your registrar allowed you to just use 1 public IP for your 2 required dns?
For F sake I wanted to play with a domain for dnsssec that nobody uses and I still put up 2 ns in different parts of the world - one in LV and the other in Luxembourg. why you would host your own production off a 2k12 is nuts.. And then you didn't even bother to setup any sort of rate limiting you said. What safe guards did you put in place other than trying to block china and russia ips? I can send queries to your "authoritative" dns from spoofed IP asking for something your authoritative for, you then send reply to spoofed IP.. So your dns is now the amplifier in an attack - be it your allowing recursive or not.
What is this domain btw, love to see the report on the dns and what it shows for ns, etc. PM it to me please if you don't want to make it public.
-
It's not blocking that traffic if states are showing up. I strongly suspect the issue has nothing to do with the DNS traffic from Russian IPs. No telling though given you have no idea what that traffic actually was.
johnpoz is right on with the complete lack of any useful info on actually troubleshooting the problem. What works around the problem isn't useful in troubleshooting the root cause, as what you're doing to "fix" it will fix a variety of network problems outside and completely unrelated to the firewall. Packet capture on WAN is your best bet here. Traffic coming in? Correct destination MAC? Then moving on to seeing if the traffic is being passed or blocked, checking the firewall log and state table. Are the interface IPs affected, or only the CARP IPs? Did you change the VHIDs on the CARP IPs to something less common than the lower numbers as I mentioned previously?
