Our Sites become unavailable randomly
-
Who said it was blocking anything.. Did you sniff on your dns server and validate that traffic was not getting there? Was pfsense logging that it was blocking? Without know what the traffic was you have no idea what it was asking your ns to do - its quite possible those queries were preventing it from doing anything else, etc.
You have no DETAILS yet you want to know why something happened?? So my car stopped running - what was the problem? Can't you tell me with that amount of information? BTW it was red and the tires where new - isn't that enough to tell my why it stopped running?
You don't know what your rules are, we don't know the setup of your network, we don't know the setup of your websites - we don't even know that they were down to be honest.. We have no details of anything to even take a guess to what your problem might have been. For all we know query was causing high load on your windows machine and that is why the websites didn't load, etc. etc..
-
No I did not do those things. However like I already stated, once I shut off our main PFSense box and then CARP kicks in the secondary box, the dns requests are instantly working again.
That proves it's not a problem with our DNS server.
-
No it doesn't it proves those queries are no longer going to your dns server..
Dude without any details there is NO WAY to even guess to what your actual problem is/was will be…
-
it proves those queries are no longer going to your dns server..
Yeah that's exactly what I'm trying to say. PFSense is crapping out and no longer sends through the port 53 udp traffic.
If this problem creeps up again some time in the future I will dig deeper. It just that our setup is so generic without any fancy packages or special settings.
-
Here's a quick advise: Stop running public DNS servers. Especially stupid idea on shitty lines.
-
"It just that our setup is so generic without any fancy packages"
Well then lets see it.. It can not be too generic since you have a carp setup. So what is in front of pfsense? How is your webserver/dns/whateverelse server connected to both of your pfsense boxes?
What just blows me away is how someone could be in charge of a network and having what I assume is a production down issue and not even get the IP that is supposedly causing your problem that you thought you were blocking, etc. etc. But your not sure since you don't even know what IPs are in the list your blocking nor what the IPs was.. You could of clearly just looked in pfsense to see the number of connections to your dns server from the outside, etc.
Why would you shut off your main pfsense without some basic details of what was going on, etc.
Just blows my freaking mind!!!
Also does one configure such setup, so where is your other ns for these domains. Also behind pfsense pointing to the same 2k12 box? So your registrar allowed you to just use 1 public IP for your 2 required dns?
For F sake I wanted to play with a domain for dnsssec that nobody uses and I still put up 2 ns in different parts of the world - one in LV and the other in Luxembourg. why you would host your own production off a 2k12 is nuts.. And then you didn't even bother to setup any sort of rate limiting you said. What safe guards did you put in place other than trying to block china and russia ips? I can send queries to your "authoritative" dns from spoofed IP asking for something your authoritative for, you then send reply to spoofed IP.. So your dns is now the amplifier in an attack - be it your allowing recursive or not.
What is this domain btw, love to see the report on the dns and what it shows for ns, etc. PM it to me please if you don't want to make it public.
-
It's not blocking that traffic if states are showing up. I strongly suspect the issue has nothing to do with the DNS traffic from Russian IPs. No telling though given you have no idea what that traffic actually was.
johnpoz is right on with the complete lack of any useful info on actually troubleshooting the problem. What works around the problem isn't useful in troubleshooting the root cause, as what you're doing to "fix" it will fix a variety of network problems outside and completely unrelated to the firewall. Packet capture on WAN is your best bet here. Traffic coming in? Correct destination MAC? Then moving on to seeing if the traffic is being passed or blocked, checking the firewall log and state table. Are the interface IPs affected, or only the CARP IPs? Did you change the VHIDs on the CARP IPs to something less common than the lower numbers as I mentioned previously?
-
@cmb:
You also can't bring down a system with a few Mbps DDoS IF it's sized and configured accordingly to handle that kind of resource exhaustion attack.
Unless the system is flawed (or has a bug) of course.
Also, what a nice out/excuse. If a few Mbps DDoS takes down a system then just say, "it's not sized or configured accordingly to handle that kind of resource exhaustion attack", and magically there is no issue with the system.
-
@cmb:
You also can't bring down a system with a few Mbps DDoS IF it's sized and configured accordingly to handle that kind of resource exhaustion attack.
Unless the system is flawed (or has a bug) of course.
Also, what a nice out/excuse. If a few Mbps DDoS takes down a system then just say, "it's not sized or configured accordingly to handle that kind of resource exhaustion attack", and magically there is no issue with the system.
Why don't you pony up and contribute a line or two of code to fix the issue.
-
@cmb:
You also can't bring down a system with a few Mbps DDoS IF it's sized and configured accordingly to handle that kind of resource exhaustion attack.
Unless the system is flawed (or has a bug) of course.
Also, what a nice out/excuse. If a few Mbps DDoS takes down a system then just say, "it's not sized or configured accordingly to handle that kind of resource exhaustion attack", and magically there is no issue with the system.
Why don't you pony up and contribute a line or two of code to fix the issue.
What issue? CMB said it's not possible to do. So what would there be to fix then?.
And besides, that is a pathetic copout statement. Don't point out any issues or problems unless you can or will fix it too. Totally bogus. How would you like to be a passenger of a cross oceanic flight on a plane that had just been inspected by a crew with that philosophy?
-
@cmb:
You also can't bring down a system with a few Mbps DDoS IF it's sized and configured accordingly to handle that kind of resource exhaustion attack.
Unless the system is flawed (or has a bug) of course.
Also, what a nice out/excuse. If a few Mbps DDoS takes down a system then just say, "it's not sized or configured accordingly to handle that kind of resource exhaustion attack", and magically there is no issue with the system.
Why don't you pony up and contribute a line or two of code to fix the issue.
What issue? CMB said it's not possible to do. So what would there be to fix then?.
And besides, that is a pathetic copout statement. Don't point out any issues or problems unless you can or will fix it too. Totally bogus. How would you like to be a passenger of a cross oceanic flight on a plane that had just been inspected by a crew with that philosophy?
Sorry, I must have confused you with a technologist. My bad. Won't happen again.
Oh, and flight crews regularly inspect the plane and prior to taking off. And, yes, they are required to actually troubleshoot issues and supply ground crews with tangible information if they detect anomalies. They don't just bitch to the ground crews and expect answers.
-
What is there to fix? We have no details to even guess with to what the problem actually is
-
Here's a quick advise: Stop running public DNS servers. Especially stupid idea on shitty lines.
We have hardly any traffic on a 100/100 Fiber line.
-
"It just that our setup is so generic without any fancy packages"
Well then lets see it.. It can not be too generic since you have a carp setup. So what is in front of pfsense? How is your webserver/dns/whateverelse server connected to both of your pfsense boxes?
Not sure how they are connected has anything to do with it. Regarless of any of the info you are asking for, the boxes worked fine for two years which wouldn't have happened had they been misconfigured. The carp setup is nothing outside of default. There is a Cisco router in front of the PFsense boxes that is supplied and controlled by our ISP. The PFsense have three network cards each, LAN, WAN and CARP.
What just blows me away is how someone could be in charge of a network and having what I assume is a production down issue and not even get the IP that is supposedly causing your problem that you thought you were blocking, etc. etc. But your not sure since you don't even know what IPs are in the list your blocking nor what the IPs was.. You could of clearly just looked in pfsense to see the number of connections to your dns server from the outside, etc.
Why would you shut off your main pfsense without some basic details of what was going on, etc.
Just blows my freaking mind!!!
I have not claimed to be a PFSense expert, I am a PFSense Noob. Not sure why you are personally attacking me here? I checked PFSense and could not find out the IP. Perhaps I was not looking in the right spot, sure, like I said I am not an expert on PFSense.
My job is to keep out connection alive and that's what I did. You think I should let the connection be down while I poke around trying to understand the problem?
Also does one configure such setup, so where is your other ns for these domains. Also behind pfsense pointing to the same 2k12 box? So your registrar allowed you to just use 1 public IP for your 2 required dns?
For F sake I wanted to play with a domain for dnsssec that nobody uses and I still put up 2 ns in different parts of the world - one in LV and the other in Luxembourg. why you would host your own production off a 2k12 is nuts..
We are a small company without the need to change our setup that has been working for us for many years. We have multiple public IPs. And we have multiple DNS servers. Really not sure why you are making assumptions, you know how that make you look right? Yeah we have a single Pipe, but if our connection drops then so does our webserver, so having secondary dns hosted somewhere else wouldn't help.
And then you didn't even bother to setup any sort of rate limiting you said. What safe guards did you put in place other than trying to block china and russia ips? I can send queries to your "authoritative" dns from spoofed IP asking for something your authoritative for, you then send reply to spoofed IP.. So your dns is now the amplifier in an attack - be it your allowing recursive or not.
I already said we are going to move DNS offsite for more security. Before this happened I thought PFSense was offering protection against such things out of the box. Forgive me for thinking PFSense could protect me from DDOS attacks as a default.
What is this domain btw, love to see the report on the dns and what it shows for ns, etc. PM it to me please if you don't want to make it public.
With how unprofessional you sound I would not want to share this info. And besides if there was any issues with how our DNS is configured then our email would experience problems sending places. We have it all configured just fine. Like I said intodns.com says it's all good.
-
whatever dude - good luck then.
Where have you shown its a ddos attack that is your issue? Where have you shown what the problem is, there is no firewall on the planet that stops ddos attacks btw.
If the pipe is filled up to your firewall, firewall can do nothing about that. If firewall is told to allow traffic to something, and you then overload that system or send it bad queries or whatever that causes an issue with that service. the firewall did what you told it to do - allow the traffic, etc
You don't even know the IP address of what you thought was the source of your problem.
All you have told us if you reboot one of your pfsense boxes the issue seems to go away.. What other info have you given that I might have missed?
-
The problem is solved already and both boxes have been back to humming along fine.
After our ISP found the IP they blocked it upstream. Our ISP said it was a 55 connection DDOS attack on port 53 udp.
Our DNS server was never overloaded or affected itself. It was just PFSense that was somehow not allowing requests through.
-
Strangely this morning we had both websites go down again, with "the connection has been reset" error.
During this time I did check DNS and it was responding no problem. It seems as if port 80 was not making it through PFSense.
I tried to enter Persistent Carp Maintenance Mode, however our sites still would not load.
After turning off PCMM and rebooting the first firewall it came back up immediately.
-
So lets see your firewall rules, and lets see sniffing on wan and lan that traffic is forwarded.
Your seeing nothing in pfsense logs any errors? anything? Nothing in firewall logs that traffic was blocked?
So your mail was working while your sites where down?
So from the PM you sent I see 3 IPs public.156, .157 and .158 and .158 and .157 are your mx boxes.. Do you have 2 of them or is this all being served up off your 1 2k12 box? And your just forwarding multiple IPs to the same private IP behind pfsense?
-
I am not sure I am looking in the right place to find relevant info. Maybe I can give you access to PFSense directly.
I did not check our mail ports specifically. I just checked if dns was resolving and it was.
Yeah we have three VMs we are running DNS on. They are not pointed to the same windows server.
-
so that changed? Where you not before running dns off your 2k12 box?
if you pm me login info I will take a look see..
