UDP Advanced Rules - Under UDP DDoS Attack
-
Hi guys.
We've been under a
massiveUDP DDoS for the past 24 hours. They're basically targeting every DNS server behind the firewall.I have limits for maximum connections per host on but they don't appear to be working, unless maybe I need to specify max table states per host too. Basically what's happening every 15 minutes or so, the states will shoot up until packet loss occurs, LAN errors show up on the interface, then the packet loss appears to reset there attack. It's been repeating like this for the past day.
Does anyone know if the advanced rules in 2.0-RC1 applies to UDP traffic? Thank you for the insight.
-
It can be very hard, if you're passing the traffic used for DDoS.
Check my post at http://forum.pfsense.org/index.php/topic,46897.0.html
-
It can be very hard, if you're passing the traffic used for DDoS.
Check my post at http://forum.pfsense.org/index.php/topic,46897.0.html
Very interesting, I have to read through that in detail.
There is an advanced rule in pfsense called "new connections per second" is this per host or for the entire rule? Depending, might be able to use it…
-
In this case, the traffic itself is not high, it's the amount of packets. It's a very interesting attack, every 15 minutes a massive storm of packets comes in, the firewall states keep going up until packet loss occurs, then it resets the attack. In most DDoS attacks I've seen, the attackers don't stop sending traffic.
Question is, under Advanced Rules, how much of those options work for UDP? It appears as though the option I have set are not working. In particular, max connections per host and connections per second.
One thing I don't know a lot about is UDP traffic. Any thoughts would be greatly appreciated. Once I can figure something out I will open up a topic on it in detail. Thanks guys.
-
With my limited knowledge and experience with pfSense here's my two cents:
-
If you change a rule it only applies on new connections. Maybe a reset of the state table helps.
-
I think an combination of the last 4 options under advanced should help, but only by trial and error.
If the attack comes from a specific country you could block ip's from that region if you don't have legitimate traffic from those countries with the pfBlocker package.
Good luck.
-
-
Even if you block the packets, they must still be processed by the firewall (if only a little) to figure that out. So pps is still an issue even if you don't allow anything at all.
It's difficult for any firewall to fend off a DDoS of that sort. You can throw more hardware at it, but really a DDoS must be cut off closer to the source (read: your ISP, or theirs)
-
Even if you block the packets, they must still be processed by the firewall (if only a little) to figure that out. So pps is still an issue even if you don't allow anything at all.
It's difficult for any firewall to fend off a DDoS of that sort. You can throw more hardware at it, but really a DDoS must be cut off closer to the source (read: your ISP, or theirs)
Thanks Jimp. I don't think the attack is as big as I thought. I think maybe someone could be spoofing the udp packets and just sending a storm of them. the thing is, the attacks are coming in on 2 of our networks, the transparent bridge (pfsense) can take it and has been ok but the pfsense firewall running nat w/ carp+virtual IPs is the one going down every so often, it's crashing long before the states run out.
can you think of any thing in pfsense under udp floods that would cause the state table to reset like this? the firewall doesn't crash, just the state table is resetting… the wan logs indicate that nothing is going down, just the lan side is what's crashing. the wan looks like it takes it.
-
intersting… I see what you're saying. even if I block it, it still requires resources. Adjusting the settings within pfsense for the UDP/53 rule did help, but it still resets the tables after enough pour in. yet, the cpu usage doesn't even spike that much, very little.
I was thinking about using the new connections per second, but it doesn't appear to work on UDP, does it?
-
Since it's easy to spoof UDP traffic, a firewall rule that would add "offending" IPs to your blacklist may become a double-edged sword …
Had it been only TCP traffic, you could use "max new connections per unit of time" rules.
-
Thanks jimp, dhatz and everyone.
Something doesn't appear right. When the packet loss occurs, it resets all of the states but the interupt cpu usage only hits no more then 8%.
I should add, this is a watchguard firewall with those realtek nics on there. The lan errors only occur when the DDoS hits, and it's only on the lan side, the wan never goes down.
There attack appears to be timed every 10 minutes, then sends a storm of packets on UDP/53.
-
curious how many packets are we talking? And what is in the traffic - you sure its just not just a storm of dns requests, maybe because of a really low ttl (say 10 minutes). On a zone that is seeing heavy use?
As stated it hard to stop a DOS at the endpoint – you need to stop the traffic upstream from your device.
-
curious how many packets are we talking? And what is in the traffic - you sure its just not just a storm of dns requests, maybe because of a really low ttl (say 10 minutes). On a zone that is seeing heavy use?
As stated it hard to stop a DOS at the endpoint – you need to stop the traffic upstream from your device.
Hi Johnpoz, thanks for the reply. There not legit DNS requests because I was watching the zone queries, it's more like they just send a udp packet. some type of UDP flood, though I don't know much about UDP. It was much higher yesterday the traffic but I blocked so many networks in China. Now its lower, I will average 1000 states to the DNS, but out of no where the states shoot to about 8000, the interrupts only reach 8.0% on the firewall then I drop packets on the lan side, but the wan connection doesn't get interrupted. It's very strange. It's been running for a year with no major issues, until this attack. They're also targeting 7 different DNS servers on the network at the same time. They hit from multiple sources, but it's all in sync.
They're also hitting another network here (different WAN), but that firewall is transparent yet it can handle it. Any reason why a NAT firewall w/carp and VIP's would drop all of the states long before the interrupts reach 100% or max state size?
As always, thank you all for your expertise!
-
Do these 7 servers host the same zone(s)?
Can you capture some of these packets they are sending and post them.. Just curious to what they are sending..
Normally in a attack against dns you would send valid queries, to get the server working trying to work extra hard in working on responding to them - just sending gibberish would be less effective. Unless is was some sort of exploit type packet.
Grab capture on the firewall for a few packets and post it up - maybe we can glean something from it that will help. More information is never a bad thing ;)
Do these servers respond to recursive queries, or only authoritative for your zones?
-
Hi Johnpoz: They were recursive but I disabled that, didn't help much. The only thing keep me up is the fact that pfsense resets tables when its crashing the lan interface.
I am going to try to capture that traffic now for you thanks
-
got it… seeing a lot of these repeated:
I replaced our actual host name with server (server.com) and replaced the first of our ip with 0
202.216.0.21.45553 > 0.41.228.2.53: [bad udp cksum 1!] 92% [1au] AAAA? ns3.server.com. ar: . OPT UDPsize=512 OK (44)
22:34:57.809364 IP (tos 0x0, ttl 251, id 41584, offset 0, flags [DF], proto UDP (17), length 72)
202.216.0.21.44340 > 0.41.228.2.53: [bad udp cksum 1!] 9243% [1au] AAAA? nr1.server.com. ar: . OPT UDPsize=512 OK (44)
22:34:57.841478 IP (tos 0x0, ttl 253, id 12585, offset 0, flags [DF], proto TCP (6), length 48) -
I don't know if this is the cause.
Normally, the DNS traffic flows from outside to inside.
Right before the states in pfsense crash, there are a lot of requests going out of the DNS. But I know the name servers are not rooted. Is it possible they're using our dns servers to launch an attack?
-
I wonder how I might expire UDP faster then the "aggressive" mode which is 30-60s.
-
so they are actual queries… Not gibberish.
And you would expect if going to be over the udp limit then yes you switch to TCP. And those are just AAAA requests for what look like NS..
So how again are you sure this an attack and not just not some dns storm?
A longer snag would be helpful.. I am not convinced its an actual directed attack.. When BP had their spill, prob could of looked like an attack on their dns if there TTLs were to small.
What is the TTL of those records? Do they keep doing the same query over and over - and you are responding?
Again -- I know these suggests that can help you with your box not crashing.. But understanding the traffic might lead you to other ways to remove the problem.
How many zones are these servers serving up? You sure one of the zones just don't have an issue with their TTL on the NS record, etc. Again - looking at the actual traffic might point us in the right direction.
-
I appreciate the help you've given me, and others. When things get stressful and people call me I tend to not put as much thought in to things, I apologize for the little info I gave you.
To be honest, I don't know that it's not a DNS storm, at this point I'm not sure of anything :-) . The only thing that appeared off, I did a test and shutdown one of the name servers but new states were still appearing to it. When I watched the dns logs there was the regular amount of queries, it's almost like empty UDP traffic. Maybe it's some other issue causing pfsense to just reset states. Is there a log in pfsense somewhere that would indicate any of this?
DNS Zones on these servers, only about 500 zones.
-
To make things even more confusing, I just noticed the packet loss on the inside is only from the LAN switch to pfsense, yet not 1 error on the switch port just on the lan interface for pfsense. I've tried different cables, even tried a different interface. From the outside in there is no packet loss at all. I am starting to think it's something in pfsense. I don't know anymore, going insane.
I don't want to waist your time, it could be something else I will keep digging… Thanks.
