UDP Advanced Rules - Under UDP DDoS Attack
-
intersting… I see what you're saying. even if I block it, it still requires resources. Adjusting the settings within pfsense for the UDP/53 rule did help, but it still resets the tables after enough pour in. yet, the cpu usage doesn't even spike that much, very little.
I was thinking about using the new connections per second, but it doesn't appear to work on UDP, does it?
-
Since it's easy to spoof UDP traffic, a firewall rule that would add "offending" IPs to your blacklist may become a double-edged sword …
Had it been only TCP traffic, you could use "max new connections per unit of time" rules.
-
Thanks jimp, dhatz and everyone.
Something doesn't appear right. When the packet loss occurs, it resets all of the states but the interupt cpu usage only hits no more then 8%.
I should add, this is a watchguard firewall with those realtek nics on there. The lan errors only occur when the DDoS hits, and it's only on the lan side, the wan never goes down.
There attack appears to be timed every 10 minutes, then sends a storm of packets on UDP/53.
-
curious how many packets are we talking? And what is in the traffic - you sure its just not just a storm of dns requests, maybe because of a really low ttl (say 10 minutes). On a zone that is seeing heavy use?
As stated it hard to stop a DOS at the endpoint – you need to stop the traffic upstream from your device.
-
curious how many packets are we talking? And what is in the traffic - you sure its just not just a storm of dns requests, maybe because of a really low ttl (say 10 minutes). On a zone that is seeing heavy use?
As stated it hard to stop a DOS at the endpoint – you need to stop the traffic upstream from your device.
Hi Johnpoz, thanks for the reply. There not legit DNS requests because I was watching the zone queries, it's more like they just send a udp packet. some type of UDP flood, though I don't know much about UDP. It was much higher yesterday the traffic but I blocked so many networks in China. Now its lower, I will average 1000 states to the DNS, but out of no where the states shoot to about 8000, the interrupts only reach 8.0% on the firewall then I drop packets on the lan side, but the wan connection doesn't get interrupted. It's very strange. It's been running for a year with no major issues, until this attack. They're also targeting 7 different DNS servers on the network at the same time. They hit from multiple sources, but it's all in sync.
They're also hitting another network here (different WAN), but that firewall is transparent yet it can handle it. Any reason why a NAT firewall w/carp and VIP's would drop all of the states long before the interrupts reach 100% or max state size?
As always, thank you all for your expertise!
-
Do these 7 servers host the same zone(s)?
Can you capture some of these packets they are sending and post them.. Just curious to what they are sending..
Normally in a attack against dns you would send valid queries, to get the server working trying to work extra hard in working on responding to them - just sending gibberish would be less effective. Unless is was some sort of exploit type packet.
Grab capture on the firewall for a few packets and post it up - maybe we can glean something from it that will help. More information is never a bad thing ;)
Do these servers respond to recursive queries, or only authoritative for your zones?
-
Hi Johnpoz: They were recursive but I disabled that, didn't help much. The only thing keep me up is the fact that pfsense resets tables when its crashing the lan interface.
I am going to try to capture that traffic now for you thanks
-
got it… seeing a lot of these repeated:
I replaced our actual host name with server (server.com) and replaced the first of our ip with 0
202.216.0.21.45553 > 0.41.228.2.53: [bad udp cksum 1!] 92% [1au] AAAA? ns3.server.com. ar: . OPT UDPsize=512 OK (44)
22:34:57.809364 IP (tos 0x0, ttl 251, id 41584, offset 0, flags [DF], proto UDP (17), length 72)
202.216.0.21.44340 > 0.41.228.2.53: [bad udp cksum 1!] 9243% [1au] AAAA? nr1.server.com. ar: . OPT UDPsize=512 OK (44)
22:34:57.841478 IP (tos 0x0, ttl 253, id 12585, offset 0, flags [DF], proto TCP (6), length 48) -
I don't know if this is the cause.
Normally, the DNS traffic flows from outside to inside.
Right before the states in pfsense crash, there are a lot of requests going out of the DNS. But I know the name servers are not rooted. Is it possible they're using our dns servers to launch an attack?
-
I wonder how I might expire UDP faster then the "aggressive" mode which is 30-60s.
-
so they are actual queries… Not gibberish.
And you would expect if going to be over the udp limit then yes you switch to TCP. And those are just AAAA requests for what look like NS..
So how again are you sure this an attack and not just not some dns storm?
A longer snag would be helpful.. I am not convinced its an actual directed attack.. When BP had their spill, prob could of looked like an attack on their dns if there TTLs were to small.
What is the TTL of those records? Do they keep doing the same query over and over - and you are responding?
Again -- I know these suggests that can help you with your box not crashing.. But understanding the traffic might lead you to other ways to remove the problem.
How many zones are these servers serving up? You sure one of the zones just don't have an issue with their TTL on the NS record, etc. Again - looking at the actual traffic might point us in the right direction.
-
I appreciate the help you've given me, and others. When things get stressful and people call me I tend to not put as much thought in to things, I apologize for the little info I gave you.
To be honest, I don't know that it's not a DNS storm, at this point I'm not sure of anything :-) . The only thing that appeared off, I did a test and shutdown one of the name servers but new states were still appearing to it. When I watched the dns logs there was the regular amount of queries, it's almost like empty UDP traffic. Maybe it's some other issue causing pfsense to just reset states. Is there a log in pfsense somewhere that would indicate any of this?
DNS Zones on these servers, only about 500 zones.
-
To make things even more confusing, I just noticed the packet loss on the inside is only from the LAN switch to pfsense, yet not 1 error on the switch port just on the lan interface for pfsense. I've tried different cables, even tried a different interface. From the outside in there is no packet loss at all. I am starting to think it's something in pfsense. I don't know anymore, going insane.
I don't want to waist your time, it could be something else I will keep digging… Thanks.
-
I see a TON of the VRRP announcments being sent out of pfsense to some other devices on the network. About 50 a second, is this normal?
-
Solved… Except you called it when you said your not sure that it is an attack.
You're never going to believe what happened. I can't even explain it.
Something with CARP caused it to send out so many VRRP announcements that it would take the LAN interface down almost every 15 minutes, but first it slowed everything so much that all the states built up making it look like I was under an attack. I have no idea why UDP states would build like that, and not others.
This firewall has been running without issues for over a year, why all of a sudden would there be a VRRP loop. I suspect that our upstream carrier might have used VRRP on the same vhid causing the loop. In any case, I didn't even have to restart pfsense, I just disabled carp then reenabled carp and the announcements stopped.
Maybe this is worth me opening a seperate topic on just to help others who might have had an issue like this. I appreciate all of your help, you guys sure know your stuff. I really do apologize for waisting your time.
-Fred
-
"About 50 a second, is this normal?"
No I doubt that could ever be considered normal?
So now that you have cleared up your crash issue - are you stilling seeing bursts of dns requests every 10 minutes? I would assume that if your seeing bursts of dns at such a small interval that someone has a ttl too small for the amount of traffic the zone(s) see.
So are these all internal zones that are managed by you or someone else in your company.. Or are they just zones you host for other people? If hosted for other people, so many users do not understand dns in the slightest – very possible to have funked up configurations that cause way too much traffic.
-
Hi Johnpoz: 50/sec was the VRRP announcements being sent out by CARP. When I reset the carp they stopped and all of the problems cleared up, until several hours later when various VIP's started going offline and the announcements started again. I have a feeling the upstream carrier is running VRRP on the same ID's as nothing else was added to that network. This firewall is part of an older network of ours and is segmented from our others so it didnt change at all, no new switches or anything. As jimp mentioned, probably a layer 2 loop.
the odd part of it all, this caused so many strange issues with the states that the UDP states would not close and they kept piling up so it looked like a DDoS when infact it was CARP.
I had to take down CARP all together and use IPAlias instead. Working fine for a year until the other day.
**If anyone else see's Out errors on the lan interface for pfsense yet it's passing traffic but having issues with states building/resetting (causing repeated LAN packet loss/drops every so often), don't assume it's a bad cable or nic. Check your system log and also use tcpdump. You would be surprised at what could happen. :-)**Maybe this will help others too.
-
A VHID conflict wouldn't cause that many CARP advertisements in a short period, only a real layer 2 loop could have done that. CARP advertisements are constant: 1 per second (+skew) per VIP on a segment. You'd either see a constant 50 per second every second with or without a problem, or in the case of a single vhid conflict, you'd only see an extra 2 or so.
But if something is looping/bridging the traffic and it's just turning circles on the network, it would cause exactly the behavior you're seeing. If you are mixing CARP+bridging, that's a very bad thing do to.
-
Hi jimp: This firewall is only for NAT and had CARP runnning for 2 WANs, other then that no special configurations. The constant advertisments only started recently and the CARP IPs became unstable. Some would work, some wouldn't. I agree with your thought about the loop but I'm not sure where or when that would have happened. The only reason I used CARP is when I started out I was on pfsense 1.2.3 and if I can remmeber IPAlias was not available at that point so when I did the upgrade, it was still working and I left it as is. a year later the problems start, something probably gave it a kick. Maybe it was a bad config, but something pushed it. If there was a loop somewhere all along and a VHID conflict occured would that do it?
-
A VHID conflict would not have any impact on the amount of traffic. The number of advertisements would be constant no matter what the case is.
The only side effect of a VHID conflict would be loss of connectivity (or intermittent connectivity) to only the IP involved in the conflict.
Check Interfaces > (assign) on the bridge tab to make sure you don't have any defined, and "ifconfig -a" would show it for sure.
If you have any other servers or equipment that plugs into two separate parts of the network, any of them could cause such a loop.
You can also get some loop-like behavior in certain other cases: http://redmine.pfsense.org/issues/2073
