Strange: per-host UDP block rule works only for ~1hr then let traffic through
-
I'm a novice at pfsense and would welcome outside eyeballs on a strange thing that's been biting me for the last 24 hours ..
I've got a box on my network that must be an authoritative DNS server for a domain used for some service auto-discovery used in the life sciences - lots of legit external requests for the SVR records associated with this particular domain so we need to serve DNS to the outside world for this domain.
Sadly at some point in time this DNS server was configured to allow recursive queries and (of course!) our IP ended up on a list that someone is using to conduct DNS Amplification Attacks against others. We've since turned off recursive queries but the requests still keep flooding in so we used the new BIND 9.10 to rate-limit and throttle all responses. Works fine but the inbound traffic is still annoying and is cluttering up our snort/securityOnion alerts …
I've got 2 IPs still flooding me with UDP requests; here is the BIND server complaining endlessly about the responses it is dropping:
Sep 27 07:17:10 dev named[26403]: client 23.246.236.18#21324 (paypal.de): rate limit drop all response to 23.246.236.0/24
Sep 27 07:17:10 dev named[26403]: client 85.204.22.40#21324 (paypal.de): rate limit drop all response to 85.204.22.0/24
Sep 27 07:17:10 dev named[26403]: client 85.204.22.40#21324 (paypal.de): rate limit drop all response to 85.204.22.0/24
Sep 27 07:17:10 dev named[26403]: client 208.43.26.191#21324 (paypal.de): rate limit drop all response to 208.43.26.0/24The deluge of DNS Amplification Attack streams has stopped but those 2 IPs are sending a continuous flood of packets
This is where the strangeness begins …
I configured very simple pfsense rules: "block all UDP traffic from (source IP) to destination/port ANY"
And because the streams were active I reset the firewall states after the rule went in. Appeared to work perfectly. The UDP flood to the DNS server stopped instantly.
Then ... between 1-2 hours later the DNS traffic started getting through to the DNS server again. Endless logs from BIND concerning rate throttling of the same IP addresses that pfsense had been configured to reject all UDP traffic from.
This is a repeatable cycle -- the IPs get blocked for a few hours and then their UDP traffic seems to get past pfsense. After resetting the states the IPs get blocked for a few hours before the cycle repeats.
I'm sort of at a loss to explain what is going on. The only thing I can come up with is that DNS Amplification Traffic is usually spoofed so that the big DNS packets go back to the victim. Maybe upstream the attackers are switching hosts that are doing the spoofing or something else about the presumably spoofed-source UDP packets is changing enough that pfsense gets confused and lets the stream past the block rules again.
Any hints / tips / URLs / documentation advice appreciated. I've got gold subscription and access to the pfsense PDF but that is about it.
Regards,
Chris -
Is it the exact same two IPs, or do they change every hour?
-
Is it the exact same two IPs, or do they change every hour?
According to pfsense and what my DNS server is seeing in the queries it is the exact same 2 IP addresses all the time although they will sometimes change up the zone record they are requesting. However I'm not sure if I can trust the actual IPs presented as those are usually spoofed as part of the DNS Amplification Attack? My guesstimate was that maybe there is a botnet involved and every hour or so the actual system that is sending the UDP packets my way with the spoofed victim address is changing and something is "different" enough about the flow that they seem to sneak on by the pfsense block rule. Still not sure what is happening. Resetting the firewall states will work but generally for not much longer than an hour or so.
Now that I've got a modern BIND server that refuses recursive queries and has super aggressive rate-limit-reply throttles it's actually more of a curiosity than a problem; it was just strange to see block rules fail to "stick".
-
"block all UDP traffic from (source IP) to destination/port ANY"
And where is the rule that allows traffic to your DNS, does that have a NOT on the source for that IP?
My guess would be that they are doing a query form that IP to your other dns server. And now that state is set, etc.
Can you post up your wan rules?
