Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    UDP Advanced Rules - Under UDP DDoS Attack

    Firewalling
    6
    29
    14.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      FJSchrankJr
      last edited by

      Hi guys.

      We've been under a massive UDP DDoS for the past 24 hours. They're basically targeting every DNS server behind the firewall.

      I have limits for maximum connections per host on but they don't appear to be working, unless maybe I need to specify max table states per host too. Basically what's happening every 15 minutes or so, the states will shoot up until packet loss occurs, LAN errors show up on the interface, then the packet loss appears to reset there attack. It's been repeating like this for the past day.

      Does anyone know if the advanced rules in 2.0-RC1 applies to UDP traffic? Thank you for the insight.

      FJS - Embedded Systems Engineer
      Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
      ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

      1 Reply Last reply Reply Quote 0
      • D
        dhatz
        last edited by

        It can be very hard, if you're passing the traffic used for DDoS.

        Check my post at http://forum.pfsense.org/index.php/topic,46897.0.html

        1 Reply Last reply Reply Quote 0
        • F
          FJSchrankJr
          last edited by

          @dhatz:

          It can be very hard, if you're passing the traffic used for DDoS.

          Check my post at http://forum.pfsense.org/index.php/topic,46897.0.html

          Very interesting, I have to read through that in detail.

          There is an advanced rule in pfsense called "new connections per second" is this per host or for the entire rule? Depending, might be able to use it…

          FJS - Embedded Systems Engineer
          Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
          ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

          1 Reply Last reply Reply Quote 0
          • F
            FJSchrankJr
            last edited by

            In this case, the traffic itself is not high, it's the amount of packets. It's a very interesting attack, every 15 minutes a massive storm of packets comes in, the firewall states keep going up until packet loss occurs, then it resets the attack. In most DDoS attacks I've seen, the attackers don't stop sending traffic.

            Question is, under Advanced Rules, how much of those options work for UDP? It appears as though the option I have set are not working. In particular, max connections per host and connections per second.

            One thing I don't know a lot about is UDP traffic. Any thoughts would be greatly appreciated. Once I can figure something out I will open up a topic on it in detail. Thanks guys.

            FJS - Embedded Systems Engineer
            Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
            ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

            1 Reply Last reply Reply Quote 0
            • G
              Goliathxo
              last edited by

              With my limited knowledge and experience with pfSense here's my two cents:

              • If you change a rule it only applies on new connections. Maybe a reset of the state table helps.

              • I think an combination of the last 4 options under advanced should help, but only by trial and error.

              If the attack comes from a specific country you could block ip's from that region if you don't have legitimate traffic from those countries with the pfBlocker package.

              Good luck.

              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                Even if you block the packets, they must still be processed by the firewall (if only a little) to figure that out. So pps is still an issue even if you don't allow anything at all.

                It's difficult for any firewall to fend off a DDoS of that sort. You can throw more hardware at it, but really a DDoS must be cut off closer to the source (read: your ISP, or theirs)

                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • F
                  FJSchrankJr
                  last edited by

                  @jimp:

                  Even if you block the packets, they must still be processed by the firewall (if only a little) to figure that out. So pps is still an issue even if you don't allow anything at all.

                  It's difficult for any firewall to fend off a DDoS of that sort. You can throw more hardware at it, but really a DDoS must be cut off closer to the source (read: your ISP, or theirs)

                  Thanks Jimp. I don't think the attack is as big as I thought. I think maybe someone could be spoofing the udp packets and just sending a storm of them. the thing is, the attacks are coming in on 2 of our networks, the transparent bridge (pfsense) can take it and has been ok but the pfsense firewall running nat w/ carp+virtual IPs is the one going down every so often, it's crashing long before the states run out.

                  can you think of any thing in pfsense under udp floods that would cause the state table to reset like this? the firewall doesn't crash, just the state table is resetting… the wan logs indicate that nothing is going down, just the lan side is what's crashing. the wan looks like it takes it.

                  FJS - Embedded Systems Engineer
                  Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                  ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                  1 Reply Last reply Reply Quote 0
                  • F
                    FJSchrankJr
                    last edited by

                    intersting… I see what you're saying. even if I block it, it still requires resources. Adjusting the settings within pfsense for the UDP/53 rule did help, but it still resets the tables after enough pour in. yet, the cpu usage doesn't even spike that much, very little.

                    I was thinking about using the new connections per second, but it doesn't appear to work on UDP, does it?

                    FJS - Embedded Systems Engineer
                    Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                    ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                    1 Reply Last reply Reply Quote 0
                    • D
                      dhatz
                      last edited by

                      Since it's easy to spoof UDP traffic, a firewall rule that would add "offending" IPs to your blacklist may become a double-edged sword …

                      Had it been only TCP traffic, you could use "max new connections per unit of time" rules.

                      1 Reply Last reply Reply Quote 0
                      • F
                        FJSchrankJr
                        last edited by

                        Thanks jimp, dhatz and everyone.

                        Something doesn't appear right. When the packet loss occurs, it resets all of the states but the interupt cpu usage only hits no more then 8%.

                        I should add, this is a watchguard firewall with those realtek nics on there. The lan errors only occur when the DDoS hits, and it's only on the lan side, the wan never goes down.

                        There attack appears to be timed every 10 minutes, then sends a storm of packets on UDP/53.

                        FJS - Embedded Systems Engineer
                        Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                        ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                        1 Reply Last reply Reply Quote 0
                        • johnpozJ
                          johnpoz LAYER 8 Global Moderator
                          last edited by

                          curious how many packets are we talking?  And what is in the traffic - you sure its just not just a storm of dns requests, maybe because of a really low ttl (say 10 minutes).  On a zone that is seeing heavy use?

                          As stated it hard to stop a DOS at the endpoint – you need to stop the traffic upstream from your device.

                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                          If you get confused: Listen to the Music Play
                          Please don't Chat/PM me for help, unless mod related
                          SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                          1 Reply Last reply Reply Quote 0
                          • F
                            FJSchrankJr
                            last edited by

                            @johnpoz:

                            curious how many packets are we talking?  And what is in the traffic - you sure its just not just a storm of dns requests, maybe because of a really low ttl (say 10 minutes).  On a zone that is seeing heavy use?

                            As stated it hard to stop a DOS at the endpoint – you need to stop the traffic upstream from your device.

                            Hi Johnpoz, thanks for the reply. There not legit DNS requests because I was watching the zone queries, it's more like they just send a udp packet. some type of UDP flood, though I don't know much about UDP. It was much higher yesterday the traffic but I blocked so many networks in China. Now its lower,  I will average 1000 states to the DNS, but out of no where the states shoot to about 8000, the interrupts only reach 8.0% on the firewall then I drop packets on the lan side, but the wan connection doesn't get interrupted. It's very strange. It's been running for a year with no major issues, until this attack. They're also targeting 7 different DNS servers on the network at the same time. They hit from multiple sources, but it's all in sync.

                            They're also hitting another network here (different WAN), but that firewall is transparent yet it can handle it. Any reason why a NAT firewall w/carp and VIP's would drop all of the states long before the interrupts reach 100% or max state size?

                            As always, thank you all for your expertise!

                            FJS - Embedded Systems Engineer
                            Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                            ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                            1 Reply Last reply Reply Quote 0
                            • johnpozJ
                              johnpoz LAYER 8 Global Moderator
                              last edited by

                              Do these 7 servers host the same zone(s)?

                              Can you capture some of these packets they are sending and post them..  Just curious to what they are sending..

                              Normally in a attack against dns you would send valid queries, to get the server working trying to work extra hard in working on responding to them - just sending gibberish would be less effective.  Unless is was some sort of exploit type packet.

                              Grab capture on the firewall for a few packets and post it up - maybe we can glean something from it that will help.  More information is never a bad thing ;)

                              Do these servers respond to recursive queries, or only authoritative for your zones?

                              An intelligent man is sometimes forced to be drunk to spend time with his fools
                              If you get confused: Listen to the Music Play
                              Please don't Chat/PM me for help, unless mod related
                              SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                              1 Reply Last reply Reply Quote 0
                              • F
                                FJSchrankJr
                                last edited by

                                Hi Johnpoz:  They were recursive but I disabled that, didn't help much. The only thing keep me up is the fact that pfsense resets tables when its crashing the lan interface.

                                I am going to try to capture that traffic now for you thanks

                                FJS - Embedded Systems Engineer
                                Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                                ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                                1 Reply Last reply Reply Quote 0
                                • F
                                  FJSchrankJr
                                  last edited by

                                  got it… seeing a lot of these repeated:

                                  I replaced our actual host name with server (server.com) and replaced the first of our ip with 0

                                  202.216.0.21.45553 > 0.41.228.2.53: [bad udp cksum 1!] 92% [1au] AAAA? ns3.server.com. ar: . OPT UDPsize=512 OK (44)
                                  22:34:57.809364 IP (tos 0x0, ttl 251, id 41584, offset 0, flags [DF], proto UDP (17), length 72)
                                     202.216.0.21.44340 > 0.41.228.2.53: [bad udp cksum 1!] 9243% [1au] AAAA? nr1.server.com. ar: . OPT UDPsize=512 OK (44)
                                  22:34:57.841478 IP (tos 0x0, ttl 253, id 12585, offset 0, flags [DF], proto TCP (6), length 48)

                                  FJS - Embedded Systems Engineer
                                  Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                                  ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                                  1 Reply Last reply Reply Quote 0
                                  • F
                                    FJSchrankJr
                                    last edited by

                                    I don't know if this is the cause.

                                    Normally, the DNS traffic flows from outside to inside.

                                    Right before the states in pfsense crash, there are a lot of requests going out of the DNS. But I know the name servers are not rooted. Is it possible they're using our dns servers to launch an attack?

                                    FJS - Embedded Systems Engineer
                                    Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                                    ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                                    1 Reply Last reply Reply Quote 0
                                    • F
                                      FJSchrankJr
                                      last edited by

                                      I wonder how I might expire UDP faster then the "aggressive" mode which is 30-60s.

                                      FJS - Embedded Systems Engineer
                                      Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                                      ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                                      1 Reply Last reply Reply Quote 0
                                      • johnpozJ
                                        johnpoz LAYER 8 Global Moderator
                                        last edited by

                                        so they are actual queries… Not gibberish.

                                        And you would expect if going to be over the udp limit then yes you switch to TCP.  And those are just AAAA requests for what look like NS..

                                        So how again are you sure this an attack and not just not some dns storm?

                                        A longer snag would be helpful..  I am not convinced its an actual directed attack..  When BP had their spill, prob could of looked like an attack on their dns if there TTLs were to small.

                                        What is the TTL of those records?  Do they keep doing the same query over and over - and you are responding?

                                        Again -- I know these suggests that can help you with your box not crashing.. But understanding the traffic might lead you to other ways to remove the problem.

                                        How many zones are these servers serving up?  You sure one of the zones just don't have an issue with their TTL on the NS record, etc.  Again - looking at the actual traffic might point us in the right direction.

                                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                                        If you get confused: Listen to the Music Play
                                        Please don't Chat/PM me for help, unless mod related
                                        SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                        1 Reply Last reply Reply Quote 0
                                        • F
                                          FJSchrankJr
                                          last edited by

                                          I appreciate the help you've given me, and others. When things get stressful and people call me I tend to not put as much thought in to things, I apologize for the little info I gave you.

                                          To be honest, I don't know that it's not a DNS storm, at this point I'm not sure of anything :-) . The only thing that appeared off, I did a test and shutdown one of the name servers but new states were still appearing to it. When I watched the dns logs there was the regular amount of queries, it's almost like empty UDP traffic. Maybe it's some other issue causing pfsense to just reset states. Is there a log in pfsense somewhere that would indicate any of this?

                                          DNS Zones on these servers, only about 500 zones.

                                          FJS - Embedded Systems Engineer
                                          Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                                          ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            FJSchrankJr
                                            last edited by

                                            To make things even more confusing, I just noticed the packet loss on the inside is only from the LAN switch to pfsense, yet not 1 error on the switch port just on the lan interface for pfsense. I've tried different cables, even tried a different interface. From the outside in there is no packet loss at all. I am starting to think it's something in pfsense. I don't know anymore, going insane.

                                            I don't want to waist your time, it could be something else I will keep digging… Thanks.

                                            FJS - Embedded Systems Engineer
                                            Pictures are worth a thousand words, but <u>posting config.xml backups are worth 10,000</u>.  Alter the IPs, change anything revealing but leave subnets intact. Use find and replace. Please try to keep it brief on the description.
                                            ALWAYS disable TSO  & LRO EXCEPT CHKSUM IF SUPPORTED. TSO/LRO breaks traffic, pf scrub and this goes for any passive device inline

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.