Limit PPS for UDP Traffic



  • We have had a few situations recently where customers VMs behind our pfSense boxes have been hijacked or misused to generate massive UDP and TCP floods.

    These attacks are sometimes DoS amplification attacks originating from outside but other times they are started from hosts inside our network and generating up to 140k PPS which pretty much takes down our firewalls and results in a service outage for everyone we host behind that firewall.

    We have been able to restrict the TCP floods by limiting states and new connections per second but UDP floods are a lot trickier to contain. We just had a situation where a single VM generated a UDP DNS flood to an external ip with over 40k PPS hitting our LAN port on our firewall and causing connectivity issues. The IP was tracked down and blocked and states killed but it took a few minutes and some customers were annoyed.

    Is there any way in pfSense to limit the amount of PPS a host on our LAN can create or is there a way we can create a rule to limit the amount of UDP connections or more specifically DNS connections per second so no one host can overwhelm our firewalls resulting in an outage?

    Any help would be greatly appreciated.



  • Limiting PPS will reduce bandwidth consumed but it won't reduce load on your firewall. The firewall still needs to process each packet before deciding to drop it.



  • Is there any other way we can control this traffic?



  • Hopefully someone with more experience could answer, but there are two issues going on

    1. Effectively an internal Denial of Service
    2. 140k PPS isn't that much, depending on the kind of traffic. Performance tuning?

    My CPU made it all the way up to 1.3ghz of 3.3ghz total during this test. CPU load displayed is relative the the clock frequency, remember to multiply the cpu load by 0.39. Turns out to be about 5.5%.

    Packets: sent=9844822, rcvd=9844822, error=0, lost=0 (0.0% loss) in 136.621570 sec
    RTTs in ms: min/avg/max/dev: 0.009 / 0.349 / 16.692 / 0.335
    Bandwidth in kbytes/sec: sent=4323.543, rcvd=4323.543

    9844822 divided by 136.621570 seconds is 72,059 pps one way, not including the ping response.


    This is probably apples and orange to some degree. I noticed my total states did not change in a meaningful way during this test, which is probably not the same for you depending on the type of attack the compromised boxes are doing.

    Kind of funny that my Windows Desktop was around 60%-70% CPU usage to run that PPS test, yet PFSense was around 5%. 9us min ping. Ahh yeah.



  • ICMP can be limited by sysctl entries, maybe there is a UDP entry as well?



  • According to CERT limiting clients DNS to 5 resolutions per second is how to handle this.
    https://www.us-cert.gov/ncas/alerts/TA13-088A

    http://www.symantec.com/connect/blogs/using-rrl-prevent-dns-amplification-attacks

    Also assuming you are using pfSense 2.2 you can restrict the size of DNS packets allowed in unbound.

    https://unbound.net/pipermail/unbound-users/2013-April/002887.html



  • @Harvy66:

    1. 140k PPS isn't that much, depending on the kind of traffic. Performance tuning?

    No, in general 140k PPS is not that much but many of our racks only have 100Mb uplinks in our data centre so 80-100k PPS is all thats needed to saturate this link making the servers behind it unavailable.



  • Well then, it's a bandwidth issue, not a PPS issue. Just do normal traffic shaping?



  • We have put traffic shaping in place and used a rule to force all UDP traffic into a lower priority queue with a hard limit of 50mb upload so this helps to prevet the uplink being saturated but i'm not sure if its the best way to do it.

    Ideally we would like something similar to the TCP option where we have limited TCP traffic on Lan to max 200 new connections per second which works very well to stop TCP floods dead in their tracks but there doesn't seem to be a similar feature for UDP traffic.
    Probably because UDP doesnt use states in the same way TCP does so it's harder to control.





  • So it looks like the feature now exists in freeBSD, I wonder can it be used in pfSense or if it will be added as an option to the GUI!



  • @craggy:

    So it looks like the feature now exists in freeBSD, I wonder can it be used in pfSense or if it will be added as an option to the GUI!

    Is it implemented?



  • How about putting your upload traffic under a dummynet limiter and setting two queues, one for DNS weighted for whatever you need (maybe 10?) And the other for everything else weighted at the remainder of 100 (in this case 90).

    It would allow all 140kpps out while the network is idle, but as soon as other traffic came on it will be prioritized over the DNS - never allowing it to override the rest of your traffic.



  • @belt9:

    How about putting your upload traffic under a dummynet limiter and setting two queues, one for DNS weighted for whatever you need (maybe 10?) And the other for everything else weighted at the remainder of 100 (in this case 90).

    Clients use my own DNS server running on pfSense.  I want to rate-limit other UDP transmits.

    What about pfSense developers would demand this feature from pf developers or (maybe better) make pf's fork and then implement UDP rate-limiting?
    PF misses MAC filtering and UDP rate-limiting. This is a must these days.



  • @mdes:

    Clients use my own DNS server running on pfSense.  I want to rate-limit other UDP transmits.

    I'm not seeing why you couldn't do that?

    If you can write a firewall rule for the traffic then you can apply a dummynet limiter to the traffic.



  • @belt9:

    If you can write a firewall rule for the traffic then you can apply a dummynet limiter to the traffic.

    Limiters work with bits/s and I need packets/s. There may be 1000 UDP packets per second producing only 100 kB/s.



  • @mdes:

    @belt9:

    How about putting your upload traffic under a dummynet limiter and setting two queues, one for DNS weighted for whatever you need (maybe 10?) And the other for everything else weighted at the remainder of 100 (in this case 90).

    Clients use my own DNS server running on pfSense.  I want to rate-limit other UDP transmits.

    What about pfSense developers would demand this feature from pf developers or (maybe better) make pf's fork and then implement UDP rate-limiting?
    PF misses MAC filtering and UDP rate-limiting. This is a must these days.

    UDP rate limit and MAC filtering are a must for switches, not edge firewalls. UDP rate limiting is primarily used for broadcast storm preventing and MAC filtering is used to prevent unknown devices from even gaining access to the physical network. Not that it's very effective, but it is another layer.

    Technically, MAC filtering on a firewall is implemented as a layering "violation". Not to mention that MAC addresses should NEVER be trusted in the first place. Filtering on MAC addresses is just extra work for weak obscurification with virtually no additional security.

    In general, many common "must haves" are just simplistic bandaids for a problem the end user does not fully understand.

    That being said, UDP rate limiting does have some useful characteristics, but mostly in trying to help protect others from themselves. DNS amp attacks, which should not happen if DNS is configured correctly.

    My home pfSense router is able to fairly easily NAT+route+shape 1.4Mpps of UDP. If your firewall is dying from it, you may have something wrong.



  • @Harvy66:

    UDP rate limit and MAC filtering are a must for switches, not edge firewalls.

    Linux iptables has it all.

    @Harvy66:

    DNS amp attacks, which should not happen if DNS is configured correctly.

    Yes, but DNS is not only protocol which uses UDP.

    @Harvy66:

    My home pfSense router is able to fairly easily NAT+route+shape 1.4Mpps of UDP.

    I have different use case, huge difference is that my router does not fit in "home" scale.



  • Exactly what UDP traffic problems are you having?



  • harvy66 I know not on topic, but how are you collecting those stats in your graphs? the supplied bsnmp in its stock config doesnt supply the data, me and some others asked in another thread but noone with the knowledge wanted to help.

    Although your answer could still help the OP as well as monitoring may help him manage his situation better.



  • @belt9:

    Exactly what UDP traffic problems are you having?

    Since PF and consequently pfSense lack limit pps for UDP feature, I had to do it on an upstream router.
    Well I think, unless proper MAC filtering, NetFlow MAC exporting and UDP PPS limiting is implemented, I cannot use pfSense for my customers (SMB and small enterprises).
    It has many great features and I really appreciate all work that developers have done.