Limit PPS for UDP Traffic

Harvy66

Limiting PPS will reduce bandwidth consumed but it won't reduce load on your firewall. The firewall still needs to process each packet before deciding to drop it.

cr_hyland

Is there any other way we can control this traffic?

Harvy66

Hopefully someone with more experience could answer, but there are two issues going on

Effectively an internal Denial of Service
140k PPS isn't that much, depending on the kind of traffic. Performance tuning?

My CPU made it all the way up to 1.3ghz of 3.3ghz total during this test. CPU load displayed is relative the the clock frequency, remember to multiply the cpu load by 0.39. Turns out to be about 5.5%.

Packets: sent=9844822, rcvd=9844822, error=0, lost=0 (0.0% loss) in 136.621570 sec
RTTs in ms: min/avg/max/dev: 0.009 / 0.349 / 16.692 / 0.335
Bandwidth in kbytes/sec: sent=4323.543, rcvd=4323.543

9844822 divided by 136.621570 seconds is 72,059 pps one way, not including the ping response.

This is probably apples and orange to some degree. I noticed my total states did not change in a meaningful way during this test, which is probably not the same for you depending on the type of attack the compromised boxes are doing.

Kind of funny that my Windows Desktop was around 60%-70% CPU usage to run that PPS test, yet PFSense was around 5%. 9us min ping. Ahh yeah.

Nullity

ICMP can be limited by sysctl entries, maybe there is a UDP entry as well?

mcwtim

According to CERT limiting clients DNS to 5 resolutions per second is how to handle this.
https://www.us-cert.gov/ncas/alerts/TA13-088A

http://www.symantec.com/connect/blogs/using-rrl-prevent-dns-amplification-attacks

Also assuming you are using pfSense 2.2 you can restrict the size of DNS packets allowed in unbound.

https://unbound.net/pipermail/unbound-users/2013-April/002887.html

cr_hyland

@Harvy66:

140k PPS isn't that much, depending on the kind of traffic. Performance tuning?

No, in general 140k PPS is not that much but many of our racks only have 100Mb uplinks in our data centre so 80-100k PPS is all thats needed to saturate this link making the servers behind it unavailable.

Harvy66

Well then, it's a bandwidth issue, not a PPS issue. Just do normal traffic shaping?

cr_hyland

We have put traffic shaping in place and used a rule to force all UDP traffic into a lower priority queue with a hard limit of 50mb upload so this helps to prevet the uplink being saturated but i'm not sure if its the best way to do it.

Ideally we would like something similar to the TCP option where we have limited TCP traffic on Lan to max 200 new connections per second which works very well to stop TCP floods dead in their tracks but there doesn't seem to be a similar feature for UDP traffic.
Probably because UDP doesnt use states in the same way TCP does so it's harder to control.

Nullity

https://forums.freebsd.org/threads/udp-blocking-with-rate-limits.42933/
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=189721

cr_hyland

So it looks like the feature now exists in freeBSD, I wonder can it be used in pfSense or if it will be added as an option to the GUI!

mdes

@craggy:

So it looks like the feature now exists in freeBSD, I wonder can it be used in pfSense or if it will be added as an option to the GUI!

Is it implemented?

belt9

How about putting your upload traffic under a dummynet limiter and setting two queues, one for DNS weighted for whatever you need (maybe 10?) And the other for everything else weighted at the remainder of 100 (in this case 90).

It would allow all 140kpps out while the network is idle, but as soon as other traffic came on it will be prioritized over the DNS - never allowing it to override the rest of your traffic.

mdes

@belt9:

How about putting your upload traffic under a dummynet limiter and setting two queues, one for DNS weighted for whatever you need (maybe 10?) And the other for everything else weighted at the remainder of 100 (in this case 90).

Clients use my own DNS server running on pfSense. I want to rate-limit other UDP transmits.

What about pfSense developers would demand this feature from pf developers or (maybe better) make pf's fork and then implement UDP rate-limiting?
PF misses MAC filtering and UDP rate-limiting. This is a must these days.

belt9

@mdes:

Clients use my own DNS server running on pfSense. I want to rate-limit other UDP transmits.

I'm not seeing why you couldn't do that?

If you can write a firewall rule for the traffic then you can apply a dummynet limiter to the traffic.

mdes

@belt9:

If you can write a firewall rule for the traffic then you can apply a dummynet limiter to the traffic.

Limiters work with bits/s and I need packets/s. There may be 1000 UDP packets per second producing only 100 kB/s.

Harvy66

@mdes:

@belt9:

How about putting your upload traffic under a dummynet limiter and setting two queues, one for DNS weighted for whatever you need (maybe 10?) And the other for everything else weighted at the remainder of 100 (in this case 90).

Clients use my own DNS server running on pfSense. I want to rate-limit other UDP transmits.

What about pfSense developers would demand this feature from pf developers or (maybe better) make pf's fork and then implement UDP rate-limiting?
PF misses MAC filtering and UDP rate-limiting. This is a must these days.

UDP rate limit and MAC filtering are a must for switches, not edge firewalls. UDP rate limiting is primarily used for broadcast storm preventing and MAC filtering is used to prevent unknown devices from even gaining access to the physical network. Not that it's very effective, but it is another layer.

Technically, MAC filtering on a firewall is implemented as a layering "violation". Not to mention that MAC addresses should NEVER be trusted in the first place. Filtering on MAC addresses is just extra work for weak obscurification with virtually no additional security.

In general, many common "must haves" are just simplistic bandaids for a problem the end user does not fully understand.

That being said, UDP rate limiting does have some useful characteristics, but mostly in trying to help protect others from themselves. DNS amp attacks, which should not happen if DNS is configured correctly.

My home pfSense router is able to fairly easily NAT+route+shape 1.4Mpps of UDP. If your firewall is dying from it, you may have something wrong.

mdes

@Harvy66:

UDP rate limit and MAC filtering are a must for switches, not edge firewalls.

Linux iptables has it all.

@Harvy66:

DNS amp attacks, which should not happen if DNS is configured correctly.

Yes, but DNS is not only protocol which uses UDP.

@Harvy66:

My home pfSense router is able to fairly easily NAT+route+shape 1.4Mpps of UDP.

I have different use case, huge difference is that my router does not fit in "home" scale.

belt9

Exactly what UDP traffic problems are you having?

chrcoluk

harvy66 I know not on topic, but how are you collecting those stats in your graphs? the supplied bsnmp in its stock config doesnt supply the data, me and some others asked in another thread but noone with the knowledge wanted to help.

Although your answer could still help the OP as well as monitoring may help him manage his situation better.

mdes

@belt9:

Exactly what UDP traffic problems are you having?

Since PF and consequently pfSense lack limit pps for UDP feature, I had to do it on an upstream router.
Well I think, unless proper MAC filtering, NetFlow MAC exporting and UDP PPS limiting is implemented, I cannot use pfSense for my customers (SMB and small enterprises).
It has many great features and I really appreciate all work that developers have done.