Kernel: [zone: pf frag entries] PF frag entries limit reached

ltctech

Hi,

I was doing some performance testing with iperf using large UDP datagrams (63KB) on our internet connection yesterday and noticed an issue. Once in a while, the iperf UDP test would fail and IPsec tunnels would drop. At the same time the following gets logged: kernel: [zone: pf frag entries] PF frag entries limit reached

It seems that by default the fragmented packet limit is 5000:


[2.2.1-RELEASE][root@####]/root: pfctl -sm
states        hard limit   402000
src-nodes     hard limit   402000
frags         hard limit     5000
table-entries hard limit  2000000

This limit is way too low in my opinion as our connection can easily outstrip it. We have 50M/50M on a 1G circuit, it is also over provisioned by a few M and has some burst.

Here's the math:
Let's say most or all packets are fragments.
50Mbits / 8 * 1024 * 1024 = 6,553,600 bytes/s
6,553,600 bytes/s / 1500 = 4369
Add a little burst into the rate and 5000 limit is exceeded.

I know I can increase this limit with pfctl but how do I get it to stick across reboots?:
set limit frags 25000

Thanks,
LTCtech

ltctech

I guess I could stick it into /etc/inc/filter.inc. Though wouldn't it make sense to add this to "System: Advanced: Firewall and NAT" tab?

/etc/inc/filter.inc mod for 25000 frags:


	$limitrules = "";

	/* Set fragments limit to 25000 */
		$limitrules .= "set limit frags 25000\n";

	/* User defined maximum table entries in Advanced menu. */
	if ($config['system']['maximumtableentries'] <> "" && is_numeric($config['system']['maximumtableentries']))
		$limitrules .= "set limit table-entries {$config['system']['maximumtableentries']}\n";

	if ($config['system']['optimization'] <> "") {
		$limitrules .= "set optimization {$config['system']['optimization']}\n";
		if($config['system']['optimization'] == "conservative") {
			$limitrules .= "set timeout { udp.first 300, udp.single 150, udp.multiple 900 }\n";
		}
	} else {
		$limitrules .= "set optimization normal\n";
	}

After rules reload:


[2.2.1-RELEASE][root@####]/root: pfctl -sm
states        hard limit   402000
src-nodes     hard limit   402000
frags         hard limit    25000
table-entries hard limit  2000000

jimp

The frag limit is the same now as it has been in previous versions. Ideally you should be located the cause of the high fragmentation and fixing that, rather than upping the limit.

For example if you are sending far too many "too large" packets via IPsec then it will easily spill over, but if you setup MSS clamping for the IPsec VPN to help reduce the packet size of VPN traffic, things will be much better off.

ltctech

The source of the high fragmentation was iperf testing, artificial traffic. Our IPsec connections normally don't generate that much fragmented traffic, the testing was not done through IPsec.

I think this value should be adjustable, as the firewall should be able to handle the amount of fragmentation that is within norms for the connection without cutting connections off.