System Tunables on Netgate Hardware - Not Active?

[RobbieTT]

Ahead of an effort to tune my PPPoE WAN performance I noticed:

root: sysctl -a | grep 'harvest'
kern.random.harvest.mask_symbolic: PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,[INTERRUPT],NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
kern.random.harvest.mask_bin: 00000000010000000101011111
kern.random.harvest.mask: 65887

Yet on the GUI 'System Tunables' it is set to a more performance orientated 351, which seems to be the default setting on my 6100:

I've also noticed that flow control appears to be set when I would normally expect it to be disabled on a router (I use igc3 for my PPPoE WAN and ix1 for LAN & VLAN):

hw.ix.flow_control: 3
dev.ix.1.fc: 3
dev.igc.3.fc: 3
dev.igc.2.fc: 3
dev.igc.1.fc: 3
dev.igc.0.fc: 3

I was also puzzled at to why BSD process limits are set on a router:

hw.igc.rx_process_limit: 100

Are elements of 'System Tunables' and the BSD config removed, masked or handled elsewhere on Netgate's own hardware?

️

[stephenw10]

Some of those are read-only in the sysctls and have to be set as loader values in /boot/loader.conf.local.

hw.ix.flow_control definitely does. hw.igc.rx_process_limit probably does. In general hw.x values are usually loader only. dev.x can be set at runtime so can be added to System Tunables.

I'm unsure where kern.random.harvest.mask would need to be set. Or if it can be.

[jimp]

The tunables are respected so long as the OS respects the changes and allows them.

Some things can't be changed at runtime, however, and must be added as loader tunables.

That one doesn't appear to allow changes at all. It acts like it will take it but doesn't.

$ sysctl kern.random.harvest.mask=351
kern.random.harvest.mask: 65887 -> 65887
$ sysctl kern.random.harvest.mask
kern.random.harvest.mask: 65887

It also doesn't work as a loader tunable.

The man page suggests that should be configurable but it may require matching some bits of the original mask exactly and not just taking a number from another system.

If it can't be changed at all, that's something that would need to be addressed upstream in FreeBSD.

[RobbieTT]

@stephenw10 said in System Tunables on Netgate Hardware - Not Active?:

Some of those are read-only in the sysctls and have to be set as loader values in /boot/loader.conf.local.

I don't have that file, only a /boot/loader.conf file, or is the implication there to create a new one rather than vi the existing loader.conf?

️

[jimp]

@RobbieTT said in System Tunables on Netgate Hardware - Not Active?:

@stephenw10 said in System Tunables on Netgate Hardware - Not Active?:

Some of those are read-only in the sysctls and have to be set as loader values in /boot/loader.conf.local.

I don't have that file, only a /boot/loader.conf file, or is the implication there to create a new one rather than vi the existing loader.conf?

️

https://docs.netgate.com/pfsense/en/latest/config/advanced-tunables.html#managing-loader-tunables

[RobbieTT]

@jimp said in System Tunables on Netgate Hardware - Not Active?:

The man page suggests that should be configurable but it may require matching some bits of the original mask exactly and not just taking a number from another system.

If it can't be changed at all, that's something that would need to be addressed upstream in FreeBSD.

It might be seen upstream as a contentious change as the 351 setting does trade some security for performance.

It stood-out to me because the default pfSense System Tunables includes that option and shows the 351 as the default setting. It probably requires a low-priority tweak to pfSense to remove the option from the menu, if it is non-functional.

️

[RobbieTT]

@jimp said in System Tunables on Netgate Hardware - Not Active?:

https://docs.netgate.com/pfsense/en/latest/config/advanced-tunables.html#managing-loader-tunables

Thanks - I've been working hard to avoid an RTFM moment but off I go and run straight into one!

️

[RobbieTT]

@stephenw10
I've looked as best as I can but I still don't understand why a process limit is set on a firewall/router:

hw.igc.rx_process_limit: 100

Clearly this would be set as-is for a general compute use of FreeBSD but it looks like an artificial limitation for a router application and is probably a bottleneck for pushing packets.

️

[stephenw10]

As far as I know we don't set that so it's the driver default value.

Try increasing it as see. We've not had to do that to get the expected throughput, but.....

I do know we have looked at increasing hw.igc.max_interrupt_rate. Setting that to 20000 provided gains in some scenarios.

Steve

[RobbieTT]

@stephenw10

Thanks Steve,

It makes sense as a default for FreeBSD but I will try disabling it on pfSense to see if it helps performance for PPPoE.

For non-PPPoE users it has been tested 'disabled' previously and shown to improve performance on Intel NICs. The first white paper I read on the subject happened to included a pfSense router as part of the bench-testing. I think Orange produced the original paper for a FreeBSD conference. I'll see if I can find a link.

I'll have a look at increasing the hw.igc.max_interrupt_rate as I am unfamiliar with it. I already have net.isr.dispatch set to deferred but any improvement I could detect was probably within the margin of error. It certainly didn't make anything worse though.

As a fellow PPPoE sufferer is there anything else I should consider beyond the pfSense guide?

Rob

️

[stephenw10]

I'm stuck at g.fast(!) speeds and I'm at the long end of the wire for that so ~140Mbps. Hence PPPoE isnt really a problem for me. Unfortunately.

Setting net.isr.dispatch to deferred usually makes a significant difference if you are hitting that as a limit. So you may not be.

[RobbieTT]

@stephenw10

The testing is somewhat synthetic for me on FTTH (930/115) as I am not running anything demanding other than FQ_Codel and pfBlocker, so I am currently within the 6100 CPU limits. But I will be switching to the 1.8 Gbit package when launched by Openreach later this year and this is probably beyond the 6100 with default settings.

Ideally I would run ntopng and Suricata too but that looks out of reach for high-bandwidth PPPoE users. In fact, high-bandwidth PPPoE is becoming a significant bottleneck for those hanging off an Openreach network.

I don't miss G.fast at all, so hope you get upgraded to FTTH in the near future!

Rob

️

[stephenw10]

@RobbieTT said in System Tunables on Netgate Hardware - Not Active?:

hope you get upgraded to FTTH in the near future!

Me too but it doesn't look like it will happen any time soon.