Performance regression 2.7.2 to 2.8
-
@stephenw10
With or without patch mr1226.diff
No traffic on any VIPs and cpu is high.
kea-dhcp6 php-fpm.
kea-dhcp6 is using almost about 3% when 10.0.0.1/32 IP Alias is set on the wan, set it to lan kea-dhcp6 uses 0.00% -
Block private networks and loopback addresses
Is enabled on wan, turning that off is all the same high cpu. -
Oh this is on the PPPoE WAN?
That's a known issue: https://redmine.pfsense.org/issues/16235
Try the patch refferenced there.
-
@stephenw10 Yes the PPPoE WAN.
Is this the patch?
This patch does fix high cpu, however when a VIP is set on wan, it breaks the whole nat 64:ff9b::/96 address space, or is a reload/restart needed? -
Yup without that when you add a VIP on a PPPoE WAN and have if_pppoe enabled then the connection loops continually. The logs will have shown it reconnecting every few seconds which obviously load the CPU significantly.
So how exactly does NAT64 fail?
What are you using that VIP for?
-
So how exactly does NAT64 fail?
Native v6 traffic is normal.
Outside NAT64 so far is not working with a VIP set on wan.
The VIPs them selves are reachable example 64:ff9b::10.0.0.3WAN ipv6-icmp 10.0.0.4:1 (fdbb::8[1]) -> 77.47.127.138:8 (64:ff9b::100:1[1]) NO_TRAFFIC:NO_TRAFFIC 2 / 2 160 B / 120 BWhat are you using that VIP for?
v4 VIPs for ping, v6 VIPs for DNS.
-
@fathead said in Performance regression 2.7.2 to 2.8:
Outside NAT64 so far is not working with a VIP set on wan.
OK so you are setting that VIP just as something to ping from a V6 only client device?
Can I assume it still responds to ping from an internal IPv4 client?
-
OK so you are setting that VIP just as something to ping from a V6 only client device?
Can I assume it still responds to ping from an internal IPv4 client?
Yes and yes.
-
Furthermore 64:ff9b::c0a8:100/120 is passed on the lan, at some point in the past two days this stopped working.
-
Oh so dhcpv6 stopped working you mean by passed?
-
@stephenw10
Firewall rule set to pass LAN subnets to 64:ff9b::c0a8:100/120
64:ff9b::c0a8:101(pfSense it self) was the only address replying.tcpdump on pfSense show the wrong ip(the VIP) and not the interface IP(192.168.1.1).
Removing all v4 VIPs from lan restores reachabilit. v4 VIPs on localhost are different some how.
Before:16:52:35.520044 IP 10.0.0.1 > 192.168.1.100: ICMP echo request, id 179, seq 1, length 64After v4 VIPs removed:
17:08:30.209222 IP 192.168.1.1 > 192.168.1.100: ICMP echo request, id 182, seq 10740, length 64 17:08:30.209351 IP 192.168.1.100 > 192.168.1.1: ICMP echo reply, id 182, seq 10740, length 64The 64:ff9b::c0a8:100/120 rule is above the 64:ff9b::/96 rule.
NAT64 and v4 VIPs alias on wan and/or lan are not playing nicely. -
The issue is that the "Automatic" NAT64 source option in the rule leaves things up to the OS itself. Hence the source tends to be the first address on the interface (e.g. 10.0.0.1). To address that issue a GUI option exists to override the source. For example:
-
@marcosm The source has always been set to "LAN subnets".
-
It's the 'Address Family Translation' source that must be set to 'LAN address' rather than the default 'Automatic' where it's choosing the VIP.
-
@stephenw10
'Address Family Translation' is also set to 'LAN address'. -
Hmm, then maybe it's not matching the test traffic somehow?
-
Hmm, then maybe it's not matching the test traffic somehow?
How to see what it is matching with?
For testing, made a do nothing OPT1, setting v4 VIPs on loopback and OPT1 does not seem break lan to lan or lan to wan.
-
Mmm, I'd only expect the VIP to make any difference on LAN. For some reason it's choosing to use the first IP on the interface (the VIP) as the source for the translation.
If you run
pfctl -vssyou can see the states with the rules that created them.Then if you run
pfctl -vvsryou can see the ruleset with numbers to see what the rule is.However since this is a NAT64 rule I'm not 100% sure how it will appear!
-
Mmm, I'd only expect the VIP to make any difference on LAN.
Exactly, 64:ff9b::c0a8:100/120 is affected by ipv4 LAN VIPs.
pfctl -vss
igb1 ipv6-icmp 10.0.0.4:1 (fd22::8[1]) -> 192.168.1.100:8 (64:ff9b::c0a8:164[1]) NO_TRAFFIC:NO_TRAFFIC age 00:00:18, expires in 00:00:07, 4:4 pkts, 320:240 bytes, rule 117pfctl -vvsr
@117 pass in quick on igb1 inet6 from fd22::/64 to 64:ff9b::c0a8:100/120 flags S/SA keep state (if-bound) label "USER_RULE" label "id:1748759142" ridentifier 1748759142 af-to inet from (igb1) [ Evaluations: 760 Packets: 52 Bytes: 3640 States: 1 ] -
Ok I think we see a problem here. Digging...
