Performance regression 2.7.2 to 2.8
-
And you only see this for ping6? Internal IPv4 traffic is unaffected?
-
I was having this issue as well and can confirm the diff has solved my 6rd issues in 2.8.0
If you'd like I have a pcap from when I was having issues I can provide.
-
@stephenw10 Unable to reproduce with v4 and the link local, fe80::1:1 is always reachable.
It also affects Virtual IPs.
Is it expected behavior that the cpu usage is high with an VIPs 10.0.0.1/32 is on wan?
10.0.0.1/32 has been reassigned to lan and cpu can idle.
For supplementary information if a ping6 64:ff9b::a00:3 is started it will fail, restarting pfSense while the ping6 remains undisturbed; when pfSense boots up ping6 is successful for about 10 minutes.
Restarted pfSense 3 times testing VIPs, ping6 64:ff9b::7f00:1 can work if it is just one ping, if two or more lan IPs ping6 at the same time it does not work; this all may or may not be correct behavior if pfSense is seeing all pings from the same ip.ping6: Warning: time of day goes back (-16520us), taking countermeasures 64 bytes from 64:ff9b::7f00:1: icmp_seq=269 ttl=64 time=0.000 ms (DUP!) ping6: Warning: time of day goes back (-16536us), taking countermeasures 64 bytes from 64:ff9b::7f00:1: icmp_seq=270 ttl=64 time=0.000 ms (DUP!) ping6: Warning: time of day goes back (-16529us), taking countermeasures 64 bytes from 64:ff9b::7f00:1: icmp_seq=271 ttl=64 time=0.000 ms (DUP!) 64 bytes from 64:ff9b::7f00:1: icmp_seq=770 ttl=64 time=0.207 ms ping6: Warning: time of day goes back (-16541us), taking countermeasures 64 bytes from 64:ff9b::7f00:1: icmp_seq=272 ttl=64 time=0.000 ms (DUP!) ping6: Warning: time of day goes back (-16471us), taking countermeasures 64 bytes from 64:ff9b::7f00:1: icmp_seq=273 ttl=64 time=0.000 ms (DUP!) ping6: Warning: time of day goes back (-16543us), taking countermeasures 64 bytes from 64:ff9b::7f00:1: icmp_seq=274 ttl=64 time=0.000 ms (DUP!) ping6: Warning: time of day goes back (-16523us), taking countermeasures -
@fathead said in Performance regression 2.7.2 to 2.8:
Is it expected behavior that the cpu usage is high with an VIPs 10.0.0.1/32 is on wan?
No, not just that. It might if it's having to deal with a lot of traffic to that VIP that otherwise gets blocked.
Can I assume that applying that patch has not changed this new problem? Just that it too is new in 2.8?
-
@stephenw10
With or without patch mr1226.diff
No traffic on any VIPs and cpu is high.
kea-dhcp6 php-fpm.
kea-dhcp6 is using almost about 3% when 10.0.0.1/32 IP Alias is set on the wan, set it to lan kea-dhcp6 uses 0.00% -
Block private networks and loopback addresses
Is enabled on wan, turning that off is all the same high cpu. -
Oh this is on the PPPoE WAN?
That's a known issue: https://redmine.pfsense.org/issues/16235
Try the patch refferenced there.
-
@stephenw10 Yes the PPPoE WAN.
Is this the patch?
This patch does fix high cpu, however when a VIP is set on wan, it breaks the whole nat 64:ff9b::/96 address space, or is a reload/restart needed? -
Yup without that when you add a VIP on a PPPoE WAN and have if_pppoe enabled then the connection loops continually. The logs will have shown it reconnecting every few seconds which obviously load the CPU significantly.
So how exactly does NAT64 fail?
What are you using that VIP for?
-
So how exactly does NAT64 fail?
Native v6 traffic is normal.
Outside NAT64 so far is not working with a VIP set on wan.
The VIPs them selves are reachable example 64:ff9b::10.0.0.3WAN ipv6-icmp 10.0.0.4:1 (fdbb::8[1]) -> 77.47.127.138:8 (64:ff9b::100:1[1]) NO_TRAFFIC:NO_TRAFFIC 2 / 2 160 B / 120 BWhat are you using that VIP for?
v4 VIPs for ping, v6 VIPs for DNS.
-
@fathead said in Performance regression 2.7.2 to 2.8:
Outside NAT64 so far is not working with a VIP set on wan.
OK so you are setting that VIP just as something to ping from a V6 only client device?
Can I assume it still responds to ping from an internal IPv4 client?
-
OK so you are setting that VIP just as something to ping from a V6 only client device?
Can I assume it still responds to ping from an internal IPv4 client?
Yes and yes.
-
Furthermore 64:ff9b::c0a8:100/120 is passed on the lan, at some point in the past two days this stopped working.
-
Oh so dhcpv6 stopped working you mean by passed?
-
@stephenw10
Firewall rule set to pass LAN subnets to 64:ff9b::c0a8:100/120
64:ff9b::c0a8:101(pfSense it self) was the only address replying.tcpdump on pfSense show the wrong ip(the VIP) and not the interface IP(192.168.1.1).
Removing all v4 VIPs from lan restores reachabilit. v4 VIPs on localhost are different some how.
Before:16:52:35.520044 IP 10.0.0.1 > 192.168.1.100: ICMP echo request, id 179, seq 1, length 64After v4 VIPs removed:
17:08:30.209222 IP 192.168.1.1 > 192.168.1.100: ICMP echo request, id 182, seq 10740, length 64 17:08:30.209351 IP 192.168.1.100 > 192.168.1.1: ICMP echo reply, id 182, seq 10740, length 64The 64:ff9b::c0a8:100/120 rule is above the 64:ff9b::/96 rule.
NAT64 and v4 VIPs alias on wan and/or lan are not playing nicely. -
The issue is that the "Automatic" NAT64 source option in the rule leaves things up to the OS itself. Hence the source tends to be the first address on the interface (e.g. 10.0.0.1). To address that issue a GUI option exists to override the source. For example:
-
@marcosm The source has always been set to "LAN subnets".
-
It's the 'Address Family Translation' source that must be set to 'LAN address' rather than the default 'Automatic' where it's choosing the VIP.
-
@stephenw10
'Address Family Translation' is also set to 'LAN address'. -
Hmm, then maybe it's not matching the test traffic somehow?
