IPv6 WAN Gateway monitoring reports 100% packet loss

A Former User

@vortex21

Hi, upgraded to release 22.05.r.20220614.0600 today and IPv6 WAN monitoring again failed requiring WAN interface having to be saved and then Apply Changes for it to start working again.

A Former User

@vortex21

Upgraded to 22.05.r.20220614.1944 and experienced the same problem. Also worth noting that unless Prefer to use IPv4 even if IPv6 is available (System -> Advanced -> Networking ) is enabled then upgrade will not complete.

A Former User

@vortex21

Just upgraded to 22.05.r.20220617.0613 but only after ensuring that Prefer IPv4 even if IPv6 is available is enabled in System -> Advanced -> Networking.
After applying update, I still lose connectivity to the IPv6 gateway. So that I have to save the WAN settings again to get the IPv6 gateway monitoring to work, this is with out changing any settings. From a monitoring perspective the RTT and RTTsd times are lower for IPv6 compared to IPv4.

luckman212

@vortex21 I think you might be seeing the same issue as I was here. You could try the linked PR #4595 to see if it helps your issue.

A Former User

@luckman212

Hi, tried applying the fix in the latest update of https://github.com/pfsense/pfsense/pull/4595. Unfortunately it did not fix the problem, I had to re-save the WAN interface settings and IPv6 GW Montoring worked

luckman212

@vortex21 How did you apply the fix? As it states in the PR notes, it probably won't work with the System Patches package alone due to the number of changes and the differences between the original files in pfS+ vs CE. So did you manually apply the changes to all the related files?

A Former User

@luckman212
Hi, followed the steps below

1 install cmdwatch:

 pkg add https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/cmdwatch-0.2.0_2.txz

download the script:

fetch https://gist.githubusercontent.com/luckman212/0fdea1cbdc0a561d781a52c7d34fb60d/raw/ffd321ef196fb1c919dd66700acdd4acc02b3e63/dpinger_static_routes.php
cmdwatch --interval=2 'php -q dpinger_static_routes.php'
php -a
include("config.inc");
install_cron_job('/usr/bin/nice -n20 /etc/rc.checkv6addrchange', true, "/1", '', '', '', '*', 'root', true);
After reboot, ran via ssh cmdwatch --interval=2 'php -q dpinger_static_routes.php'
Then checked GUI, IPv6 monitoring was offline, and I had to save WAN interface to fix monitoring issue.

luckman212

@vortex21 You are missing most of the important steps. You just downloaded the little helper script from the other PR which does nothing but display some info. You need to apply the patches in the linked commit that actually change the behavior. I know it might be a bit complicated- so I'll try to post a step by step.

Are you using pfSense+ or CE?

A Former User

@luckman212

Hi,

Currently running pfsense+ 22.05.r.20220617.0613

luckman212

@vortex21 I posted some new instructions on the PR#4595. I hope you're able to give them a try.

A Former User

@luckman212

Hi, I followed the instructions, applying the system patches and then the new patch. After rebooting, and login the IPv6 GW Monitoring was reporting 70% packet loss and as I watched it increased to 77% before I re-saved the WAN interface which fixed the problem.

luckman212

@vortex21 If your IPv6 WAN is down immediately after a fresh boot then something different is going on here. Can you send some more details?

how is your WAN6 configured- DHCP6, SLAAC, etc?
can you ssh in after rebooting your system and run ifconfig -v -- copy the output.
then, edit your interface and hit Save, and run ifconfig -v again and copy that too. Paste those outputs here (or if you don't want to post publicly, PM it to me)
what happens if you manually run /etc/rc.checkv6addrchange ? Does it give you an error? Does anything change after running that?

A Former User

@luckman212

Hi,

I captured the output of ifconfig -v pasting it into a txt file after-reboot.txt , saved the WAN interface and repeated ifconfig -v saving it into after-save.txt. Then I used diff to compare the after-reboot.txt and after-save.txt and found no change in the configuration.

ifconfig -v

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: LAN
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether :::::a6
inet6 fe80::bbbb:bbbb:bbbb:bbbb%igb0 prefixlen 64 scopeid 0x1
inet6 2a02:SSSS:SSSS::SSSS prefixlen 64
inet XXX:XXX:XXX.254 netmask 0xffffff00 broadcast XXX:XXX:XXX.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether :::::a7
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
ether ££:££:££:££:££:6e
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether &&:&&:&&:&&:e4
inet6 fe80::dddd:dddd:dddd:dddd%igb2 prefixlen 64 scopeid 0x4
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether &&:&&:&&:&&:e5
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
groups: enc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
groups: pflog
igb2.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: WAN
options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether &&:&&:&&:&&:e4
inet6 fe80::dddd:dddd:dddd:dddd%igb2.3 prefixlen 64 scopeid 0xa
inet6 2a02:LLLL:LLLL::LLLL prefixlen 64
inet YYY:YYY:YYY10 netmask 0xffffff00 broadcast YYY:YYY:YYY255
groups: vlan
vlan: 3 vlanpcp: 0 parent interface: igb2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: LANWORK
options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ::::**:a6
inet6 fe80::bbbb:bbbb:bbbb:bbbb%igb0.2 prefixlen 64 scopeid 0xb
inet KKK:KKK:KKK.1 netmask 0xffffff00 broadcast KKK:KKK:KKK.255
groups: vlan
vlan: 2 vlanpcp: 0 parent interface: igb0
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

luckman212

@vortex21 Ok so you have VLANs on both LAN (igb0) and WAN (igb2) interfaces?

Please answer these other questions:

how is your IPv6 configured on WAN & LAN interfaces (DHCP6, SLAAC etc)
are you using PPPoE?
what is the result of manually running /etc/rc.checkv6addrchange
please also paste the output of pgrep -lf dpinger

A Former User

@luckman212 said in IPv6 WAN Gateway monitoring reports 100% packet loss:

-lf dpinger

lease answer these other questions:

how is your IPv6 configured on WAN & LAN interfaces (DHCP6, SLAAC etc)
       WAN and LAN are both statically assigned IPv6 address
       DHCPv6 is running within my internal network but is being handled by raspberry pi running ISC Kea

are you using PPPoE?
       No, PPPoE is not configured on firewall

what is the result of manually running /etc/rc.checkv6addrchange
          no output, no change in IPv6 gateway monitoring in GUI     

please also paste the output of pgrep -lf dpinger

8312 /usr/local/bin/dpinger -S -r 0 -i 
             WANGWv6 -B 2a02:yyyy:yyyy:y:yyyy:yyyy:yyyy:yyyy -p /var/run/dpinger_
             WANGWv6~fa5faaa6~2a02:xxxx:xxxxx:x:xxxx:xxxx:xxxx:xxxx.pid -u /var/run/dpinger_
             WANGWv6~fa5faaa6~2a02:xxxx:xxxxx:x:xxxx:xxxx:xxxx:xxxx.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2a02:8xxxx:xxxxx:x:xxxx:xxxx:xxxx:xxxx
   7987 /usr/local/bin/dpinger -S -r 0 -i 
             WANGWv4 -B yyy.yyy.yyy.10 -p /var/run/dpinger_WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.pid -u /var/run/dpinger_
             WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 nnn.nnn.n.1

luckman212

@vortex21 Ok so you have static IPv6's configured -- well then this appears to be a different problem, not really the one that my PR is designed to solve!

The pgrep -lf dpinger output you pasted above, is that from before or after you re-saved your interface config? Hard to tell, but looking at it, I would guess after (because it appears to be bound [-B 2a02:] to the correct IP). Can you post the "before" output as well?

A Former User

@luckman212

Immediately after reboot
pgrep -lf dpinger

43507 /usr/local/bin/dpinger -S -r 0 -i
WANGWv6 -B 2a02::22 -p /var/run/dpinger_
WANGWv6~fa5faaa6~2a02::38.pid -u /var/run/dpinger_
WANGWv6~fa5faaa6~2a02::38.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2a02::38

42959 /usr/local/bin/dpinger -S -r 0 -i
WANGWv4 -B -p /var/run/dpinger_
WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.pid -u /var/run/dpinger_
WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 nnn.nnn.nnn.1
[22.05-RC][admin@pfsense]/root:

Reporting GUI login
Message from syslogd@gw at Jun 21 16:46:30 ...
php-fpm[384]: /index.php: Successful login for user 'admin' from: 2a02::1 (Local Database)

Immediately after WAN interface save
[22.05-RC][admin@pfsense]/root: pgrep -lf dpinger
63333 /usr/local/bin/dpinger -S -r 0 -i
WANGWv6 -B 2a02::22 -p /var/run/dpinger_
WANGWv6~fa5faaa6~2a02::38.pid -u /var/run/dpinger_
WANGWv6~fa5faaa6~2a02::38.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2a02:38
63257 /usr/local/bin/dpinger -S -r 0 -i WANGWv4 -B yyy.yyy.yyy.10 -p /var/run/dpinger_WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.pid -u /var/run/dpinger_WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 nnn.nnn.nnn.1
[22.05-RC][admin@pfsense]/root:

luckman212

@vortex21 Makes no sense- the dpinger process and args from "before" are identical to the "after". So there must be a difference in the interface config.

Can you post before & after of ifconfig -v and ndp -a?

It would help if you didn't redact the info, if you're worried about privacy use a password protected pastebin, PM, etc...

A Former User

@luckman212
Hi,

Took the output of ifconfig -v from before and after and use diff to find the differences

diff after.txt before.txt
2c2
< igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
23c23
< igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
50c50
< igb2.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492

igb2.3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
62c62
< igb0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492

igb0.2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
78,79c78

When looking at ndp -a and comparing before and after saving the WAN interface, the difference is four link local interfaces

fe80::21c:ffff:fef0:b5e%igb0 00:0c:29:f0:0b:5e igb0 2s R
fe80::21c:ffff:febe:772b%igb0 00:0c:29:be:77:2b igb0 13s R
fe80::21d:bbff:fec9:5938%igb2.3 00:1d:aa:f9:59:38 igb2.3 24s R R
fe80::21c:ffff:fe38:293e%igb0 00:0c:29:88:39:3e igb0 23h59m39s S

luckman212

@vortex21 The forum is mangling your output. Can you please put it on a private pastebin instead of just posting the diff output which is not easy to decipher.