IPv6 WAN Gateway monitoring reports 100% packet loss

A Former User

I had a power outage this morning due to a toaster tripping the earth leakage circuit breaker. When pfsense was booted the WAN IPv6 connection reported 100% packet loss while IPv4 reported everything fine. I tried using the diagnostic IPv6 ping and they all failed, in fact so did all IPv4 diagnostic pings. Went to the WAN interface, saved the settings, and applied them and everthing was working correctly. Pfsense is configured as a router, as I have a public ::/48, this is divided into multiple ::/64 so internal traffic passes through pfsense to my internet firewall router and then hits my ISP's network.

A Former User

@vortex21

Just upgraded to Version 22.05.r.20220609.1919 and after rebooting the IPv6 WAN Gateway monitoring failed again. So again I had to go to Interfaces -> WAN , then press Save and Apply Changes for the WAN gateway monitoring to work otherwise IPv6 traffic is blocked

Bob.Dig

@vortex21 Is your pfSense behind another router doing DHCP with IPv6? In this situation it was normal to fail for me too but a reboot of pfSense would solve that, not provoke that behavior. What monitoring IP are you using?

A Former User

@bob-dig

Hi,

No, DHCPv6 is not being used. If I reboot my firewall the IPV6 Gateway also fails so I have to manually re-save the WAN settings and then IPv6 will begin to work again.

I am using a static IP on my edge router both interfaces are statically assigned, I am using the private interface on the router as my monitoring IPv6 address.

A Former User

@vortex21

Hi, upgraded to release 22.05.r.20220614.0600 today and IPv6 WAN monitoring again failed requiring WAN interface having to be saved and then Apply Changes for it to start working again.

A Former User

@vortex21

Upgraded to 22.05.r.20220614.1944 and experienced the same problem. Also worth noting that unless Prefer to use IPv4 even if IPv6 is available (System -> Advanced -> Networking ) is enabled then upgrade will not complete.

A Former User

@vortex21

Just upgraded to 22.05.r.20220617.0613 but only after ensuring that Prefer IPv4 even if IPv6 is available is enabled in System -> Advanced -> Networking.
After applying update, I still lose connectivity to the IPv6 gateway. So that I have to save the WAN settings again to get the IPv6 gateway monitoring to work, this is with out changing any settings. From a monitoring perspective the RTT and RTTsd times are lower for IPv6 compared to IPv4.

luckman212

@vortex21 I think you might be seeing the same issue as I was here. You could try the linked PR #4595 to see if it helps your issue.

A Former User

@luckman212

Hi, tried applying the fix in the latest update of https://github.com/pfsense/pfsense/pull/4595. Unfortunately it did not fix the problem, I had to re-save the WAN interface settings and IPv6 GW Montoring worked

luckman212

@vortex21 How did you apply the fix? As it states in the PR notes, it probably won't work with the System Patches package alone due to the number of changes and the differences between the original files in pfS+ vs CE. So did you manually apply the changes to all the related files?

A Former User

@luckman212
Hi, followed the steps below

1 install cmdwatch:

 pkg add https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/cmdwatch-0.2.0_2.txz

download the script:

fetch https://gist.githubusercontent.com/luckman212/0fdea1cbdc0a561d781a52c7d34fb60d/raw/ffd321ef196fb1c919dd66700acdd4acc02b3e63/dpinger_static_routes.php
cmdwatch --interval=2 'php -q dpinger_static_routes.php'
php -a
include("config.inc");
install_cron_job('/usr/bin/nice -n20 /etc/rc.checkv6addrchange', true, "/1", '', '', '', '*', 'root', true);
After reboot, ran via ssh cmdwatch --interval=2 'php -q dpinger_static_routes.php'
Then checked GUI, IPv6 monitoring was offline, and I had to save WAN interface to fix monitoring issue.

luckman212

@vortex21 You are missing most of the important steps. You just downloaded the little helper script from the other PR which does nothing but display some info. You need to apply the patches in the linked commit that actually change the behavior. I know it might be a bit complicated- so I'll try to post a step by step.

Are you using pfSense+ or CE?

A Former User

@luckman212

Hi,

Currently running pfsense+ 22.05.r.20220617.0613

luckman212

@vortex21 I posted some new instructions on the PR#4595. I hope you're able to give them a try.

A Former User

@luckman212

Hi, I followed the instructions, applying the system patches and then the new patch. After rebooting, and login the IPv6 GW Monitoring was reporting 70% packet loss and as I watched it increased to 77% before I re-saved the WAN interface which fixed the problem.

luckman212

@vortex21 If your IPv6 WAN is down immediately after a fresh boot then something different is going on here. Can you send some more details?

how is your WAN6 configured- DHCP6, SLAAC, etc?
can you ssh in after rebooting your system and run ifconfig -v -- copy the output.
then, edit your interface and hit Save, and run ifconfig -v again and copy that too. Paste those outputs here (or if you don't want to post publicly, PM it to me)
what happens if you manually run /etc/rc.checkv6addrchange ? Does it give you an error? Does anything change after running that?

A Former User

@luckman212

Hi,

I captured the output of ifconfig -v pasting it into a txt file after-reboot.txt , saved the WAN interface and repeated ifconfig -v saving it into after-save.txt. Then I used diff to compare the after-reboot.txt and after-save.txt and found no change in the configuration.

ifconfig -v

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: LAN
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether :::::a6
inet6 fe80::bbbb:bbbb:bbbb:bbbb%igb0 prefixlen 64 scopeid 0x1
inet6 2a02:SSSS:SSSS::SSSS prefixlen 64
inet XXX:XXX:XXX.254 netmask 0xffffff00 broadcast XXX:XXX:XXX.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether :::::a7
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
ether ££:££:££:££:££:6e
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether &&:&&:&&:&&:e4
inet6 fe80::dddd:dddd:dddd:dddd%igb2 prefixlen 64 scopeid 0x4
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether &&:&&:&&:&&:e5
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
groups: enc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
groups: pflog
igb2.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: WAN
options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether &&:&&:&&:&&:e4
inet6 fe80::dddd:dddd:dddd:dddd%igb2.3 prefixlen 64 scopeid 0xa
inet6 2a02:LLLL:LLLL::LLLL prefixlen 64
inet YYY:YYY:YYY10 netmask 0xffffff00 broadcast YYY:YYY:YYY255
groups: vlan
vlan: 3 vlanpcp: 0 parent interface: igb2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: LANWORK
options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ::::**:a6
inet6 fe80::bbbb:bbbb:bbbb:bbbb%igb0.2 prefixlen 64 scopeid 0xb
inet KKK:KKK:KKK.1 netmask 0xffffff00 broadcast KKK:KKK:KKK.255
groups: vlan
vlan: 2 vlanpcp: 0 parent interface: igb0
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

luckman212

@vortex21 Ok so you have VLANs on both LAN (igb0) and WAN (igb2) interfaces?

Please answer these other questions:

how is your IPv6 configured on WAN & LAN interfaces (DHCP6, SLAAC etc)
are you using PPPoE?
what is the result of manually running /etc/rc.checkv6addrchange
please also paste the output of pgrep -lf dpinger

A Former User

@luckman212 said in IPv6 WAN Gateway monitoring reports 100% packet loss:

-lf dpinger

lease answer these other questions:

how is your IPv6 configured on WAN & LAN interfaces (DHCP6, SLAAC etc)
       WAN and LAN are both statically assigned IPv6 address
       DHCPv6 is running within my internal network but is being handled by raspberry pi running ISC Kea

are you using PPPoE?
       No, PPPoE is not configured on firewall

what is the result of manually running /etc/rc.checkv6addrchange
          no output, no change in IPv6 gateway monitoring in GUI     

please also paste the output of pgrep -lf dpinger

8312 /usr/local/bin/dpinger -S -r 0 -i 
             WANGWv6 -B 2a02:yyyy:yyyy:y:yyyy:yyyy:yyyy:yyyy -p /var/run/dpinger_
             WANGWv6~fa5faaa6~2a02:xxxx:xxxxx:x:xxxx:xxxx:xxxx:xxxx.pid -u /var/run/dpinger_
             WANGWv6~fa5faaa6~2a02:xxxx:xxxxx:x:xxxx:xxxx:xxxx:xxxx.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 2a02:8xxxx:xxxxx:x:xxxx:xxxx:xxxx:xxxx
   7987 /usr/local/bin/dpinger -S -r 0 -i 
             WANGWv4 -B yyy.yyy.yyy.10 -p /var/run/dpinger_WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.pid -u /var/run/dpinger_
             WANGWv4~yyy.yyy.yyy.10~nnn.nnn.nnn.1.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 nnn.nnn.n.1

luckman212

@vortex21 Ok so you have static IPv6's configured -- well then this appears to be a different problem, not really the one that my PR is designed to solve!

The pgrep -lf dpinger output you pasted above, is that from before or after you re-saved your interface config? Hard to tell, but looking at it, I would guess after (because it appears to be bound [-B 2a02:] to the correct IP). Can you post the "before" output as well?