Gigaclear & ip6 - lose of connectivity after *exactly* 5 minutes

Gertjan

@Irata said in Gigaclear & ip6 - lose of connectivity after *exactly* 5 minutes:

However, there should not be a need for a router to have a GUA with IPv6 .

By no means I'm an IPv6 expert, but what about this : If you want to use (Open)VPN over IPv6 to connect to your pfSense OpenVPN server, you'll need a WAN GUA ....

Irata

@Gertjan Well of course, but IPv6 routers don't need a GUA and they don't need to be VPN servers.

If you did want to run a VPN server in this setup, you'd just address it to a LAN address which has been delegated a GUA /64. Then setup the firewall to allow OpenVPN to that specific IPv6 GUA.

However, this is not related to this issue at all.

kprovost

It's not at all clear to me why the neighbor solicitation doesn't get a reply.
Our ruleset includes pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136}, which ought to be enough for it to be allowed.
So far I've been unable to reproduce such issues.

It'd be interesting to look at the pflog output to see where and why the packet is dropped. (I.e. is the NS dropped on the way on, or the NA on the way out? Due to what rule?)

As for the OPNSense commit it's ... well, leaving the editorialising aside, it seems to be trying to do three different things at once, at least two of which it does wrong. (Hint: look at where pd->hdr is copied out of the packet for ICMPv6. It'll never contain anything other than 0 for nd->nd_ns_target or mld->mld_addr. At least it'll always be 0 so it's mostly harmless, but it is still wrong.)

bmeeks

@kprovost said in Gigaclear & ip6 - lose of connectivity after *exactly* 5 minutes:

It's not at all clear to me why the neighbor solicitation doesn't get a reply.
Our ruleset includes pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136}, which ought to be enough for it to be allowed.
So far I've been unable to reproduce such issues.

It'd be interesting to look at the pflog output to see where and why the packet is dropped. (I.e. is the NS dropped on the way on, or the NA on the way out? Due to what rule?)

As for the OPNSense commit it's ... well, leaving the editorialising aside, it seems to be trying to do three different things at once, at least two of which it does wrong. (Hint: look at where pd->hdr is copied out of the packet for ICMPv6. It'll never contain anything other than 0 for nd->nd_ns_target or mld->mld_addr. At least it'll always be 0 so it's mostly harmless, but it is still wrong.)

My ISP does not offer native IPv6, so I do not have an IPv6 configuration to test with. I was simply providing the user with some information from the older FreeBSD bug report that may or may not be relevant to his situation.

I agree the linked thread contained a lot of back and forth, hence the use of quotes around the word discussion . I periodically check in at the other *sense forum as a guest to monitor for any potential Suricata issues that might be applicable to my pfSense package. That's how I encountered the linked threads.

Irata

Could it be a routing issue? Because it requires responding to a GUA address from a link-local address? This appears to be what other implementations allow, plus the Juniper switch itself. In all the captures I've taken, I've never once seen pfSense send a Neighbor solicitation/advertisement to a GUA from its link-local source address.

If I do both these things, it resolves the issue:

Add a static route to IPv6 router GUA on WAN
Add an Virtual IP/Alias to the WAN, picking any /128 from within the delegated /48 from the WAN

I think that static route is an important clue. If I just add the static route then pfsense picks some GUA from any LAN (that could be outside of the delegated /48) for the Neighbor Advertisement from pfSense. Again, it refuses to respond to the GUA from it's link-local, pfsense will do anything but use it's link-local even if that means picking some unrelated LAN GUA that it should not be sending out this WAN link.

Then adding the IP Alias stops the random picking of a LAN GUA and it now picks the Virtual IP.

Without the Virtual IP, I can't Ping that GUA from the WAN link-local address, so would the same logic be blocking these Neighbor Advertisements too?

I trawled days of syslogs and I don't see anything Neighbor Advertisements/Solicitations that hit any block rule. nothing showed up.

Sadly, the specifiation for Neghbor Solications/Advertisements are ambiguous here but it can be read that a Link-Local address can respond to a GUA and that's what others have implemented.

Having searched internet forums for the Virtual IP/static route trick, I've found several examples of this issue with pfSense and several ISP's around the globe and interestingly they post a very similar workaround.

JKnott

@Irata said in Gigaclear & ip6 - lose of connectivity after *exactly* 5 minutes:

I beleive the root cause of this is when no GUA is provided for the WAN, so it is left with only a link-local address and a deligated prefix. But the key here is, it has no GUA bound to its interface.

The WAN interface doesn't need a GUA. All it needs is a link local address. Even if it has a GUA, that address will have nothing to do with the delegated prefix.

JKnott

@Gertjan said in Gigaclear & ip6 - lose of connectivity after *exactly* 5 minutes:

By no means I'm an IPv6 expert, but what about this : If you want to use (Open)VPN over IPv6 to connect to your pfSense OpenVPN server, you'll need a WAN GUA ....

Use the interface address on the LAN side.

ahxcjay

My v6 just failed.

Apr 7 19:53:53 dhcp6c 20400 got an expected reply, sleeping.
Apr 7 19:53:53 dhcp6c 20400 removing an event on igb0, state=RENEW
Apr 7 19:53:53 dhcp6c 20400 script "/var/etc/dhcp6c_wan_dhcp6withoutra_script.sh" terminated
Apr 7 19:53:53 dhcp6c 7146 dhcp6c renew, no change - bypassing update on igb0
Apr 7 19:53:53 dhcp6c 20400 executes /var/etc/dhcp6c_wan_dhcp6withoutra_script.sh
Apr 7 19:53:53 dhcp6c 20400 update a prefix 2a06:61c1:ba35::/48 pltime=86400, vltime=86400
Apr 7 19:53:53 dhcp6c 20400 update an IA: PD-0
Apr 7 19:53:53 dhcp6c 20400 nameserver[1] 2a02:fb8:3:300::3
Apr 7 19:53:53 dhcp6c 20400 nameserver[0] 2a02:fb8:2:200::3
Apr 7 19:53:53 dhcp6c 20400 dhcp6c Received INFO
Apr 7 19:53:53 dhcp6c 20400 get DHCP option DNS, len 32
Apr 7 19:53:53 dhcp6c 20400 IA_PD prefix: 2a06:61c1:ba35::/48 pltime=86400 vltime=86400
Apr 7 19:53:53 dhcp6c 20400 get DHCP option IA_PD prefix, len 25
Apr 7 19:53:53 dhcp6c 20400 IA_PD: ID=0, T1=43200, T2=69120
Apr 7 19:53:53 dhcp6c 20400 get DHCP option IA_PD, len 41
Apr 7 19:53:53 dhcp6c 20400 unknown or unexpected DHCP6 option opt_20, len 0
Apr 7 19:53:53 dhcp6c 20400 get DHCP option opt_20, len 0
Apr 7 19:53:53 dhcp6c 20400 DUID: 00:02:00:00:05:83:34:63:3a:36:64:3a:35:38:3a:38:37:3a:31:37:3a:66:38:00:00:00
Apr 7 19:53:53 dhcp6c 20400 get DHCP option server ID, len 26
Apr 7 19:53:53 dhcp6c 20400 DUID: 00:01:00:01:2f:75:c3:80:64:62:66:21:6b:74
Apr 7 19:53:53 dhcp6c 20400 get DHCP option client ID, len 14
Apr 7 19:53:53 dhcp6c 20400 receive reply from fe80::4e6d:58ff:fe87:1017%igb0 on igb0
Apr 7 19:53:53 dhcp6c 20400 send renew to ff02::1:2%igb0
Apr 7 19:53:53 dhcp6c 20400 set IA_PD
Apr 7 19:53:53 dhcp6c 20400 set IA_PD prefix
Apr 7 19:53:53 dhcp6c 20400 set option request (len 4)
Apr 7 19:53:53 dhcp6c 20400 set elapsed time (len 2)
Apr 7 19:53:53 dhcp6c 20400 set server ID (len 26)
Apr 7 19:53:53 dhcp6c 20400 set client ID (len 14)
Apr 7 19:53:53 dhcp6c 20400 a new XID (5c85c4) is generated
Apr 7 19:53:53 dhcp6c 20400 Sending Renew
Apr 7 19:53:53 dhcp6c 20400 reset a timer on igb0, state=RENEW, timeo=0, retrans=9557
Apr 7 19:53:53 dhcp6c 20400 IA timeout for PD-0, state=ACTIVE

5 minutes later after this the connection dies. This is despite assigning a WAN VIP & static route to the first v6 hope.

ahxcjay

The only way I can have v6 connectivity last >24h05m00s is by:

route -6 add -host 2a02:fb8::11 -interface igb0
ndp -s 2a02:fb8::11 4e:6d:58:87:10:17 permanent

Without adding the permanent ndp entry, it defaults to 24 hours, then 5 minutes after the logs you see above, v6 dies. Once I add the permanent ndp entry I can go past 24 hours.

marcosm

@Irata

When a WAN only has a local-link IPv6 address, pfsense should be using this as the source address of neighbor advertisement messages, not picking a random LAN interface as the source address.

I tested this on 25.03 and pfSense always sent NA's sourced with the correct link-local address. This sounds more like there may be some routing/NAT issue. There's a default rule that allows any source/destination though so it should go through regardless.

Regarding https://redmine.pfsense.org/issues/16123#note-3
I see the ISP device sending NS for pfSense's link-local address without a response from pfSense:

138	2025-04-04 01:51:01.778710	0.020680	2a02:fb8::11	ff02::1:ff21:6b74	ICMPv6	86	Neighbor Solicitation for fe80::6662:66ff:fe21:6b74 from 4e:6d:58:87:10:17
145	2025-04-04 01:51:04.730835	0.972840	2a02:fb8::11	ff02::1:ff21:6b74	ICMPv6	86	Neighbor Solicitation for fe80::6662:66ff:fe21:6b74 from 4e:6d:58:87:10:17
152	2025-04-04 01:51:07.730711	0.359749	2a02:fb8::11	ff02::1:ff21:6b74	ICMPv6	86	Neighbor Solicitation for fe80::6662:66ff:fe21:6b74 from 4e:6d:58:87:10:17
209	2025-04-04 01:51:11.150835	0.015817	2a02:fb8::11	ff02::1:ff21:6b74	ICMPv6	86	Neighbor Solicitation for fe80::6662:66ff:fe21:6b74 from 4e:6d:58:87:10:17
214	2025-04-04 01:51:14.138711	1.003684	2a02:fb8::11	ff02::1:ff21:6b74	ICMPv6	86	Neighbor Solicitation for fe80::6662:66ff:fe21:6b74 from 4e:6d:58:87:10:17

While this is happening, check that pfSense has joined the respective multicast group with ifmcstat -i igc0 -f inet6. If it has joined as expected then it should show something like:

inet6 fe80::6662:66ff:fe21:6b74%igc0 scopeid 0x3
mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
...
        group ff02::1:ff21:6b74%igc0 scopeid 0x3 mode exclude
                mcast-macaddr 33:33:ff:21:6b:74

A similar issue which has since been fixed exists in old versions:
https://redmine.pfsense.org/issues/13423

Irata

When this is happening I see this:

vtnet0.3:
inet6 fe80::1c1e:54ff:fe8a:705%vtnet0.3 scopeid 0x19
mldv2 flags=2<USEALLOW> rv 2 qi 125 qri 10 uri 3
group ff02::1:ff00:1%vtnet0.3 scopeid 0x19 mode exclude
mcast-macaddr 33:33:ff:00:00:01
group ff01::1%vtnet0.3 scopeid 0x19 mode exclude
mcast-macaddr 33:33:00:00:00:01
group ff02::2:1861:20ce%vtnet0.3 scopeid 0x19 mode exclude
mcast-macaddr 33:33:18:61:20:ce
group ff02::2:ff18:6120%vtnet0.3 scopeid 0x19 mode exclude
mcast-macaddr 33:33:ff:18:61:20
group ff02::1%vtnet0.3 scopeid 0x19 mode exclude
mcast-macaddr 33:33:00:00:00:01
group ff02::1:ff8a:705%vtnet0.3 scopeid 0x19 mode exclude
mcast-macaddr 33:33:ff:8a:07:05

The only clue I have is, just the same as others I must add a static route in to the GUA of the ISP router to get pfsense to reply out of the correct WAN interface, and I have to manually add a permanent ndp entry for that device too. I am running multi-wan. I also don't understand why the ndp table is not being updated for that ISP device, even when I ping that device.

It's really strange, but pfsense definately does not send NA without the above encouragement. I can recreate this issue so easily, every time without fail.

The only other oddity I've noticed is that running:

ndp -s 2a02:fb8::32 4a:5a:0d:5a:f2:b7

gives this sometimes:

ndp: delete: cannot locate 2a02:fb8::32

But I'm not trying to delete, I'm trying to add? Whereas, running this command just after adding the static route will make the command work.

marcosm

@Irata Would you share the output of uname -a?

Irata

FreeBSD pfSense 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_25_03-n256497-da24eca0fcd2: Mon Apr 14 19:32:49 UTC 2025     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/obj/amd64/ILoDLiJx/var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/sources/FreeBSD-src-plus-RELENG_25_03/amd64.amd64/sys/pfSense amd64

Thought I should add some info of my setup:

I have 4 WAN interfaces, all dual-stack IPv4 and IPv6 and all with a delegated /48 subnet. I use the IPv6 prefix of the first WAN for all LAN interfaces, I use NPT to translate those to the other 3 WAN prefixes if used. I use a failover group, with each WAN interfaces in a different tier.

The ISP link causing this issue is not the first interface, so i am not using that /48 prefix at the LAN. This troublesome ISP link is the only one where IPv6 is native on the WAN interface and they send NS from a GUA. The other IPv6 interfaces are either L2TP tunnels, or the ISP uses a link-local address for it's Network Solicitation/Advertisements.

It appears others have a much simpler setup to myself and still see the issue.

If I shutdown my pfsense VM, and start up a simple OpenWrt VM, everything is stable and the IPv6 interface just works for this ISP, NA's are sent correctly to the GUA. All I do is swap one VM to another VM of a different OS, no other network changes are made.

To me, the only common factor when NA's are not sent (without some tinkering) is when I'm using pfsense and the ISP sends NS from a GUA. This does appear to be typical behavour from Juniper switches, as I also see this issue on a different test bed.

I have also just discovered, during this issue on this version of pfsense, that opening diagnostics->NDP table on pfsense UI causes the UI to lockup and eventually crash. It sends me to a crash report but that just says

Crash report begins.  Anonymous machine information:

amd64
15.0-CURRENT
FreeBSD 15.0-CURRENT #0 plus-RELENG_25_03-n256497-da24eca0fcd2: Mon Apr 14 19:32:49 UTC 2025     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/obj/amd64/ILoDLiJx/var/jenkins/workspace/pfSense-Plus-snapshots-25_03-main/sources/FreeB

Crash report details:

No PHP errors found.

No FreeBSD crash data found.

I'm unsure if this is related.

marcosm

I was able to reproduce the issue. The behavior is intended and a workaround exists - see:
https://redmine.pfsense.org/issues/16146

I expect this to work with the ISP Gigaclear as well.

ahxcjay

@marcosm thank you!

(5 minutes later) Confirmed: it works!

Irata

@ahxcjay Check after 24 hours, as even with this change I still see the NDP entry for first hop going stale and it may get deleted after 24 hours.

ahxcjay

@Irata yes - 2h53m left…

ahxcjay

@ahxcjay NDP renewed for another 24h…! 15+ minutes in. No drops.

Irata

Unfortunately, it does not work for me on 25.03-BETA. The setting has not changed behaviour - i have tried a reboot after adding the setting into system->advanced>system tunables.

I did try this setting weeks ago, but since it made no difference I disgarded it.

I still need to add the static NDP entry, even with this setting.

I wonder if this setting is no longer working on 25.03-BETA. However, that NDP entry should not be going stale for 24 hours anyway and so something still isn't right. It's responding to NS, so why isn't the NDP entry updating every minute whenever they are responded to?

There is something additional I've spotted with the NA from the ISP, the source IP is GUA but the target address is link-local. I don't mean destination ip, I do mean target address within the ICMPv6 payload - before the (rte, sol) below:

4	13:45:21.68	2a02:fb8::32	fe80::1c1e:54ff:fe8a:705	ICMPv6	82	Neighbor Advertisement fe80::4a5a:dff:fe5a:f2b7 (rtr, sol)

The NDP table is being updated with the target address, but it does not update the source ip into the NDP table. That might be correct behaviour, but if so then what is updating the source ip entry in the NDP table after 24 hours (for the situation it does).

I have also seen the situation with where the NDP entry was stale for 24 hours and strangely was updated, which kept it working and when using my previous tricks to get it to work. However, it was not consistent after every reboot and connectiviy was still unreliable.

I still beleive we are walking around some other root cause here, possibly two issues.

I do admit the spec is ambiguious and this is not prohibited within the spec, but this implementation is a good example of exactly not what the spec intended. However, it should still be working.

We should not be seeing the first hop GUA going stale for 24 hours anyway.