IPv6 Gateway Issue (see issue 11454)

g.shaffer

The 2.5.1-RC did not resolve the issue described in #11454.

Upon reboot, gateway is still showing "dynamic" and and "pending for "RTT", "RRTsd" and "Loss". As stated in https://forum.netgate.com/topic/160952/ipv6-no-gateway-after-2-5-upgrade and https://redmine.pfsense.org/issues/11454 the issue cascades into further issues with Track Interfaces, routing, FW rules, PBR, etc.

My WAN setup is a generic DHCP6 which worked in 2.4.5-p1.

Screen Shot 2021-03-16 at 10.07.29 AM.png

jimp

Going to need some more info, starting with:

The contents of /var/etc/rtsold_<your WAN interface>_script.sh
The contents or /tmp/*_routerv6 files for the WAN interface
The contents of the system log entries for rtsold from the most recent boot

g.shaffer

@jimp said in IPv6 Gateway Issue (see issue 11454):

The contents of /var/etc/rtsold_<your WAN interface>_script.sh
The contents or /tmp/*_routerv6 files for the WAN interface
The contents of the system log entries for rtsold from the most recent boot

Contents of /var/etc/rtsold_em0_script.sh

This shell script launches dhcp6c and configured gateways for this interface.

if [ -n "$2" ]; then
echo $2 > /tmp/em0_routerv6
echo $2 > /tmp/em0_defaultgwv6
/usr/bin/logger -t rtsold "Received RA specifying route $2 for interface wan(em0)"
fi
if [ ! -f /tmp/dhcp6c_em0_lock ]; then
/usr/bin/touch /tmp/dhcp6c_em0_lock
if [ -f /var/run/dhcp6c_em0.pid ]; then
/bin/pkill -F /var/run/dhcp6c_em0.pid
/bin/rm -f /var/run/dhcp6c_em0.pid
/bin/sleep 1
fi
/usr/local/sbin/dhcp6c -D -n -c /var/etc/dhcp6c_wan.conf -p /var/run/dhcp6c_em0.pid em0
/usr/bin/logger -t rtsold "Starting dhcp6 client for interface wan(em0)"
else
/usr/bin/logger -t rtsold "RTSOLD Lock in place - sending SIGHUP to dhcp6c"
dhcp6c_pid=$(cat "/var/run/dhcp6c_em0.pid")
/bin/kill -1 ${dhcp6c_pid}
fi

The file /var/tmp/em0_routerv6 does not exist.

Contents of syslog for rtsold entries:
Mar 16 12:24:37 egis php[409]: rc.bootup: Starting rtsold process
Mar 16 12:24:38 egis php-fpm[348]: /rc.newwanip: Starting rtsold process
Mar 16 12:24:39 egis rtsold[50722]: <rtsock_input_ifannounce> interface tun2 removed
Mar 16 12:24:39 egis rtsold[50722]: <rtsock_input_ifannounce> interface tun3 removed
Mar 16 12:24:39 egis rtsold[50722]: <rtsock_input_ifannounce> interface tun4 removed
Mar 16 12:24:39 egis rtsold[44680]: Received RA specifying route for interface wan(em0)
Mar 16 12:24:39 egis rtsold[45172]: Starting dhcp6 client for interface wan(em0)
Mar 16 12:24:42 egis rtsold[21498]: Received RA specifying route for interface wan(em0)
Mar 16 12:24:42 egis rtsold[21642]: RTSOLD Lock in place - sending SIGHUP to dhcp6c
Mar 16 12:45:22 egis php[409]: rc.bootup: Starting rtsold process
Mar 16 12:45:24 egis php-fpm[348]: /rc.newwanip: Starting rtsold process
Mar 16 12:45:24 egis rtsold[48543]: <rtsock_input_ifannounce> interface tun2 removed
Mar 16 12:45:25 egis rtsold[48543]: <rtsock_input_ifannounce> interface tun3 removed
Mar 16 12:45:25 egis rtsold[48543]: <rtsock_input_ifannounce> interface tun4 removed
Mar 16 12:45:25 egis rtsold[40425]: Starting dhcp6 client for interface wan(em0)
Mar 16 12:45:28 egis rtsold[16284]: RTSOLD Lock in place - sending SIGHUP to dhcp6c

jimp

That's a different issue than the one I've seen then, as yours is never getting a gateway via rtsold for some reason.

The issue I hit (on several systems) was that the gateway was received but then subsequently removed by later runs of the script.

I don't see any indication in the output you posted that it ever sees a gateway value though.

I'm not sure how that might have worked before unless something about the request/response is different now vs before.

Is yours DHCP6 on a plain Ethernet Interface or is it on top of something else like PPPoE?

g.shaffer

@jimp

Plain old ethernet, no PPPoE.

Here are my settings for the WAN DHCP6 config:

Screen Shot 2021-03-16 at 10.07.29 AM.png

What other debug/output can I get you?

jimp

First, uncheck all those boxes and save/apply, then reboot and see if the behavior changes.

At a minimum you won't need that first box as that's just for PPPoE weirdness.

About the only two you might need are the prefix hint and release option but those may not affect it either.

It's entirely possible it's a difference in your upstream as well, there was an rtsold.c patch on the Redmine issue which may be related to your case but unrelated to many of the others affected. We might be getting two different bugs with similar results confused here.

g.shaffer

@jimp

Unchecked all of the boxes and rebooted, the IPv6 behavior did not change.

My WAN IPv6 address changed, my PD did not. Neither the /tmp/em0_routerv6 or the /tmp/em0_defaultv6 exist. The IPv6 default route exists in the routing table.

mloiterman

I think I'm seeing the same issue and almost any change to WAN interface triggers this error:

 There were error(s) loading the rules: /tmp/rules.debug:311: no routing address with matching address family found. - The line in question reads [311]: pass in quick on $LAN1_PRIMARY $GWDefaultIPv6 inet6 from 2601:xxx:xxx:xxx::/64 to any tracker 1436634070 keep state label &quot;USER_RULE: LAN1 -&gt; any Default IPv6&quot;
@ 2021-03-16 16:30:47

On reboot, the IPv6 gateway appears up, both WAN and LAN get IPv6 address, but there is no routing of IPv6 traffic. pfSense can ping IPv6 hosts, but no LAN side clients can.

Dashboard shows Pending, Pending, Pending, Unknown for the Gateway,

If I mark the IPv6 gateway as down, Apply, and then Mark it is Up I can get it to route IPv6 traffic correctly. Until I reboot and then it reverts to not routing IPv6.

jimp

Copying some info back here from Redmine for good measure:

There were two components that we found and fixed:

A gateway learned via RA (without M or O) could be clobbered by later invocations of the script that lacked gateway information
The script when invoked from M flags needed the gateway parameter added to pass the gateway through to the script processing the gateway data

The fix for #1 is in current RC snapshots, the fix for #2 was just committed and will make it into builds overnight.

I've been testing a patched rtsold with the fix for #2 and have been able to confirm it works for me when RA is set to M+O and it did not work without the patched binary. I replicated the conditions and tested the fix on three separate lab systems.

Hopefully this fixes it for you and others experiencing one or both of these problems.

jimp

FYI- This fix is in the current builds and is working for me as mentioned above.

bobbenheim

It did fix the gateway problem, but now i don't have any states registered on Floating, LAN and WAN rules despite having traffic from WAN to LAN. The rules on WAN and LAN still seems to work as i can get online and the port forward i have seems to work, but fq_codel set up with floating pass rules doesn't work after upgrade to 2.6.0.a.20210318.0100. Anyone else seeing the same?

mloiterman

@jimp Thanks, Jim. As always, you’re awesome.

jimp

@bobbenheim said in IPv6 Gateway Issue (see issue 11454):

It did fix the gateway problem, but now i don't have any states registered on Floating, LAN and WAN rules despite having traffic from WAN to LAN. The rules on WAN and LAN still seems to work as i can get online and the port forward i have seems to work, but fq_codel set up with floating pass rules doesn't work after upgrade to 2.6.0.a.20210318.0100. Anyone else seeing the same?

No, please start your own thread with more information in the 2.6.0 category, this one is for 2.5.1/21.02.2 and any problems on 2.6.0 are likely unrelated to this.

g.shaffer

@jimp
It has fixed the IPv6 Gateway issue, Thanks! But, I'm now seeing issues with Track Interface, I'll open up another thread.