Wireguard client Gateway disabled after reboot - service not starting

rpotter28 · Oct 16, 2023, 10:41 PM

This is still happening on 23.09 beta, as posted here:

https://forum.netgate.com/topic/177255/wireguard-site-to-site-gateways-disabled-after-reboot-service-not-starting

I followed all that at the time on 23.05, and the common thread is PPPoE WAN which I have. Solution has been to restart dpinger and wireguard after reboot, either manually or with cron.

This is not critical for me, as wireguard is only used for a SID so my tenants can circumvent geo-restrictions on some streaming service.

Just wanted to report it for the record!

stephenw10 · Oct 23, 2023, 4:04 PM

Hmm, can you see in the boot log what order those things are starting in? Something are disabled during boot so I could believe, for example, PPPoE comes up after WG has tried to start and failed but it doesn't re-start WG because boot hasn't completed.

Does it start correctly if you reconnect PPPoE after boot?

Steve

rpotter28 · Oct 23, 2023, 6:41 PM

@stephenw10 Sorry, but I lost patience with wireguard and uninstalled it. Switched to OpenVPN client with DCO and all is good.

More reboots than usual because of the Beta track, is what did me in!
_
Richard

4o4rh · Oct 24, 2023, 2:28 PM

@stephenw10 i added wireguard to the service watchdog which seems to have fixed it for me. so should be an order of execution or a delay needed for pppoe to come up first.

rpotter28 · Oct 27, 2023, 9:53 PM

@stephenw10 Steve, I felt bad that I didn't have wireguard installed when you replied, so I changed back from OpenVPN yesterday.

Today, after the upgrade to RC built on Thu Oct 26 21:51:00 EDT 2023, Wireguard started right up, no issues. Has this been worked on, the pppoe part?

Here are the log snippets, which shows wg0 going up after pppoe:

Oct 27 13:12:42 php-cgi 504 rc.bootup: Default gateway setting WAN Gateway PPPoE as default.
Oct 27 13:12:41 php-fpm 408 /rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: 0_WAN[wan]) (real interface: pppoe0).
Oct 27 13:12:41 php-fpm 408 /rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Oct 27 13:12:40 ppp 75227 [wan] IFACE: Rename interface ng0 to pppoe0
Oct 27 13:12:40 check_reload_status 448 rc.newwanip starting pppoe0
Oct 27 13:12:36 ppp 75227 [wan_link0] PPPoE: connection successful
Oct 27 13:12:36 ppp 75227 PPPoE: rec'd ACNAME "KGTNON0881W"
Oct 27 13:12:36 ppp 75227 [wan_link0] PPPoE: Connecting to ''
Oct 27 13:12:36 kernel ng0: changing name to 'pppoe0

Oct 27 13:13:00 kernel tun_wg0: link state changed to UP
Oct 27 13:13:00 kernel wg0: changing name to 'tun_wg0'
Oct 27 13:12:58 kernel tun_wg0: link state changed to DOWN
Oct 27 13:12:35 kernel tun_wg0: link state changed to UP
Oct 27 13:12:35 kernel wg0: changing name to 'tun_wg0'
Oct 27 13:11:38 kernel tun_wg0: link state changed to DOWN

Thanks!
Richard

stephenw10 · Oct 27, 2023, 9:58 PM

Yes there have been some changes to the system aliases used there. I wonder if it was trying to start with something undefined.

Thanks for the feedback!

rpotter28 · Oct 27, 2023, 10:56 PM

@stephenw10 Thank you for seeing my initial post and giving it some traction. There has been only this one reboot. I am old school, never invite trouble by rebooting something that's working.

I will follow up if anything changes after the next reboot, possibly another RC update?
Thanks!

stephenw10 · Oct 27, 2023, 11:16 PM

Unless we find something show-stopping this will be the last build before release.

rpotter28 · Oct 27, 2023, 11:55 PM

@stephenw10 That makes sense, thanks. This would not be a show stopper anyways, and it looks good IMHO.
Thanks for your efforts!

rpotter28 · Oct 30, 2023, 4:44 PM

@stephenw10 I have been running the BETA as a vm on Hyper-v, but this morning I had the opportunity to switch over to a bare metal white box, to get it up to RC.

Guess what, wireguard didn't start. There is a difference in the logs:

Oct 30 10:46:03 php-fpm 411 /rc.newwanip: Default gateway setting WAN Gateway PPPoE as default.
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: 0_WAN[wan]) (real interface: pppoe0).
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Oct 30 10:46:01 ppp 40734 [wan] IFACE: Rename interface ng0 to pppoe0
Oct 30 10:46:01 check_reload_status 443 rc.newwanip starting pppoe0
Oct 30 10:45:57 ppp 40734 [wan_link0] PPPoE: connection successful
Oct 30 10:45:57 ppp 40734 PPPoE: rec'd ACNAME "KGTNON0881W"
Oct 30 10:45:55 ppp 40734 [wan_link0] PPPoE: Connecting to ''
Oct 30 10:45:55 kernel ng0: changing name to 'pppoe0'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 -ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 fe80::baca:3aff:fe8d:70b2%em0.35 delete' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist

Do you have any idea where those first three rc.bootup lines are coming from? They are not there on the vm, and I don't have ipv6 enabled on any interfaces. I think I have same configs on both, but apparently not.

stephenw10 · Oct 30, 2023, 4:59 PM

Hmm, looks like it's disabling the interface in order to remove a V6 address but failing because pppoe0 doesn't exist yet.

Is em0.35 the VLAN pppoe0 is on?

What is shown just before those lines?

rpotter28 · Oct 30, 2023, 5:08 PM

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

Is em0.35 the VLAN pppoe0 is on?

Yes,

I will log at the logs,

rpotter28 · Oct 30, 2023, 5:24 PM

@stephenw10 This box has a 10GB LAGG, ix0 and ix1, with 7 vlans and 2 wg tunnels. I rebooted, so different log here.

In the logs I also see this: which is vlan 90 and I have no ipv6 config on any interfaces.

php-cgi 477 rc.bootup: The command '/sbin/ifconfig 'lagg0.90' inet6 delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'

And this:
Oct 30 12:58:46 kernel lagg0: IPv6 addresses on ix1 have been removed before adding it as a member to prevent IPv6 address scope violation.

Oct 30 12:58:45 kernel lagg0: IPv6 addresses on ix0 have been removed before adding it as a member to prevent IPv6 address scope violation.

All my vlans and interfaces are working as normal, just wireguard doesn't start on boot. Disables the gateways,

stephenw10 · Oct 30, 2023, 6:31 PM

Hmm, maybe unrelated then.

I see similar lines for interfaces of that type where no IPv6 address is defined:

Oct 30 17:38:13 	kernel 		vlan0: changing name to 'lagg0.100'
Oct 30 17:38:13 	kernel 		lagg0: IPv6 addresses on igc1 have been removed before adding it as a member to prevent IPv6 address scope violation.
Oct 30 17:38:13 	php-cgi 	575 	rc.bootup: The command '/sbin/ifconfig 'lagg0.100' inet6 delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'

That shouldn't itself be an issue.

rpotter28 · Nov 2, 2023, 9:49 PM

@stephenw10 OK thanks.

So, I am back to these 3 lines (in bold) do not show-up on the vm, but they do on the bare metal:

Oct 30 10:46:03 php-fpm 411 /rc.newwanip: Default gateway setting WAN Gateway PPPoE as default.
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: 0_WAN[wan]) (real interface: pppoe0).
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Oct 30 10:46:01 ppp 40734 [wan] IFACE: Rename interface ng0 to pppoe0
Oct 30 10:46:01 check_reload_status 443 rc.newwanip starting pppoe0
Oct 30 10:45:57 ppp 40734 [wan_link0] PPPoE: connection successful
Oct 30 10:45:57 ppp 40734 PPPoE: rec'd ACNAME "KGTNON0881W"
Oct 30 10:45:55 ppp 40734 [wan_link0] PPPoE: Connecting to ''
Oct 30 10:45:55 kernel ng0: changing name to 'pppoe0'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 -ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 fe80::baca:3aff:fe8d:70b2%em0.35 delete' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist

And I have no idea why that is :-)

stephenw10 · Oct 30, 2023, 8:58 PM

Do you have a bridge configured on the bare metal box only?

rpotter28 · Oct 30, 2023, 9:03 PM

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

Do you have a bridge configured on the bare metal box only?

I am not bridging... It's a LACP LAGG.
And no, I don't have a LAGG on the vm, no need to.

stephenw10 · Oct 30, 2023, 9:26 PM

Hmm, could be the lagg. That message is the system removing v6 addresses so they don't appear in the same layer 2. That could be a bridge or I guess a lagg. pfSense doesn't allow that for lagg interfaces but in FreeBSD it could be an issue. Either way that shouldn't be an issue.
However I'm not sure why that would be trying to do it to a PPPoE interface. I imagine it may have inherited that from the interface it's on in some way Is em0 is use for something else?

rpotter28 · Oct 30, 2023, 9:36 PM

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

em0 is use for something else?

No sir, em0 just has vlan 35 for the pppoe connection. ISP requirement.

All the vlans which includes the LAN are on the LAGG.

This is similar to the vm, which also has 2 interfaces. WAN-vlan35-pppoe on one, and the other trunked for all vlans. Not a lagg, just one trunked hyper-v virtual nic.

rpotter28 · Nov 1, 2023, 2:43 AM

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

However I'm not sure why that would be trying to do it to a PPPoE interface

I have given up, spent too much time on this. I am very sure I tripled checked everything, comparing the working vm to the bare metal settings in the GUI. I can find no rhyme or reason why.

Following that thought, I analyzed and diffed the configs, still nothing stands out. So one has to conclude the issue in my bare metal install. My problem nobody else has I guess, but I can't find it.

I thought of eliminating the em0 interface and just do it all on the LAGG, but that doesn't really make sense either for this issue?

And I can't reinstall to test because it's on a HL licence. However, the vm is working perfectly, after 5 reboots now :-) So that proves it does work, and I am embarrassed that I can't make it work on my bare metal install.

Richard