Wireguard client Gateway disabled after reboot - service not starting

[stephenw10]

Unless we find something show-stopping this will be the last build before release.

[rpotter28]

@stephenw10 That makes sense, thanks. This would not be a show stopper anyways, and it looks good IMHO.
Thanks for your efforts!

[rpotter28]

@stephenw10 I have been running the BETA as a vm on Hyper-v, but this morning I had the opportunity to switch over to a bare metal white box, to get it up to RC.

Guess what, wireguard didn't start. There is a difference in the logs:

Oct 30 10:46:03 php-fpm 411 /rc.newwanip: Default gateway setting WAN Gateway PPPoE as default.
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: 0_WAN[wan]) (real interface: pppoe0).
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Oct 30 10:46:01 ppp 40734 [wan] IFACE: Rename interface ng0 to pppoe0
Oct 30 10:46:01 check_reload_status 443 rc.newwanip starting pppoe0
Oct 30 10:45:57 ppp 40734 [wan_link0] PPPoE: connection successful
Oct 30 10:45:57 ppp 40734 PPPoE: rec'd ACNAME "KGTNON0881W"
Oct 30 10:45:55 ppp 40734 [wan_link0] PPPoE: Connecting to ''
Oct 30 10:45:55 kernel ng0: changing name to 'pppoe0'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 -ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 fe80::baca:3aff:fe8d:70b2%em0.35 delete' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist

Do you have any idea where those first three rc.bootup lines are coming from? They are not there on the vm, and I don't have ipv6 enabled on any interfaces. I think I have same configs on both, but apparently not.

[stephenw10]

Hmm, looks like it's disabling the interface in order to remove a V6 address but failing because pppoe0 doesn't exist yet.

Is em0.35 the VLAN pppoe0 is on?

What is shown just before those lines?

[rpotter28]

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

Is em0.35 the VLAN pppoe0 is on?

Yes,

I will log at the logs,

[rpotter28]

@stephenw10 This box has a 10GB LAGG, ix0 and ix1, with 7 vlans and 2 wg tunnels. I rebooted, so different log here.

In the logs I also see this: which is vlan 90 and I have no ipv6 config on any interfaces.

php-cgi 477 rc.bootup: The command '/sbin/ifconfig 'lagg0.90' inet6 delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'

And this:
Oct 30 12:58:46 kernel lagg0: IPv6 addresses on ix1 have been removed before adding it as a member to prevent IPv6 address scope violation.

Oct 30 12:58:45 kernel lagg0: IPv6 addresses on ix0 have been removed before adding it as a member to prevent IPv6 address scope violation.

All my vlans and interfaces are working as normal, just wireguard doesn't start on boot. Disables the gateways,

[stephenw10]

Hmm, maybe unrelated then.

I see similar lines for interfaces of that type where no IPv6 address is defined:

Oct 30 17:38:13 	kernel 		vlan0: changing name to 'lagg0.100'
Oct 30 17:38:13 	kernel 		lagg0: IPv6 addresses on igc1 have been removed before adding it as a member to prevent IPv6 address scope violation.
Oct 30 17:38:13 	php-cgi 	575 	rc.bootup: The command '/sbin/ifconfig 'lagg0.100' inet6 delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'

That shouldn't itself be an issue.

[rpotter28]

@stephenw10 OK thanks.

So, I am back to these 3 lines (in bold) do not show-up on the vm, but they do on the bare metal:

Oct 30 10:46:03 php-fpm 411 /rc.newwanip: Default gateway setting WAN Gateway PPPoE as default.
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: 0_WAN[wan]) (real interface: pppoe0).
Oct 30 10:46:02 php-fpm 411 /rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Oct 30 10:46:01 ppp 40734 [wan] IFACE: Rename interface ng0 to pppoe0
Oct 30 10:46:01 check_reload_status 443 rc.newwanip starting pppoe0
Oct 30 10:45:57 ppp 40734 [wan_link0] PPPoE: connection successful
Oct 30 10:45:57 ppp 40734 PPPoE: rec'd ACNAME "KGTNON0881W"
Oct 30 10:45:55 ppp 40734 [wan_link0] PPPoE: Connecting to ''
Oct 30 10:45:55 kernel ng0: changing name to 'pppoe0'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 -ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 fe80::baca:3aff:fe8d:70b2%em0.35 delete' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist

And I have no idea why that is :-)

[stephenw10]

Do you have a bridge configured on the bare metal box only?

[rpotter28]

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

Do you have a bridge configured on the bare metal box only?

I am not bridging... It's a LACP LAGG.
And no, I don't have a LAGG on the vm, no need to.

[stephenw10]

Hmm, could be the lagg. That message is the system removing v6 addresses so they don't appear in the same layer 2. That could be a bridge or I guess a lagg. pfSense doesn't allow that for lagg interfaces but in FreeBSD it could be an issue. Either way that shouldn't be an issue.
However I'm not sure why that would be trying to do it to a PPPoE interface. I imagine it may have inherited that from the interface it's on in some way Is em0 is use for something else?

[rpotter28]

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

em0 is use for something else?

No sir, em0 just has vlan 35 for the pppoe connection. ISP requirement.

All the vlans which includes the LAN are on the LAGG.

This is similar to the vm, which also has 2 interfaces. WAN-vlan35-pppoe on one, and the other trunked for all vlans. Not a lagg, just one trunked hyper-v virtual nic.

[rpotter28]

@stephenw10 said in Wireguard client Gateway disabled after reboot - service not starting:

However I'm not sure why that would be trying to do it to a PPPoE interface

I have given up, spent too much time on this. I am very sure I tripled checked everything, comparing the working vm to the bare metal settings in the GUI. I can find no rhyme or reason why.

Following that thought, I analyzed and diffed the configs, still nothing stands out. So one has to conclude the issue in my bare metal install. My problem nobody else has I guess, but I can't find it.

I thought of eliminating the em0 interface and just do it all on the LAGG, but that doesn't really make sense either for this issue?

And I can't reinstall to test because it's on a HL licence. However, the vm is working perfectly, after 5 reboots now :-) So that proves it does work, and I am embarrassed that I can't make it work on my bare metal install.

Richard

[rpotter28]

Today I removed vlan 35 from pfSense, and tagged it on the switch with a port pvid 35 back to pfSense. That got rid of the below errors, but wg gateways were still disabled. I find that interesting, but it did not fix the problem.

Restarting dpinger with cron at boot does fix it, as posted in a thread here somewhere.
@reboot root sleep 30 && /usr/local/sbin/pfSsh.php playback svc restart dpinger

So, I am back to these 3 lines (in bold) do not show-up on the vm, but they do on the bare metal:
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 -ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 fe80::baca:3aff:fe8d:70b2%em0.35 delete' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist'
Oct 30 10:45:55 php-cgi 478 rc.bootup: The command '/sbin/ifconfig 'pppoe0' inet6 ifdisabled' returned exit code '1', the output was 'ifconfig: interface pppoe0 does not exist