No internet connection after upgrade from April 20 snapshot to May 19

w0w

@piba
Thats fun! This comes on system boot BEFORE PPPoE connection is established :

Jun 10 19:49:56 	php-cgi 		rc.bootup: gateways_arr:Array ( [WAN_PPPOE] => Array ( [dynamic] => [ipprotocol] => inet [gateway] => [interface] => pppoe0 [friendlyiface] => wan [name] => WAN_PPPOE [attribute] => system [descr] => Interface WAN_PPPOE Gateway ) [Null4] => Array ( [name] => Null4 [interface] => lo0 [ipprotocol] => inet [gateway] => 127.0.0.1 ) [Null6] => Array ( [name] => Null6 [interface] => lo0 [ipprotocol] => inet6 [gateway] => ::1 ) )
Jun 10 19:49:56 	php-cgi 		rc.bootup: fixup_default_gateway dfltgwdown: upgw: dfltgwname: , gwdefault:
Jun 10 19:49:56 	php-cgi 		rc.bootup: gateways_arr:Array ( [WAN_PPPOE] => Array ( [dynamic] => [ipprotocol] => inet [gateway] => [interface] => pppoe0 [friendlyiface] => wan [name] => WAN_PPPOE [attribute] => system [descr] => Interface WAN_PPPOE Gateway ) [Null4] => Array ( [name] => Null4 [interface] => lo0 [ipprotocol] => inet [gateway] => 127.0.0.1 ) [Null6] => Array ( [name] => Null6 [interface] => lo0 [ipprotocol] => inet6 [gateway] => ::1 ) )

PPPoE is established 2 seconds later just start of session!

Jun 10 19:49:58 	ppp 		[wan_link0] PPPoE: connection successful

And this coming when I manually re-connect PPPoE

Jun 10 19:53:28 	php-fpm 	335 	/rc.newwanip: gateways_arr:Array ( [WAN_PPPOE] => Array ( [dynamic] => 1 [ipprotocol] => inet [gateway] => 212.7.29.236 [interface] => pppoe0 [friendlyiface] => wan [name] => WAN_PPPOE [attribute] => system [monitor] => 212.7.29.236 [descr] => Interface WAN_PPPOE Gateway ) [Null4] => Array ( [name] => Null4 [interface] => lo0 [ipprotocol] => inet [gateway] => 127.0.0.1 ) [Null6] => Array ( [name] => Null6 [interface] => lo0 [ipprotocol] => inet6 [gateway] => ::1 ) )
Jun 10 19:53:28 	php-fpm 	335 	/rc.newwanip: fixup_default_gateway dfltgwdown: upgw: dfltgwname: , gwdefault:
Jun 10 19:53:28 	php-fpm 	335 	/rc.newwanip: Default gateway setting Interface WAN_PPPOE Gateway as default.
Jun 10 19:53:28 	php-fpm 	335 	/rc.newwanip: gateways_arr:Array ( [WAN_PPPOE] => Array ( [dynamic] => 1 [ipprotocol] => inet [gateway] => 212.7.29.236 [interface] => pppoe0 [friendlyiface] => wan [name] => WAN_PPPOE [attribute] => system [monitor] => 212.7.29.236 [descr] => Interface WAN_PPPOE Gateway ) [Null4] => Array ( [name] => Null4 [interface] => lo0 [ipprotocol] => inet [gateway] => 127.0.0.1 ) [Null6] => Array ( [name] => Null6 [interface] => lo0 [ipprotocol] => inet6 [gateway] => ::1 ) )

PPPoE session is established completely 1 second before:

Jun 10 19:53:27 	ppp 		[wan] IFACE: Rename interface ng0 to pppoe0

PiBa

I suppose between the 'Rename interface ng0 to pppoe0' and the manual re-connect you do have these 2 lines?:

Jun 10 19:49:13	php-fpm	322	/rc.newwanip: rc.newwanip: on (IP address: 10.30.20.163) (interface: WAN[wan]) (real interface: pppoe0).
Jun 10 19:49:13	php-fpm	322	/rc.newwanip: rc.newwanip: Info: starting on pppoe0.

And then the "rc.newwanip" just stops running .? While it should log a few more lines and set the gateway like in previous posts.? I think there is a race condition there where the script exit's during bootup...

w0w

@piba
Here is more:

PiBa

Yep thats the 2 lines right there..

Jun 10 19:50:00 	php-fpm 	334 	/rc.newwanip: rc.newwanip: on (IP address: 84.52.**.200) (interface: WAN[wan]) (real interface: pppoe0).
Jun 10 19:50:00 	php-fpm 	334 	/rc.newwanip: rc.newwanip: Info: starting on pppoe0.

So indeed in your case it is 'aborting' that script before its done running.
Seems like this 'exit' is supposed to be avoiding a race condition where its actually causing one...:
https://github.com/pfsense/pfsense/blob/d84eec807d6216cfbc073438ce57e01f1c52a2f4/src/etc/rc.newwanip#L192

@jimp any insights on what would be the right order to call things?

initialize PPPOE-client while booting
initialize routes and packages and rules and stuff after PPPOE gets connected
finish the rc.newwanip to configure routes and other things that depends on the newly assigned wan-ip..
finish booting..

I think it was previously working because every call to return_gateway_groups_array() would again try and fix the gateways.. and eventually one did catch.. Now the return_gateway_groups_array() most of the time skips these actions as fixing the gateway should not happen as a 'side-effect' of 'requesting information'.. But that does seem to cause this problem..

PiBa

Perhaps the fix could be as simple as removing that 'exit' from the rc.newwanip .? (in addition to the PR made)

w0w

@piba
I've deleted this line and this fixes the problem with connection on boot, really.

PiBa

@w0w
I've discussed with jimp, and he didn't like removing the exit completely to avoid bringing the old previous race condition back. we agreed upon a specific 'ppp check' to avoid the exit. Can you put the exit back in, but apply this extra check instead.? That should still fix the issue.. does it.? :

https://github.com/pfsense/pfsense/pull/3947/commits/627b0941889f4b19ad419ddfe01d329bef2c2bd6

w0w

@piba
Looks like with BOTH (gwlib.inc and rc.newwanip) fixes applied — I now have internet connection. One thing is still need to be fixed — after upgrade from previous version and after applying those patches I need also to select default gateway in drop-down list on system_gateways.php page.
If I restore old configuration on patched system the same thing happens also, I need manually select gateway on system_gateways.php via GUI, is it feature or bug?

PiBa

Yes the first link https://github.com/pfsense/pfsense/pull/3947 contains 3 fixes/commits, the 2 you refer to, and one that fixes the displaying of the IPv6 for a IPv4 gateway..

The gui 'none' selection is is still something to fix. Lets call that a 'unwanted feature' ;) . Ill probably work on that soon..

w0w

@piba
Thank you, let me know if you need further testing.

PiBa

@w0w
Thank you for testing and reporting back each time :).

Can you try latest snapshot? Its build with these (slightly modified) fixes included.

p.s. not the 'none' selection yet though, still working on that.

w0w

@piba
I have tried this one:

2.4.4-DEVELOPMENT (amd64)
built on Sat Jun 23 01:57:55 EDT 2018
FreeBSD 11.2-RC3

I've found that internet not working with this snapshot on reboot, but both fixes are present.
I think it's modified line that was looked like

if (platform_booting() && strpos($interface_real, "ppp") !== 0) {

before,
and now it's

if (platform_booting() && in_array(substr($interface_real, 0, 3), array("ppp", "ppt", "l2t"))) {

because if I change it back, it works again.

PiBa

@w0w
Ah crap, i forgot to do the inverse, '!'.

if (platform_booting() && !in_array(substr($interface_real, 0, 3), array("ppp", "ppt", "l2t"))) {

sorry, thanks for testing!

new PR made: https://github.com/pfsense/pfsense/pull/3956

w0w

@piba This one is much better
It works!

w0w

As expected, latest snapshot containing this PR boots normally.

PiBa

@w0w
Thanks, added your confirmation to the redmine ticked so it can be closed.

w0w

@piba
Please let me know when you need testing gui 'none' selection fix

PiBa

@w0w
The commit was merged, but not build yet. next snapshot should contain a new 'automatic' option which would be the default for the gateway selection, also a option to move gateways up/down on the list with checkboxes and anchor click. The option 'none' should now also actually be none.. I guess all of those things can use a little testing once it gets build ..

w0w

@piba
Here is my test result. If I select "Automatic" option in GUI then everything looks OK, I have internet and default gateway, but when I try reboot the firewall, on next boot there is no internet and default gateway anymore, sounds familiar?
This easy to fix, just pressing "save" on system_gateways.php and "Apply changes", it immediately starts working and I see this line in log:

/system_gateways.php: Default gateway setting Interface WAN_PPPOE Gateway as default.

There is no such/similar line during boot, so automatic selection does not trig for some reason.
How can I help to debug this?

PiBa

@w0w
Thanks again for testing. We seem to have hit catch22 issue here.

'Automatic' selection takes the gateway 'status' into account which depends on dpinger, which depends on routes to the monitor-targets being configured before hand so to the monitor target will be using the correct gateway, which while configuring routes it also configures the default route which depends on the dpinger already running..

Hmm... This needs a bit more thought..