pfSense 2.4 wont reboot after interface removal - route cannot be added, network unreachable

mix_room

I am cross posting this from reddit hoping that people here might help me.

After removing a machine from my network, I disabled the interface it was connected to. I unassigned the interface from the web-UI. The network card is still physically attached.
When I reboot the firewall it does not come up properly.

During interface creation stage, in particular "Configuring IPsec VTI interfaces ..." step, it fails with the error message "route: writing to routing socket: Network is unreachable"

Pressing Ctrl-C and dropping to the default shell I can assign an IP to WAN device (igb0) ( ifconfig igb0 ... ) which is statically assigned. Pressing Ctrl-D to continue boot, the entire boot process now works without issue. Firewall is configured correctly.

Before I set the IP I cannot assign a default route, with the same error message about the network being unreachable. It seems to me that the routes are being set too early. But I do not understand why this has started happening after I unassigned an interface.

I have tried to restore the interface portion of an old backup, but that does not change the symptoms.

Has anyone experienced something similiar, or have any suggestions to remedy the problem?

tl;dr: Inactivated and unassigned an interface in webui. On reboot I get failures with the message "route: writing to routing socket: Network is unreachable"

kiokoman

@mix_room said in pfSense 2.4 wont reboot after interface removal - route cannot be added, network unreachable:

Configuring IPsec VTI interfaces

function interfaces_ipsec_vti_configure() {
        global $config;
        if (platform_booting()) {
                echo gettext("Configuring IPsec VTI interfaces...");
        }
        if (is_array($config['ipsec']) && is_array($config['ipsec']['phase1']) && is_array($config['ipsec']['phase2'])) {
                foreach ($config['ipsec']['phase1'] as $ph1ent) {
                        if ($ph1ent['disabled']) {
                                continue;
                        }
                        interface_ipsec_vti_configure($ph1ent);
                }
        }
        if (platform_booting()) {
                echo gettext("done.") . "\n";
        }
}

can you try to disable ipsec and restart to see if the problem persist ?
i don't think that removing an interface have anything to do with your problem.
this function check for enabled/disabled ipsec on your config

if it is enabled function interface_ipsec_vti_configure read your config and create the tunnel for the ipsec

probably something went wrong on your config. you can also try to delete the ipsec configuration / reboot / recreate the ipsec / reboot

could it be that on phase2 you have the old ip from that interface you removed?

stephenw10

By far the most likely issue there is that you have something still configured to run on the interface you removed, I would guess an IPSec tunnel. I would search your config for references to either the removed interface or the subnet.

Steve

mix_room

I removed all Phase 2 entries that might have been affected. The only ones that were present were disabled, but they were removed nonetheless.
That didn't make a difference.

I started looking around where else there might have been an issue. Next two were ~~Unbound~~ DNSmasq and NTP portions of the XML. These two were supposed to bind to the inactivated interface. Now it boots, but there are still error messages in the start-up log.

stephenw10

Can we see those errors?

mix_room

See for example the last bootup log:

Sep 8 14:13:03	php-cgi		rc.bootup: Default gateway setting Interface wan Gateway as default.
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec5000' inet tunnel '85.XX.XX.XX' 'vpn.example2.com' up' returned exit code '1', the output was 'ifconfig: error in parsing address string: hostname nor servname provided, or not known'
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec5000' destroy' returned exit code '1', the output was 'ifconfig: interface ipsec5000 does not exist'
Sep 8 14:13:03	php-cgi		rc.bootup: Default gateway setting Interface wan Gateway as default.
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec4000' inet tunnel '85.XX.XX.XX' 'vpn.example.com' up' returned exit code '1', the output was 'ifconfig: error in parsing address string: hostname nor servname provided, or not known'
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec4000' destroy' returned exit code '1', the output was 'ifconfig: interface ipsec4000 does not exist'
Sep 8 14:13:03	php-cgi		rc.bootup: Default gateway setting Interface wan Gateway as default.
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec3000' inet tunnel '85.XX.XX.XX' 'vpn.example3.com' up' returned exit code '1', the output was 'ifconfig: error in parsing address string: hostname nor servname provided, or not known'
**Sep 8 14:13:03	kernel		route: writing to routing socket: Network is unreachable**
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec3000' destroy' returned exit code '1', the output was 'ifconfig: interface ipsec3000 does not exist'
Sep 8 14:13:03	php-cgi		rc.bootup: Default gateway setting Interface wan Gateway as default.
Sep 8 14:13:03	kernel		vlan2: changing name to 'igb1.3646'
Sep 8 14:13:03	php-cgi		rc.bootup: The command '/sbin/ifconfig 'ipsec2000' destroy' returned exit code '1', the output was 'ifconfig: interface ipsec2000 does not exist'
Sep 8 14:13:02	kernel		vlan1: changing name to 'igb7.466'
Sep 8 14:13:02	kernel		vlan0: changing name to 'igb7.1052'
Sep 8 14:13:02	kernel		aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard

The rest of the messages do not seem to make it into the log, and I do not feel like manually transcribing a video of the boot process.

mix_room

The ** are not actually in the log, but I get flagged as SPAM when trying to post the log.

kiokoman

are that ipsec configured or pfSense is "remembering" some old configuration ?

stephenw10

Mmm, you removed all the P2s but are any of those P1s on the removed interface?

I assume vpn.example3.com etc are substitutes for the real fqdns there? Are the real values resolvable?

Steve

mix_room

@kiokoman said in pfSense 2.4 wont reboot after interface removal - route cannot be added, network unreachable:

are that ipsec configured or pfSense is "remembering" some old configuration ?

They are configured. I have about 8 tunnels that are configured and active.

@stephenw10 said in pfSense 2.4 wont reboot after interface removal - route cannot be added, network unreachable:

Mmm, you removed all the P2s but are any of those P1s on the removed interface?

I assume vpn.example3.com etc are substitutes for the real fqdns there? Are the real values resolvable?

The removed interface was internal, there were no P1s configured on it. I checked the full config XML just to be sure.
Yes. They are substitutes. The real values are normally resolvable, they just do not seem to be resolvable during this step.

stephenw10

Do those tunnels come up correctly after bootup is complete? If so that might just have to be considered log spam if those fqdns actually need to be resolved.
Otherwise you could just use IPs there or maybe add static entries for them.

Steve

mix_room

@stephenw10 said in pfSense 2.4 wont reboot after interface removal - route cannot be added, network unreachable:

Do those tunnels come up correctly after bootup is complete? If so that might just have to be considered log spam if those fqdns actually need to be resolved.
Otherwise you could just use IPs there or maybe add static entries for them.

The endpoints are on dynamic IPs, hence the need for hostnames. They resolve to dynamic ones.
The tunnels come up, I would consider it log spam.