OpenVPN server - how to fix apparent race condition with cable modem startup?

mrneutron

My system consists of an Arris cable modem connected to Comcast, followed by a Netgate SG-1100 pfsense router.
SG-1100 firmware: 22.01-RELEASE (arm64)
OpenVPN version: openvpn-2.5.4_1

While I was traveling a week ago, I was unable to establish an OpenVPN connection with the OpenVPN server running inside my Netgate SG-1100 pfsense router. I tried with the OpenVPN client on my Android phone and on my Windows 10 laptop.
I asked my wife to power-cycle the pfsense router (and the cable modem at the same time), but I still could not connect.
In the past I found that they needed to be power cycled at the same time in order for the SG-1100 to get assigned a Comcast IP address.

When I returned home, I looked into the pfsense router OpenVPN status screen and it had a green checkmark for a status but I could still not log into it via my phone (with the wifi turned off so it would force a connection through the cell phone network rather than the home wifi.

Then, I looked into the OpenVPN system log and I saw that when the OpenVPN server started up after the power cycle of the pfsense router & cable modem that my wife performed while I was traveling, it was listening for connections on the cable modem's local IP address of 192.168.100.10, rather than the Comcast external IP address, which has been 73.5.x.x for the last year.

Next, I clicked the restart button (circular arrow button) in the pfsense OpenVPN screen, and I was instantly able to connect to the OpenVPN server via my cell phone.
With the system working, the OpenVPN log showed that the OpenVPN server was listening for connections on the Comcast IP address 73.5.x.x.

It looks like this IP address problem started right after a power failure we had on May 31, 2022! When power was restored, I guess the OpenVPN server in the pfsense router started up before the cable modem made connection with Comcast and got an external IP?

I think the evidence shows there is a startup race condition of the cable modem and the OpenVPN server?
I believe the cable modem needs to come up first, so there is an active connection to Comcast before the OpenVPN server starts up?

What is the best method of working around this apparent race condition in the future?

delay the start of the OpenVPN server for several minutes following a boot-up?
forcing a restart of the OpenVPN server several minutes after boot-up?

johnpoz

@mrneutron couple suggestions.. 1st would be to put your cable modem and even pfsense box on ups.. So little power outages (shorter than your runtime on your ups).. Don't cause issues like this. Never a good idea to just have pfsense loose power either way.

2nd option, while personally have never ran into it - is set advanced options in wan dhcp client not to accept dhcp from 192.168.100.1 - this is common IP for cable modems for web gui, and they will hand out IPs in that range when they don't have a internet connection.

I do believe there are some threads or even doc on blocking that 192.168.100 address issue.

edit: see here under reject dhcp from
https://docs.netgate.com/pfsense/en/latest/interfaces/configure-ipv4.html#dhcp

On your wan interface

mrneutron

@johnpoz, thank you for that reject DHCP suggestion! I will try that immediately.

On the UPS topic - total agreement.
Both the Netgate SG-1100 and the cable modem are plugged into a 1500VA UPS. The run time on that is about 4 hours and it will outlast most power outages.
The problem I've seen with longer power outages is that Comcast's Internet connection drops, at the coax cable level. So, even though my cable modem and Netgate SG-1100 stay powered up, the Comcast connection is lost. When power comes back up, I often have to repower various IP-connected devices (like the smart tv) to force a reconnect to the Internet. In such situations I have usually forced a power cycle of the cable modem and Netgate SG-1100 as part of the reconnection process. Hopefully, this reject DHCP step will fix my reported problem with WAN IP address assignment.

johnpoz

@mrneutron hmm I have never seen that, but I don't recall an outage of that long in very long time..

Normally the outages here are very short, like 1 hour is really long outage.. We had pretty bad storm last night in the area, lots of people in the area out for really long time (downed trees taking out lines I think) - still out I think for some, but we were lucky my power bounced, it was maybe 10 seconds if that.. Long enough to reset all the clocks etc. But I didn't even hear my upses start beeping that they were off Ac.

Just long enough for all my smart lights to turn on because of the outage.. You know power bounced in my house because all the smart lights turn on when it comes back - hehe.. I have one of my alexas on ups so when it bounces like that I can turn off house without having to wait for all the alexas to reboot ;) And since network and internet are still up when have a power outage can normally still control stuff from the one alexa..

But if you loose internet like that, you should prob just need to reboot your modem and not even worry about rebooting pfsense.

But hope the reject thing helps.. Heres hoping though you don't have to see if works for a long time.. Power outages suck ;)

We did have one long time ago where we were out for 3 some days, but electric company even paid for food we lost in frig, etc.