Debugging 2.3-RC IPSEC Startup Issues

[Zaphon]

Since I upgraded to 2.3-RC a few nights ago,  I have had an issue where upon reboot I end up with a non-functioning IPSEC.  Further investigation leads me to find that for whatever reason, IPSEC was started up twice, and of course that leads to bad things.  I see basically the following (NOTE THESE ARE NOT REAL)

root    39914  0.0  0.1  30380  3896  -  Is  10:41PM    0:00.00 /usr/local/libexec/ipsec/starter –daemon charon
root    39920  0.0  0.1  30380  3896  -  Is  10:41PM    0:00.00 /usr/local/libexec/ipsec/starter --daemon charon
root    39972  0.0  0.4 222300 16380  -  Ss  10:41PM    0:08.02 /usr/local/libexec/ipsec/charon --use-syslog
root    39990  0.0  0.4 222300 16380  -  Ss  10:41PM    0:08.02 /usr/local/libexec/ipsec/charon --use-syslog

This is completely hypothetical right now as I don't feel like rebooting to reproduce it (and pissing off the teenager)..  But basically on every reboot, until I SSH in and kill these by hand (because hitting stop from the Interface doesn't fix this), I'm dead in the water.  Once I do kill them I can restart from the interface and my IPSEC tunnels come right up.  I've been running 2.3 for months now, and I've never encountered this bug until I updated to RC.  So this is something relatively VERY new.  I've rebooted about 10 times in the past 2 days since going to 2.3-RC and it still is occurring as well (as I've had an occasional oddity on one boot be cleared up before)

I'd be happy to debug this and try and figure it out, just not sure how to debug startup items like this.  Any pointers, etc.

[Gram]

I'm not using IPsec but after the upgrade to 2.3 RC the IPsec service was running. There was a tunnel created with no endpoints. I deleted the tunnel, rebooted, and the service/tunnel remain gone for me.

At some points ages ago I was testing IPsec tunnels but I don't see any reference to them in my pre-upgrade backup, and I'm sure there was no ipsec service running in my install of 2.2.6. So I'm not sure where that tunnel came from.

[cmb]

This is the code from /etc/inc/vpn.inc that runs 'ipsec start'.

        /* manage process */
        if ($restart === true) {
                mwexec("/usr/local/sbin/ipsec restart", false);
        } else {
                if (isvalidpid("{$g['varrun_path']}/starter.charon.pid")) {
                        /* Update configuration changes */
                        /* Read secrets */
                        mwexec("/usr/local/sbin/ipsec rereadall", false);
                        mwexec("/usr/local/sbin/ipsec reload", false);
                } else {
                        mwexec("/usr/local/sbin/ipsec start", false);
                }
        }

If you try to start it while it's already running, it sees the PID file and just exits.

# ipsec start 
Starting strongSwan 5.4.0 IPsec [starter]...
charon is already running (/var/run/charon.pid exists) -- skipping daemon start

So I'm not sure how it could end up running twice. Maybe if you run 'ipsec start' twice really close to the same time, so both run before the first gets the PID in place.

I'd turn those lines above into something like this:

$ts = microtime(true);
log_error("DEBUG: vpn_ipsec_configure manage process. $ts");
        /* manage process */
        if ($restart === true) {
                mwexec("/usr/local/sbin/ipsec restart", false);
log_error("DEBUG: $ts - ipsec reload");
        } else {
                if (isvalidpid("{$g['varrun_path']}/starter.charon.pid")) {
                        /* Update configuration changes */
                        /* Read secrets */
log_error("DEBUG: $ts - rereadall and reload");
                        mwexec("/usr/local/sbin/ipsec rereadall", false);
                        mwexec("/usr/local/sbin/ipsec reload", false);
                } else {
log_error("DEBUG: $ts - ipsec start");
                        mwexec("/usr/local/sbin/ipsec start", false);
                }
        }

and replicate, then look at the log to see what's happening ("clog /var/log/system.log|grep DEBUG" in SSH).