Bootup configuration not loading from USB

stephenw10

It didn't load a config via the ECL at that boot:

External config loader 1.0 is now starting... mmcsd0s1 mmcsd0s1a mmcsd0s1b

That would list the USB drive as da0sX and show a config found and loaded if it had.

And it looks like that's because it didn't find the USB device until after the ECL had run. Later in the boot we see:

da0 at umass-sim0 bus 0 scbus5 target 0 lun 0
da0: <PNY USB 2.0 FD PMAP> Removable Direct Access SPC-2 SCSI device
da0: Serial Number 071917E1598E6375
da0: 40.000MB/s transfers
da0: 29604MB (60628992 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

Was it inserted after boot? Was the the first boot?

When you restore a config file it loads the complete config or nothing. It would only not load it if it's invalid.

Does it boot completely without the USB drive?

Try entering ctl+t at the console at that point. It should respond with whatever it's waiting for.

Steve

emmdee

@stephenw10 Thank you

The USB drive is permanently installed in the device. It's never removed. This is not the first boot, the device is operational, I'm just trying to make minor changes to the config (dns servers and add some fw rules).

Getting physical access to the device would require me booking a flight :) In an emergency I would create a ticket with on-site COLO personnel to access the physical device if anything was needed like pulling the drive or patching cables/etc. I am currently accessing it via serial console cable from another system that I'm remotely connected to.

A little background:
Our company has dozens of these netgate devices all over the world so the goal is to program them remotely and in batches using configuration management with no need for GUI clicks since there are so many of them and we need to keep their configs in-sync. Using ECL seems like the best way to do that unless we write a whole SDK from scratch around the php-shell since there is no API which is a bit unfeasible at this time.

PS - I did a diff on my config.xml and the current running config and it looks good and valid.

Try entering ctl+t at the console at that point. It should respond with whatever it's waiting for.

At what point?

Thanks again

stephenw10

Ok, well it will try to pull in a config via the ECL at every boot so be aware of that.

It isn't pulling in config there because the USB device is not detected early enough. It must be a very slow USB device as the 7100 has a loader line to wait for it and works with most devices.

One way around that is to re-root instead of rebooting. That doesn't disconnect the USB so will almost always work.

Entering ctl+t when the console appears unresponsive can often show why.

Steve

emmdee

@stephenw10

it will try to pull in a config via the ECL at every boot so be aware of that.

Yep the idea is to use the USB as "startup config" -- as the source of truth for all boot ups.

It must be a very slow USB device

It's a USB 3.0 brand new PNY stick. Purchased in 2021. About as fast as you can get unfortunately. Not sure what else to buy.

re-root instead of rebooting

I'll look into this, thanks.

stephenw10

It's not the transfer speed of the interface it's the time it takes to initialise the drive. Which seems random!

But you might check that /boot/loader.conf contains the kern.cam.boot_delay line. It should be present by default.
It could have been overridden by a value in /boot/loader.conf.local if you're using that.

Steve

emmdee

Interestingly there are two identical lines in /boot/loader.conf:

$ grep cam.boot_delay /boot/loader.conf
kern.cam.boot_delay=10000
kern.cam.boot_delay=10000

/boot/loader.conf.local doesn't exist.

So are you saying I could create a /boot/loader.conf.local and add the line kern.cam.boot_delay=30000 to make it wait longer for the device?

stephenw10

Yes, though I wouldn't expect 30000 to have much effect there over 10000. It's worth trying.

emmdee

@stephenw10 Re-root worked! I'll go with this route moving forward.

Appreciate the help.

bingo600

@stephenw10 said in Bootup configuration not loading from USB:

One way around that is to re-root instead of rebooting. That doesn't disconnect the USB so will almost always work.

Any quick hint to what a re-root is & how2 do ?

/Bingo

bigsy

@bingo600 It's covered here.

Reroot is one of the options presented on selecting reboot from the GUI or the console.

"Performs a “reroot” style reboot, which is faster than a traditional reboot but does not restart the entire operating system. All running processes are killed, all filesystems are remounted, and then the system startup sequence is run again. This type of restart is much faster as it does not reset the hardware, reload the kernel, or need to go through the hardware detection process."

emmdee

@bingo600

reroot is covered in docs (link) -- although it says it's faster than a reboot it still took a few mins.
To perform a reroot, choose option r when triggering a reboot from the terminal menu.
Since everything in my stack is automated (no human touch where we can help it), I just made a simple php script that triggers like this (php) and it's called through our automation engine:
```
require_once("functions.inc");
system_reboot_sync(true);
```
the true in system_reboot_sync(true) tells it to do a reroot instead of a reboot. It's not documented, but I found it in the source code here