20 minute pause on reboot

[dennypage]

I am experiencing an odd restart delay. At first I thought it was associated with updating to new snapshots, but I had time this morning to conduct a reboot test.

I am running the current RC snapshot, although I've had this problem for a while. Unfortunately I cannot identify when it started.

When the router reboots, either from new firmware or simply invoking reboot via the web interface, the system will reboot, and pause (hang) for 20 minutes during the startup process. The syslog for one of these reboots is below.

During the pause, the web interface is unresponsive, and the system does not route packets. The ssh interface is responsive to the local lan.

During the pause, if I go in via the ssh interface and select "Restart webConfigurator", the system will recover immediately.

I have also included ps output while the system is hung.

Please let me know if there is additional information that I can offer.

–---

USER  PID  %CPU %MEM    VSZ  RSS TT  STAT STARTED    TIME COMMAND
root    11 400.0  0.0      0    64  -  RL  10:33AM 79:41.94 [idle]
root    0  0.0  0.0      0  288  -  DLs  10:33AM  0:00.01 [kernel]
root    1  0.0  0.0  9472  740  -  ILs  10:33AM  0:00.01 /sbin/init –
root    2  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [crypto]
root    3  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [crypto returns]
root    4  0.0  0.0      0    32  -  DL  10:33AM  0:00.00 [cam]
root    5  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [fdc0]
root    6  0.0  0.0      0    16  -  DL  10:33AM  0:00.49 [pf purge]
root    7  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [sctp_iterator]
root    8  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [enc_daemon0]
root    9  0.0  0.0      0    16  -  DL  10:33AM  0:00.03 [pagedaemon]
root    10  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [audit]
root    12  0.0  0.0      0  608  -  WL  10:33AM  0:00.87 [intr]
root    13  0.0  0.0      0    64  -  DL  10:33AM  0:00.00 [ng_queue]
root    14  0.0  0.0      0    48  -  DL  10:33AM  0:00.01 [geom]
root    15  0.0  0.0      0    16  -  DL  10:33AM  0:00.18 [rand_harvestq]
root    16  0.0  0.0      0  448  -  DL  10:33AM  0:00.04 [usb]
root    17  0.0  0.0      0    16  -  DL  10:33AM  0:00.03 [acpi_thermal]
root    18  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [acpi_cooling0]
root    19  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [vmdaemon]
root    20  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [idlepoll]
root    21  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [pagezero]
root    22  0.0  0.0      0    16  -  DL  10:33AM  0:00.01 [bufdaemon]
root    23  0.0  0.0      0    16  -  DL  10:33AM  0:00.00 [vnlru]
root    24  0.0  0.0      0    16  -  DL  10:33AM  0:00.02 [syncer]
root    50  0.0  0.0      0    16  -  DL  10:33AM  0:00.01 [md0]
root    55  0.0  0.0      0    16  -  DL  10:33AM  0:00.06 [md1]
root  282  0.0  0.6 224268 23304  -  Ss  10:34AM  0:00.04 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root  297  0.0  0.1  19028  2432  -  INs  10:34AM  0:00.01 /usr/local/sbin/check_reload_status
root  299  0.0  0.1  19028  2304  -  IN  10:34AM  0:00.00 check_reload_status: Monitoring daemon of check_reload_status
root  309  0.0  0.1  13164  4404  -  Ss  10:34AM  0:00.00 /sbin/devd -q
root  5991  0.0  0.1  14664  2240  -  Ss  10:34AM  0:00.08 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf
root 12011  0.0  0.1  21724  5044  -  Ss  10:34AM  0:00.02 /usr/local/sbin/openvpn –config /var/etc/openvpn/server1.conf
root 13283  0.0  0.1  16812  2084  -  Ss  10:34AM  0:00.02 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root 13735  0.0  0.1  21724  5108  -  Ss  10:34AM  0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
root 15407  0.0  0.1  32428  5092  -  Is  10:34AM  0:00.00 /usr/sbin/sshd
root 15574  0.0  0.9 228364 35732  -  I    10:34AM  0:00.21 php-fpm: pool lighty (php-fpm)
root 15780  0.0  0.1  14756  2136  -  Is  10:34AM  0:00.01 /usr/local/sbin/sshlockout_pf 15
root 16821  0.0  0.1  55632  5576  -  Ss  10:35AM  0:00.09 sshd: root@pts/0 (sshd)
root    25  0.0  0.1  17144  2080 v0  Is+  10:33AM  0:00.03 sh /etc/rc autoboot
root  302  0.0  1.2 236456 48420 v0  S+  10:34AM  0:09.54 /usr/local/bin/php -f /etc/rc.bootup
root 17101  0.0  0.1  17144  2584  0  Is  10:35AM  0:00.01 -sh (sh)
root 17281  0.0  0.1  17144  2472  0  I    10:35AM  0:00.01 /bin/sh /etc/rc.initial
root 19211  0.0  0.1  17484  3260  0  S    10:35AM  0:00.03 /bin/tcsh
root 23003  0.0  0.1  18816  2136  0  R+  10:53AM  0:00.00 ps -axuwww

---

Dec 23 10:56:30 sshd[30026]: Accepted publickey for root from 192.168.230.21 port 56432 ssh2: RSA be:81:cc:09:1d:7e:72:99:ef:a4:89:c6:9f:14:1d:29
Dec 23 10:56:05 php-fpm[11034]: /index.php: Successful login for user 'denny' from: 192.168.230.21
Dec 23 10:56:05 php-fpm[11034]: /index.php: Successful login for user 'denny' from: 192.168.230.21
Dec 23 10:56:03 snmpd[46956]: could not encode error response
Dec 23 10:56:02 snmpd[46956]: could not encode error response
Dec 23 10:55:37 sshlockout[8936]: sshlockout/webConfigurator v3.0 starting up
Dec 23 10:55:37 login: login on ttyv0 as root
Dec 23 10:55:36 upsd[97102]: User monuser@127.0.0.1 logged into UPS [closetups]
Dec 23 10:55:36 upsmon[505]: Startup successful
Dec 23 10:55:36 upsd[97102]: Startup successful
Dec 23 10:55:36 upsd[94373]: Connected to UPS [closetups]: usbhid-ups-closetups
Dec 23 10:55:36 upsd[94373]: listening on 127.0.0.1 port 3493
Dec 23 10:55:36 upsd[94373]: listening on ::1 port 3493
Dec 23 10:55:36 usbhid-ups[94278]: Startup successful
Dec 23 10:55:34 php-fpm[15574]: /rc.start_packages: Restarting/Starting all packages.
Dec 23 10:55:34 kernel: done.
Dec 23 10:55:34 syslogd: kernel boot file is /boot/kernel/kernel
Dec 23 10:55:34 syslogd: exiting on signal 15
Dec 23 10:55:34 php: rc.bootup: Creating rrd update script
Dec 23 10:55:29 sshd[16821]: fatal: Write failed: Operation not permitted
Dec 23 10:55:29 sshd[16821]: fatal: Write failed: Operation not permitted
Dec 23 10:55:22 kernel: done
Dec 23 10:55:22 kernel: .done.
Dec 23 10:55:22 kernel: ..
Dec 23 10:55:21 kernel: ..
Dec 23 10:55:17 check_reload_status: Updating all dyndns
Dec 23 10:55:17 kernel: done.
Dec 23 10:55:17 dhcpleases: kqueue error: unkown
Dec 23 10:55:17 kernel: done.
Dec 23 10:55:17 dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) does not exist, No such file or directory.
Dec 23 10:55:17 dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) does not exist, No such file or directory.
Dec 23 10:55:17 kernel: done.
Dec 23 10:55:17 php: rc.bootup: ROUTING: setting default route to 50.193.41.78
Dec 23 10:55:16 kernel: done.
Dec 23 10:55:16 kernel: done.
Dec 23 10:55:13 kernel: .done.
Dec 23 10:55:13 php: rc.bootup: Removing static route for monitor 50.152.240.1 and adding a new route through 50.193.41.78
Dec 23 10:55:13 kernel: ..
Dec 23 10:35:47 sshd[16821]: Accepted publickey for root from 192.168.230.21 port 56315 ssh2: RSA be:81:cc:09:1d:7e:72:99:ef:a4:89:c6:9f:14:1d:29
Dec 23 10:34:16 check_reload_status: Linkup starting em1_vlan1003
Dec 23 10:34:16 kernel: em1_vlan1003: link state changed to UP
Dec 23 10:34:16 kernel: em1: link state changed to UP
Dec 23 10:34:16 check_reload_status: Linkup starting em1
Dec 23 10:34:15 kernel: em2: link state changed to UP
Dec 23 10:34:15 check_reload_status: Linkup starting em2
Dec 23 10:34:15 kernel: em0: link state changed to UP
Dec 23 10:34:15 check_reload_status: Linkup starting em0
Dec 23 10:34:13 sshlockout[15780]: sshlockout/webConfigurator v3.0 starting up
Dec 23 10:34:13 sshd[15407]: Server listening on 0.0.0.0 port 22.
Dec 23 10:34:13 sshd[15407]: Server listening on :: port 22.
Dec 23 10:34:12 check_reload_status: rc.newwanip starting ovpns2
Dec 23 10:34:12 kernel: ovpns2: link state changed to UP
Dec 23 10:34:12 kernel:
Dec 23 10:34:12 check_reload_status: rc.newwanip starting ovpns1
Dec 23 10:34:12 kernel: pflog0: promiscuous mode enabled
Dec 23 10:34:12 kernel: ovpns1: link state changed to UP
Dec 23 10:34:12 kernel: tun2: changing name to 'ovpns2'
Dec 23 10:34:12 kernel: tun1: changing name to 'ovpns1'
Dec 23 10:34:12 kernel:
Dec 23 10:34:12 kernel: done.
Dec 23 10:34:12 php: rc.bootup: Resyncing OpenVPN instances.
Dec 23 10:34:12 kernel: vlan0: changing name to 'em1_vlan1003'
Dec 23 10:34:12 kernel:
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Trying to mount root from ufs:/dev/ufsid/4fd123aa21bb6b8f [rw,sync,noatime]…
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Timecounter "TSC" frequency 1795537980 Hz quality 1000
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: SMP: AP CPU #2 Launched!
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: SMP: AP CPU #3 Launched!
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: SMP: AP CPU #1 Launched!
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ses0: SEMB SES Device
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ses0: <ahci sgpio="" enclosure="" 1.00="" 0001="">SEMB S-E-S 2.00 device
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0: Previously was known as ad4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0: Command Queueing enabled
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0: Serial Number 86F80723030800053818
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0: <patriot 2="" torqx="" 64gb="" ssd="" s5fam014="">ATA-8 SATA 2.x device
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen3.2: <cps>at usbus3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub5: 6 ports with 6 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub1: 4 ports with 4 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub6: 2 ports with 2 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub4: 2 ports with 2 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub3: 2 ports with 2 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub2: 2 ports with 2 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub0: 2 ports with 2 removable, self powered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub6: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus5
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen5.1: <intel>at usbus5
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub5: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus6
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen6.1: <intel>at usbus6
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub4: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen3.1: <intel>at usbus3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen4.1: <intel>at usbus4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub2: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen1.1: <intel>at usbus1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen2.1: <intel>at usbus2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ugen0.1: <intel>at usbus0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus6: 480Mbps High Speed USB v2.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus5: 12Mbps Full Speed USB v1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus4: 12Mbps Full Speed USB v1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus3: 12Mbps Full Speed USB v1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus2: 480Mbps High Speed USB v2.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus1: 12Mbps Full Speed USB v1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus0: 12Mbps Full Speed USB v1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: random: unblocking device.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: IPsec: Initialized Security Association Processing.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Timecounters tick every 1.000 msec
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ppc0: cannot reserve I/O port range
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: fdc0: <enhanced floppy="" controller="">at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: sc0: <system console="">at flags 0x100 on isa0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: orm0: <isa option="" rom="">at iomem 0xd0000-0xd37ff on isa0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: atkbd0: [GIANT-LOCKED]
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: kbd0 at atkbd0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: atkbd0: <at keyboard="">irq 1 on atkbdc0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi_tz0: <thermal zone="">on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ahciem0: <ahci enclosure="" management="" bridge="">on ahci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ahcich2: <ahci channel="">at channel 2 on ahci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ahcich1: <ahci channel="">at channel 1 on ahci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ahcich0: <ahci channel="">at channel 0 on ahci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ahci0: AHCI v1.10 with 3 3Gbps ports, Port Multiplier not supported
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ahci0: <intel ich8m="" ahci="" sata="" controller="">port 0xf800-0xf807,0xf700-0xf703,0xf600-0xf607,0xf500-0xf503,0xf400-0xf41f mem 0xfdffd000-0xfdffd7ff irq 19 at device 31.2 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ata0: <ata channel="">at channel 0 on atapci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: atapci0: <intel ich8m="" udma100="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf900-0xf90f at device 31.1 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: isa0: <isa bus="">on isab0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: isab0: <pci-isa bridge="">at device 31.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pci5: <acpi pci="" bus="">on pcib5
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pcib5: <acpi pci-pci="" bridge="">at device 30.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus6 on ehci1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus6: EHCI version 1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ehci1: <intel 82801h="" (ich8)="" usb="" 2.0="" controller="" usb2-a="">mem 0xfdffe000-0xfdffe3ff irq 23 at device 29.7 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus5 on uhci4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhci4: <intel 82801h="" (ich8)="" usb="" controller="" usb-c="">port 0xfa00-0xfa1f irq 18 at device 29.2 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus4 on uhci3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhci3: <intel 82801h="" (ich8)="" usb="" controller="" usb-b="">port 0xfb00-0xfb1f irq 19 at device 29.1 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus3 on uhci2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhci2: <intel 82801h="" (ich8)="" usb="" controller="" usb-a="">port 0xfc00-0xfc1f irq 23 at device 29.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em3: Using MSIX interrupts with 3 vectors
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em3: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xbf00-0xbf1f mem 0xfdac0000-0xfdadffff,0xfdafc000-0xfdafffff irq 19 at device 0.0 on pci4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pci4: <acpi pci="" bus="">on pcib4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pcib4: <acpi pci-pci="" bridge="">irq 19 at device 28.3 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em2: Using MSIX interrupts with 3 vectors
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em2: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xdf00-0xdf1f mem 0xfdcc0000-0xfdcdffff,0xfdcfc000-0xfdcfffff irq 18 at device 0.0 on pci3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pci3: <acpi pci="" bus="">on pcib3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pcib3: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em1: Using MSIX interrupts with 3 vectors
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xef00-0xef1f mem 0xfd3c0000-0xfd3dffff,0xfd3fc000-0xfd3fffff irq 17 at device 0.0 on pci2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pci2: <acpi pci="" bus="">on pcib2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.1 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em0: Using MSIX interrupts with 3 vectors
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xaf00-0xaf1f mem 0xfd5c0000-0xfd5dffff,0xfd5fc000-0xfd5fffff irq 16 at device 0.0 on pci1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pci1: <acpi pci="" bus="">on pcib1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pcib1: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: hdac0: <intel 82801h="" hda="" controller="">mem 0xfdff4000-0xfdff7fff irq 22 at device 27.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus2 on ehci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus2: EHCI version 1.0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ehci0: <intel 82801h="" (ich8)="" usb="" 2.0="" controller="" usb2-b="">mem 0xfdfff000-0xfdfff3ff irq 18 at device 26.7 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus1 on uhci1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhci1: <intel 82801h="" (ich8)="" usb="" controller="" usb-e="">port 0xfd00-0xfd1f irq 21 at device 26.1 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: usbus0 on uhci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: uhci0: <intel 82801h="" (ich8)="" usb="" controller="" usb-d="">port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: vgapci1: <vga-compatible display="">mem 0xfde80000-0xfdefffff at device 2.1 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: vgapci0: Boot video device
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: agp0: aperture size is 256M, detected 8188k stolen memory
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: agp0: <intel pineview="" svga="" controller="">on vgapci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: vgapci0: <vga-compatible display="">port 0xff00-0xff07 mem 0xfdf00000-0xfdf7ffff,0xd0000000-0xdfffffff,0xfd800000-0xfd8fffff irq 16 at device 2.0 on pci0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pci0: <acpi pci="" bus="">on pcib0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi_button0: <power button="">on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Event timer "RTC" frequency 32768 Hz quality 0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: atrtc0: <at realtime="" clock="">port 0x70-0x73 on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: device_attach: hpet0 attach returned 12
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: attimer0: <at timer="">port 0x40-0x43 on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu3: <acpi cpu="">on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu2: <acpi cpu="">on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu1: <acpi cpu="">on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu0: <acpi cpu="">on acpi0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi0: reservation of 100000, bf490000 (3) failed
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi0: reservation of 0, a0000 (3) failed
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi0: Power Button (fixed)
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: acpi0: <intelr awrdacpi="">on motherboard
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: padlock0: No ACE support.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cryptosoft0: <software crypto="">on motherboard
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: kbd1 at kbdmux0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: random: <software, yarrow="">initialized
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8062def0, 0) error 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8062de40, 0) error 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8062dd90, 0) error 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80606720, 0) error 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80606670, 0) error 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806065c0, 0) error 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: wlan: mac acl policy registered
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ioapic0 <version 2.0="">irqs 0-23 on motherboard
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ioapic0: Changing APIC ID to 4
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu3 (AP/HT): APIC ID: 3
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu2 (AP): APIC ID: 2
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu1 (AP/HT): APIC ID: 1
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: cpu0 (BSP): APIC ID: 0
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: ACPI APIC Table: <intelr awrdacpi="">Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Event timer "LAPIC" quality 400
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: avail memory = 4087549952 (3898 MB)
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: real memory = 4294967296 (4096 MB)
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: TSC: P-state invariant, performance statistics
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: AMD Features2=0x1 <lahf>Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: AMD Features=0x20100800 <syscall,nx,lm>Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Origin = "GenuineIntel" Id = 0x106ca Family = 0x6 Model = 0x1c Stepping = 10
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: CPU: Intel(R) Atom(TM) CPU D525 @ 1.80GHz (1795.54-MHz K8-class CPU)
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: root@pfsense-22-amd64-builder:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10 amd64
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: FreeBSD 10.1-RELEASE-p2 #0 d1c9830(releng/10.1)-dirty: Tue Dec 23 05:56:14 CST 2014
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: The Regents of the University of California. All rights reserved.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Copyright 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 kernel: Copyright (c) 1992-2014 The FreeBSD Project.
Dec 23 10:34:12 syslogd: sendto: Network is unreachable
Dec 23 10:34:12 syslogd: kernel boot file is /boot/kernel/kernel
Dec 23 10:34:12 syslogd: sendto: Network is unreachable</fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></syscall,nx,lm></lahf></intelr></version></software,></software></intelr></acpi></acpi></acpi></acpi></at></high></at></power></acpi></acpi></vga-compatible></intel></vga-compatible></intel></intel></intel></intel></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></intel></intel></intel></intel></acpi></acpi></pci-isa></isa></intel></ata></intel></ahci></ahci></ahci></ahci></thermal></keyboard></at></isa></system></generic></enhanced></cpu></cpu></cpu></cpu></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></cps></patriot></ahci></cpu></cpu></cpu></cpu>

[cmb]

Odd that restarting the web interface does anything there. In prior versions, that'd kill off all PHP which could fix a variety of things, but in 2.2 that doesn't touch PHP.

That a system where you can mess with things a bit? I'm curious if you uninstall all packages and reboot, if it still does that.

[dennypage]

One moment please…

@cmb:

That a system where you can mess with things a bit? I'm curious if you uninstall all packages and reboot, if it still does that.

[dennypage]

Good guess, and somewhat curious results.

[Note that I had done several reboot tests previously, including removing all but one last package.]

If I remove all packages, then the reboot works as expected. Further, after the clean reboot with no packages installed, I am able to re-add packages and reboot continues to work as expected. Perhaps there is a flag set regarding packages on firmware install that isn't being cleared?

When the next snapshot update is available, I will try uninstalling all packages prior to performing the firmware update.

Is there anything else I can do to diagnose?

@cmb:

I'm curious if you uninstall all packages and reboot, if it still does that.

[dennypage]

Chris, I was able to confirm this with the last firmware update.

1. Remove all packages
2. Invoke firmware auto update
    [System updates and reboots in a timely manner]
3. Initiate reboot via the web interface
    [System reboots in a timely manner]
4. Add a package
5. Initiate reboot via the web interface
    [System reboots in a timely manner]

So it does appear that it is associated with installing packages.

I can leave a package installed next time before the firmware upgrade to reconfirm if you like.

[cmb]

Can you narrow it down to a specific package, or does it matter which?

[dennypage]

I have two packages installed (NUT and pfBlockerNG). It doesn't appear to matter which is enabled. My testing with a single package was with NUT. pfBlockerNG was completely removed prior to the tests.

Is there a package you would consider safe that you prefer I test with? I'd like something relatively easy to clean up if possible as this is a non-vm production install.

[wcrowder]

I had this problem with 2.2 alpha a few months ago, I don't remember what I did to fix it. I just tested this and it works now, and rebooted normally after update and reboot.

Avahi Network Management 0.6.31 pkg v1.09
Cron Services 0.1.8
nmap Security nmap-6.47 pkg v1.3
ntopng Network Management 1.2.1 v0.2
Open-VM-Tools Services 1280544_9
pfBlockerNG Firewall Current: 0.99
Installed: 1.0
snort Security 2.9.7.0 pkg v3.2.1
System Patches System 1.0.2

[cmb]

anything other than nut would suffice as a test. nmap's an easy one that requires no config and would at least determine whether it's specific to having a package installed, or having nut installed. I suspect it's specific to nut.

[dennypage]

I think you are correct. I installed nmap and removed NUT. The problem did not occur.

I submitted a patch to nut.inc in the bug tracking system do fix the NUT rc file. Could the firmware upgrade problem be related to the problem with NUT's rc file?

[dennypage]

I need to do more testing to be 100% sure, but what I am currently thinking is that the problem occurs if anything gets in the way of successful package re-installation and activation.

This morning's fix to NUT solves the problem if the package manager is using the standard server. However, if a non standard package server is set and NUT cannot be re-installed, then the problem still occurs.

[The reason for the non-standard package server is that I'm beta testing pfBlockerNG and I had neglected to clear the server before the firmware update.]

[cmb]

So is it only during reboots where packages are reinstalled? Thought earlier it was every reboot.

[dennypage]

Once it occurs, it's every reboot.

I believe that what causes it to occur is associated with package reinstall (or lack thereof) on reboot. However, once the problem starts, the 20 minute delay appears to happen every reboot until all packages are removed.

Still working on what exactly triggers it.

[dennypage]

So, I have not had sufficient time to vigorously test to be 100% what certain what triggers the problem…

However, I currently have a system that experiences the 20 minute pause on each reboot. During the pause, the system will not route packets, DNS/DHCP do not function, and the web interface cannot be connected to. The system is responsive to ping and ssh. This is true regardless of whether it's the first reboot after install or subsequent reboots.

If I go in via ssh and look at top, the system is completely idle. If I choose "Restart webConfigurator" during the pause, the system will recover within a few seconds and complete the boot process normally.

Are there any file system contents that would be useful in diagnosing the problem? Things you would like me to check, either during a pause or after? I will try to leave the system in it's current state as long as possible.

More information:

At the point of the firmware upgrade, the status was as follows:

• Two packages installed: NUT and pfBlockerNG
• The package server was set to a non standard server (pfBlockerNG beta server)
• The system had previously been through a reboot cleanly
• Auto update with fill backup was selected

During the pause resulting from the first reboot after firmware install:

• The system was left alone for 5 minutes or more after boot
• Top was checked via ssh--system was idle
• Package installation had not begun
• The webConfigurator was restarted via the ssh menu
• The system recovers and begins normal startup, including installing packages
• After pfBlocker is installed, the system declares itself to be up and all is normal
• Note that NUT cannot be installed because the system is pointing to a non-standard package server which does not have the NUT package available

During the pause resulting from the second reboot after firmware install:

• The system was left alone for 5 minutes or more after boot
• Top was checked via ssh--system was idle
• The webConfigurator was restarted via the ssh menu
• The system recovers and competes normal startup within a few seconds

In previous testing I have determined that the following will fix the pause:

• Remove all packages via the UI
• Reboot
• Reinstall all packages via the UI
• Reboot

Please let me know what I can do to help diagnose

[dennypage]

Chris, anything I can get you from the existing system?

I'm currently holding off on the next snapshot to preserve the problem.

[cmb]

What's the last log entry you see in the system log (SSH in, run "clog /var/log/system.log") while it's hung? That'll hopefully help narrow down where to look.

[dennypage]

I did a full dump during and after for a reboot yesterday. You can see my login at 12:27:35. Nothing else in the system log until 12:40:38.

Dec 29 12:26:37 fw kernel: vlan0: changing name to 'em1_vlan1003'
Dec 29 12:26:37 fw php: rc.bootup: Resyncing OpenVPN instances.
Dec 29 12:26:37 fw kernel: done.
Dec 29 12:26:37 fw kernel:
Dec 29 12:26:37 fw kernel: tun1: changing name to 'ovpns1'
Dec 29 12:26:37 fw kernel: tun2: changing name to 'ovpns2'
Dec 29 12:26:37 fw kernel: ovpns1: link state changed to UP
Dec 29 12:26:37 fw kernel: pflog0: promiscuous mode enabled
Dec 29 12:26:37 fw check_reload_status: rc.newwanip starting ovpns1
Dec 29 12:26:37 fw kernel:
Dec 29 12:26:37 fw kernel: ovpns2: link state changed to UP
Dec 29 12:26:37 fw check_reload_status: rc.newwanip starting ovpns2
Dec 29 12:26:38 fw sshd[14700]: Server listening on :: port 22.
Dec 29 12:26:38 fw sshd[14700]: Server listening on 0.0.0.0 port 22.
Dec 29 12:26:38 fw sshlockout[14992]: sshlockout/webConfigurator v3.0 starting up
Dec 29 12:26:40 fw check_reload_status: Linkup starting em2
Dec 29 12:26:40 fw kernel: em2: link state changed to UP
Dec 29 12:26:41 fw check_reload_status: Linkup starting em0
Dec 29 12:26:41 fw kernel: em0: link state changed to UP
Dec 29 12:26:41 fw check_reload_status: Linkup starting em1
Dec 29 12:26:41 fw kernel: em1: link state changed to UP
Dec 29 12:26:41 fw kernel: em1_vlan1003: link state changed to UP
Dec 29 12:26:41 fw check_reload_status: Linkup starting em1_vlan1003
Dec 29 12:27:35 fw sshd[16156]: Accepted publickey for root from 192.168.230.21 port 59902 ssh2: RSA be:81:cc:09:1d:7e:72:99:ef:a4:89:c6:9f:14:1d:29
Dec 29 12:30:47 fw sshd[19907]: Accepted publickey for root from 192.168.230.21 port 59978 ssh2: RSA be:81:cc:09:1d:7e:72:99:ef:a4:89:c6:9f:14:1d:29

Dec 29 12:40:38 fw kernel: ..
Dec 29 12:40:39 fw php: rc.restart_webgui: Creating rrd update script
Dec 29 12:40:39 fw sshd[19907]: fatal: Write failed: Operation not permitted
Dec 29 12:40:39 fw sshd[19907]: fatal: Write failed: Operation not permitted
Dec 29 12:40:39 fw php: rc.bootup: Removing static route for monitor 50.152.240.1 and adding a new route through 50.193.41.78
Dec 29 12:40:39 fw kernel: .done.
Dec 29 12:40:39 fw sshd[16156]: fatal: Write failed: Operation not permitted
Dec 29 12:40:39 fw sshd[16156]: fatal: Write failed: Operation not permitted
Dec 29 12:40:41 fw kernel: done.
Dec 29 12:40:41 fw lighttpd[31353]: (server.c.1558) server stopped by UID = 0 PID = 81972
Dec 29 12:40:41 fw kernel: done.
Dec 29 12:40:42 fw php: rc.bootup: ROUTING: setting default route to 50.193.41.78
Dec 29 12:40:42 fw kernel: done.
Dec 29 12:40:44 fw kernel: done.
Dec 29 12:40:44 fw kernel: done.
Dec 29 12:40:44 fw kernel: done.
Dec 29 12:40:44 fw dhcpleases: kqueue error: unkown
Dec 29 12:40:44 fw check_reload_status: Updating all dyndns
Dec 29 12:40:49 fw kernel: ..
Dec 29 12:40:49 fw kernel: ..
Dec 29 12:40:49 fw kernel: .done.
Dec 29 12:40:49 fw kernel: done
Dec 29 12:40:51 fw kernel: done.
Dec 29 12:40:58 fw php-fpm[6277]: /index.php: Successful login for user 'denny' from: 192.168.230.21
Dec 29 12:40:58 fw php-fpm[6277]: /index.php: Successful login for user 'denny' from: 192.168.230.21
Dec 29 12:41:02 fw php: rc.bootup: Creating rrd update script
Dec 29 12:41:02 fw kernel: done.
Dec 29 12:41:02 fw syslogd: exiting on signal 15
Dec 29 12:41:02 fw syslogd: kernel boot file is /boot/kernel/kernel
Dec 29 12:41:02 fw php-fpm[14317]: /rc.start_packages: Restarting/Starting all packages.
Dec 29 12:41:02 fw php-fpm[14317]: /rc.start_packages: [pfBlockerNG] Sync terminated during boot process.
Dec 29 12:41:02 fw php-fpm[14317]: /rc.start_packages: [pfBlockerNG] Sync terminated during boot process.
Dec 29 12:41:02 fw php-fpm[14317]: /rc.start_packages: [pfBlockerNG] Sync terminated during boot process.
Dec 29 12:41:02 fw php-fpm[14317]: /rc.start_packages: [pfBlockerNG] Sync terminated during boot process.
Dec 29 12:41:04 fw login: login on ttyv0 as root
Dec 29 12:41:04 fw sshlockout[71340]: sshlockout/webConfigurator v3.0 starting up
Dec 29 12:41:20 fw sshd[12868]: Accepted publickey for root from 192.168.230.21 port 60066 ssh2: RSA be:81:cc:09:1d:7e:72:99:ef:a4:89:c6:9f:14:1d:29
Dec 29 12:41:27 fw snmpd[1924]: could not encode error response
Dec 29 12:41:28 fw snmpd[1924]: could not encode error response

[dennypage]

Of note is that the problem hasn't occurred in the last several updates… was anything done specifically to address this?

[cmb]

@dennypage:

Of note is that the problem hasn't occurred in the last several updates… was anything done specifically to address this?

I'm pretty sure the problem with the nut package was fixed, which seemed to be the root cause of the delay.