Embedded Services Stopping

[kewing75]

We have multiple PFSense Embedded units out in the field. But the problem we are having is after about 3-6 months we get a call from the customer that the unit isnt working. We remote into it and that unit is up and working fine except the DHCP DNS NTP and SIP Proxy services are all stopped and wont start back up. We then go down to the customer put in a new unit and all is good for 3-6 months then the same thing happens again. Anyone have anyidea on how to get the services to start back up remotely or are we looking at a bigger issue. This will be the 4th Embedded unit at this location.

[jimp]

What version of pfSense? 1.2.1? 1.2.2? 1.2.3?

1.2.3 has only been out for about three months, and it's based on NanoBSD, which is a big step up from the old embedded methodology.

If you haven't yet reflashed these devices to 1.2.3, you'd be best served by doing that first.

The contents of the system log(s) would also help, there may be some info there as to why some services died.

[kewing75]

Yes we are running 1.2.3. We were running 1.2.2 and 1.2.1 and 1.2.3 is alot better but we keep having this issue with the system. Here is the Sys Log

Last 50 system log entries
Feb 26 21:22:08 kernel: sio0 at port 0x3f8-0x3ff irq 4 flags 0x30 on isa0
Feb 26 21:22:08 kernel: sio0: type 16550A, console
Feb 26 21:22:08 kernel: sio0: [FILTER]
Feb 26 21:22:08 kernel: sio1 at port 0x2f8-0x2ff irq 3 on isa0
Feb 26 21:22:08 kernel: sio1: type 16550A
Feb 26 21:22:08 kernel: sio1: [FILTER]
Feb 26 21:22:08 kernel: Timecounter "TSC" frequency 498054665 Hz quality 800
Feb 26 21:22:08 kernel: Timecounters tick every 10.000 msec
Feb 26 21:22:08 kernel: IPsec: Initialized Security Association Processing.
Feb 26 21:22:08 kernel: ad0: 488MB <sandisk sdcfb-512="" hdx="" 4.03="">at ata0-master PIO4
Feb 26 21:22:08 kernel: GEOM_LABEL: Label for provider ad0a is ufs/pfSense.
Feb 26 21:22:08 kernel: GEOM_LABEL: Label for provider ad0d is ufs/pfSenseCfg.
Feb 26 21:22:08 kernel: Trying to mount root from ufs:/dev/ufs/pfSense
Feb 26 21:22:09 kernel: vr0: link state changed to UP
Feb 26 21:22:09 kernel: vr2: link state changed to UP
Feb 26 21:22:09 kernel: vr1: link state changed to DOWN
Feb 26 21:22:11 kernel: pflog0: promiscuous mode enabled
Feb 26 21:22:12 pftpx[380]: cannot drop privileges: Unknown error: 0
Feb 26 21:22:12 pftpx[380]: cannot drop privileges: Unknown error: 0
Feb 26 21:22:17 dhcpd: Internet Systems Consortium DHCP Server V3.0.5
Feb 26 21:22:17 dhcpd: Copyright 2004-2006 Internet Systems Consortium.
Feb 26 21:22:17 dhcpd: All rights reserved.
Feb 26 21:22:17 dhcpd: For info, please visit http://www.isc.org/sw/dhcp/
Feb 26 21:22:17 dhcpd: no such user: dhcpd
Feb 26 21:22:17 dhcpd: no such user: dhcpd
Feb 26 21:22:17 dnsmasq[485]: unknown user or group: nobody
Feb 26 21:22:17 dnsmasq[485]: unknown user or group: nobody
Feb 26 21:22:17 dnsmasq[485]: FAILED to start up
Feb 26 21:22:17 dnsmasq[485]: FAILED to start up
Feb 26 21:22:20 pftpx[567]: cannot drop privileges: Unknown error: 0
Feb 26 21:22:20 pftpx[567]: cannot drop privileges: Unknown error: 0
Feb 26 21:22:21 pftpx[606]: cannot drop privileges: Unknown error: 0
Feb 26 21:22:21 pftpx[606]: cannot drop privileges: Unknown error: 0
Feb 26 21:22:22 php: : Creating rrd update script
Feb 26 21:22:22 dhcpd: Internet Systems Consortium DHCP Server V3.0.5
Feb 26 21:22:22 dhcpd: Copyright 2004-2006 Internet Systems Consortium.
Feb 26 21:22:22 dhcpd: All rights reserved.
Feb 26 21:22:22 dhcpd: For info, please visit http://www.isc.org/sw/dhcp/
Feb 26 21:22:22 dhcpd: no such user: dhcpd
Feb 26 21:22:22 dhcpd: no such user: dhcpd
Feb 26 21:22:24 php: : Resyncing configuration for all packages.
Feb 26 21:22:29 check_reload_status: check_reload_status is starting
Feb 26 21:22:29 login: login on console as root
Feb 26 21:22:29 sshlockout[826]: sshlockout starting up
Feb 26 21:22:29 sshlockout[826]: sshlockout starting up
Feb 26 23:23:34 check_reload_status: reloading filter
Feb 26 23:23:39 login: login on console as root
Feb 26 23:23:44 pftpx[13052]: cannot drop privileges: Unknown error: 0
Feb 26 23:23:41 login: login on console as root
Feb 26 23:23:44 pftpx[13052]: cannot drop privileges: Unknown error: 0</sandisk>

[jimp]

Feb 26 21:22:17    dhcpd: no such user: dhcpd
Feb 26 21:22:17    dnsmasq[485]: unknown user or group: nobody
Feb 26 21:22:20    pftpx[567]: cannot drop privileges: Unknown error: 0

Those errors are not normal. It implies something is broken in such as /etc/group or /etc/passwd

I've never seen them on any box I've used, and I have >10 embedded units in the field.

[kewing75]

Thats what i was figuring what was wrong but wasnt sure. I have 10 Units out in the field running on Alix hardware along with wireless and they all work great. But this location always seems to be an issue. Could it be an issue with the power resetting? Does anyone know can i recreate the user and or password to get the services started back up?

[jimp]

If you're on nanobsd, you could set the boot slice to the alternate and reboot, and it would be back to normal (but with your config, of course). That's under Diagnostics > NanoBSD

Power loss normally isn't an issue with embedded, as the device is read-only most of the time. Even so, it's rare that a file gets corrupted.