Issue: dhcpd fails to start

getbluefrog · Sep 22, 2008, 6:30 PM

Problem: dhcpd fails to start on LAN

environment: 1.2-RELEASE
ALIX 2c3
Embedded

History:
Has been working for about a year without changes

Tshooting done:
rebooted firewall.
Tried manually clicking Start next to the dhcpd service. (under Status: Services)

getbluefrog · Sep 22, 2008, 8:08 PM

update:

Logs###########

Sep 22 13:00:16 dhcpd: no such user: dhcpd
Sep 22 13:00:16 dhcpd: no such user: dhcpd
Sep 22 13:00:16 dhcpd: For info, please visit http://www.isc.org/sw/dhcp/
Sep 22 13:00:16 dhcpd: All rights reserved.
Sep 22 13:00:16 dhcpd: Copyright 2004-2006 Internet Systems Consortium.
Sep 22 13:00:16 dhcpd: Internet Systems Consortium DHCP Server V3.0.5

getbluefrog · Sep 24, 2008, 4:42 PM

Ping! Anybody?

jobah · Sep 25, 2008, 9:07 AM

have you solved the problem? I have removed my minipci wifi card yesterday from the router (alix) pfsense 1.2, booted and removed ath0 interface and have these errors in log:

Sep 25 10:48:32 php: : New alert found: SSHD failed to start.
…
Sep 25 10:49:04 pftpx[853]: cannot drop privileges: Unknown error: 0
…
Sep 25 10:49:21 dhcpd: no such user: dhcpd

so my dhcpd & sshd are down...everything else works?!

Oroboros · Nov 12, 2008, 3:03 AM

We've had a similar type of failure on the same platform. Something has apparently corrupted /etc/passwd and related files.

Since I can't ssh in, I've been using the www GUI "Diagnostics:Command" to do some investigation. The output of cat /etc/passwd was all on one line but I've broken it for 'readability' sake here:


$ cat /etc/passwd
???????????????????????????????????????????????????????????????????????????????????
??????????????????????????????????????????????????????????????????????????????????
??????????????????????????????????????????????????????????????????????????????????
???????????????????????????????????????????????????????????????????????????????????
????????????????????????????????????????????????????????????????_dhcp?*?A???A????????dhcp 
programs?/var/empty?/usr/sbin/nologin?????ß??3A???_dhcp?*????A???A?????dhcp programs?
/var/empty?/usr/sbin/nologin???????ßC???Adhcpd?*?ê??ê???????DHCP 
Daemon?/nonexistent?/sbin/nologin?????ß??1dhcpddhcpd?*???ê??ê?????DHCP 
Daemon?/nonexistent?/sbin/nologin???????ßAdhcpdnobody?*?þÿ??þÿ???????Unprivileged 
user?/nonexistent?/usr/sbin/nologin?????ß??2???nobody?*?þÿ??þÿ???????Unprivileged 
user?/nonexistent?/usr/sbin/nologin?????ß??1nobodynobody?*???ÿþ??ÿþ?????Unprivileged 
user?/nonexistent?/usr/sbin/nologin???????ßB???nobody?*???ÿþ??ÿþ?????Unprivileged 
user?/nonexistent?/usr/sbin/nologin???????ßAnobody_pflogd?*?@???@????????pflogd privsep 
user?/var/empty?/usr/sbin/nologin?????ß??1_pflogd_pflogd?*????@???@?????pflogd privsep 
user?/var/empty?/usr/sbin/nologin???????ßA_pflogdsmmsp?*????????????Sendmail Submission 
User?/var/spool/clientmqueue?/usr/sbin/nologin?????ß??3???smmsp?*????????????Sendmail Submission 
User?/var/spool/clientmqueue?/usr/sbin/nologin???????ßC???man?*?	???	????????Mister Man 
Pages?/usr/share/man?/usr/sbin/nologin?????ß??3	???man?*?	
???	????????Mister Man P

My system is not running dhcpd, though I'm sure that would fail just as sshd and pftpx have for the same reason:


Nov 11 15:08:28 	pftpx[439]: cannot drop privileges: Unknown error: 0
Nov 11 15:08:28 	pftpx[439]: cannot drop privileges: Unknown error: 0
Nov 11 15:08:27 	pftpx[413]: cannot drop privileges: Unknown error: 0
Nov 11 15:08:27 	pftpx[413]: cannot drop privileges: Unknown error: 0
Nov 11 15:08:27 	inetd[401]: 19010/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19010/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19009/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19009/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19008/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19008/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19007/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19007/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19006/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19006/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19005/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19005/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19004/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19004/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19003/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19003/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19002/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19002/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19001/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19001/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19000/tcp: no such user 'nobody', service ignored
Nov 11 15:08:27 	inetd[401]: 19000/tcp: no such user 'nobody', service ignored
Nov 11 15:08:22 	php: : New alert found: SSHD failed to start.

I'm somewhat nervous of a possible intrusion, although both ssh and web administration access were not exposed to the WAN at all. The leading theory is that a power failure somehow caused corruption, except that doesn't make much sense on a flash-based device…

Oroboros · Nov 12, 2008, 3:02 AM

I've restored both pftpx and sshd services on my system. I did so by downloading the following files from the "Diagnostics: Command" web page on a known working system (and just for good measure, I picked one with an identical firmware build date of the embedded 1.2-RELEASE):


/etc/passwd
/etc/master.passwd
/etc/pwd.db
/etc/spwd.db

I uploaded those files on the broken system via the same page, "Diagnostics:Command". That put them all into /tmp so I executed the following four commands to move them into /etc:


cp /tmp/passwd /etc/passwd
cp /tmp/master.passwd /etc/master.passwd
cp /tmp/pwd.db /etc/pwd.db
cp /tmp/spwd.db /etc/spwd.db

Then rebooted, and the pftpx, sshd, and port forwarding services all came up as expected.

I also satisified my curiosity about the mysterious inetd services on ports 19000+ It looks like the port forwarding is handled by netcat….


fw:/etc#  cat /var/etc/inetd.conf
19000   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.3 25
19001   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.3 80
19002   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.3 110
19003   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.3 8383
19004   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.4 80
19005   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.5 80
19006   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.7 25
19007   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.7 80
19008   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.7 110
19009   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.7 443
19010   stream  tcp     nowait/0        nobody  /usr/bin/nc nc -w 20 192.169.1.21 80
fw:/etc#

Also verified that from an untrusted host on the WAN, the only open ports are the two proxies I expect to see:


(The 65533 ports scanned but not shown below are in state: filtered)
PORT     STATE SERVICE
21/tcp   open  ftp
1723/tcp open  pptp

Nmap run completed -- 1 IP address (1 host up) scanned in 180.733 seconds

So, if there was a remote compromise it would have likely been via one of those services.

My process for finding different files was mainly to run md5 /etc/* via the web command line, and then diff'd the results against a known good system.