pfSense rebooted by root?
-
I noticed this evening that I lost my internet connectivity for a few minutes. I looked at my pfSense router and noticed it had been rebooted.
This is all that I see in the logs:
Nov 19 22:51:59 kernel root@buildbot3:/crossbuild/ce-244/obj/amd64/WvDslnYb/crossbuild/ce-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64 Nov 19 22:51:59 kernel FreeBSD 11.2-RELEASE-p3 #17 e6b497fa0a3(RELENG_2_4_4): Thu Sep 20 09:04:45 EDT 2018 Nov 19 22:51:59 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Nov 19 22:51:59 kernel The Regents of the University of California. All rights reserved. Nov 19 22:51:59 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Nov 19 22:51:59 kernel Copyright (c) 1992-2018 The FreeBSD Project. Nov 19 22:51:59 kernel Uptime: 34d21h9m31s Nov 19 22:51:59 kernel All buffers synced. Nov 19 22:51:59 kernel Syncing disks, vnodes remaining... 0 0 done Nov 19 22:51:59 kernel Waiting (max 60 seconds) for system process `syncer' to stop... Nov 19 22:51:59 kernel Waiting (max 60 seconds) for system process `bufdaemon' to stop... done Nov 19 22:51:59 kernel Waiting (max 60 seconds) for system process `vnlru' to stop... done Nov 19 22:51:59 kernel ovpnc3: link state changed to DOWN Nov 19 22:51:59 kernel pflog0: promiscuous mode disabled Nov 19 22:51:59 syslogd kernel boot file is /boot/kernel/kernel Nov 19 22:51:00 syslogd exiting on signal 15 Nov 19 22:51:00 reboot rebooted by root Nov 19 22:00:00 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Nov 19 22:00:00 php [pfBlockerNG] Starting cron process. Nov 19 21:00:00 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Nov 19 21:00:00 php [pfBlockerNG] Starting cron process.``` It is strange that it seemed to reboot on its own. Is it possible it has been compromised?
-
Check your logs for logins.
It could also have been someone at the console.
Steve
-
It was definitely not someone at the physical console.
All of my recent logins look legitimate:
Nov 10 13:51:23 router php-fpm[77178]: /index.php: Successful login for user 'admin' from: 192.168.1.100 (Local Database) Nov 10 20:05:42 router php-fpm[62086]: /index.php: Successful login for user 'admin' from: 192.168.1.230 (Local Database) Nov 12 08:38:58 router php-fpm[49422]: /index.php: Successful login for user 'admin' from: 192.168.1.230 (Local Database) Nov 17 19:52:04 router php-fpm[62086]: /index.php: Successful login for user 'admin' from: 192.168.1.161 (Local Database) Nov 19 22:52:31 router login: login on ttyv0 as root Nov 19 22:53:14 router php-fpm[72550]: /index.php: Successful login for user 'admin' from: 192.168.1.161 (Local Database) Nov 20 11:58:18 router php-fpm[72550]: /index.php: Successful login for user 'admin' from: 192.168.1.230 (Local Database) Nov 20 17:54:16 router php-fpm[342]: /index.php: Successful login for user 'admin' from: 192.168.1.230 (Local Database) Nov 20 17:55:46 router sshd[97425]: user root login class [preauth] Nov 20 17:55:46 router sshd[97425]: user root login class [preauth] Nov 20 17:55:46 router sshd[97425]: user root login class [preauth]
-
Is the console physically protected then?
Steve
-
@stephenw10 said in pfSense rebooted by root?:
Is the console physically protected then?
Steve
Yes, it is in my crawlspace at my home and it is just me and my wife here. There was no one else in the home when this happened.
I wasn't sure if this was an indication that I was hacked somehow.
-
Hmm, hard to say then.
Hard to imagine some hacker logged into your box (somehow), removed the logs of logging in but not the reboot.You have packages installed?
Watchdog enabled?
Steve
-
Have 2 packages installed: iperf and pfBlockerNG. I don't believe I have Watchdog enabled.
-
Does that box have IPMI or any other out of band access like that?
That could only really have been caused by some script that ran or something at the console directly since they apparently didn't login.Steve
-
Do you have any internet facing services enabled or is the WAN interface completely locked down from the outside? If you do have services open to the broad internet, what are they, and have you checked their access logs?
Also, can you elaborate on the first line in your logs, what is "root@buildbot3" trying to do?
Hope this helps.
-
That's not doing anything it's first logged output from the kernel. Buildbot3 is our internal buildbox that built the kernel.
Steve
-
@stephenw10 - thanks for the clarification, just looked a bit suspicious to me at first glance.
-
My machine does not support IPMI. A fuller copy of the log is available here: https://docs.google.com/spreadsheets/d/1JC-8D-RQ44JRlXGF4PXnRid32aUxdZ8RrHdE7Z4GQnk/edit?usp=sharing
I had avoided posting the whole thing because I didn't want to cleanse it.
The reboot is on row 549.
I do have some open ports on the WAN with forwarding rules:
Port - Description
80, 81 - HTTP
2222 - SSH
1194, 1196 - OpenVPN
8080 - Docker / Rancher
32400, 32444 - Plex -
Is port 2222 just to your firewall?
-
Nothing looks very out of place. Something on your network (192.168.1.4) is trying to open pages on the firewall that don't exist:
Nov 15 16:37:48 ***REMOVED_ROUTER_HOSTNAME*** nginx: 2018/11/15 16:37:48 [error] 20989#100149: *217580 "/usr/local/www/JNAP/index.php" is not found (2: No such file or directory), client: 192.168.1.4, server: , request: "POST /JNAP/ HTTP/1.1", host: "192.168.1.1" Nov 15 16:37:48 ***REMOVED_ROUTER_HOSTNAME*** nginx: 2018/11/15 16:37:48 [error] 20989#100149: *217578 "/usr/local/www/HNAP1/index.php" is not found (2: No such file or directory), client: 192.168.1.4, server: , request: "GET /HNAP1/ HTTP/1.1", host: "192.168.1.1"
But that's pretty common, probably just something misconfigured on the client.
You could certainly make it more secure by setting ssh to key only and restricting gui access to limited source IPs. If that's not already the case.
Steve
-
Port 2222 actually goes to an internal Ubuntu server.
192.168.1.4 is a Linksys 4200v2 WiFi router configured as a bridge/access point.
I will look into using Key only and restricting the GUI access.
-
Assuming that only you should have SSH access to that server, not the whole Internet, would it not be better to use OpenVPN to pfSense and then SSH to the Ubuntu server?
-
I may have to look into this a bit deeper. My router just rebooted again on it's own today:
Nov 28 12:01:59 syslogd kernel boot file is /boot/kernel/kernel Nov 28 12:01:00 syslogd exiting on signal 15 Nov 28 12:01:00 reboot rebooted by root Nov 28 12:00:00 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Nov 28 12:00:00 php [pfBlockerNG] Starting cron process. Nov 28 11:00:00 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Nov 28 11:00:00 php [pfBlockerNG] Starting cron process.
Could this be caused by some kind of scheduled job trying to reboot my router periodically?
-
Ok, I just locked down some ports, included SSH (port 2222). Still, this feels like something on the router and causing causing it to reboot. Either a scheduled process, or something causing a fault that results in a reboot.
-
ssh is not port 2222, 22 yes not 2222
-
They are forwarding port 2222 to ssh here. I assume you locked it down to known source IPs only.
Has it rebooted again since?
Steve
-
Yes, my isp blocks port 22 so I was forwarding port 2222 from wan to port 22 to the internal machine. I turned this off. The router rebooted another 2.5 hours later. I don’t think it has rebooted since.
-
I just updated to 2.4.4-RELEASE-p1. I will continue to monitor and see if this makes a difference.
-
@RyanM @stephenw10 Did you ever find a solution to this? I'm experiencing the same exact thing and my logs read the same with random daily reboots.
-
Exactly the same?
Let's see logs covering the run up to the reboot.
Steve
-
I guess not "exactly" but very similar. For no reason each morning I lose the internet connection to see pfsense is rebooting. I see this in the logs:
Oct 30 10:18:34 gateway syslogd: kernel boot file is /kernel
Oct 30 10:18:34 gateway kernel: Copyright (c) 1992-2018 The FreeBSD Project.
Oct 30 10:18:34 gateway kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Oct 30 10:18:34 gateway kernel: The Regents of the University of California. All rights reserved.
Oct 30 10:18:34 gateway kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Oct 30 10:18:34 gateway kernel: FreeBSD 11.2-RELEASE-p10 #21 10fea60fdde(factory-RELENG_2_4_4): Thu May 16 06:26:11 EDT 2019
Oct 30 10:18:34 gateway kernel: root@buildbot1-nyi.netgate.com:/build/factory-crossbuild-244/obj/aarch64/upm8hD25/arm64.aarch64/build/factory-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense arm64
Oct 30 10:18:34 gateway kernel: FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
Oct 30 10:18:34 gateway kernel: VT: init without driver.
Oct 30 10:18:34 gateway kernel: Starting CPU 1 (1)
Oct 30 10:18:34 gateway kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Oct 30 10:18:34 gateway kernel: random: entropy device external interface
Oct 30 10:18:34 gateway kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Oct 30 10:18:34 gateway kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.Oct 30 10:18:48 gateway php-cgi: rc.bootup:
Oct 30 10:19:07 gateway kernel: .done.
Oct 30 10:19:56 gateway php-cgi: rc.bootup: Creating rrd update script
ct 30 10:19:57 gateway root: /etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Oct 30 10:20:00 gateway syslogd: exiting on signal 15
Oct 30 10:20:00 gateway syslogd: kernel boot file is /kernel
Oct 30 10:20:00 gateway kernel: done.
Oct 30 10:20:00 gateway kernel: done.
Oct 30 10:20:06 gateway php-fpm[363]: /rc.start_packages: Restarting/Starting all packages.
Oct 30 10:20:06 gateway php-fpm[363]: [pfBlockerNG] Starting cron process.
Oct 30 10:20:08 gateway getty[7801]: open /dev/ttyv0: No such file or directory
Oct 30 10:20:08 gateway login: login on ttyu0 as rootAnd I was not logged into the console and was sitting right next to the box so nobody was physically here.
-
@RyanM @stephenw10 I just updated with some logs. Thanks for the help.
-
Ok, importantly you are not seeing
rebooted by root
there. Rather it just appears to reboot.That's an SG-1100 I assume?
Are you able to connect to the serial console and log the output there across a reboot? That would show what's happening.
Steve
-
yep its an 1100. Are you thinking maybe its rebooting due to some overload on the hardware or potential hardware issue then?
I should be able to connect to the serial console directly. How would I log the output from there and then do a reboot?
-
If you are using putty in Windows (or Linux) you can just enable logging there to get a file directly. Most terminal clients will have enough scroll back anyway to just copy and paste it out.
You would need to just leave it connected and wait for it to reboot unless you are able to predict when it will happen.Steve