[SOLVED] Router reboots in a day after upgrade to new snapshot



  • I have latest snapshot of pfSense 2.4 installed with RAM Disk enabled.

    After upgrade to the next snapshot pfSense works fine for approximately one day and reboots itself. All of the logs and monitoring data during this period (after upgrade and before reboot) are lost. However, after reboot it works fine for many days and keeps all logs properly.

    Here is how system log looks like:

    
    Jun 14 15:04:00     kernel         Copyright (c) 1992-2016 The FreeBSD Project.
    Jun 14 15:04:00     syslogd         kernel boot file is /boot/kernel/kernel
    Jun 13 16:40:45     php-fpm     74160     /rc.start_packages: Restarting/Starting all packages.
    Jun 13 16:40:44     check_reload_status         Starting packages 
    

    As you can see, everything that happened after Jun 13 16:40:45 and before Jun 14 15:04:00 do not exist anymore.

    I'm using amd64 version with UEFI, the issue happened to me twice already. First time I thought it was me doing something wrong, but it appears to be pfSense.

    I've originally posted this to bug tracker, but was redirected here: https://redmine.pfsense.org/issues/7647
    I was suggested to

    capture console output when the crash or reboot happens
    

    Could anyone explain me how to do this in pfSense? I don't have a capture card and reboot typically happens when I'm not looking at router and even HDMI cable is not attached.



  • I've upgraded system today (2.4.0.b.20170612.1823 -> 2.4.0.b.20170621.0152) once again and this time rebooted it immediately after upgrade:

    Jun 21 11:21:17 	sshlockout 	90899 	sshlockout/webConfigurator v3.0 starting up
    Jun 21 11:21:17 	php-fpm 	320 	/index.php: Successful login for user 'admin' from: 192.168.1.2
    Jun 21 11:21:13 	php-fpm 	320 	/rc.start_packages: Restarting/Starting all packages.
    Jun 21 11:21:13 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 21 11:21:13 	syslogd 		exiting on signal 15
    Jun 21 11:21:11 	root 		/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
    Jun 21 11:21:11 	php-cgi 		rc.bootup: Creating rrd update script
    Jun 21 11:21:09 	rtsold 	15820 	 <sendpacket>sendmsg on re0: Permission denied
    Jun 21 11:21:06 	check_reload_status 		Updating all dyndns
    Jun 21 11:21:06 	dhcpleases 		kqueue error: unkown
    Jun 21 11:21:06 	php-cgi 		rc.bootup: NTPD is starting up.
    Jun 21 11:21:05 	rtsold 	15820 	 <sendpacket>sendmsg on re0: Permission denied
    Jun 21 11:21:05 	kernel 		em0: link state changed to UP
    Jun 21 11:21:05 	check_reload_status 		Linkup starting em0
    Jun 21 11:21:03 	kernel 		bridge0: link state changed to UP
    Jun 21 11:21:03 	kernel 		ue0: link state changed to UP
    Jun 21 11:21:03 	check_reload_status 		Linkup starting ue0
    Jun 21 11:21:01 	rtsold 	15820 	 <sendpacket>sendmsg on re0: Permission denied
    Jun 21 11:21:01 	php-cgi 		rc.bootup: ROUTING: setting default route to 176.122.88.1
    Jun 21 11:21:01 	kernel 		pflog0: promiscuous mode enabled
    Jun 21 11:21:01 	kernel 		bridge0: link state changed to DOWN
    Jun 21 11:21:01 	kernel 		ue0: promiscuous mode enabled
    Jun 21 11:21:01 	kernel 		bridge0: link state changed to UP
    Jun 21 11:21:01 	kernel 		em0: promiscuous mode enabled
    Jun 21 11:21:01 	kernel 		bridge0: Ethernet address: 02:78:94:8f:a5:00
    Jun 21 11:21:01 	php-cgi 		rc.bootup: Resyncing OpenVPN instances.
    Jun 21 11:21:01 	kernel 		ue0: link state changed to DOWN
    Jun 21 11:21:01 	check_reload_status 		Linkup starting ue0
    Jun 21 11:21:01 	check_reload_status 		Linkup starting ue0
    Jun 21 11:21:01 	kernel 		ue0: link state changed to UP
    Jun 21 11:21:01 	check_reload_status 		Linkup starting em0
    Jun 21 11:21:01 	kernel 		em0: link state changed to DOWN
    Jun 21 11:21:01 	kernel 		em0: link state changed to UP
    Jun 21 11:21:01 	check_reload_status 		Linkup starting em0
    Jun 21 11:21:00 	php-fpm 	320 	/rc.newwanip: rc.newwanip: on (IP address: 176.122.88.128) (interface: WAN[wan]) (real interface: re0).
    Jun 21 11:21:00 	php-fpm 	320 	/rc.newwanip: rc.newwanip: Info: starting on re0.
    Jun 21 11:20:58 	php-cgi 		rc.bootup: Starting rtsold process
    Jun 21 11:20:58 	php-cgi 		rc.bootup: Accept router advertisements on interface re0
    Jun 21 11:20:58 	check_reload_status 		rc.newwanip starting re0
    Jun 21 11:20:56 	kernel 		re0: link state changed to UP
    Jun 21 11:20:56 	check_reload_status 		Linkup starting re0
    Jun 21 11:20:54 	kernel 		re0: link state changed to DOWN
    Jun 21 11:20:54 	check_reload_status 		Linkup starting re0
    Jun 21 11:20:54 	kernel 		re0: link state changed to UP
    Jun 21 11:20:54 	check_reload_status 		Linkup starting re0
    Jun 21 11:20:54 	kernel 		coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
    Jun 21 11:20:54 	kernel 		coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
    Jun 21 11:20:54 	kernel 		coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
    Jun 21 11:20:54 	kernel 		coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
    Jun 21 11:20:54 	kernel 		aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
    Jun 21 11:20:54 	kernel 		padlock0: No ACE support.
    Jun 21 11:20:54 	kernel 		da0: quirks=0x2 <no_6_byte>Jun 21 11:20:54 	kernel 		da0: 15237MB (31205621 512 byte sectors)
    Jun 21 11:20:54 	kernel 		da0: 400.000MB/s transfers
    Jun 21 11:20:54 	kernel 		da0: Serial Number 20170606
    Jun 21 11:20:54 	kernel 		da0: <ssd is917="" 1.00="">Removable Direct Access SPC-4 SCSI device
    Jun 21 11:20:54 	kernel 		da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
    Jun 21 11:20:54 	kernel 		mountroot: waiting for device /dev/gpt/pfSense...
    Jun 21 11:20:54 	kernel 		umass0: <is917 0="" 3="" ssd-16gb,="" class="" 0,="" rev="" 3.00="" 0.01,="" addr="">on usbus0
    Jun 21 11:20:54 	kernel 		ugen0.4: <is917>at usbus0
    Jun 21 11:20:54 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:20:54 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:20:54 	kernel 		ue0: Ethernet address: 10:13:50:a4:51:70
    Jun 21 11:20:54 	kernel 		ue0: <usb ethernet="">on udav0
    Jun 21 11:20:54 	kernel 		ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    Jun 21 11:20:54 	kernel 		ukphy0: <generic ieee="" 802.3u="" media="" interface="">PHY 0 on miibus1
    Jun 21 11:20:54 	kernel 		miibus1: <mii bus="">on udav0
    Jun 21 11:20:54 	kernel 		udav0: <vendor 0="" 2="" 0x0fe6="" usb="" network="" controller,="" class="" 0,="" rev="" 1.10="" 1.01,="" addr="">on usbus0
    Jun 21 11:20:54 	kernel 		ugen0.3: <vendor 0x0fe6="">at usbus0
    Jun 21 11:20:54 	kernel 		uhub1: 4 ports with 4 removable, self powered
    Jun 21 11:20:54 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:20:54 	kernel 		uhub1: <vendor 1="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 88.31,="" addr="">on usbus0
    Jun 21 11:20:54 	kernel 		ugen0.2: <vendor 0x05e3="">at usbus0
    Jun 21 11:20:54 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:20:54 	kernel 		uhub0: 13 ports with 13 removable, self powered
    Jun 21 11:20:54 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:20:54 	kernel 		Trying to mount root from ufs:/dev/gpt/pfSense [rw]...
    Jun 21 11:20:54 	kernel 		Timecounter "TSC" frequency 1600052360 Hz quality 1000
    Jun 21 11:20:54 	kernel 		SMP: AP CPU #2 Launched!
    Jun 21 11:20:54 	kernel 		SMP: AP CPU #1 Launched!
    Jun 21 11:20:54 	kernel 		SMP: AP CPU #3 Launched!
    Jun 21 11:20:54 	kernel 		uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    Jun 21 11:20:54 	kernel 		ugen0.1: <0x8086> at usbus0
    Jun 21 11:20:54 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
    Jun 21 11:20:54 	kernel 		nvme cam probe device init
    Jun 21 11:20:54 	kernel 		Timecounters tick every 1.000 msec
    Jun 21 11:20:54 	kernel 		est3: <enhanced speedstep="" frequency="" control="">on cpu3
    Jun 21 11:20:54 	kernel 		est2: <enhanced speedstep="" frequency="" control="">on cpu2
    Jun 21 11:20:54 	kernel 		est1: <enhanced speedstep="" frequency="" control="">on cpu1
    Jun 21 11:20:54 	kernel 		est0: <enhanced speedstep="" frequency="" control="">on cpu0
    Jun 21 11:20:54 	kernel 		ppc0: cannot reserve I/O port range
    Jun 21 11:20:54 	kernel 		acpi_tz0: <thermal zone="">on acpi0
    Jun 21 11:20:54 	kernel 		acpi_button0: <sleep button="">on acpi0
    Jun 21 11:20:54 	kernel 		isa0: <isa bus="">on isab0
    Jun 21 11:20:54 	kernel 		isab0: <pci-isa bridge="">at device 31.0 on pci0
    Jun 21 11:20:54 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
    Jun 21 11:20:54 	kernel 		em0: Ethernet address: 68:05:ca:1a:e5:0a
    Jun 21 11:20:54 	kernel 		em0: Using MSIX interrupts with 3 vectors
    Jun 21 11:20:54 	kernel 		em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xd000-0xd01f mem 0x812c0000-0x812dffff,0x81200000-0x8127ffff,0x812e0000-0x812e3fff at device 0.0 on pci2
    Jun 21 11:20:54 	kernel 		pci2: <acpi pci="" bus="">on pcib2
    Jun 21 11:20:54 	kernel 		pcib2: [GIANT-LOCKED]
    Jun 21 11:20:54 	kernel 		pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0
    Jun 21 11:20:54 	kernel 		re0: netmap queues/slots: TX 1/256, RX 1/256
    Jun 21 11:20:54 	kernel 		re0: Ethernet address: d8:cb:8a:98:90:0a
    Jun 21 11:20:54 	kernel 		re0: Using defaults for TSO: 65518/35/2048
    Jun 21 11:20:54 	kernel 		rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    Jun 21 11:20:54 	kernel 		rgephy0: <rtl8251 1000base-t="" media="" interface="">PHY 1 on miibus0
    Jun 21 11:20:54 	kernel 		miibus0: <mii bus="">on re0
    Jun 21 11:20:54 	kernel 		re0: MAC rev. 0x00000000
    Jun 21 11:20:54 	kernel 		re0: Chip rev. 0x4c000000
    Jun 21 11:20:54 	kernel 		re0: Using 1 MSI-X message
    Jun 21 11:20:54 	kernel 		re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0x81300000-0x81300fff,0xa0000000-0xa0003fff at device 0.0 on pci1
    Jun 21 11:20:54 	kernel 		pci1: <acpi pci="" bus="">on pcib1
    Jun 21 11:20:54 	kernel 		pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0
    Jun 21 11:20:54 	kernel 		pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
    Jun 21 11:20:54 	kernel 		usbus0 on xhci0
    Jun 21 11:20:54 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
    Jun 21 11:20:54 	kernel 		xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81400000-0x8140ffff at device 20.0 on pci0
    Jun 21 11:20:54 	kernel 		ahcich1: <ahci channel="">at channel 1 on ahci0
    Jun 21 11:20:54 	kernel 		ahcich0: <ahci channel="">at channel 0 on ahci0
    Jun 21 11:20:54 	kernel 		ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
    Jun 21 11:20:54 	kernel 		ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81411000-0x814117ff at device 19.0 on pci0
    Jun 21 11:20:54 	kernel 		vgapci0: Boot video device
    Jun 21 11:20:54 	kernel 		vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff at device 2.0 on pci0
    Jun 21 11:20:54 	kernel 		pci0: <acpi pci="" bus="">on pcib0
    Jun 21 11:20:54 	kernel 		pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    Jun 21 11:20:54 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    Jun 21 11:20:54 	kernel 		Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    Jun 21 11:20:54 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
    Jun 21 11:20:54 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
    Jun 21 11:20:54 	kernel 		attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Jun 21 11:20:54 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
    Jun 21 11:20:54 	kernel 		atrtc0: Warning: Couldn't map I/O.
    Jun 21 11:20:54 	kernel 		atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
    Jun 21 11:20:54 	kernel 		cpu3: <acpi cpu="">on acpi0
    Jun 21 11:20:54 	kernel 		cpu2: <acpi cpu="">on acpi0
    Jun 21 11:20:54 	kernel 		cpu1: <acpi cpu="">on acpi0
    Jun 21 11:20:54 	kernel 		cpu0: <acpi cpu="">on acpi0
    Jun 21 11:20:54 	kernel 		unknown: I/O range not supported
    Jun 21 11:20:54 	kernel 		acpi0: Power Button (fixed)
    Jun 21 11:20:54 	kernel 		acpi0: <alaska a="" m="" i="">on motherboard
    Jun 21 11:20:54 	kernel 		padlock0: No ACE support.
    Jun 21 11:20:54 	kernel 		cryptosoft0: <software crypto="">on motherboard
    Jun 21 11:20:54 	kernel 		random: fast provider: "Intel Secure Key RNG"
    Jun 21 11:20:54 	kernel 		random: registering fast source Intel Secure Key RNG
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (vesa, 0xffffffff8122cb10, 0) error 19
    Jun 21 11:20:54 	kernel 		netmap: loaded module
    Jun 21 11:20:54 	kernel 		kbd1 at kbdmux0
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069ebb0, 0) error 1
    Jun 21 11:20:54 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 21 11:20:54 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069eb00, 0) error 1
    Jun 21 11:20:54 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 21 11:20:54 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069ea50, 0) error 1
    Jun 21 11:20:54 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 21 11:20:54 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 21 11:20:54 	kernel 		wlan: mac acl policy registered
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80675620, 0) error 1
    Jun 21 11:20:54 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 21 11:20:54 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80675570, 0) error 1
    Jun 21 11:20:54 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 21 11:20:54 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 21 11:20:54 	kernel 		random: entropy device external interface
    Jun 21 11:20:54 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806754c0, 0) error 1
    Jun 21 11:20:54 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 21 11:20:54 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 21 11:20:54 	kernel 		ioapic0 <version 2.0="">irqs 0-114 on motherboard
    Jun 21 11:20:54 	kernel 		ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
    Jun 21 11:20:54 	kernel 		random: unblocking device.
    Jun 21 11:20:54 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
    Jun 21 11:20:54 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    Jun 21 11:20:54 	kernel 		0 < 1
    Jun 21 11:20:54 	kernel 		WARNING: L1 data cache covers less APIC IDs than a core
    Jun 21 11:20:54 	kernel 		ACPI APIC Table: <alaska a="" m="" i="">Jun 21 11:20:54 	kernel 		Event timer "LAPIC" quality 600
    Jun 21 11:20:54 	kernel 		avail memory = 1933852672 (1844 MB)
    Jun 21 11:20:54 	kernel 		real memory = 2147483648 (2048 MB)
    Jun 21 11:20:54 	kernel 		TSC: P-state invariant, performance statistics
    Jun 21 11:20:54 	kernel 		VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
    Jun 21 11:20:54 	kernel 		Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>Jun 21 11:20:54 	kernel 		AMD Features2=0x101 <lahf,prefetch>Jun 21 11:20:54 	kernel 		AMD Features=0x28100800 <syscall,nx,rdtscp,lm>Jun 21 11:20:54 	kernel 		Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>Jun 21 11:20:54 	kernel 		Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jun 21 11:20:54 	kernel 		Origin="GenuineIntel" Id=0x406c3 Family=0x6 Model=0x4c Stepping=3
    Jun 21 11:20:54 	kernel 		CPU: Intel(R) Celeron(R) CPU N3150 @ 1.60GHz (1600.05-MHz K8-class CPU)
    Jun 21 11:20:54 	kernel 		VT(vga): resolution 640x480
    Jun 21 11:20:54 	kernel 		FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
    Jun 21 11:20:54 	kernel 		root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
    Jun 21 11:20:54 	kernel 		FreeBSD 11.0-RELEASE-p10 #138 981956f70fd(RELENG_2_4): Wed Jun 21 01:59:51 CDT 2017
    Jun 21 11:20:54 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
    Jun 21 11:20:54 	kernel 		The Regents of the University of California. All rights reserved.
    Jun 21 11:20:54 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    Jun 21 11:20:54 	kernel 		Copyright (c) 1992-2016 The FreeBSD Project.
    Jun 21 11:20:54 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 21 11:19:42 	php-fpm 	18842 	/diag_reboot.php: Stopping all packages.
    Jun 21 11:17:15 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 21 11:17:15 	syslogd 		exiting on signal 15
    Jun 21 11:17:13 	root 		/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
    Jun 21 11:17:13 	check_reload_status 		Syncing firewall
    Jun 21 11:11:55 	sshlockout 	88395 	sshlockout/webConfigurator v3.0 starting up
    Jun 21 11:11:55 	pfsense.localdomain 		nginx: 2017/06/21 11:11:55 [error] 30876#100083: send() failed (54: Connection reset by peer)
    Jun 21 11:11:55 	php-fpm 	18574 	/index.php: Successful login for user 'admin' from: 192.168.1.2
    Jun 21 11:11:46 	php-fpm 	18574 	/rc.start_packages: Restarting/Starting all packages.
    Jun 21 11:11:46 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 21 11:11:45 	syslogd 		exiting on signal 15
    Jun 21 11:11:43 	root 		/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
    Jun 21 11:11:43 	php-cgi 		rc.bootup: Creating rrd update script
    Jun 21 11:11:39 	check_reload_status 		Updating all dyndns
    Jun 21 11:11:39 	dhcpleases 		kqueue error: unkown
    Jun 21 11:11:39 	php-cgi 		rc.bootup: NTPD is starting up.
    Jun 21 11:11:21 	php-cgi 		rc.bootup: ROUTING: setting default route to 176.122.88.1
    Jun 21 11:11:21 	kernel 		pflog0: promiscuous mode enabled
    Jun 21 11:11:00 	check_reload_status 		Linkup starting em0
    Jun 21 11:11:00 	check_reload_status 		Linkup starting ue0
    Jun 21 11:10:47 	kernel 		em0: link state changed to UP
    Jun 21 11:10:45 	kernel 		bridge0: link state changed to UP
    Jun 21 11:10:45 	kernel 		ue0: link state changed to UP
    Jun 21 11:10:43 	kernel 		bridge0: link state changed to DOWN
    Jun 21 11:10:43 	kernel 		ue0: promiscuous mode enabled
    Jun 21 11:10:43 	kernel 		bridge0: link state changed to UP
    Jun 21 11:10:43 	kernel 		em0: promiscuous mode enabled
    Jun 21 11:10:43 	php-cgi 		rc.bootup: Resyncing OpenVPN instances.
    Jun 21 11:10:43 	kernel 		bridge0: Ethernet address: 02:81:a9:9c:3c:00
    Jun 21 11:10:43 	check_reload_status 		Linkup starting ue0
    Jun 21 11:10:43 	check_reload_status 		Linkup starting ue0
    Jun 21 11:10:43 	kernel 		ue0: link state changed to DOWN
    Jun 21 11:10:43 	check_reload_status 		Linkup starting ue0
    Jun 21 11:10:43 	kernel 		ue0: link state changed to UP
    Jun 21 11:10:43 	kernel 		ue0: link state changed to DOWN
    Jun 21 11:10:43 	kernel 		ue0: link state changed to UP
    Jun 21 11:10:43 	check_reload_status 		Linkup starting ue0
    Jun 21 11:10:43 	check_reload_status 		Linkup starting em0
    Jun 21 11:10:43 	kernel 		em0: link state changed to DOWN
    Jun 21 11:10:43 	kernel 		em0: link state changed to UP
    Jun 21 11:10:43 	check_reload_status 		Linkup starting em0
    Jun 21 11:10:42 	php-fpm 	8220 	/rc.newwanip: rc.newwanip: on (IP address: 176.122.88.128) (interface: WAN[wan]) (real interface: re0).
    Jun 21 11:10:42 	php-fpm 	8220 	/rc.newwanip: rc.newwanip: Info: starting on re0.
    Jun 21 11:10:41 	php-cgi 		rc.bootup: Starting rtsold process
    Jun 21 11:10:41 	php-cgi 		rc.bootup: Accept router advertisements on interface re0
    Jun 21 11:10:41 	check_reload_status 		rc.newwanip starting re0
    Jun 21 11:10:38 	check_reload_status 		Linkup starting re0
    Jun 21 11:10:35 	kernel 		re0: link state changed to UP
    Jun 21 11:10:33 	check_reload_status 		Linkup starting re0
    Jun 21 11:10:33 	kernel 		re0: link state changed to DOWN
    Jun 21 11:10:33 	kernel 		re0: link state changed to UP
    Jun 21 11:10:33 	check_reload_status 		Linkup starting re0
    Jun 21 11:10:33 	check_reload_status 		starting sshd
    Jun 21 11:10:33 	root 		RAM disk restore succeeded.
    Jun 21 11:10:33 	root 		RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/rrd.tgz
    Jun 21 11:10:33 	root 		RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/logs.tgz
    Jun 21 11:10:33 	root 		RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/dhcpleases.tgz
    Jun 21 11:10:33 	root 		Restoring contents of RAM disk store...
    Jun 21 11:10:33 	kernel 		coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
    Jun 21 11:10:33 	kernel 		coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
    Jun 21 11:10:33 	kernel 		coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
    Jun 21 11:10:33 	kernel 		coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
    Jun 21 11:10:33 	kernel 		aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
    Jun 21 11:10:33 	kernel 		padlock0: No ACE support.
    Jun 21 11:10:33 	kernel 		da0: quirks=0x2 <no_6_byte>Jun 21 11:10:33 	kernel 		da0: 15237MB (31205621 512 byte sectors)
    Jun 21 11:10:33 	kernel 		da0: 400.000MB/s transfers
    Jun 21 11:10:33 	kernel 		da0: Serial Number 20170606
    Jun 21 11:10:33 	kernel 		da0: <ssd is917="" 1.00="">Removable Direct Access SPC-4 SCSI device
    Jun 21 11:10:33 	kernel 		da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
    Jun 21 11:10:33 	kernel 		mountroot: waiting for device /dev/gpt/pfSense...
    Jun 21 11:10:33 	kernel 		umass0: <is917 0="" 3="" ssd-16gb,="" class="" 0,="" rev="" 3.00="" 0.01,="" addr="">on usbus0
    Jun 21 11:10:33 	kernel 		ugen0.4: <is917>at usbus0
    Jun 21 11:10:33 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:10:33 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:10:33 	kernel 		ue0: Ethernet address: 10:13:50:a4:51:70
    Jun 21 11:10:33 	kernel 		ue0: <usb ethernet="">on udav0
    Jun 21 11:10:33 	kernel 		ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    Jun 21 11:10:33 	kernel 		ukphy0: <generic ieee="" 802.3u="" media="" interface="">PHY 0 on miibus1
    Jun 21 11:10:33 	kernel 		miibus1: <mii bus="">on udav0
    Jun 21 11:10:33 	kernel 		udav0: <vendor 0="" 2="" 0x0fe6="" usb="" network="" controller,="" class="" 0,="" rev="" 1.10="" 1.01,="" addr="">on usbus0
    Jun 21 11:10:33 	kernel 		ugen0.3: <vendor 0x0fe6="">at usbus0
    Jun 21 11:10:33 	kernel 		uhub1: 4 ports with 4 removable, self powered
    Jun 21 11:10:33 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:10:33 	kernel 		uhub1: <vendor 1="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 88.31,="" addr="">on usbus0
    Jun 21 11:10:33 	kernel 		ugen0.2: <vendor 0x05e3="">at usbus0
    Jun 21 11:10:33 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:10:33 	kernel 		uhub0: 13 ports with 13 removable, self powered
    Jun 21 11:10:33 	kernel 		Root mount waiting for: usbus0
    Jun 21 11:10:33 	kernel 		Trying to mount root from ufs:/dev/gpt/pfSense [rw]...
    Jun 21 11:10:33 	kernel 		Timecounter "TSC" frequency 1600052380 Hz quality 1000
    Jun 21 11:10:33 	kernel 		SMP: AP CPU #3 Launched!
    Jun 21 11:10:33 	kernel 		SMP: AP CPU #1 Launched!
    Jun 21 11:10:33 	kernel 		SMP: AP CPU #2 Launched!
    Jun 21 11:10:33 	kernel 		uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    Jun 21 11:10:33 	kernel 		ugen0.1: <0x8086> at usbus0
    Jun 21 11:10:33 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
    Jun 21 11:10:33 	kernel 		nvme cam probe device init
    Jun 21 11:10:33 	kernel 		Timecounters tick every 1.000 msec
    Jun 21 11:10:33 	kernel 		est3: <enhanced speedstep="" frequency="" control="">on cpu3
    Jun 21 11:10:33 	kernel 		est2: <enhanced speedstep="" frequency="" control="">on cpu2
    Jun 21 11:10:33 	kernel 		est1: <enhanced speedstep="" frequency="" control="">on cpu1
    Jun 21 11:10:33 	kernel 		est0: <enhanced speedstep="" frequency="" control="">on cpu0
    Jun 21 11:10:33 	kernel 		ppc0: cannot reserve I/O port range
    Jun 21 11:10:33 	kernel 		acpi_tz0: <thermal zone="">on acpi0
    Jun 21 11:10:33 	kernel 		acpi_button0: <sleep button="">on acpi0
    Jun 21 11:10:33 	kernel 		isa0: <isa bus="">on isab0
    Jun 21 11:10:33 	kernel 		isab0: <pci-isa bridge="">at device 31.0 on pci0
    Jun 21 11:10:33 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
    Jun 21 11:10:33 	kernel 		em0: Ethernet address: 68:05:ca:1a:e5:0a
    Jun 21 11:10:33 	kernel 		em0: Using MSIX interrupts with 3 vectors
    Jun 21 11:10:33 	kernel 		em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xd000-0xd01f mem 0x812c0000-0x812dffff,0x81200000-0x8127ffff,0x812e0000-0x812e3fff at device 0.0 on pci2
    Jun 21 11:10:33 	kernel 		pci2: <acpi pci="" bus="">on pcib2
    Jun 21 11:10:33 	kernel 		pcib2: [GIANT-LOCKED]
    Jun 21 11:10:33 	kernel 		pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0
    Jun 21 11:10:33 	kernel 		re0: netmap queues/slots: TX 1/256, RX 1/256
    Jun 21 11:10:33 	kernel 		re0: Ethernet address: d8:cb:8a:98:90:0a
    Jun 21 11:10:33 	kernel 		re0: Using defaults for TSO: 65518/35/2048
    Jun 21 11:10:33 	kernel 		rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    Jun 21 11:10:33 	kernel 		rgephy0: <rtl8251 1000base-t="" media="" interface="">PHY 1 on miibus0
    Jun 21 11:10:33 	kernel 		miibus0: <mii bus="">on re0
    Jun 21 11:10:33 	kernel 		re0: MAC rev. 0x00000000
    Jun 21 11:10:33 	kernel 		re0: Chip rev. 0x4c000000
    Jun 21 11:10:33 	kernel 		re0: Using 1 MSI-X message
    Jun 21 11:10:33 	kernel 		re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0x81300000-0x81300fff,0xa0000000-0xa0003fff at device 0.0 on pci1
    Jun 21 11:10:33 	kernel 		pci1: <acpi pci="" bus="">on pcib1
    Jun 21 11:10:33 	kernel 		pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0
    Jun 21 11:10:33 	kernel 		pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
    Jun 21 11:10:33 	kernel 		usbus0 on xhci0
    Jun 21 11:10:33 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
    Jun 21 11:10:33 	kernel 		xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81400000-0x8140ffff at device 20.0 on pci0
    Jun 21 11:10:33 	kernel 		ahcich1: <ahci channel="">at channel 1 on ahci0
    Jun 21 11:10:33 	kernel 		ahcich0: <ahci channel="">at channel 0 on ahci0
    Jun 21 11:10:33 	kernel 		ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
    Jun 21 11:10:33 	kernel 		ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81411000-0x814117ff at device 19.0 on pci0
    Jun 21 11:10:33 	kernel 		vgapci0: Boot video device
    Jun 21 11:10:33 	kernel 		vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff at device 2.0 on pci0
    Jun 21 11:10:33 	kernel 		pci0: <acpi pci="" bus="">on pcib0
    Jun 21 11:10:33 	kernel 		pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    Jun 21 11:10:33 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    Jun 21 11:10:33 	kernel 		Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    Jun 21 11:10:33 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
    Jun 21 11:10:33 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
    Jun 21 11:10:33 	kernel 		attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Jun 21 11:10:33 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
    Jun 21 11:10:33 	kernel 		atrtc0: Warning: Couldn't map I/O.
    Jun 21 11:10:33 	kernel 		atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
    Jun 21 11:10:33 	kernel 		cpu3: <acpi cpu="">on acpi0
    Jun 21 11:10:33 	kernel 		cpu2: <acpi cpu="">on acpi0
    Jun 21 11:10:33 	kernel 		cpu1: <acpi cpu="">on acpi0
    Jun 21 11:10:33 	kernel 		cpu0: <acpi cpu="">on acpi0
    Jun 21 11:10:33 	kernel 		unknown: I/O range not supported
    Jun 21 11:10:33 	kernel 		acpi0: Power Button (fixed)
    Jun 21 11:10:33 	kernel 		acpi0: <alaska a="" m="" i="">on motherboard
    Jun 21 11:10:33 	kernel 		padlock0: No ACE support.
    Jun 21 11:10:33 	kernel 		cryptosoft0: <software crypto="">on motherboard
    Jun 21 11:10:33 	kernel 		random: fast provider: "Intel Secure Key RNG"
    Jun 21 11:10:33 	kernel 		random: registering fast source Intel Secure Key RNG
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (vesa, 0xffffffff8122cb10, 0) error 19
    Jun 21 11:10:33 	kernel 		netmap: loaded module
    Jun 21 11:10:33 	kernel 		kbd1 at kbdmux0
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069ebb0, 0) error 1
    Jun 21 11:10:33 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 21 11:10:33 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069eb00, 0) error 1
    Jun 21 11:10:33 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 21 11:10:33 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069ea50, 0) error 1
    Jun 21 11:10:33 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 21 11:10:33 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 21 11:10:33 	kernel 		wlan: mac acl policy registered
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80675620, 0) error 1
    Jun 21 11:10:33 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 21 11:10:33 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80675570, 0) error 1
    Jun 21 11:10:33 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 21 11:10:33 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 21 11:10:33 	kernel 		random: entropy device external interface
    Jun 21 11:10:33 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806754c0, 0) error 1
    Jun 21 11:10:33 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 21 11:10:33 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 21 11:10:33 	kernel 		ioapic0 <version 2.0="">irqs 0-114 on motherboard
    Jun 21 11:10:33 	kernel 		ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
    Jun 21 11:10:33 	kernel 		random: unblocking device.
    Jun 21 11:10:33 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
    Jun 21 11:10:33 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    Jun 21 11:10:33 	kernel 		0 < 1
    Jun 21 11:10:33 	kernel 		WARNING: L1 data cache covers less APIC IDs than a core
    Jun 21 11:10:33 	kernel 		ACPI APIC Table: <alaska a="" m="" i="">Jun 21 11:10:33 	kernel 		Event timer "LAPIC" quality 600
    Jun 21 11:10:33 	kernel 		avail memory = 1933852672 (1844 MB)
    Jun 21 11:10:33 	kernel 		real memory = 2147483648 (2048 MB)
    Jun 21 11:10:33 	kernel 		TSC: P-state invariant, performance statistics
    Jun 21 11:10:33 	kernel 		VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
    Jun 21 11:10:33 	kernel 		Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>Jun 21 11:10:33 	kernel 		AMD Features2=0x101 <lahf,prefetch>Jun 21 11:10:33 	kernel 		AMD Features=0x28100800 <syscall,nx,rdtscp,lm>Jun 21 11:10:33 	kernel 		Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>Jun 21 11:10:33 	kernel 		Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jun 21 11:10:33 	kernel 		Origin="GenuineIntel" Id=0x406c3 Family=0x6 Model=0x4c Stepping=3
    Jun 21 11:10:33 	kernel 		CPU: Intel(R) Celeron(R) CPU N3150 @ 1.60GHz (1600.05-MHz K8-class CPU)
    Jun 21 11:10:33 	kernel 		VT(vga): resolution 640x480
    Jun 21 11:10:33 	kernel 		FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
    Jun 21 11:10:33 	kernel 		root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
    Jun 21 11:10:33 	kernel 		FreeBSD 11.0-RELEASE-p10 #138 981956f70fd(RELENG_2_4): Wed Jun 21 01:59:51 CDT 2017
    Jun 21 11:10:33 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
    Jun 21 11:10:33 	kernel 		The Regents of the University of California. All rights reserved.
    Jun 21 11:10:33 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    Jun 21 11:10:33 	kernel 		Copyright (c) 1992-2016 The FreeBSD Project.
    Jun 21 11:10:33 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 21 11:01:35 	php-fpm 	91025 	/rc.start_packages: Restarting/Starting all packages.
    Jun 21 11:01:34 	check_reload_status 		Starting packages
    Jun 21 11:01:34 	check_reload_status 		Reloading filter
    Jun 21 11:01:32 	pkg 		pfSense-kernel-pfSense upgraded: 2.4.0.b.20170612.1823 -> 2.4.0.b.20170621.0152
    Jun 21 11:00:52 	pkg 		pfSense-rc upgraded: 2.4.0.b.20170612.1823 -> 2.4.0.b.20170621.0152
    Jun 21 10:59:43 	check_reload_status 		Syncing firewall
    Jun 21 10:50:39 	pkg 		pfSense-repo upgraded: 2.4.0.b.20170620.1018 -> 2.4.0.b.20170620.2258
    Jun 21 10:50:39 	pkg 		pfSense-upgrade upgraded: 0.23 -> 0.24</fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></syscall,nx,rdtscp,lm></lahf,prefetch></tscadj,smep,erms,nfpusg></alaska></version></software></alaska></acpi></acpi></acpi></acpi></at></at></acpi></acpi></vga-compatible></ahci></ahci></ahci></xhci></encrypt></acpi></acpi></realtek></mii></rtl8251></acpi></acpi></intel(r)></pci-isa></isa></sleep></thermal></enhanced></enhanced></enhanced></enhanced></vendor></vendor></vendor></vendor></mii></generic></usb></is917></is917></ssd></no_6_byte></aes-cbc,aes-xts,aes-gcm,aes-icm></cpu></cpu></cpu></cpu></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></syscall,nx,rdtscp,lm></lahf,prefetch></tscadj,smep,erms,nfpusg></alaska></version></software></alaska></acpi></acpi></acpi></acpi></at></at></acpi></acpi></vga-compatible></ahci></ahci></ahci></xhci></encrypt></acpi></acpi></realtek></mii></rtl8251></acpi></acpi></intel(r)></pci-isa></isa></sleep></thermal></enhanced></enhanced></enhanced></enhanced></vendor></vendor></vendor></vendor></mii></generic></usb></is917></is917></ssd></no_6_byte></aes-cbc,aes-xts,aes-gcm,aes-icm></cpu></cpu></cpu></cpu></sendpacket></sendpacket></sendpacket> 
    

    There is something going wrong here, since right after upgrade system memory was 21% of 1916 MiB, however, after reboot it went back to normal 11% of 1916 MiB.

    So:

    1. There is something consuming 10% of memory after reboot that doesn't consume memory otherwise
    2. Logs are lost (not persisted) if system is not rebooted after upgrade manually, but instead crashes on its owm (reboots actually)


  • Loosing logs on a unexpected reboot and having "RAM Disk enabled" likely explains that part..
    What packages do you have installed.?.



  • @PiBa:

    Loosing logs on a unexpected reboot and having "RAM Disk enabled" likely explains that part..
    What packages do you have installed.?.

    I've installed ntopng in past long time ago, but it didn't work well with RAM disk, so I've removed it and do not have any packages besides stock pfSense.



  • Could you disable the ram disks before the next update.? That should hopefully keep logs around to tell something more about the 'crash'.



  • @PiBa:

    Could you disable the ram disks before the next update.? That should hopefully keep logs around to tell something more about the 'crash'.

    It slows down the system significantly, but I'll try. I suspect, however, that it is actually caused by enabling RAM Disk (partially because of increased RAM usage right after upgrade).



  • One more think that indicates RAM Disk problem is that Periodic RAM Disk Data Backups is configured to 1 hour (all 3 options there) and the fact that no system logs and no RRD data were saved makes me wondering what happened to this feature and how to check if it was/is properly scheduled.



  • I've disabled RAM Disk, rebooted and upgraded the system.
    It is running for 2 days and 17 hours now without any crashes and issues. Here are relevant system logs:

    Jun 23 00:12:17 	php-fpm 	35108 	/rc.start_packages: Restarting/Starting all packages.
    Jun 23 00:12:17 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 23 00:12:17 	syslogd 		exiting on signal 15
    Jun 23 00:12:15 	root 		/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
    Jun 23 00:12:14 	php-cgi 		rc.bootup: Creating rrd update script
    Jun 23 00:12:09 	check_reload_status 		Updating all dyndns
    Jun 23 00:12:06 	rtsold 	14452 	 <sendpacket>sendmsg on re0: Permission denied
    Jun 23 00:12:04 	php-cgi 		rc.bootup: NTPD is starting up.
    Jun 23 00:12:02 	rtsold 	14452 	 <sendpacket>sendmsg on re0: Permission denied
    Jun 23 00:12:01 	kernel 		em0: link state changed to UP
    Jun 23 00:12:01 	check_reload_status 		Linkup starting em0
    Jun 23 00:12:00 	check_reload_status 		Linkup starting ue0
    Jun 23 00:11:59 	kernel 		bridge0: link state changed to UP
    Jun 23 00:11:59 	kernel 		ue0: link state changed to UP
    Jun 23 00:11:58 	rtsold 	14452 	 <sendpacket>sendmsg on re0: Permission denied
    Jun 23 00:11:58 	php-cgi 		rc.bootup: ROUTING: setting default route to 176.122.88.1
    Jun 23 00:11:58 	kernel 		pflog0: promiscuous mode enabled
    Jun 23 00:11:57 	kernel 		bridge0: link state changed to DOWN
    Jun 23 00:11:57 	kernel 		ue0: promiscuous mode enabled
    Jun 23 00:11:57 	kernel 		bridge0: link state changed to UP
    Jun 23 00:11:57 	kernel 		em0: promiscuous mode enabled
    Jun 23 00:11:57 	php-cgi 		rc.bootup: Resyncing OpenVPN instances.
    Jun 23 00:11:57 	kernel 		bridge0: Ethernet address: 02:83:1c:1d:71:00
    Jun 23 00:11:57 	kernel 		ue0: link state changed to DOWN
    Jun 23 00:11:57 	check_reload_status 		Linkup starting ue0
    Jun 23 00:11:57 	kernel 		ue0: link state changed to UP
    Jun 23 00:11:57 	check_reload_status 		Linkup starting ue0
    Jun 23 00:11:57 	check_reload_status 		Linkup starting em0
    Jun 23 00:11:57 	kernel 		em0: link state changed to DOWN
    Jun 23 00:11:57 	kernel 		em0: link state changed to UP
    Jun 23 00:11:57 	check_reload_status 		Linkup starting em0
    Jun 23 00:11:56 	php-fpm 	348 	/rc.newwanip: rc.newwanip: on (IP address: 176.122.88.128) (interface: WAN[wan]) (real interface: re0).
    Jun 23 00:11:56 	php-fpm 	348 	/rc.newwanip: rc.newwanip: Info: starting on re0.
    Jun 23 00:11:55 	php-cgi 		rc.bootup: Starting rtsold process
    Jun 23 00:11:55 	php-cgi 		rc.bootup: Accept router advertisements on interface re0
    Jun 23 00:11:55 	check_reload_status 		rc.newwanip starting re0
    Jun 23 00:11:55 	check_reload_status 		Linkup starting re0
    Jun 23 00:11:55 	kernel 		re0: link state changed to UP
    Jun 23 00:11:53 	check_reload_status 		Linkup starting re0
    Jun 23 00:11:53 	check_reload_status 		Linkup starting re0
    Jun 23 00:11:53 	kernel 		re0: link state changed to DOWN
    Jun 23 00:11:53 	kernel 		re0: link state changed to UP
    Jun 23 00:11:53 	kernel 		coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
    Jun 23 00:11:53 	kernel 		coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
    Jun 23 00:11:53 	kernel 		coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
    Jun 23 00:11:53 	kernel 		coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
    Jun 23 00:11:53 	kernel 		aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
    Jun 23 00:11:53 	kernel 		padlock0: No ACE support.
    Jun 23 00:11:53 	kernel 		da0: quirks=0x2 <no_6_byte>Jun 23 00:11:53 	kernel 		da0: 15237MB (31205621 512 byte sectors)
    Jun 23 00:11:53 	kernel 		da0: 400.000MB/s transfers
    Jun 23 00:11:53 	kernel 		da0: Serial Number 20170606
    Jun 23 00:11:53 	kernel 		da0: <ssd is917="" 1.00="">Removable Direct Access SPC-4 SCSI device
    Jun 23 00:11:53 	kernel 		da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
    Jun 23 00:11:53 	kernel 		mountroot: waiting for device /dev/gpt/pfSense...
    Jun 23 00:11:53 	kernel 		umass0: <is917 0="" 3="" ssd-16gb,="" class="" 0,="" rev="" 3.00="" 0.01,="" addr="">on usbus0
    Jun 23 00:11:53 	kernel 		ugen0.4: <is917>at usbus0
    Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
    Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
    Jun 23 00:11:53 	kernel 		ue0: Ethernet address: 10:13:50:a4:51:70
    Jun 23 00:11:53 	kernel 		ue0: <usb ethernet="">on udav0
    Jun 23 00:11:53 	kernel 		ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    Jun 23 00:11:53 	kernel 		ukphy0: <generic ieee="" 802.3u="" media="" interface="">PHY 0 on miibus1
    Jun 23 00:11:53 	kernel 		miibus1: <mii bus="">on udav0
    Jun 23 00:11:53 	kernel 		udav0: <vendor 0="" 2="" 0x0fe6="" usb="" network="" controller,="" class="" 0,="" rev="" 1.10="" 1.01,="" addr="">on usbus0
    Jun 23 00:11:53 	kernel 		ugen0.3: <vendor 0x0fe6="">at usbus0
    Jun 23 00:11:53 	kernel 		uhub1: 4 ports with 4 removable, self powered
    Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
    Jun 23 00:11:53 	kernel 		uhub1: <vendor 1="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 88.31,="" addr="">on usbus0
    Jun 23 00:11:53 	kernel 		ugen0.2: <vendor 0x05e3="">at usbus0
    Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
    Jun 23 00:11:53 	kernel 		uhub0: 13 ports with 13 removable, self powered
    Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
    Jun 23 00:11:53 	kernel 		Trying to mount root from ufs:/dev/gpt/pfSense [rw]...
    Jun 23 00:11:53 	kernel 		Timecounter "TSC" frequency 1600053120 Hz quality 1000
    Jun 23 00:11:53 	kernel 		SMP: AP CPU #3 Launched!
    Jun 23 00:11:53 	kernel 		SMP: AP CPU #1 Launched!
    Jun 23 00:11:53 	kernel 		SMP: AP CPU #2 Launched!
    Jun 23 00:11:53 	kernel 		uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    Jun 23 00:11:53 	kernel 		ugen0.1: <0x8086> at usbus0
    Jun 23 00:11:53 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
    Jun 23 00:11:53 	kernel 		nvme cam probe device init
    Jun 23 00:11:53 	kernel 		Timecounters tick every 1.000 msec
    Jun 23 00:11:53 	kernel 		est3: <enhanced speedstep="" frequency="" control="">on cpu3
    Jun 23 00:11:53 	kernel 		est2: <enhanced speedstep="" frequency="" control="">on cpu2
    Jun 23 00:11:53 	kernel 		est1: <enhanced speedstep="" frequency="" control="">on cpu1
    Jun 23 00:11:53 	kernel 		est0: <enhanced speedstep="" frequency="" control="">on cpu0
    Jun 23 00:11:53 	kernel 		ppc0: cannot reserve I/O port range
    Jun 23 00:11:53 	kernel 		acpi_tz0: <thermal zone="">on acpi0
    Jun 23 00:11:53 	kernel 		acpi_button0: <sleep button="">on acpi0
    Jun 23 00:11:53 	kernel 		isa0: <isa bus="">on isab0
    Jun 23 00:11:53 	kernel 		isab0: <pci-isa bridge="">at device 31.0 on pci0
    Jun 23 00:11:53 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
    Jun 23 00:11:53 	kernel 		em0: Ethernet address: 68:05:ca:1a:e5:0a
    Jun 23 00:11:53 	kernel 		em0: Using MSIX interrupts with 3 vectors
    Jun 23 00:11:53 	kernel 		em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xd000-0xd01f mem 0x812c0000-0x812dffff,0x81200000-0x8127ffff,0x812e0000-0x812e3fff at device 0.0 on pci2
    Jun 23 00:11:53 	kernel 		pci2: <acpi pci="" bus="">on pcib2
    Jun 23 00:11:53 	kernel 		pcib2: [GIANT-LOCKED]
    Jun 23 00:11:53 	kernel 		pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0
    Jun 23 00:11:53 	kernel 		re0: netmap queues/slots: TX 1/256, RX 1/256
    Jun 23 00:11:53 	kernel 		re0: Ethernet address: d8:cb:8a:98:90:0a
    Jun 23 00:11:53 	kernel 		re0: Using defaults for TSO: 65518/35/2048
    Jun 23 00:11:53 	kernel 		rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    Jun 23 00:11:53 	kernel 		rgephy0: <rtl8251 1000base-t="" media="" interface="">PHY 1 on miibus0
    Jun 23 00:11:53 	kernel 		miibus0: <mii bus="">on re0
    Jun 23 00:11:53 	kernel 		re0: MAC rev. 0x00000000
    Jun 23 00:11:53 	kernel 		re0: Chip rev. 0x4c000000
    Jun 23 00:11:53 	kernel 		re0: Using 1 MSI-X message
    Jun 23 00:11:53 	kernel 		re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0x81300000-0x81300fff,0xa0000000-0xa0003fff at device 0.0 on pci1
    Jun 23 00:11:53 	kernel 		pci1: <acpi pci="" bus="">on pcib1
    Jun 23 00:11:53 	kernel 		pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0
    Jun 23 00:11:53 	kernel 		pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
    Jun 23 00:11:53 	kernel 		usbus0 on xhci0
    Jun 23 00:11:53 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
    Jun 23 00:11:53 	kernel 		xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81400000-0x8140ffff at device 20.0 on pci0
    Jun 23 00:11:53 	kernel 		ahcich1: <ahci channel="">at channel 1 on ahci0
    Jun 23 00:11:53 	kernel 		ahcich0: <ahci channel="">at channel 0 on ahci0
    Jun 23 00:11:53 	kernel 		ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
    Jun 23 00:11:53 	kernel 		ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81411000-0x814117ff at device 19.0 on pci0
    Jun 23 00:11:53 	kernel 		vgapci0: Boot video device
    Jun 23 00:11:53 	kernel 		vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff at device 2.0 on pci0
    Jun 23 00:11:53 	kernel 		pci0: <acpi pci="" bus="">on pcib0
    Jun 23 00:11:53 	kernel 		pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    Jun 23 00:11:53 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    Jun 23 00:11:53 	kernel 		Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    Jun 23 00:11:53 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
    Jun 23 00:11:53 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
    Jun 23 00:11:53 	kernel 		attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Jun 23 00:11:53 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
    Jun 23 00:11:53 	kernel 		atrtc0: Warning: Couldn't map I/O.
    Jun 23 00:11:53 	kernel 		atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
    Jun 23 00:11:53 	kernel 		cpu3: <acpi cpu="">on acpi0
    Jun 23 00:11:53 	kernel 		cpu2: <acpi cpu="">on acpi0
    Jun 23 00:11:53 	kernel 		cpu1: <acpi cpu="">on acpi0
    Jun 23 00:11:53 	kernel 		cpu0: <acpi cpu="">on acpi0
    Jun 23 00:11:53 	kernel 		unknown: I/O range not supported
    Jun 23 00:11:53 	kernel 		acpi0: Power Button (fixed)
    Jun 23 00:11:53 	kernel 		acpi0: <alaska a="" m="" i="">on motherboard
    Jun 23 00:11:53 	kernel 		padlock0: No ACE support.
    Jun 23 00:11:53 	kernel 		cryptosoft0: <software crypto="">on motherboard
    Jun 23 00:11:53 	kernel 		random: fast provider: "Intel Secure Key RNG"
    Jun 23 00:11:53 	kernel 		random: registering fast source Intel Secure Key RNG
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (vesa, 0xffffffff8122cb10, 0) error 19
    Jun 23 00:11:53 	kernel 		netmap: loaded module
    Jun 23 00:11:53 	kernel 		kbd1 at kbdmux0
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069ebb0, 0) error 1
    Jun 23 00:11:53 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 23 00:11:53 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069eb00, 0) error 1
    Jun 23 00:11:53 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 23 00:11:53 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069ea50, 0) error 1
    Jun 23 00:11:53 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    Jun 23 00:11:53 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    Jun 23 00:11:53 	kernel 		wlan: mac acl policy registered
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80675620, 0) error 1
    Jun 23 00:11:53 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 23 00:11:53 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80675570, 0) error 1
    Jun 23 00:11:53 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 23 00:11:53 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 23 00:11:53 	kernel 		random: entropy device external interface
    Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806754c0, 0) error 1
    Jun 23 00:11:53 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    Jun 23 00:11:53 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    Jun 23 00:11:53 	kernel 		ioapic0 <version 2.0="">irqs 0-114 on motherboard
    Jun 23 00:11:53 	kernel 		ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
    Jun 23 00:11:53 	kernel 		random: unblocking device.
    Jun 23 00:11:53 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
    Jun 23 00:11:53 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    Jun 23 00:11:53 	kernel 		0 < 1
    Jun 23 00:11:53 	kernel 		WARNING: L1 data cache covers less APIC IDs than a core
    Jun 23 00:11:53 	kernel 		ACPI APIC Table: <alaska a="" m="" i="">Jun 23 00:11:53 	kernel 		Event timer "LAPIC" quality 600
    Jun 23 00:11:53 	kernel 		avail memory = 1933852672 (1844 MB)
    Jun 23 00:11:53 	kernel 		real memory = 2147483648 (2048 MB)
    Jun 23 00:11:53 	kernel 		TSC: P-state invariant, performance statistics
    Jun 23 00:11:53 	kernel 		VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
    Jun 23 00:11:53 	kernel 		Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>Jun 23 00:11:53 	kernel 		AMD Features2=0x101 <lahf,prefetch>Jun 23 00:11:53 	kernel 		AMD Features=0x28100800 <syscall,nx,rdtscp,lm>Jun 23 00:11:53 	kernel 		Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>Jun 23 00:11:53 	kernel 		Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jun 23 00:11:53 	kernel 		Origin="GenuineIntel" Id=0x406c3 Family=0x6 Model=0x4c Stepping=3
    Jun 23 00:11:53 	kernel 		CPU: Intel(R) Celeron(R) CPU N3150 @ 1.60GHz (1600.05-MHz K8-class CPU)
    Jun 23 00:11:53 	kernel 		VT(vga): resolution 640x480
    Jun 23 00:11:53 	kernel 		FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
    Jun 23 00:11:53 	kernel 		root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
    Jun 23 00:11:53 	kernel 		FreeBSD 11.0-RELEASE-p10 #139 981956f70fd(RELENG_2_4): Thu Jun 22 03:49:45 CDT 2017
    Jun 23 00:11:53 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
    Jun 23 00:11:53 	kernel 		The Regents of the University of California. All rights reserved.
    Jun 23 00:11:53 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    Jun 23 00:11:53 	kernel 		Copyright (c) 1992-2016 The FreeBSD Project.
    Jun 23 00:11:53 	syslogd 		kernel boot file is /boot/kernel/kernel 
    Jun 23 00:02:01 	syslogd 		exiting on signal 15
    Jun 23 00:02:01 	reboot 		rebooted by root
    Jun 23 00:01:51 	php-fpm 	88532 	/rc.start_packages: Restarting/Starting all packages.
    Jun 23 00:01:50 	check_reload_status 		Starting packages
    Jun 23 00:01:50 	check_reload_status 		Reloading filter
    Jun 23 00:01:39 	pkg 		pfSense-kernel-pfSense upgraded: 2.4.0.b.20170621.0152 -> 2.4.0.b.20170622.0342
    Jun 23 00:00:46 	pkg 		pfSense-rc upgraded: 2.4.0.b.20170621.0152 -> 2.4.0.b.20170622.0342</fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></syscall,nx,rdtscp,lm></lahf,prefetch></tscadj,smep,erms,nfpusg></alaska></version></software></alaska></acpi></acpi></acpi></acpi></at></at></acpi></acpi></vga-compatible></ahci></ahci></ahci></xhci></encrypt></acpi></acpi></realtek></mii></rtl8251></acpi></acpi></intel(r)></pci-isa></isa></sleep></thermal></enhanced></enhanced></enhanced></enhanced></vendor></vendor></vendor></vendor></mii></generic></usb></is917></is917></ssd></no_6_byte></aes-cbc,aes-xts,aes-gcm,aes-icm></cpu></cpu></cpu></cpu></sendpacket></sendpacket></sendpacket> 
    

    And once again, even with RAM Disk disabled right after ugrade I've seen 20% of RAM usage, while now I've just rebooted pfSense and it only consumes 10% of the RAM, so this aspect persists.



  • It happened for a third time, definitetly not a coincidense.
    This time it freezed while I was at my working machine. Didn't have monitor attached to the router, so I've hard rebooted it (maybe it would reboot itself after some time, I don't know).

    This time it survived a little bit over 3 days, all the logs from the moment upgrade started and till freeze are lost again:

    Jul 1 02:58:15 	kernel 		Copyright (c) 1992-2016 The FreeBSD Project.
    Jul 1 02:58:15 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jun 28 22:31:13 	php-fpm 	68463 	/rc.start_packages: Restarting/Starting all packages.
    Jun 28 22:31:12 	check_reload_status 		Starting packages
    Jun 28 22:31:12 	check_reload_status 		Reloading filter
    Jun 28 22:31:08 	pkg 		pfSense-kernel-pfSense upgraded: 2.4.0.b.20170622.0342 -> 2.4.0.b.20170627.1443 
    

    I don't know what happens, but there is an issue here for sure:

    • memory consumption right after upgrade is higher than it is otherwise
    • logs are not persisted to disk each hour as they should
    • system freezes/reboots after some time


  • Memory usage going to 20% instead of 10% by itself shouldn't cause trouble.. It might be interesting to see what processes are using that memory.. Though actually i wouldn't worry about that itself to much.. During bootup after a upgrade some extra steps are made to check/upgrade packages though once that finishes probably should expect memory usage to drop a little..

    Perhaps try a MemTest image and check if any of the memory sticks is faulty.? Or maybe remove some of its memory dimms as it has plenty of memory it seems.?.

    Backup job
    When the upgrade is done it could be nice to check if 'ps -A | grep cron' shows that thats running.. As that should call the log backup scripts.. Check under 'cat /etc/crontab | grep rc.backup' if the 3 rc.backup jobs are scheduled.
    And check after a hour if the '/cf/conf/RAM_Disk_Store/' is getting filled with new tgz files..
    If not try running them manually from console? To see if that succeeds..

    Freeze/reboot
    Might be caused by faulty memory, though could be lots of other things..
    Really difficult to tell something usefull here without some log/error to indicate what is actually going wrong..

    RAM Disk
    Please do try to attach a monitor when it happens or leave the RAM disk disabled it shouldnt affect performance much as pretty much everything a stock pfSense install does is move packets around, and that shouldn't need any disk access.. Anything specific there? Big captive portal with usage voucher counting or something?

    Overall you could re-install pfSense from scratch.? And restore the config file.. That way at least we know for sure that all binaries/filestructures arent a problem..



  • For the cron backup job it seems like the crontab itself is actually not properly written..



  • Ok so the backup job itself should get fixed once the pull-request is approved/pulled: https://github.com/pfsense/pfsense/pull/3771 .

    For the crashes/hang issue.. I still don't have a clue..



  • I've tested memory with Windows memory testing available in boot menu (just because it works with UEFI and memtest86 does not), after 2 loops it didn't find any issues, so memory should be fine.
    And it doesn't have plenty of memory (well, comparing to regular routers it does), just one 2G SO-DIMM stick of DDR3L.

    Also I've checked memory stats before and after upgrade, it seems that Memory usage shows Total memory - Free, while I was expecting just Active. So it is confusing (should be changed?) but generally memory consumption is fine after upgrade.

    It seems you've found the issue with backup job, so it should also be fixed at some point.

    I've initially enabled RAM Disk because of extremely slow USB 2.0 flash drive where pfSense lived, it was painfully slow and affected router performance a lot on any disk activity. Now I've upgraded to much faster USB 3.0 flash drive, but it is still not a proper SSD, so I'd like to avoid unnecessary writes there.

    I'll try to keep display connected and get something more about those freezes/reboots when it happens next time.

    Thank you for your help so far!



  • Ok, I've being tweaking UI with some Wi-Fi settings and then let it alone for few minutes and discovered that it feezed again and then rebooted itself (after a bit over 5 days uptime).
    RAM Disk was disabled, nothing in logs that would indicate why the system is rebooting:

    
    Jul 10 17:26:23 	syslogd 		kernel boot file is /boot/kernel/kernel
    Jul 10 17:17:52 	check_reload_status 		Syncing firewall
    Jul 10 17:17:43 	check_reload_status 		Syncing firewall
    Jul 10 17:17:36 	check_reload_status 		Reloading filter
    Jul 10 17:17:36 	php-fpm 	89258 	/rc.linkup: Ignoring link event for bridge member without IP config
    Jul 10 17:17:35 	kernel 		em0: link state changed to UP
    Jul 10 17:17:35 	check_reload_status 		Linkup starting em0
    Jul 10 17:17:32 	check_reload_status 		Reloading filter
    Jul 10 17:17:32 	php-fpm 	89258 	/rc.linkup: Ignoring link event for bridge member without IP config
    Jul 10 17:17:32 	dhcpleases 		/etc/hosts changed size from original!
    Jul 10 17:17:32 	check_reload_status 		Syncing firewall
    Jul 10 17:17:31 	kernel 		run0_wlan0: promiscuous mode enabled
    Jul 10 17:17:31 	kernel 		ue0: promiscuous mode enabled
    Jul 10 17:17:31 	kernel 		bridge0: link state changed to UP
    Jul 10 17:17:31 	kernel 		em0: promiscuous mode enabled
    Jul 10 17:17:31 	check_reload_status 		Linkup starting em0
    Jul 10 17:17:31 	kernel 		bridge0: Ethernet address: 02:0d:b2:50:df:00
    Jul 10 17:17:31 	kernel 		em0: promiscuous mode disabled
    Jul 10 17:17:31 	kernel 		bridge0: link state changed to DOWN
    Jul 10 17:17:31 	kernel 		ue0: promiscuous mode disabled
    Jul 10 17:17:31 	kernel 		em0: link state changed to DOWN
    Jul 10 17:17:31 	kernel 		run0_wlan0: promiscuous mode disabled
    Jul 10 17:15:19 	php-fpm 	42940 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
    Jul 10 17:15:19 	php-fpm 	42940 	/interfaces.php: Cloning new wireless interface run0_wlan1
    Jul 10 17:14:32 	php-fpm 	42940 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
    Jul 10 17:14:32 	php-fpm 	42940 	/interfaces.php: Cloning new wireless interface run0_wlan1
    Jul 10 17:12:44 	php-fpm 	51717 	/interfaces.php: Creating rrd update script
    Jul 10 17:12:44 	check_reload_status 		Reloading filter
    Jul 10 17:12:42 	check_reload_status 		updating dyndns opt3
    Jul 10 17:12:41 	dhcpleases 		kqueue error: unkown
    Jul 10 17:12:40 	dhcpleases 		Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Jul 10 17:12:40 	dhcpleases 		/etc/hosts changed size from original!
    Jul 10 17:12:38 	kernel 		run0_wlan0: promiscuous mode enabled
    Jul 10 17:12:38 	check_reload_status 		Restarting ipsec tunnels
    Jul 10 17:12:38 	kernel 		run0: firmware RT3071 ver. 0.33 loaded
    Jul 10 17:12:36 	kernel 		run0: firmware RT3071 ver. 0.33 loaded
    Jul 10 17:12:33 	check_reload_status 		Syncing firewall
    Jul 10 17:12:02 	php-fpm 	26730 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
    Jul 10 17:12:02 	php-fpm 	26730 	/interfaces.php: Cloning new wireless interface run0_wlan1
    Jul 10 17:11:57 	kernel 		wlan0: changing name to 'run0_wlan0'
    Jul 10 17:11:57 	kernel 		wlan0: Ethernet address: c8:3a:35:ca:f8:90
    Jul 10 17:11:57 	php-fpm 	26730 	/interfaces.php: Cloning new wireless interface run0_wlan0
    Jul 10 17:11:43 	kernel 		run0: MAC/BBP RT3572 (rev 0x0223), RF RT3052 (MIMO 2T2R), address c8:3a:35:ca:f8:90
    Jul 10 17:11:43 	kernel 		run0: <1.0> on usbus0
    Jul 10 17:11:43 	kernel 		ugen0.6: <ralink>at usbus0
    Jul 10 17:11:40 	kernel 		run0: at uhub1, port 3, addr 8 (disconnected)
    Jul 10 17:11:40 	kernel 		ugen0.6: <ralink>at usbus0 (disconnected)
    Jul 10 17:11:29 	php-fpm 	42425 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
    Jul 10 17:11:29 	php-fpm 	42425 	/interfaces.php: Cloning new wireless interface run0_wlan0</ralink></ralink> 
    


  • Without RAM disk, if the kernel was crashing it should write a dump file to /var/crash/  .. But then dashboard should show a big banner that tells it found that dump file.

    Unless of-course there is a problem with the storage / usb connections / usb controllers that causes the crash, then it cant write anything.. I'm just wildly guessing here though.. Sorry..



  • I thinks I've found the reason. It was faulty flash drive, it was returning read/write operations.