[SOLVED] Router reboots in a day after upgrade to new snapshot

nazar-pc · Jun 21, 2017, 11:50 PM

One more think that indicates RAM Disk problem is that Periodic RAM Disk Data Backups is configured to 1 hour (all 3 options there) and the fact that no system logs and no RRD data were saved makes me wondering what happened to this feature and how to check if it was/is properly scheduled.

nazar-pc · Jun 25, 2017, 2:29 PM

I've disabled RAM Disk, rebooted and upgraded the system.
It is running for 2 days and 17 hours now without any crashes and issues. Here are relevant system logs:

Jun 23 00:12:17 	php-fpm 	35108 	/rc.start_packages: Restarting/Starting all packages.
Jun 23 00:12:17 	syslogd 		kernel boot file is /boot/kernel/kernel
Jun 23 00:12:17 	syslogd 		exiting on signal 15
Jun 23 00:12:15 	root 		/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Jun 23 00:12:14 	php-cgi 		rc.bootup: Creating rrd update script
Jun 23 00:12:09 	check_reload_status 		Updating all dyndns
Jun 23 00:12:06 	rtsold 	14452 	 <sendpacket>sendmsg on re0: Permission denied
Jun 23 00:12:04 	php-cgi 		rc.bootup: NTPD is starting up.
Jun 23 00:12:02 	rtsold 	14452 	 <sendpacket>sendmsg on re0: Permission denied
Jun 23 00:12:01 	kernel 		em0: link state changed to UP
Jun 23 00:12:01 	check_reload_status 		Linkup starting em0
Jun 23 00:12:00 	check_reload_status 		Linkup starting ue0
Jun 23 00:11:59 	kernel 		bridge0: link state changed to UP
Jun 23 00:11:59 	kernel 		ue0: link state changed to UP
Jun 23 00:11:58 	rtsold 	14452 	 <sendpacket>sendmsg on re0: Permission denied
Jun 23 00:11:58 	php-cgi 		rc.bootup: ROUTING: setting default route to 176.122.88.1
Jun 23 00:11:58 	kernel 		pflog0: promiscuous mode enabled
Jun 23 00:11:57 	kernel 		bridge0: link state changed to DOWN
Jun 23 00:11:57 	kernel 		ue0: promiscuous mode enabled
Jun 23 00:11:57 	kernel 		bridge0: link state changed to UP
Jun 23 00:11:57 	kernel 		em0: promiscuous mode enabled
Jun 23 00:11:57 	php-cgi 		rc.bootup: Resyncing OpenVPN instances.
Jun 23 00:11:57 	kernel 		bridge0: Ethernet address: 02:83:1c:1d:71:00
Jun 23 00:11:57 	kernel 		ue0: link state changed to DOWN
Jun 23 00:11:57 	check_reload_status 		Linkup starting ue0
Jun 23 00:11:57 	kernel 		ue0: link state changed to UP
Jun 23 00:11:57 	check_reload_status 		Linkup starting ue0
Jun 23 00:11:57 	check_reload_status 		Linkup starting em0
Jun 23 00:11:57 	kernel 		em0: link state changed to DOWN
Jun 23 00:11:57 	kernel 		em0: link state changed to UP
Jun 23 00:11:57 	check_reload_status 		Linkup starting em0
Jun 23 00:11:56 	php-fpm 	348 	/rc.newwanip: rc.newwanip: on (IP address: 176.122.88.128) (interface: WAN[wan]) (real interface: re0).
Jun 23 00:11:56 	php-fpm 	348 	/rc.newwanip: rc.newwanip: Info: starting on re0.
Jun 23 00:11:55 	php-cgi 		rc.bootup: Starting rtsold process
Jun 23 00:11:55 	php-cgi 		rc.bootup: Accept router advertisements on interface re0
Jun 23 00:11:55 	check_reload_status 		rc.newwanip starting re0
Jun 23 00:11:55 	check_reload_status 		Linkup starting re0
Jun 23 00:11:55 	kernel 		re0: link state changed to UP
Jun 23 00:11:53 	check_reload_status 		Linkup starting re0
Jun 23 00:11:53 	check_reload_status 		Linkup starting re0
Jun 23 00:11:53 	kernel 		re0: link state changed to DOWN
Jun 23 00:11:53 	kernel 		re0: link state changed to UP
Jun 23 00:11:53 	kernel 		coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
Jun 23 00:11:53 	kernel 		coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
Jun 23 00:11:53 	kernel 		coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
Jun 23 00:11:53 	kernel 		coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
Jun 23 00:11:53 	kernel 		aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
Jun 23 00:11:53 	kernel 		padlock0: No ACE support.
Jun 23 00:11:53 	kernel 		da0: quirks=0x2 <no_6_byte>Jun 23 00:11:53 	kernel 		da0: 15237MB (31205621 512 byte sectors)
Jun 23 00:11:53 	kernel 		da0: 400.000MB/s transfers
Jun 23 00:11:53 	kernel 		da0: Serial Number 20170606
Jun 23 00:11:53 	kernel 		da0: <ssd is917="" 1.00="">Removable Direct Access SPC-4 SCSI device
Jun 23 00:11:53 	kernel 		da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
Jun 23 00:11:53 	kernel 		mountroot: waiting for device /dev/gpt/pfSense...
Jun 23 00:11:53 	kernel 		umass0: <is917 0="" 3="" ssd-16gb,="" class="" 0,="" rev="" 3.00="" 0.01,="" addr="">on usbus0
Jun 23 00:11:53 	kernel 		ugen0.4: <is917>at usbus0
Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
Jun 23 00:11:53 	kernel 		ue0: Ethernet address: 10:13:50:a4:51:70
Jun 23 00:11:53 	kernel 		ue0: <usb ethernet="">on udav0
Jun 23 00:11:53 	kernel 		ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
Jun 23 00:11:53 	kernel 		ukphy0: <generic ieee="" 802.3u="" media="" interface="">PHY 0 on miibus1
Jun 23 00:11:53 	kernel 		miibus1: <mii bus="">on udav0
Jun 23 00:11:53 	kernel 		udav0: <vendor 0="" 2="" 0x0fe6="" usb="" network="" controller,="" class="" 0,="" rev="" 1.10="" 1.01,="" addr="">on usbus0
Jun 23 00:11:53 	kernel 		ugen0.3: <vendor 0x0fe6="">at usbus0
Jun 23 00:11:53 	kernel 		uhub1: 4 ports with 4 removable, self powered
Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
Jun 23 00:11:53 	kernel 		uhub1: <vendor 1="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 88.31,="" addr="">on usbus0
Jun 23 00:11:53 	kernel 		ugen0.2: <vendor 0x05e3="">at usbus0
Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
Jun 23 00:11:53 	kernel 		uhub0: 13 ports with 13 removable, self powered
Jun 23 00:11:53 	kernel 		Root mount waiting for: usbus0
Jun 23 00:11:53 	kernel 		Trying to mount root from ufs:/dev/gpt/pfSense [rw]...
Jun 23 00:11:53 	kernel 		Timecounter "TSC" frequency 1600053120 Hz quality 1000
Jun 23 00:11:53 	kernel 		SMP: AP CPU #3 Launched!
Jun 23 00:11:53 	kernel 		SMP: AP CPU #1 Launched!
Jun 23 00:11:53 	kernel 		SMP: AP CPU #2 Launched!
Jun 23 00:11:53 	kernel 		uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Jun 23 00:11:53 	kernel 		ugen0.1: <0x8086> at usbus0
Jun 23 00:11:53 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
Jun 23 00:11:53 	kernel 		nvme cam probe device init
Jun 23 00:11:53 	kernel 		Timecounters tick every 1.000 msec
Jun 23 00:11:53 	kernel 		est3: <enhanced speedstep="" frequency="" control="">on cpu3
Jun 23 00:11:53 	kernel 		est2: <enhanced speedstep="" frequency="" control="">on cpu2
Jun 23 00:11:53 	kernel 		est1: <enhanced speedstep="" frequency="" control="">on cpu1
Jun 23 00:11:53 	kernel 		est0: <enhanced speedstep="" frequency="" control="">on cpu0
Jun 23 00:11:53 	kernel 		ppc0: cannot reserve I/O port range
Jun 23 00:11:53 	kernel 		acpi_tz0: <thermal zone="">on acpi0
Jun 23 00:11:53 	kernel 		acpi_button0: <sleep button="">on acpi0
Jun 23 00:11:53 	kernel 		isa0: <isa bus="">on isab0
Jun 23 00:11:53 	kernel 		isab0: <pci-isa bridge="">at device 31.0 on pci0
Jun 23 00:11:53 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
Jun 23 00:11:53 	kernel 		em0: Ethernet address: 68:05:ca:1a:e5:0a
Jun 23 00:11:53 	kernel 		em0: Using MSIX interrupts with 3 vectors
Jun 23 00:11:53 	kernel 		em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xd000-0xd01f mem 0x812c0000-0x812dffff,0x81200000-0x8127ffff,0x812e0000-0x812e3fff at device 0.0 on pci2
Jun 23 00:11:53 	kernel 		pci2: <acpi pci="" bus="">on pcib2
Jun 23 00:11:53 	kernel 		pcib2: [GIANT-LOCKED]
Jun 23 00:11:53 	kernel 		pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0
Jun 23 00:11:53 	kernel 		re0: netmap queues/slots: TX 1/256, RX 1/256
Jun 23 00:11:53 	kernel 		re0: Ethernet address: d8:cb:8a:98:90:0a
Jun 23 00:11:53 	kernel 		re0: Using defaults for TSO: 65518/35/2048
Jun 23 00:11:53 	kernel 		rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
Jun 23 00:11:53 	kernel 		rgephy0: <rtl8251 1000base-t="" media="" interface="">PHY 1 on miibus0
Jun 23 00:11:53 	kernel 		miibus0: <mii bus="">on re0
Jun 23 00:11:53 	kernel 		re0: MAC rev. 0x00000000
Jun 23 00:11:53 	kernel 		re0: Chip rev. 0x4c000000
Jun 23 00:11:53 	kernel 		re0: Using 1 MSI-X message
Jun 23 00:11:53 	kernel 		re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0x81300000-0x81300fff,0xa0000000-0xa0003fff at device 0.0 on pci1
Jun 23 00:11:53 	kernel 		pci1: <acpi pci="" bus="">on pcib1
Jun 23 00:11:53 	kernel 		pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0
Jun 23 00:11:53 	kernel 		pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
Jun 23 00:11:53 	kernel 		usbus0 on xhci0
Jun 23 00:11:53 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
Jun 23 00:11:53 	kernel 		xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81400000-0x8140ffff at device 20.0 on pci0
Jun 23 00:11:53 	kernel 		ahcich1: <ahci channel="">at channel 1 on ahci0
Jun 23 00:11:53 	kernel 		ahcich0: <ahci channel="">at channel 0 on ahci0
Jun 23 00:11:53 	kernel 		ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
Jun 23 00:11:53 	kernel 		ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81411000-0x814117ff at device 19.0 on pci0
Jun 23 00:11:53 	kernel 		vgapci0: Boot video device
Jun 23 00:11:53 	kernel 		vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff at device 2.0 on pci0
Jun 23 00:11:53 	kernel 		pci0: <acpi pci="" bus="">on pcib0
Jun 23 00:11:53 	kernel 		pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
Jun 23 00:11:53 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
Jun 23 00:11:53 	kernel 		Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Jun 23 00:11:53 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
Jun 23 00:11:53 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
Jun 23 00:11:53 	kernel 		attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Jun 23 00:11:53 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
Jun 23 00:11:53 	kernel 		atrtc0: Warning: Couldn't map I/O.
Jun 23 00:11:53 	kernel 		atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
Jun 23 00:11:53 	kernel 		cpu3: <acpi cpu="">on acpi0
Jun 23 00:11:53 	kernel 		cpu2: <acpi cpu="">on acpi0
Jun 23 00:11:53 	kernel 		cpu1: <acpi cpu="">on acpi0
Jun 23 00:11:53 	kernel 		cpu0: <acpi cpu="">on acpi0
Jun 23 00:11:53 	kernel 		unknown: I/O range not supported
Jun 23 00:11:53 	kernel 		acpi0: Power Button (fixed)
Jun 23 00:11:53 	kernel 		acpi0: <alaska a="" m="" i="">on motherboard
Jun 23 00:11:53 	kernel 		padlock0: No ACE support.
Jun 23 00:11:53 	kernel 		cryptosoft0: <software crypto="">on motherboard
Jun 23 00:11:53 	kernel 		random: fast provider: "Intel Secure Key RNG"
Jun 23 00:11:53 	kernel 		random: registering fast source Intel Secure Key RNG
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (vesa, 0xffffffff8122cb10, 0) error 19
Jun 23 00:11:53 	kernel 		netmap: loaded module
Jun 23 00:11:53 	kernel 		kbd1 at kbdmux0
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069ebb0, 0) error 1
Jun 23 00:11:53 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Jun 23 00:11:53 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069eb00, 0) error 1
Jun 23 00:11:53 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Jun 23 00:11:53 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069ea50, 0) error 1
Jun 23 00:11:53 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Jun 23 00:11:53 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Jun 23 00:11:53 	kernel 		wlan: mac acl policy registered
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80675620, 0) error 1
Jun 23 00:11:53 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Jun 23 00:11:53 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80675570, 0) error 1
Jun 23 00:11:53 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Jun 23 00:11:53 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Jun 23 00:11:53 	kernel 		random: entropy device external interface
Jun 23 00:11:53 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806754c0, 0) error 1
Jun 23 00:11:53 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Jun 23 00:11:53 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Jun 23 00:11:53 	kernel 		ioapic0 <version 2.0="">irqs 0-114 on motherboard
Jun 23 00:11:53 	kernel 		ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
Jun 23 00:11:53 	kernel 		random: unblocking device.
Jun 23 00:11:53 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
Jun 23 00:11:53 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Jun 23 00:11:53 	kernel 		0 < 1
Jun 23 00:11:53 	kernel 		WARNING: L1 data cache covers less APIC IDs than a core
Jun 23 00:11:53 	kernel 		ACPI APIC Table: <alaska a="" m="" i="">Jun 23 00:11:53 	kernel 		Event timer "LAPIC" quality 600
Jun 23 00:11:53 	kernel 		avail memory = 1933852672 (1844 MB)
Jun 23 00:11:53 	kernel 		real memory = 2147483648 (2048 MB)
Jun 23 00:11:53 	kernel 		TSC: P-state invariant, performance statistics
Jun 23 00:11:53 	kernel 		VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Jun 23 00:11:53 	kernel 		Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>Jun 23 00:11:53 	kernel 		AMD Features2=0x101 <lahf,prefetch>Jun 23 00:11:53 	kernel 		AMD Features=0x28100800 <syscall,nx,rdtscp,lm>Jun 23 00:11:53 	kernel 		Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>Jun 23 00:11:53 	kernel 		Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jun 23 00:11:53 	kernel 		Origin="GenuineIntel" Id=0x406c3 Family=0x6 Model=0x4c Stepping=3
Jun 23 00:11:53 	kernel 		CPU: Intel(R) Celeron(R) CPU N3150 @ 1.60GHz (1600.05-MHz K8-class CPU)
Jun 23 00:11:53 	kernel 		VT(vga): resolution 640x480
Jun 23 00:11:53 	kernel 		FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
Jun 23 00:11:53 	kernel 		root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
Jun 23 00:11:53 	kernel 		FreeBSD 11.0-RELEASE-p10 #139 981956f70fd(RELENG_2_4): Thu Jun 22 03:49:45 CDT 2017
Jun 23 00:11:53 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Jun 23 00:11:53 	kernel 		The Regents of the University of California. All rights reserved.
Jun 23 00:11:53 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jun 23 00:11:53 	kernel 		Copyright (c) 1992-2016 The FreeBSD Project.
Jun 23 00:11:53 	syslogd 		kernel boot file is /boot/kernel/kernel 
Jun 23 00:02:01 	syslogd 		exiting on signal 15
Jun 23 00:02:01 	reboot 		rebooted by root
Jun 23 00:01:51 	php-fpm 	88532 	/rc.start_packages: Restarting/Starting all packages.
Jun 23 00:01:50 	check_reload_status 		Starting packages
Jun 23 00:01:50 	check_reload_status 		Reloading filter
Jun 23 00:01:39 	pkg 		pfSense-kernel-pfSense upgraded: 2.4.0.b.20170621.0152 -> 2.4.0.b.20170622.0342
Jun 23 00:00:46 	pkg 		pfSense-rc upgraded: 2.4.0.b.20170621.0152 -> 2.4.0.b.20170622.0342</fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></syscall,nx,rdtscp,lm></lahf,prefetch></tscadj,smep,erms,nfpusg></alaska></version></software></alaska></acpi></acpi></acpi></acpi></at></at></acpi></acpi></vga-compatible></ahci></ahci></ahci></xhci></encrypt></acpi></acpi></realtek></mii></rtl8251></acpi></acpi></intel(r)></pci-isa></isa></sleep></thermal></enhanced></enhanced></enhanced></enhanced></vendor></vendor></vendor></vendor></mii></generic></usb></is917></is917></ssd></no_6_byte></aes-cbc,aes-xts,aes-gcm,aes-icm></cpu></cpu></cpu></cpu></sendpacket></sendpacket></sendpacket>

And once again, even with RAM Disk disabled right after ugrade I've seen 20% of RAM usage, while now I've just rebooted pfSense and it only consumes 10% of the RAM, so this aspect persists.

nazar-pc · Jul 1, 2017, 12:05 AM

It happened for a third time, definitetly not a coincidense.
This time it freezed while I was at my working machine. Didn't have monitor attached to the router, so I've hard rebooted it (maybe it would reboot itself after some time, I don't know).

This time it survived a little bit over 3 days, all the logs from the moment upgrade started and till freeze are lost again:

Jul 1 02:58:15 	kernel 		Copyright (c) 1992-2016 The FreeBSD Project.
Jul 1 02:58:15 	syslogd 		kernel boot file is /boot/kernel/kernel
Jun 28 22:31:13 	php-fpm 	68463 	/rc.start_packages: Restarting/Starting all packages.
Jun 28 22:31:12 	check_reload_status 		Starting packages
Jun 28 22:31:12 	check_reload_status 		Reloading filter
Jun 28 22:31:08 	pkg 		pfSense-kernel-pfSense upgraded: 2.4.0.b.20170622.0342 -> 2.4.0.b.20170627.1443

I don't know what happens, but there is an issue here for sure:

memory consumption right after upgrade is higher than it is otherwise
logs are not persisted to disk each hour as they should
system freezes/reboots after some time

PiBa · Jul 1, 2017, 2:31 PM

Memory usage going to 20% instead of 10% by itself shouldn't cause trouble.. It might be interesting to see what processes are using that memory.. Though actually i wouldn't worry about that itself to much.. During bootup after a upgrade some extra steps are made to check/upgrade packages though once that finishes probably should expect memory usage to drop a little..

Perhaps try a MemTest image and check if any of the memory sticks is faulty.? Or maybe remove some of its memory dimms as it has plenty of memory it seems.?.

Backup job
When the upgrade is done it could be nice to check if 'ps -A | grep cron' shows that thats running.. As that should call the log backup scripts.. Check under 'cat /etc/crontab | grep rc.backup' if the 3 rc.backup jobs are scheduled.
And check after a hour if the '/cf/conf/RAM_Disk_Store/' is getting filled with new tgz files..
If not try running them manually from console? To see if that succeeds..

Freeze/reboot
Might be caused by faulty memory, though could be lots of other things..
Really difficult to tell something usefull here without some log/error to indicate what is actually going wrong..

RAM Disk
Please do try to attach a monitor when it happens or leave the RAM disk disabled it shouldnt affect performance much as pretty much everything a stock pfSense install does is move packets around, and that shouldn't need any disk access.. Anything specific there? Big captive portal with usage voucher counting or something?

Overall you could re-install pfSense from scratch.? And restore the config file.. That way at least we know for sure that all binaries/filestructures arent a problem..

PiBa · Jul 1, 2017, 8:13 PM

For the cron backup job it seems like the crontab itself is actually not properly written..

PiBa · Jul 1, 2017, 9:05 PM

Ok so the backup job itself should get fixed once the pull-request is approved/pulled: https://github.com/pfsense/pfsense/pull/3771 .

For the crashes/hang issue.. I still don't have a clue..

nazar-pc · Jul 2, 2017, 2:55 AM

I've tested memory with Windows memory testing available in boot menu (just because it works with UEFI and memtest86 does not), after 2 loops it didn't find any issues, so memory should be fine.
And it doesn't have plenty of memory (well, comparing to regular routers it does), just one 2G SO-DIMM stick of DDR3L.

Also I've checked memory stats before and after upgrade, it seems that Memory usage shows Total memory - Free, while I was expecting just Active. So it is confusing (should be changed?) but generally memory consumption is fine after upgrade.

It seems you've found the issue with backup job, so it should also be fixed at some point.

I've initially enabled RAM Disk because of extremely slow USB 2.0 flash drive where pfSense lived, it was painfully slow and affected router performance a lot on any disk activity. Now I've upgraded to much faster USB 3.0 flash drive, but it is still not a proper SSD, so I'd like to avoid unnecessary writes there.

I'll try to keep display connected and get something more about those freezes/reboots when it happens next time.

Thank you for your help so far!

nazar-pc · Jul 10, 2017, 2:36 PM

Ok, I've being tweaking UI with some Wi-Fi settings and then let it alone for few minutes and discovered that it feezed again and then rebooted itself (after a bit over 5 days uptime).
RAM Disk was disabled, nothing in logs that would indicate why the system is rebooting:


Jul 10 17:26:23 	syslogd 		kernel boot file is /boot/kernel/kernel
Jul 10 17:17:52 	check_reload_status 		Syncing firewall
Jul 10 17:17:43 	check_reload_status 		Syncing firewall
Jul 10 17:17:36 	check_reload_status 		Reloading filter
Jul 10 17:17:36 	php-fpm 	89258 	/rc.linkup: Ignoring link event for bridge member without IP config
Jul 10 17:17:35 	kernel 		em0: link state changed to UP
Jul 10 17:17:35 	check_reload_status 		Linkup starting em0
Jul 10 17:17:32 	check_reload_status 		Reloading filter
Jul 10 17:17:32 	php-fpm 	89258 	/rc.linkup: Ignoring link event for bridge member without IP config
Jul 10 17:17:32 	dhcpleases 		/etc/hosts changed size from original!
Jul 10 17:17:32 	check_reload_status 		Syncing firewall
Jul 10 17:17:31 	kernel 		run0_wlan0: promiscuous mode enabled
Jul 10 17:17:31 	kernel 		ue0: promiscuous mode enabled
Jul 10 17:17:31 	kernel 		bridge0: link state changed to UP
Jul 10 17:17:31 	kernel 		em0: promiscuous mode enabled
Jul 10 17:17:31 	check_reload_status 		Linkup starting em0
Jul 10 17:17:31 	kernel 		bridge0: Ethernet address: 02:0d:b2:50:df:00
Jul 10 17:17:31 	kernel 		em0: promiscuous mode disabled
Jul 10 17:17:31 	kernel 		bridge0: link state changed to DOWN
Jul 10 17:17:31 	kernel 		ue0: promiscuous mode disabled
Jul 10 17:17:31 	kernel 		em0: link state changed to DOWN
Jul 10 17:17:31 	kernel 		run0_wlan0: promiscuous mode disabled
Jul 10 17:15:19 	php-fpm 	42940 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
Jul 10 17:15:19 	php-fpm 	42940 	/interfaces.php: Cloning new wireless interface run0_wlan1
Jul 10 17:14:32 	php-fpm 	42940 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
Jul 10 17:14:32 	php-fpm 	42940 	/interfaces.php: Cloning new wireless interface run0_wlan1
Jul 10 17:12:44 	php-fpm 	51717 	/interfaces.php: Creating rrd update script
Jul 10 17:12:44 	check_reload_status 		Reloading filter
Jul 10 17:12:42 	check_reload_status 		updating dyndns opt3
Jul 10 17:12:41 	dhcpleases 		kqueue error: unkown
Jul 10 17:12:40 	dhcpleases 		Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
Jul 10 17:12:40 	dhcpleases 		/etc/hosts changed size from original!
Jul 10 17:12:38 	kernel 		run0_wlan0: promiscuous mode enabled
Jul 10 17:12:38 	check_reload_status 		Restarting ipsec tunnels
Jul 10 17:12:38 	kernel 		run0: firmware RT3071 ver. 0.33 loaded
Jul 10 17:12:36 	kernel 		run0: firmware RT3071 ver. 0.33 loaded
Jul 10 17:12:33 	check_reload_status 		Syncing firewall
Jul 10 17:12:02 	php-fpm 	26730 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
Jul 10 17:12:02 	php-fpm 	26730 	/interfaces.php: Cloning new wireless interface run0_wlan1
Jul 10 17:11:57 	kernel 		wlan0: changing name to 'run0_wlan0'
Jul 10 17:11:57 	kernel 		wlan0: Ethernet address: c8:3a:35:ca:f8:90
Jul 10 17:11:57 	php-fpm 	26730 	/interfaces.php: Cloning new wireless interface run0_wlan0
Jul 10 17:11:43 	kernel 		run0: MAC/BBP RT3572 (rev 0x0223), RF RT3052 (MIMO 2T2R), address c8:3a:35:ca:f8:90
Jul 10 17:11:43 	kernel 		run0: <1.0> on usbus0
Jul 10 17:11:43 	kernel 		ugen0.6: <ralink>at usbus0
Jul 10 17:11:40 	kernel 		run0: at uhub1, port 3, addr 8 (disconnected)
Jul 10 17:11:40 	kernel 		ugen0.6: <ralink>at usbus0 (disconnected)
Jul 10 17:11:29 	php-fpm 	42425 	/interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error
Jul 10 17:11:29 	php-fpm 	42425 	/interfaces.php: Cloning new wireless interface run0_wlan0</ralink></ralink>

PiBa · Jul 10, 2017, 5:50 PM

Without RAM disk, if the kernel was crashing it should write a dump file to /var/crash/ .. But then dashboard should show a big banner that tells it found that dump file.

Unless of-course there is a problem with the storage / usb connections / usb controllers that causes the crash, then it cant write anything.. I'm just wildly guessing here though.. Sorry..

nazar-pc · Jul 28, 2017, 8:24 PM

I thinks I've found the reason. It was faulty flash drive, it was returning read/write operations.