[SOLVED] Router reboots in a day after upgrade to new snapshot
-
Could you disable the ram disks before the next update.? That should hopefully keep logs around to tell something more about the 'crash'.
It slows down the system significantly, but I'll try. I suspect, however, that it is actually caused by enabling RAM Disk (partially because of increased RAM usage right after upgrade).
-
One more think that indicates RAM Disk problem is that Periodic RAM Disk Data Backups is configured to 1 hour (all 3 options there) and the fact that no system logs and no RRD data were saved makes me wondering what happened to this feature and how to check if it was/is properly scheduled.
-
I've disabled RAM Disk, rebooted and upgraded the system.
It is running for 2 days and 17 hours now without any crashes and issues. Here are relevant system logs:Jun 23 00:12:17 php-fpm 35108 /rc.start_packages: Restarting/Starting all packages. Jun 23 00:12:17 syslogd kernel boot file is /boot/kernel/kernel Jun 23 00:12:17 syslogd exiting on signal 15 Jun 23 00:12:15 root /etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one Jun 23 00:12:14 php-cgi rc.bootup: Creating rrd update script Jun 23 00:12:09 check_reload_status Updating all dyndns Jun 23 00:12:06 rtsold 14452 <sendpacket>sendmsg on re0: Permission denied Jun 23 00:12:04 php-cgi rc.bootup: NTPD is starting up. Jun 23 00:12:02 rtsold 14452 <sendpacket>sendmsg on re0: Permission denied Jun 23 00:12:01 kernel em0: link state changed to UP Jun 23 00:12:01 check_reload_status Linkup starting em0 Jun 23 00:12:00 check_reload_status Linkup starting ue0 Jun 23 00:11:59 kernel bridge0: link state changed to UP Jun 23 00:11:59 kernel ue0: link state changed to UP Jun 23 00:11:58 rtsold 14452 <sendpacket>sendmsg on re0: Permission denied Jun 23 00:11:58 php-cgi rc.bootup: ROUTING: setting default route to 176.122.88.1 Jun 23 00:11:58 kernel pflog0: promiscuous mode enabled Jun 23 00:11:57 kernel bridge0: link state changed to DOWN Jun 23 00:11:57 kernel ue0: promiscuous mode enabled Jun 23 00:11:57 kernel bridge0: link state changed to UP Jun 23 00:11:57 kernel em0: promiscuous mode enabled Jun 23 00:11:57 php-cgi rc.bootup: Resyncing OpenVPN instances. Jun 23 00:11:57 kernel bridge0: Ethernet address: 02:83:1c:1d:71:00 Jun 23 00:11:57 kernel ue0: link state changed to DOWN Jun 23 00:11:57 check_reload_status Linkup starting ue0 Jun 23 00:11:57 kernel ue0: link state changed to UP Jun 23 00:11:57 check_reload_status Linkup starting ue0 Jun 23 00:11:57 check_reload_status Linkup starting em0 Jun 23 00:11:57 kernel em0: link state changed to DOWN Jun 23 00:11:57 kernel em0: link state changed to UP Jun 23 00:11:57 check_reload_status Linkup starting em0 Jun 23 00:11:56 php-fpm 348 /rc.newwanip: rc.newwanip: on (IP address: 176.122.88.128) (interface: WAN[wan]) (real interface: re0). Jun 23 00:11:56 php-fpm 348 /rc.newwanip: rc.newwanip: Info: starting on re0. Jun 23 00:11:55 php-cgi rc.bootup: Starting rtsold process Jun 23 00:11:55 php-cgi rc.bootup: Accept router advertisements on interface re0 Jun 23 00:11:55 check_reload_status rc.newwanip starting re0 Jun 23 00:11:55 check_reload_status Linkup starting re0 Jun 23 00:11:55 kernel re0: link state changed to UP Jun 23 00:11:53 check_reload_status Linkup starting re0 Jun 23 00:11:53 check_reload_status Linkup starting re0 Jun 23 00:11:53 kernel re0: link state changed to DOWN Jun 23 00:11:53 kernel re0: link state changed to UP Jun 23 00:11:53 kernel coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3 Jun 23 00:11:53 kernel coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2 Jun 23 00:11:53 kernel coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1 Jun 23 00:11:53 kernel coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0 Jun 23 00:11:53 kernel aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard Jun 23 00:11:53 kernel padlock0: No ACE support. Jun 23 00:11:53 kernel da0: quirks=0x2 <no_6_byte>Jun 23 00:11:53 kernel da0: 15237MB (31205621 512 byte sectors) Jun 23 00:11:53 kernel da0: 400.000MB/s transfers Jun 23 00:11:53 kernel da0: Serial Number 20170606 Jun 23 00:11:53 kernel da0: <ssd is917="" 1.00="">Removable Direct Access SPC-4 SCSI device Jun 23 00:11:53 kernel da0 at umass-sim0 bus 0 scbus2 target 0 lun 0 Jun 23 00:11:53 kernel mountroot: waiting for device /dev/gpt/pfSense... Jun 23 00:11:53 kernel umass0: <is917 0="" 3="" ssd-16gb,="" class="" 0,="" rev="" 3.00="" 0.01,="" addr="">on usbus0 Jun 23 00:11:53 kernel ugen0.4: <is917>at usbus0 Jun 23 00:11:53 kernel Root mount waiting for: usbus0 Jun 23 00:11:53 kernel Root mount waiting for: usbus0 Jun 23 00:11:53 kernel ue0: Ethernet address: 10:13:50:a4:51:70 Jun 23 00:11:53 kernel ue0: <usb ethernet="">on udav0 Jun 23 00:11:53 kernel ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto Jun 23 00:11:53 kernel ukphy0: <generic ieee="" 802.3u="" media="" interface="">PHY 0 on miibus1 Jun 23 00:11:53 kernel miibus1: <mii bus="">on udav0 Jun 23 00:11:53 kernel udav0: <vendor 0="" 2="" 0x0fe6="" usb="" network="" controller,="" class="" 0,="" rev="" 1.10="" 1.01,="" addr="">on usbus0 Jun 23 00:11:53 kernel ugen0.3: <vendor 0x0fe6="">at usbus0 Jun 23 00:11:53 kernel uhub1: 4 ports with 4 removable, self powered Jun 23 00:11:53 kernel Root mount waiting for: usbus0 Jun 23 00:11:53 kernel uhub1: <vendor 1="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 88.31,="" addr="">on usbus0 Jun 23 00:11:53 kernel ugen0.2: <vendor 0x05e3="">at usbus0 Jun 23 00:11:53 kernel Root mount waiting for: usbus0 Jun 23 00:11:53 kernel uhub0: 13 ports with 13 removable, self powered Jun 23 00:11:53 kernel Root mount waiting for: usbus0 Jun 23 00:11:53 kernel Trying to mount root from ufs:/dev/gpt/pfSense [rw]... Jun 23 00:11:53 kernel Timecounter "TSC" frequency 1600053120 Hz quality 1000 Jun 23 00:11:53 kernel SMP: AP CPU #3 Launched! Jun 23 00:11:53 kernel SMP: AP CPU #1 Launched! Jun 23 00:11:53 kernel SMP: AP CPU #2 Launched! Jun 23 00:11:53 kernel uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 Jun 23 00:11:53 kernel ugen0.1: <0x8086> at usbus0 Jun 23 00:11:53 kernel usbus0: 5.0Gbps Super Speed USB v3.0 Jun 23 00:11:53 kernel nvme cam probe device init Jun 23 00:11:53 kernel Timecounters tick every 1.000 msec Jun 23 00:11:53 kernel est3: <enhanced speedstep="" frequency="" control="">on cpu3 Jun 23 00:11:53 kernel est2: <enhanced speedstep="" frequency="" control="">on cpu2 Jun 23 00:11:53 kernel est1: <enhanced speedstep="" frequency="" control="">on cpu1 Jun 23 00:11:53 kernel est0: <enhanced speedstep="" frequency="" control="">on cpu0 Jun 23 00:11:53 kernel ppc0: cannot reserve I/O port range Jun 23 00:11:53 kernel acpi_tz0: <thermal zone="">on acpi0 Jun 23 00:11:53 kernel acpi_button0: <sleep button="">on acpi0 Jun 23 00:11:53 kernel isa0: <isa bus="">on isab0 Jun 23 00:11:53 kernel isab0: <pci-isa bridge="">at device 31.0 on pci0 Jun 23 00:11:53 kernel em0: netmap queues/slots: TX 1/1024, RX 1/1024 Jun 23 00:11:53 kernel em0: Ethernet address: 68:05:ca:1a:e5:0a Jun 23 00:11:53 kernel em0: Using MSIX interrupts with 3 vectors Jun 23 00:11:53 kernel em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xd000-0xd01f mem 0x812c0000-0x812dffff,0x81200000-0x8127ffff,0x812e0000-0x812e3fff at device 0.0 on pci2 Jun 23 00:11:53 kernel pci2: <acpi pci="" bus="">on pcib2 Jun 23 00:11:53 kernel pcib2: [GIANT-LOCKED] Jun 23 00:11:53 kernel pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0 Jun 23 00:11:53 kernel re0: netmap queues/slots: TX 1/256, RX 1/256 Jun 23 00:11:53 kernel re0: Ethernet address: d8:cb:8a:98:90:0a Jun 23 00:11:53 kernel re0: Using defaults for TSO: 65518/35/2048 Jun 23 00:11:53 kernel rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow Jun 23 00:11:53 kernel rgephy0: <rtl8251 1000base-t="" media="" interface="">PHY 1 on miibus0 Jun 23 00:11:53 kernel miibus0: <mii bus="">on re0 Jun 23 00:11:53 kernel re0: MAC rev. 0x00000000 Jun 23 00:11:53 kernel re0: Chip rev. 0x4c000000 Jun 23 00:11:53 kernel re0: Using 1 MSI-X message Jun 23 00:11:53 kernel re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0x81300000-0x81300fff,0xa0000000-0xa0003fff at device 0.0 on pci1 Jun 23 00:11:53 kernel pci1: <acpi pci="" bus="">on pcib1 Jun 23 00:11:53 kernel pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0 Jun 23 00:11:53 kernel pci0: <encrypt decrypt="">at device 26.0 (no driver attached) Jun 23 00:11:53 kernel usbus0 on xhci0 Jun 23 00:11:53 kernel xhci0: 32 bytes context size, 64-bit DMA Jun 23 00:11:53 kernel xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81400000-0x8140ffff at device 20.0 on pci0 Jun 23 00:11:53 kernel ahcich1: <ahci channel="">at channel 1 on ahci0 Jun 23 00:11:53 kernel ahcich0: <ahci channel="">at channel 0 on ahci0 Jun 23 00:11:53 kernel ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported Jun 23 00:11:53 kernel ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81411000-0x814117ff at device 19.0 on pci0 Jun 23 00:11:53 kernel vgapci0: Boot video device Jun 23 00:11:53 kernel vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff at device 2.0 on pci0 Jun 23 00:11:53 kernel pci0: <acpi pci="" bus="">on pcib0 Jun 23 00:11:53 kernel pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0 Jun 23 00:11:53 kernel acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 Jun 23 00:11:53 kernel Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 Jun 23 00:11:53 kernel Event timer "i8254" frequency 1193182 Hz quality 100 Jun 23 00:11:53 kernel Timecounter "i8254" frequency 1193182 Hz quality 0 Jun 23 00:11:53 kernel attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Jun 23 00:11:53 kernel Event timer "RTC" frequency 32768 Hz quality 0 Jun 23 00:11:53 kernel atrtc0: Warning: Couldn't map I/O. Jun 23 00:11:53 kernel atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0 Jun 23 00:11:53 kernel cpu3: <acpi cpu="">on acpi0 Jun 23 00:11:53 kernel cpu2: <acpi cpu="">on acpi0 Jun 23 00:11:53 kernel cpu1: <acpi cpu="">on acpi0 Jun 23 00:11:53 kernel cpu0: <acpi cpu="">on acpi0 Jun 23 00:11:53 kernel unknown: I/O range not supported Jun 23 00:11:53 kernel acpi0: Power Button (fixed) Jun 23 00:11:53 kernel acpi0: <alaska a="" m="" i="">on motherboard Jun 23 00:11:53 kernel padlock0: No ACE support. Jun 23 00:11:53 kernel cryptosoft0: <software crypto="">on motherboard Jun 23 00:11:53 kernel random: fast provider: "Intel Secure Key RNG" Jun 23 00:11:53 kernel random: registering fast source Intel Secure Key RNG Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (vesa, 0xffffffff8122cb10, 0) error 19 Jun 23 00:11:53 kernel netmap: loaded module Jun 23 00:11:53 kernel kbd1 at kbdmux0 Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069ebb0, 0) error 1 Jun 23 00:11:53 kernel iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Jun 23 00:11:53 kernel iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069eb00, 0) error 1 Jun 23 00:11:53 kernel iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Jun 23 00:11:53 kernel iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069ea50, 0) error 1 Jun 23 00:11:53 kernel iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Jun 23 00:11:53 kernel iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Jun 23 00:11:53 kernel wlan: mac acl policy registered Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80675620, 0) error 1 Jun 23 00:11:53 kernel ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Jun 23 00:11:53 kernel ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80675570, 0) error 1 Jun 23 00:11:53 kernel ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Jun 23 00:11:53 kernel ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Jun 23 00:11:53 kernel random: entropy device external interface Jun 23 00:11:53 kernel module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806754c0, 0) error 1 Jun 23 00:11:53 kernel ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Jun 23 00:11:53 kernel ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Jun 23 00:11:53 kernel ioapic0 <version 2.0="">irqs 0-114 on motherboard Jun 23 00:11:53 kernel ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650) Jun 23 00:11:53 kernel random: unblocking device. Jun 23 00:11:53 kernel FreeBSD/SMP: 1 package(s) x 4 core(s) Jun 23 00:11:53 kernel FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Jun 23 00:11:53 kernel 0 < 1 Jun 23 00:11:53 kernel WARNING: L1 data cache covers less APIC IDs than a core Jun 23 00:11:53 kernel ACPI APIC Table: <alaska a="" m="" i="">Jun 23 00:11:53 kernel Event timer "LAPIC" quality 600 Jun 23 00:11:53 kernel avail memory = 1933852672 (1844 MB) Jun 23 00:11:53 kernel real memory = 2147483648 (2048 MB) Jun 23 00:11:53 kernel TSC: P-state invariant, performance statistics Jun 23 00:11:53 kernel VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID Jun 23 00:11:53 kernel Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>Jun 23 00:11:53 kernel AMD Features2=0x101 <lahf,prefetch>Jun 23 00:11:53 kernel AMD Features=0x28100800 <syscall,nx,rdtscp,lm>Jun 23 00:11:53 kernel Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>Jun 23 00:11:53 kernel Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jun 23 00:11:53 kernel Origin="GenuineIntel" Id=0x406c3 Family=0x6 Model=0x4c Stepping=3 Jun 23 00:11:53 kernel CPU: Intel(R) Celeron(R) CPU N3150 @ 1.60GHz (1600.05-MHz K8-class CPU) Jun 23 00:11:53 kernel VT(vga): resolution 640x480 Jun 23 00:11:53 kernel FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) Jun 23 00:11:53 kernel root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64 Jun 23 00:11:53 kernel FreeBSD 11.0-RELEASE-p10 #139 981956f70fd(RELENG_2_4): Thu Jun 22 03:49:45 CDT 2017 Jun 23 00:11:53 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Jun 23 00:11:53 kernel The Regents of the University of California. All rights reserved. Jun 23 00:11:53 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jun 23 00:11:53 kernel Copyright (c) 1992-2016 The FreeBSD Project. Jun 23 00:11:53 syslogd kernel boot file is /boot/kernel/kernel Jun 23 00:02:01 syslogd exiting on signal 15 Jun 23 00:02:01 reboot rebooted by root Jun 23 00:01:51 php-fpm 88532 /rc.start_packages: Restarting/Starting all packages. Jun 23 00:01:50 check_reload_status Starting packages Jun 23 00:01:50 check_reload_status Reloading filter Jun 23 00:01:39 pkg pfSense-kernel-pfSense upgraded: 2.4.0.b.20170621.0152 -> 2.4.0.b.20170622.0342 Jun 23 00:00:46 pkg pfSense-rc upgraded: 2.4.0.b.20170621.0152 -> 2.4.0.b.20170622.0342</fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></syscall,nx,rdtscp,lm></lahf,prefetch></tscadj,smep,erms,nfpusg></alaska></version></software></alaska></acpi></acpi></acpi></acpi></at></at></acpi></acpi></vga-compatible></ahci></ahci></ahci></xhci></encrypt></acpi></acpi></realtek></mii></rtl8251></acpi></acpi></intel(r)></pci-isa></isa></sleep></thermal></enhanced></enhanced></enhanced></enhanced></vendor></vendor></vendor></vendor></mii></generic></usb></is917></is917></ssd></no_6_byte></aes-cbc,aes-xts,aes-gcm,aes-icm></cpu></cpu></cpu></cpu></sendpacket></sendpacket></sendpacket>And once again, even with RAM Disk disabled right after ugrade I've seen 20% of RAM usage, while now I've just rebooted pfSense and it only consumes 10% of the RAM, so this aspect persists.
-
It happened for a third time, definitetly not a coincidense.
This time it freezed while I was at my working machine. Didn't have monitor attached to the router, so I've hard rebooted it (maybe it would reboot itself after some time, I don't know).This time it survived a little bit over 3 days, all the logs from the moment upgrade started and till freeze are lost again:
Jul 1 02:58:15 kernel Copyright (c) 1992-2016 The FreeBSD Project. Jul 1 02:58:15 syslogd kernel boot file is /boot/kernel/kernel Jun 28 22:31:13 php-fpm 68463 /rc.start_packages: Restarting/Starting all packages. Jun 28 22:31:12 check_reload_status Starting packages Jun 28 22:31:12 check_reload_status Reloading filter Jun 28 22:31:08 pkg pfSense-kernel-pfSense upgraded: 2.4.0.b.20170622.0342 -> 2.4.0.b.20170627.1443I don't know what happens, but there is an issue here for sure:
- memory consumption right after upgrade is higher than it is otherwise
- logs are not persisted to disk each hour as they should
- system freezes/reboots after some time
-
Memory usage going to 20% instead of 10% by itself shouldn't cause trouble.. It might be interesting to see what processes are using that memory.. Though actually i wouldn't worry about that itself to much.. During bootup after a upgrade some extra steps are made to check/upgrade packages though once that finishes probably should expect memory usage to drop a little..
Perhaps try a MemTest image and check if any of the memory sticks is faulty.? Or maybe remove some of its memory dimms as it has plenty of memory it seems.?.
Backup job
When the upgrade is done it could be nice to check if 'ps -A | grep cron' shows that thats running.. As that should call the log backup scripts.. Check under 'cat /etc/crontab | grep rc.backup' if the 3 rc.backup jobs are scheduled.
And check after a hour if the '/cf/conf/RAM_Disk_Store/' is getting filled with new tgz files..
If not try running them manually from console? To see if that succeeds..Freeze/reboot
Might be caused by faulty memory, though could be lots of other things..
Really difficult to tell something usefull here without some log/error to indicate what is actually going wrong..RAM Disk
Please do try to attach a monitor when it happens or leave the RAM disk disabled it shouldnt affect performance much as pretty much everything a stock pfSense install does is move packets around, and that shouldn't need any disk access.. Anything specific there? Big captive portal with usage voucher counting or something?Overall you could re-install pfSense from scratch.? And restore the config file.. That way at least we know for sure that all binaries/filestructures arent a problem..
-
For the cron backup job it seems like the crontab itself is actually not properly written..
-
Ok so the backup job itself should get fixed once the pull-request is approved/pulled: https://github.com/pfsense/pfsense/pull/3771 .
For the crashes/hang issue.. I still don't have a clue..
-
I've tested memory with Windows memory testing available in boot menu (just because it works with UEFI and memtest86 does not), after 2 loops it didn't find any issues, so memory should be fine.
And it doesn't have plenty of memory (well, comparing to regular routers it does), just one 2G SO-DIMM stick of DDR3L.Also I've checked memory stats before and after upgrade, it seems that Memory usage shows Total memory - Free, while I was expecting just Active. So it is confusing (should be changed?) but generally memory consumption is fine after upgrade.
It seems you've found the issue with backup job, so it should also be fixed at some point.
I've initially enabled RAM Disk because of extremely slow USB 2.0 flash drive where pfSense lived, it was painfully slow and affected router performance a lot on any disk activity. Now I've upgraded to much faster USB 3.0 flash drive, but it is still not a proper SSD, so I'd like to avoid unnecessary writes there.
I'll try to keep display connected and get something more about those freezes/reboots when it happens next time.
Thank you for your help so far!
-
Ok, I've being tweaking UI with some Wi-Fi settings and then let it alone for few minutes and discovered that it feezed again and then rebooted itself (after a bit over 5 days uptime).
RAM Disk was disabled, nothing in logs that would indicate why the system is rebooting:Jul 10 17:26:23 syslogd kernel boot file is /boot/kernel/kernel Jul 10 17:17:52 check_reload_status Syncing firewall Jul 10 17:17:43 check_reload_status Syncing firewall Jul 10 17:17:36 check_reload_status Reloading filter Jul 10 17:17:36 php-fpm 89258 /rc.linkup: Ignoring link event for bridge member without IP config Jul 10 17:17:35 kernel em0: link state changed to UP Jul 10 17:17:35 check_reload_status Linkup starting em0 Jul 10 17:17:32 check_reload_status Reloading filter Jul 10 17:17:32 php-fpm 89258 /rc.linkup: Ignoring link event for bridge member without IP config Jul 10 17:17:32 dhcpleases /etc/hosts changed size from original! Jul 10 17:17:32 check_reload_status Syncing firewall Jul 10 17:17:31 kernel run0_wlan0: promiscuous mode enabled Jul 10 17:17:31 kernel ue0: promiscuous mode enabled Jul 10 17:17:31 kernel bridge0: link state changed to UP Jul 10 17:17:31 kernel em0: promiscuous mode enabled Jul 10 17:17:31 check_reload_status Linkup starting em0 Jul 10 17:17:31 kernel bridge0: Ethernet address: 02:0d:b2:50:df:00 Jul 10 17:17:31 kernel em0: promiscuous mode disabled Jul 10 17:17:31 kernel bridge0: link state changed to DOWN Jul 10 17:17:31 kernel ue0: promiscuous mode disabled Jul 10 17:17:31 kernel em0: link state changed to DOWN Jul 10 17:17:31 kernel run0_wlan0: promiscuous mode disabled Jul 10 17:15:19 php-fpm 42940 /interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error Jul 10 17:15:19 php-fpm 42940 /interfaces.php: Cloning new wireless interface run0_wlan1 Jul 10 17:14:32 php-fpm 42940 /interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error Jul 10 17:14:32 php-fpm 42940 /interfaces.php: Cloning new wireless interface run0_wlan1 Jul 10 17:12:44 php-fpm 51717 /interfaces.php: Creating rrd update script Jul 10 17:12:44 check_reload_status Reloading filter Jul 10 17:12:42 check_reload_status updating dyndns opt3 Jul 10 17:12:41 dhcpleases kqueue error: unkown Jul 10 17:12:40 dhcpleases Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process. Jul 10 17:12:40 dhcpleases /etc/hosts changed size from original! Jul 10 17:12:38 kernel run0_wlan0: promiscuous mode enabled Jul 10 17:12:38 check_reload_status Restarting ipsec tunnels Jul 10 17:12:38 kernel run0: firmware RT3071 ver. 0.33 loaded Jul 10 17:12:36 kernel run0: firmware RT3071 ver. 0.33 loaded Jul 10 17:12:33 check_reload_status Syncing firewall Jul 10 17:12:02 php-fpm 26730 /interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error Jul 10 17:12:02 php-fpm 26730 /interfaces.php: Cloning new wireless interface run0_wlan1 Jul 10 17:11:57 kernel wlan0: changing name to 'run0_wlan0' Jul 10 17:11:57 kernel wlan0: Ethernet address: c8:3a:35:ca:f8:90 Jul 10 17:11:57 php-fpm 26730 /interfaces.php: Cloning new wireless interface run0_wlan0 Jul 10 17:11:43 kernel run0: MAC/BBP RT3572 (rev 0x0223), RF RT3052 (MIMO 2T2R), address c8:3a:35:ca:f8:90 Jul 10 17:11:43 kernel run0: <1.0> on usbus0 Jul 10 17:11:43 kernel ugen0.6: <ralink>at usbus0 Jul 10 17:11:40 kernel run0: at uhub1, port 3, addr 8 (disconnected) Jul 10 17:11:40 kernel ugen0.6: <ralink>at usbus0 (disconnected) Jul 10 17:11:29 php-fpm 42425 /interfaces.php: Failed to clone interface run0 with error code 1, output ifconfig: SIOCIFCREATE2: Input/output error Jul 10 17:11:29 php-fpm 42425 /interfaces.php: Cloning new wireless interface run0_wlan0</ralink></ralink> -
Without RAM disk, if the kernel was crashing it should write a dump file to /var/crash/ .. But then dashboard should show a big banner that tells it found that dump file.
Unless of-course there is a problem with the storage / usb connections / usb controllers that causes the crash, then it cant write anything.. I'm just wildly guessing here though.. Sorry..
-
I thinks I've found the reason. It was faulty flash drive, it was returning read/write operations.
