Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense 2.7.2 in Hyper-V freezing with no crash report after reboot

    Scheduled Pinned Locked Moved
    Virtualization
    2
    16
    362
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      Techniker_ctr
      last edited by stephenw10

      Greetings,

      We are currently observing freezing of pfSense 2.7.2 systems, which are only accessible again after a reboot via Hyper-v.

      During freezing, the system is neither accessible via GUI nor via SSH. The console is also no longer accessible in some cases.

      What makes error analysis particularly difficult is that no dumps or error messages are generated. The failures can only be traced via gaps in the system logs.

      We also cannot see any spikes in resource usage during the outages. The affected systems are partly updated but there are also systems freshly installed on 2.7.2.
      Another commonality is that so far all affected systems have HA proxy installed.

      Do you have any idea how to recreate or better analyze the freezes?

      Here is an example of a log: (Gap between Nov 13 10:23:59 - Nov 13 10:18:57)

      Nov 13 10:23:59	kernel		Launching APs: 1
Nov 13 10:23:59	kernel		ioapic0 <Version 1.1> irqs 0-23
Nov 13 10:23:59	kernel		ioapic0: MADT APIC ID 0 != hw id 2
Nov 13 10:23:59	kernel		arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
Nov 13 10:23:59	kernel		random: fast provider: "Intel Secure Key RNG"
Nov 13 10:23:59	kernel		random: registering fast source Intel Secure Key RNG
Nov 13 10:23:59	kernel		FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 hardware threads
Nov 13 10:23:59	kernel		FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Nov 13 10:23:59	kernel		ACPI APIC Table: <VRTUAL MICROSFT>
Nov 13 10:23:59	kernel		Event timer "LAPIC" quality 100
Nov 13 10:23:59	kernel		avail memory = 979812352 (934 MB)
Nov 13 10:23:59	kernel		real memory = 1073741824 (1024 MB)
Nov 13 10:23:59	kernel		Hypervisor: Origin = "Microsoft Hv"
Nov 13 10:23:59	kernel		AMD Extended Feature Extensions ID EBX=0x6000d004<XSaveErPtr,IBPB,IBRS,STIBP>
Nov 13 10:23:59	kernel		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Nov 13 10:23:59	kernel		Structured Extended Features3=0x10<FSRM>
Nov 13 10:23:59	kernel		Structured Extended Features2=0x405f42<AVX512VBMI,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,RDPID>
Nov 13 10:23:59	kernel		Structured Extended Features=0xf1bf03a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,AVX512CD,SHA,AVX512BW,AVX512VL>
Nov 13 10:23:59	kernel		AMD Features2=0x4003f3<LAHF,CMP,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,Topology>
Nov 13 10:23:59	kernel		AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
Nov 13 10:23:59	kernel		Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
Nov 13 10:23:59	kernel		Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Nov 13 10:23:59	kernel		Origin="AuthenticAMD" Id=0xa10f11 Family=0x19 Model=0x11 Stepping=1
Nov 13 10:23:59	kernel		CPU: AMD EPYC 9354 32-Core Processor (3245.57-MHz K8-class CPU)
Nov 13 10:23:59	kernel		Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
Nov 13 10:23:59	kernel		Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Nov 13 10:23:59	kernel		PM Features=0x0 [C2]
Nov 13 10:23:59	kernel		Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
Nov 13 10:23:59	kernel		Hyper-V Version: 10.0.17763 [SP5]
Nov 13 10:23:59	kernel		VT(vga): text 80x25
Nov 13 10:23:59	kernel		SRAT: Ignoring memory at addr 0x1000000000
Nov 13 10:23:59	kernel		SRAT: Ignoring memory at addr 0x100000000
Nov 13 10:23:59	kernel		SRAT: Ignoring memory at addr 0x40000000
Nov 13 10:23:59	kernel		FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Nov 13 10:23:59	kernel		root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64
Nov 13 10:23:59	kernel		FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023
Nov 13 10:23:59	kernel		FreeBSD is a registered trademark of The FreeBSD Foundation.
Nov 13 10:23:59	kernel		The Regents of the University of California. All rights reserved.
Nov 13 10:23:59	kernel		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Nov 13 10:23:59	kernel		Copyright (c) 1992-2023 The FreeBSD Project.
Nov 13 10:23:59	kernel		---<<BOOT>>---
Nov 13 10:23:59	syslogd		kernel boot file is /boot/kernel/kernel
Nov 13 10:18:57	sshd	75283	Disconnected from user root xxxx:xxxx:x:x:xxx::x port 33640
Nov 13 10:18:57	sshd	75283	Received disconnect from xxxx:xxxx:x:x:xxx::x port 33640:11: disconnected by user
Nov 13 10:18:57	sshd	75283	Accepted publickey for root from xxxx:xxxx:x:x:xxx::x port 33640 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxx
Nov 13 10:18:52	sshd	72955	Disconnected from user root xxxx:xxxx:x:x:xxx::x port 40136
Nov 13 10:18:52	sshd	72955	Received disconnect from xxxx:xxxx:x:x:xxx::x port 40136:11: disconnected by user
Nov 13 10:18:50	sshd	72955	Accepted publickey for root from xxxx:xxxx:x:x:xxx::x port 40136 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxx
Nov 13 10:18:46	sshd	70399	Disconnected from user root xxxx:xxxx:x:x:xxx::x port 40132
Nov 13 10:18:46	sshd	70399	Received disconnect from xxxx:xxxx:x:x:xxx::x port 40132:11: disconnected by user
Nov 13 10:18:46	sshd	70399	Accepted publickey for root from xxxx:xxxx:x:x:xxx::x port 40132 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxx
Nov 13 10:18:39	sshd	68841	Disconnected from user root xxxx:xxxx:x:x:xxx::x port 60236
Nov 13 10:18:39	sshd	68841	Received disconnect from xxxx:xxxx:x:x:xxx::x port 60236:11: disconnected by user
Nov 13 10:18:39	sshd	68841	Accepted publickey for root from xxxx:xxxx:x:x:xxx::x port 60236 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxx
Nov 13 10:18:13	sshd	65925	Disconnected from user root xxxx:xxxx:x:x:xxx::x port 46936
Nov 13 10:18:13	sshd	65925	Received disconnect from xxxx:xxxx:x:x:xxx::x port 46936:11: disconnected by user
Nov 13 10:18:13	sshd	65925	Accepted publickey for root from xxxx:xxxx:x:x:xxx::x port 46936 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxx
Nov 13 10:17:44	sshd	31345	Connection closed by xxxx:xxxx:x:x:xxx::x port 58628 [preauth]
Nov 13 10:15:59	sshd	76242	Disconnected from user root xxxx:xxxx:x:x:xxx::x port 59520
Nov 13 10:15:59	sshd	76242	Received disconnect from xxxx:xxxx:x:x:xxx::x port 59520:11: disconnected by user

      Ressourcen:
      Memory.PNG proccesses.PNG

      stephenw10S 1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Hmm, what are those ssh connections from? Not that it should be a problem.

        T 1 Reply Last reply Reply Quote 0
        • T
          Techniker_ctr @stephenw10
          last edited by

          @stephenw10

          Hello,

          thanks for your reply.

          Those are connections from our monitoring server. No other users conncted to the system before the freeze.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Anything shown at the console? Can you setup a serial console and log that output?

            T 1 Reply Last reply Reply Quote 0
            • T
              Techniker_ctr @stephenw10
              last edited by

              @stephenw10

              Yes I checked console connection, but all you see are syslogd messages of user logins sometimes multiple days old.

              The affected systems are unfortunately all live systems, which have to be made functional again as soon as possible, so I don't have much time for longer tests.

              I have not yet been able to cause a freeze on my test systems.

              Normally a freeze occurs every 2 - 4 weeks. However, there is no guarantee of this.

              T 1 Reply Last reply Reply Quote 0
              • T
                Techniker_ctr @Techniker_ctr
                last edited by

                @Techniker_ctr

                We have currently moved the HA proxy from one of the affected systems to another system to see if the freezes are still occurring

                Perhaps this will help to narrow down the potential sources of error

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator @Techniker_ctr
                  last edited by

                  @Techniker_ctr said in pfSense 2.7.2 freezing with no crash report after reboot:

                  The console is also no longer accessible in some cases.

                  Hmm, so sometimes the console was still responsive? I assume you were not able to connect out at that point?

                  T 1 Reply Last reply Reply Quote 0
                  • T
                    Techniker_ctr @stephenw10
                    last edited by

                    @stephenw10

                    It seems so. In all the freezes I've seen, the console was no longer functional, but my colleague said he could access a device via console.

                    At the next freeze I will actively check if more info is available via console

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Make sure you try using ctl+t at the console. That can often show a stuck process when nothing else is responding.

                      T 1 Reply Last reply Reply Quote 1
                      • T
                        Techniker_ctr @stephenw10
                        last edited by

                        @stephenw10

                        Hey,

                        We had another outage today, unfortunately no further information could be retrieved via the console.

                        The system that failed is an updated system that runs on ufs and does not use a ha proxy.

                        Failure pattern is the same again. Normal load and suddenly no response from the system and total traffic failure. As alsways no crash report.

                        The following are the system loads
                        RAM
                        Memory_2.PNG

                        Proccesses:
                        proccesses_2.PNG

                        Any other Idea how we can track down the cause?

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Do you see anything in Hyper-V? Usage spikes etc?

                          T 1 Reply Last reply Reply Quote 0
                          • T
                            Techniker_ctr @stephenw10
                            last edited by Techniker_ctr

                            @stephenw10

                            I appreciate your help

                            I have checked our Grafana for the HyperV again and can indeed detect a CPU spike on the system, just before the freeze.
                            VCPU_Usage.PNG

                            And a complete loss of traffic
                            Network_per_secound(MB).PNG

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Hmm, so like some process uses all the CPU cycles until it's rebooted. Yet nothing is logged.... 🤔

                              T 2 Replies Last reply Reply Quote 0
                              • T
                                Techniker_ctr @stephenw10
                                last edited by

                                @stephenw10

                                Today we had another freeze. Same symptomps, no logs, no console. All we see is a CPU Spike right before the freeze.

                                Any other Ideas how to continue? Are we the only ones with this issue?

                                We are currently running 71 pfSense 2.7.2 and around 10 of them are showing the issue. We tried reinstalls on fresh 2.7.2 images to make sure there is no issue with the updating from 2.6.

                                All our 2.6 and 2.5.2 System run without these issues.

                                1 Reply Last reply Reply Quote 0
                                • T
                                  Techniker_ctr @stephenw10
                                  last edited by

                                  @stephenw10

                                  we just had another freeze. This time i have a couple of new errors in the logs before the logs completely stop.

                                  Maybe this helps narrowing it down.

                                  Nov 22 16:55:35	kernel		Copyright (c) 1992-2023 The FreeBSD Project.
Nov 22 16:55:35	kernel		---<<BOOT>>---
Nov 22 16:55:35	syslogd		kernel boot file is /boot/kernel/kernel
Nov 22 16:52:00	sshd	11196	Accepted publickey for root from xxx.xxx.xxx.xxx port 6014 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:51:56	sshd	8691	Accepted publickey for root from xxx.xxx.xxx.xxx port 13587 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:51:19	pfSctl	2755	could not finish read in a reasonable time. Action of event might not be completed.
Nov 22 16:51:16	sshd	53470	Disconnected from user root xxx.xxx.xxx.xxx port 52790
Nov 22 16:51:16	sshd	53470	Received disconnect from xxx.xxx.xxx.xxx port 52790:11: disconnected by user
Nov 22 16:51:16	sshd	26642	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 27224
Nov 22 16:51:16	rc.gateway_alarm	22904	>>> Gateway alarm: WANGWv6 (Addr:xxx:xxx:xxx::xxx Alarm:0 RTT:50.416ms RTTsd:167.822ms Loss:0%)
Nov 22 16:50:46	php-fpm	60747	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use GW_WAN.
Nov 22 16:50:46	check_reload_status	416	Reloading filter
Nov 22 16:50:46	check_reload_status	416	Restarting OpenVPN tunnels/interfaces
Nov 22 16:50:46	check_reload_status	416	Restarting IPsec tunnels
Nov 22 16:50:46	check_reload_status	416	updating dyndns GW_WAN
Nov 22 16:50:46	check_reload_status	416	updating dyndns WANGWv6
Nov 22 16:50:46	sshd	50953	Disconnected from user root xxx.xxx.xxx.xxx port 60047
Nov 22 16:50:46	sshd	50953	Received disconnect from xxx.xxx.xxx.xxx port 60047:11: disconnected by user
Nov 22 16:50:46	rc.gateway_alarm	10435	>>> Gateway alarm: GW_WAN (Addr:xxx.xxx.xxx.xxx Alarm:0 RTT:52.696ms RTTsd:60.464ms Loss:0%)
Nov 22 16:50:38	sshd	17812	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 33797
Nov 22 16:50:33	sshd	18507	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 39653
Nov 22 16:50:28	sshd	18455	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 21224
Nov 22 16:50:28	sshd	17705	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 1337
Nov 22 16:50:23	sshd	25669	Accepted publickey for root from xxx.xxx.xxx.xxx port 11618 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:50:01	pfSctl	17188	could not finish read in a reasonable time. Action of event might not be completed.
Nov 22 16:49:58	sshd	99832	Disconnected from user root xxx.xxx.xxx.xxx port 60268
Nov 22 16:49:58	sshd	99832	Received disconnect from xxx.xxx.xxx.xxx port 60268:11: disconnected by user
Nov 22 16:49:58	sshd	18141	Accepted publickey for root from xxx.xxx.xxx.xxx port 15849 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:49:58	sshd	1365	Disconnected from user root xxx.xxx.xxx.xxx port 53098
Nov 22 16:49:58	sshd	1365	Received disconnect from xxx.xxx.xxx.xxx port 53098:11: disconnected by user
Nov 22 16:49:58	sshd	53096	Disconnected from user root xxx.xxx.xxx.xxx port 65117
Nov 22 16:49:58	sshd	53096	Received disconnect from xxx.xxx.xxx.xxx port 65117:11: disconnected by user
Nov 22 16:49:58	sshd	97774	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 9367
Nov 22 16:49:58	rc.gateway_alarm	69423	>>> Gateway alarm: WANGWv6 (Addr:xxx:xxx:xxx::xxx Alarm:1 RTT:26.765ms RTTsd:25.516ms Loss:21%)
Nov 22 16:48:51	check_reload_status	416	Reloading filter
Nov 22 16:48:51	check_reload_status	416	Restarting OpenVPN tunnels/interfaces
Nov 22 16:48:51	check_reload_status	416	Restarting IPsec tunnels
Nov 22 16:48:51	check_reload_status	416	updating dyndns GW_WAN
Nov 22 16:48:51	sshd	50248	Disconnected from user root xxx.xxx.xxx.xxx port 25550
Nov 22 16:48:51	sshd	50248	Received disconnect from xxx.xxx.xxx.xxx port 25550:11: disconnected by user
Nov 22 16:48:51	sshd	51781	Disconnected from user root xxx.xxx.xxx.xxx port 45454
Nov 22 16:48:51	rc.gateway_alarm	59700	>>> Gateway alarm: GW_WAN (Addr:xxx.xxx.xxx.xxx Alarm:1 RTT:509.029ms RTTsd:1028.814ms Loss:0%)
Nov 22 16:48:51	sshd	51781	Received disconnect from xxx.xxx.xxx.xxx port 45454:11: disconnected by user
Nov 22 16:48:41	sshd	1365	Accepted publickey for root from xxx.xxx.xxx.xxx port 53098 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:48:35	sshd	99832	Accepted publickey for root from xxx.xxx.xxx.xxx port 60268 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:48:29	sshd	97374	Accepted publickey for root from xxx.xxx.xxx.xxx port 55569 ssh2: RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxx
Nov 22 16:48:22	sshd	53683	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 4106
Nov 22 16:48:22	sshd	52565	fatal: Timeout before authentication for xxx.xxx.xxx.xxx port 34558
Nov 22 16:47:53	sshd	78140	Disconnected from user root xxx.xxx.xxx.xxx port 49005
Nov 22 16:47:53	sshd	78140	Received disconnect from xxx.xxx.xxx.xxx port 49005:11: disconnected by user
                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Hmm, unfortunately nothing there really looks like an issue. Certainly not something that would cause it to stop responding entirely.

                                    We could try running dtrace against it to see what's using the cycles but I think we'd need some way to trigger it before it stopped responding.

                                    If you create a test instance in the same hypervisor does that also stop responding?

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S stephenw10 moved this topic from General pfSense Questions
                                    • First post
                                      Last post