Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CRASH Report: My Netgate 7100 crashes every ten to fifteen days.

    Scheduled Pinned Locked Moved
    Official Netgate® Hardware
    7100 crash
    4
    35
    2.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      smithjt @stephenw10
      last edited by

      @stephenw10

      Yes, the debug kernel was installed and I just experienced another crash.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        OK great, do you have a new crash report from that?

        S 1 Reply Last reply Reply Quote 0
        • S
          smithjt @stephenw10
          last edited by

          @stephenw10

          Here it is.

          ddb.txt
          msgbuf.txt

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            That isn't the debug kernel, it should show:

            ---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec  6 21:01:42 UTC 2023
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/sources/FreeBSD-src-plus-RELENG_23_09_1/amd64.amd64/sys/pfSense-DEBUG amd64

            Did you add the loader value to make it boot the debug kernel every time?

            S 1 Reply Last reply Reply Quote 0
            • S
              smithjt @stephenw10
              last edited by

              @stephenw10

              No, I didn't at first, but it is now and I haven't experienced a crash since I have been on the debug kernel. It's been running for five days so far without a crash.

              S 1 Reply Last reply Reply Quote 0
              • S
                smithjt @smithjt
                last edited by

                @smithjt

                ---<<BOOT>>---
                Copyright (c) 1992-2023 The FreeBSD Project.
                Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
                The Regents of the University of California. All rights reserved.
                FreeBSD is a registered trademark of The FreeBSD Foundation.
                FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 21:01:42 UTC 2023
                root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/sources/FreeBSD-src-plus-RELENG_23_09_1/amd64.amd64/sys/pfSense-DEBUG amd64

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Ok, cool so we're just waiting for a crash then? 😉

                  S 1 Reply Last reply Reply Quote 0
                  • S
                    smithjt @stephenw10
                    last edited by

                    @stephenw10

                    It's been 31 days, and I have not experienced a crash since I went to the debug kernal.

                    keyserK 1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Hmm, and you would have had a crash before that previously?

                      I have seen issues where running in debug mode actually changed the timing sufficiently to avoid it.

                      1 Reply Last reply Reply Quote 0
                      • keyserK
                        keyser Rebel Alliance @smithjt
                        last edited by

                        @smithjt Not that I’m saying it’s the case here, But I had a similar issue on my 6100 where the crashes was actually triggered by the MCE (Hardware error) mechanism in the Intel CPU. Support and the Internet told me that error could only come from actual unstable/defective hardware, but that just didn’t sound right to me. My box had been 100% stable up until I upgraded to 23.01 where it dumped anywhere between 7 and 16 days from boot. Booting the 22.05 snapshot boot environement made it completely stable again.
                        Never got it running on 23.01, and when 23.05 came i decided to start over (reflash from recovery image). Never had the issue again.
                        My conclusion was that the upgrade to 23.01 or some files on my SSD was somehow botsched, and caused the error.

                        Love the no fuss of using the official appliances :-)

                        S 1 Reply Last reply Reply Quote 1
                        • S
                          smithjt @keyser
                          last edited by

                          @keyser

                          Thanks for the info!

                          It has been forty days and counting since my last crash, so I'm still keeping an eye out for any. I recently reinstalled the "Netgate_Firmware_Upgrade" package a few days ago in an attempt to return to my previous configuration before all of this. I will wait patiently for at least a week or two to see whether that is the reason for the crash.

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            The Netgate Formare package does nothing at all until you actually update the BIOS. If you already have the current version there is no need to have it installed. It's installed by default in Plus though so I'd be surprised if it was giving any problems.

                            1 Reply Last reply Reply Quote 0
                            • W
                              wblanton
                              last edited by

                              I’m having a similar issue. It started on my 7100 after I updated all remote sites to 23.09.1 and it was left on 23.05.1. It crashed 2x since January, and I was able to update it this pas week.

                              Today I’ve got a unit at a remote site that appears to be doing something similar. Starting to wonder if I need to look at other firewall options, as they have only been in place for about 2 years.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                You have a crash report?

                                W 1 Reply Last reply Reply Quote 0
                                • W
                                  wblanton @stephenw10
                                  last edited by

                                  @stephenw10 Unfortunately no. On second thought, this may be unrelated because I'm not sure it's crashing so much as not responding on the network. Almost like it may be losing all routes. I just logged into the one that froze up over this weekend after it was rebooted and there's no banner indicating a crash report is available and there is nothing new in the /var/crash/ directory.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Anything in the system log when it stopped responding.

                                    W 1 Reply Last reply Reply Quote 0
                                    • W
                                      wblanton @stephenw10
                                      last edited by

                                      @stephenw10 Unfortunately, because we are using syslog over VPN tunnels, the system logs are spammed with "syslogd: sendto: Host is down" messages and has completely overwritten all the logs in 2/3 of the cases. There was one time that I was able to have some reboot the router in time for the logs to NOT be completely overwritten. But in that case, everything looked relatively normal until 10:45AM when suddenly all of my GWs began to experience high losses. It started with my WAN2 GW, followed by the endpoints we are monitoring over our MPLS circuits, and finally our primary WAN GW. It almost just looks to me like it just didn't know where to route anything anymore. In fact, the "gateways.log" file is filled with nothing but "sendto error: 65" messages for all gateways (this is true for all 3 instances). I've got a GW group set up for WAN1 and WAN2 and my default GW is that GW group, so it is NOT set to "automatic."

                                      I have noticed that in all three instances (2 on the primary router and this last event at a remote site), the issue occurs at the 45-minute mark of the hour.

                                      The only thing I saw that looked abnormal in the first two occurrences at the primary site, specifically the second event on 2/27, was that I started getting BGP errors at 10:45:08 saying things like "CeaseUnregonized Error Subcode," "Cease/Connection collision resolution," and "Unrecognized capability code" for 6, 70, and 71.

                                      Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.0.15 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.15(remote5.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 172.16.0.18 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.18(remote7.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.19(remote9.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.11 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.11(remote1.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.21 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.21(remote11.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.12 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.12(remote2.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.20 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.20(remote10.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.30 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.30(remote12.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.21(remote11.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.15 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.15(remote5.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:08 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.0.14 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:08 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.14(remote4.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:09 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.13(remote3.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:09 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.0.16 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:09 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.16(remote6.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:09 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 172.16.0.14 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:09 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.14(remote7.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:09 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.18 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:09 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.18(remote8.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:09 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.0.18 6/10 (CeaseUnrecognized Error Subcode) 0 bytes 
Feb 27 10:45:09 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.18(remote8.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:09 r1 bgpd[92136]: [EC 33554503] 192.168.1.19 unrecognized capability code: 70 - ignored
Feb 27 10:45:09 r1 bgpd[92136]: [EC 33554503] 192.168.1.19 unrecognized capability code: 6 - ignored
Feb 27 10:45:09 r1 bgpd[92136]: [EC 33554503] 192.168.1.19 unrecognized capability code: 71 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.19(remote9.example.com) in vrf default Up
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.1.13 unrecognized capability code: 70 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.1.13 unrecognized capability code: 6 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.1.13 unrecognized capability code: 71 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.0.21 unrecognized capability code: 70 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.0.21 unrecognized capability code: 6 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.0.21 unrecognized capability code: 71 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.13(remote3.example.com) in vrf default Up
Feb 27 10:45:10 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.21(remote11.example.com) in vrf default Up
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.0.15 unrecognized capability code: 70 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.0.15 unrecognized capability code: 6 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554503] 192.168.0.15 unrecognized capability code: 71 - ignored
Feb 27 10:45:10 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.0.15 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:10 r1 bgpd[92136]: [EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: 192.168.0.15
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.15 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.15 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.15 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.15(remote5.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 172.16.0.18 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 172.16.0.18 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 172.16.0.18 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.14 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.14 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.14 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.1.14 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: 192.168.1.14
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.11 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.11 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.11 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.12 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.12 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.12 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.20 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.20 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.20 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.14 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.14 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.14 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.12(remote2.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.15 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.15 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.15 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.18(remote7.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 172.16.0.18 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.16 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.16 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.16 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 172.16.0.14 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 172.16.0.14 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 172.16.0.14 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 172.16.0.14 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.19 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.19 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.19 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.0.19 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: 192.168.0.19
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.18 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.18 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.18 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.15(remote5.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.1.15 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.14(remote4.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.0.14 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.20(remote10.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.1.20 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.11(remote1.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.1.11 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.15(remote5.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.14(remote7.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.16(remote6.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.21 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.21 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.1.21 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.0.14 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.14(remote4.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.18(remote8.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: received from neighbor 192.168.1.20 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.20(remote10.example.com) in vrf default Down BGP Notification received
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.11(remote1.example.com) in vrf default Down Peer closed the session
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.18 unrecognized capability code: 70 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.18 unrecognized capability code: 6 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: [EC 33554503] 192.168.0.18 unrecognized capability code: 71 - ignored
Feb 27 10:45:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.0.18 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.21(remote11.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.18(remote8.example.com) in vrf default Up
Feb 27 10:45:11 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.19 in vrf default
Feb 27 10:45:11 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.13 in vrf default
Feb 27 10:45:11 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.21 in vrf default
Feb 27 10:45:11 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.0.21 in vrf default
Feb 27 10:45:12 r1 bgpd[92136]: [EC 33554503] 192.168.1.15 unrecognized capability code: 70 - ignored
Feb 27 10:45:12 r1 bgpd[92136]: [EC 33554503] 192.168.1.15 unrecognized capability code: 6 - ignored
Feb 27 10:45:12 r1 bgpd[92136]: [EC 33554503] 192.168.1.15 unrecognized capability code: 71 - ignored
Feb 27 10:45:12 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.15(remote5.example.com) in vrf default Up
Feb 27 10:45:12 r1 bgpd[92136]: [EC 33554503] 192.168.1.11 unrecognized capability code: 70 - ignored
Feb 27 10:45:12 r1 bgpd[92136]: [EC 33554503] 192.168.1.11 unrecognized capability code: 6 - ignored
Feb 27 10:45:12 r1 bgpd[92136]: [EC 33554503] 192.168.1.11 unrecognized capability code: 71 - ignored
Feb 27 10:45:12 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.11(remote1.example.com) in vrf default Up
Feb 27 10:45:12 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.0.15 in vrf default
Feb 27 10:45:12 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.15 in vrf default
Feb 27 10:45:12 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.12 in vrf default
Feb 27 10:45:12 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.0.16 in vrf default
Feb 27 10:45:12 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.18 in vrf default
Feb 27 10:45:12 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.0.18 in vrf default
Feb 27 10:45:13 r1 bgpd[92136]: [EC 33554503] 192.168.0.14 unrecognized capability code: 70 - ignored
Feb 27 10:45:13 r1 bgpd[92136]: [EC 33554503] 192.168.0.14 unrecognized capability code: 6 - ignored
Feb 27 10:45:13 r1 bgpd[92136]: [EC 33554503] 192.168.0.14 unrecognized capability code: 71 - ignored
Feb 27 10:45:13 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.14(remote4.example.com) in vrf default Up
Feb 27 10:45:13 r1 bgpd[92136]: [EC 33554503] 192.168.1.20 unrecognized capability code: 70 - ignored
Feb 27 10:45:13 r1 bgpd[92136]: [EC 33554503] 192.168.1.20 unrecognized capability code: 6 - ignored
Feb 27 10:45:13 r1 bgpd[92136]: [EC 33554503] 192.168.1.20 unrecognized capability code: 71 - ignored
Feb 27 10:45:13 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.20(remote10.example.com) in vrf default Up
Feb 27 10:45:13 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.11 in vrf default
Feb 27 10:45:14 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 172.16.0.18 in vrf default
Feb 27 10:45:14 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 172.16.0.14 in vrf default
Feb 27 10:45:14 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.0.14 in vrf default
Feb 27 10:45:14 r1 bgpd[92136]: bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 192.168.1.20 in vrf default
Feb 27 10:45:15 r1 bgpd[92136]: [EC 33554503] 192.168.1.14 unrecognized capability code: 70 - ignored
Feb 27 10:45:15 r1 bgpd[92136]: [EC 33554503] 192.168.1.14 unrecognized capability code: 6 - ignored
Feb 27 10:45:15 r1 bgpd[92136]: [EC 33554503] 192.168.1.14 unrecognized capability code: 71 - ignored
Feb 27 10:45:15 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.1.14 6/7 (Cease/Connection collision resolution) 0 bytes 
Feb 27 10:45:15 r1 bgpd[92136]: [EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: 192.168.1.14
Feb 27 10:45:28 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 172.16.0.50 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:45:28 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.50(Unknown) in vrf default Down BGP Notification send
Feb 27 10:45:31 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 172.16.0.46 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:45:31 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.46(Unknown) in vrf default Down BGP Notification send
Feb 27 10:45:35 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.254.1 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:45:35 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.254.1(Unknown) in vrf default Down BGP Notification send
Feb 27 10:45:41 r1 watchfrr[7590]: [EC 268435457] bfdd state -> unresponsive : no response yet to ping sent 30 seconds ago
Feb 27 10:45:41 r1 watchfrr[7590]: Forked background command [pid 54099]: /usr/local/etc/rc.d/frr restart bfdd
Feb 27 10:46:01 r1 watchfrr[7590]: Warning: restart bfdd child process 54099 still running after 20 seconds, sending signal 15
Feb 27 10:46:01 r1 watchfrr[7590]: restart bfdd process 54099 terminated due to signal 15
Feb 27 10:46:13 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 172.16.0.42 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:46:13 r1 bgpd[92136]: %ADJCHANGE: neighbor 172.16.0.42(Unknown) in vrf default Down BGP Notification send
Feb 27 10:46:58 r1 bgpd[92136]: [EC 33554454] 192.168.0.11 [Error] bgp_read_packet error: Operation timed out
Feb 27 10:46:58 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.11(remote1.example.com) in vrf default Down Peer closed the session
Feb 27 10:47:02 r1 watchfrr[7590]: Forked background command [pid 4051]: /usr/local/etc/rc.d/frr restart bfdd
Feb 27 10:47:04 r1 bgpd[92136]: [EC 33554454] 192.168.0.20 [Error] bgp_read_packet error: Operation timed out
Feb 27 10:47:04 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.0.20(remote10.example.com) in vrf default Down Peer closed the session
Feb 27 10:47:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.2.18 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:47:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.2.18(remote8.example.com) in vrf default Down BGP Notification send
Feb 27 10:47:11 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.3.18 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:47:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.3.18(remote8.example.com) in vrf default Down BGP Notification send
Feb 27 10:47:11 r1 bgpd[92136]: [EC 33554454] 192.168.1.16 [Error] bgp_read_packet error: Operation timed out
Feb 27 10:47:11 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.1.16(remote6.example.com) in vrf default Down Peer closed the session
Feb 27 10:47:18 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.2.14 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:47:18 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.2.14(remote4.example.com) in vrf default Down BGP Notification send
Feb 27 10:47:20 r1 bgpd[92136]: %NOTIFICATION: sent to neighbor 192.168.2.20 4/0 (Hold Timer Expired) 0 bytes 
Feb 27 10:47:20 r1 bgpd[92136]: %ADJCHANGE: neighbor 192.168.2.20(remote10.example.com) in vrf default Down BGP Notification send
Feb 27 10:47:22 r1 watchfrr[7590]: Warning: restart bfdd child process 4051 still running after 20 seconds, sending signal 15
Feb 27 10:47:22 r1 watchfrr[7590]: restart bfdd process 4051 terminated due to signal 15
                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Hmm, yeah that seems suspect. Is that gateway group failover?

                                        Can you check the routing table when this happens?

                                        W 1 Reply Last reply Reply Quote 0
                                        • W
                                          wblanton @stephenw10
                                          last edited by

                                          @stephenw10 said in CRASH Report: My Netgate 7100 crashes every ten to fifteen days.:

                                          m, yeah that seems suspect. Is that gateway group failover?

                                          Can you check the routing table when this happens?

                                          Yeah, it's a failover group. (Tier 1 and Tier 2). Since I've been remote, I have no way to check anything because I haven't had console access.

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Hmm that error sure looks like it ends up with a missing route somehow
                                            https://docs.netgate.com/pfsense/en/latest/troubleshooting/gateway-errors.html#sendto-error-65

                                            W 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post