Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    21.02 Sudden lockup

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    164 Posts 30 Posters 61.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mcury Rebel Alliance @kphillips
      last edited by mcury

      @kphillips said in 21.02 Sudden lockup:

      @mcury OK thank you. So to confirm, the webConfigurator is inaccessible and you cannot pass traffic at all when this happens? What if you ping by IP address rather than by hostname from a device attached to the LAN? If you ping 8.8.8.8, for example, does it succeed but DNS queries fail? Since pfBlockerNG ties into unbound (DNS Resolver) I'm wondering if its crashing the DNS Resolver causing your issue.

      Yes, webConfigurator is inaccessible and I cannot pass traffic.
      Ping to 8.8.8.8 fails, or any other IP on the internet (it's not DNS).

      I'll perform the tests through the console.

      I'll reproduce the problem, give me a few minutes.
      1 - enable pfblockerng-devel again
      2 - Reboot
      3 - Test again the ping to 8.8.8.8, ping the pfsense LAN interface.

      Then, I'll perform the tests @stephenw10 asked

      1 - perform ifconfig -va
      2 - netstat -rn
      3 - etherswitchcfg to check the switch

      dead on arrival, nowhere to be found.

      M 1 Reply Last reply Reply Quote 0
      • F
        ffuentes @rloeb
        last edited by

        @rloeb no output for me when it locked up.
        Once it reboots I can see the normal boot process.

        @kphillips Anything I can capture to help out?

        1 Reply Last reply Reply Quote 0
        • M
          mcury Rebel Alliance @mcury
          last edited by mcury

          After enabling pfBlockerNG, rebooted the system and the problem didn't happen.
          So I accessed the GUI, clicked in Firewall>pfBlockerNG>Update>Reload all>Run

          Then I rebooted again, and the problem happened.
          It seems that it happens hourly after the cron update from pfblockerng, or if I manually force a reload or an update, and it's triggered during the first boot after that.

          • GUI is inaccessible.
          • SSH from a device in LAN to pfsense LAN interface - Works
          • SSH from VLAN WIFI to devices in another VLAN - Don't work.
          • Ping to pfsense LAN interface from a PC in the LAN - Works.
          • Ping from a PC in the LAN to 8.8.8.8 - Don't work.

          ifconfig -va

          21.02-RELEASE][root@pfsense.local.lan]/root: ifconfig -va
          mvneta0: flags=8a02<BROADCAST,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
          	options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
          	ether 00:08:a2:0c:c4:1b
          	media: Ethernet autoselect (none)
          	status: no carrier
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          mvneta1: flags=88a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500
          	description: mgmt
          	options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
          	ether 00:08:a2:0c:c4:1c
          	inet6 fe80::208:a2ff:fe0c:c41c%mvneta1 prefixlen 64 scopeid 0x2
          	inet 172.16.200.1 netmask 0xfffffff8 broadcast 172.16.200.7
          	media: Ethernet 2500Base-KX <full-duplex>
          	status: active
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          mvneta2: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
          	options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
          	ether 00:08:a2:0c:c4:1d
          	inet6 fe80::208:a2ff:fe0c:c41d%mvneta2 prefixlen 64 scopeid 0x8
          	inet6 X:X:X:X:X:X:X:X prefixlen 64 autoconf
          	inet6 X:X:X::X prefixlen 128
          	inet X.X.X.X netmask 0xfffffc00 broadcast X.X.X.X
          	inet 192.168.100.2 netmask 0xffffffff broadcast 192.168.100.2
          	media: Ethernet autoselect (1000baseT <full-duplex>)
          	status: active
          	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
          enc0: flags=0<> metric 0 mtu 1536
          	groups: enc
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
          	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
          	inet6 ::1 prefixlen 128
          	inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
          	inet 127.0.0.1 netmask 0xff000000
          	inet 10.10.10.1 netmask 0xffffffff
          	groups: lo
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          pflog0: flags=100<PROMISC> metric 0 mtu 33184
          	groups: pflog
          pfsync0: flags=0<> metric 0 mtu 1500
          	groups: pfsync
          mvneta1.100: flags=88843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500
          	description: lan
          	options=3<RXCSUM,TXCSUM>
          	ether 00:08:a2:0c:c4:1c
          	inet6 fe80::208:a2ff:fe0c:c41c%mvneta1.100 prefixlen 64 scopeid 0xd
          	inet 192.168.255.249 netmask 0xfffffff8 broadcast 192.168.255.255
          	groups: vlan
          	vlan: 100 vlanpcp: 0 parent interface: mvneta1
          	media: Ethernet Other <full-duplex>
          	status: active
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          mvneta1.10: flags=88843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500
          	description: wifi
          	options=3<RXCSUM,TXCSUM>
          	ether 00:08:a2:0c:c4:1c
          	inet6 fe80::208:a2ff:fe0c:c41c%mvneta1.10 prefixlen 64 scopeid 0xe
          	inet6 fe80::1:1%mvneta1.10 prefixlen 64 scopeid 0xe
          	inet6 X:X:X:X:X:X:X:X prefixlen 64
          	inet 192.168.10.1 netmask 0xfffffff0 broadcast 192.168.10.15
          	groups: vlan
          	vlan: 10 vlanpcp: 0 parent interface: mvneta1
          	media: Ethernet Other <full-duplex>
          	status: active
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          mvneta1.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
          	description: guest
          	options=3<RXCSUM,TXCSUM>
          	ether 00:08:a2:0c:c4:1c
          	inet6 fe80::208:a2ff:fe0c:c41c%mvneta1.20 prefixlen 64 scopeid 0xf
          	inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
          	groups: vlan
          	vlan: 20 vlanpcp: 0 parent interface: mvneta1
          	media: Ethernet Other <full-duplex>
          	status: active
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          

          netstat -rn

          [21.02-RELEASE][root@pfsense.local.lan]/root: netstat -rn
          Routing tables
          
          Internet:
          Destination        Gateway            Flags     Netif Expire
          default            X.X.X.X      UGS     mvneta2
          10.10.10.1         link#10            UH          lo0
          127.0.0.1          link#10            UH          lo0
          172.16.200.0/29    link#2             U       mvneta1
          172.16.200.1       link#2             UHS         lo0
          X.X.X.X/22   link#8             U       mvneta2
          X.X.X.X    link#8             UHS         lo0
          192.168.10.0/28    link#14            U      mvneta1.
          192.168.10.1       link#14            UHS         lo0
          192.168.20.0/24    link#15            U      mvneta1.
          192.168.20.1       link#15            UHS         lo0
          192.168.100.2      link#8             UHS         lo0
          192.168.100.2/32   link#8             U       mvneta2
          192.168.255.248/29 link#13            U      mvneta1.
          192.168.255.249    link#13            UHS         lo0
          
          Internet6:
          Destination                       Gateway                       Flags     Netif Expire
          default                           fe80::2c8:8bff:fed8:3419%mvneta2 UG   mvneta2
          ::1                               link#10                       UH          lo0
          X:X:X::/64                link#8                        U       mvneta2
          X:X:X::X                link#8                        UHS         lo0
          X:X:X:X:X:X:X:X link#8                       UHS         lo0
          X:X:X:X::/64           link#14                       U      mvneta1.
          X:X:X:X:X:X:X:X link#14                   UHS         lo0
          fe80::%mvneta1/64                 link#2                        U       mvneta1
          fe80::208:a2ff:fe0c:c41c%mvneta1  link#2                        UHS         lo0
          fe80::%mvneta2/64                 link#8                        U       mvneta2
          fe80::208:a2ff:fe0c:c41d%mvneta2  link#8                        UHS         lo0
          fe80::%lo0/64                     link#10                       U           lo0
          fe80::1%lo0                       link#10                       UHS         lo0
          fe80::%mvneta1.100/64             link#13                       U      mvneta1.
          fe80::208:a2ff:fe0c:c41c%mvneta1.100 link#13                    UHS         lo0
          fe80::%mvneta1.10/64              link#14                       U      mvneta1.
          fe80::1:1%mvneta1.10              link#14                       UHS         lo0
          fe80::208:a2ff:fe0c:c41c%mvneta1.10 link#14                     UHS         lo0
          fe80::%mvneta1.20/64              link#15                       U      mvneta1.
          fe80::208:a2ff:fe0c:c41c%mvneta1.20 link#15                     UHS         lo0
          

          etherswitchcfg

          21.02-RELEASE][root@pfsense.local.lan]/root: etherswitchcfg 
          etherswitch0: VLAN mode: DOT1Q
          port1:
          	pvid: 100
          	state=8<FORWARDING>
          	flags=0<>
          	media: Ethernet autoselect (1000baseT <full-duplex>)
          	status: active
          port2:
          	pvid: 100
          	state=8<FORWARDING>
          	flags=0<>
          	media: Ethernet autoselect (100baseTX <full-duplex>)
          	status: active
          port3:
          	pvid: 100
          	state=8<FORWARDING>
          	flags=0<>
          	media: Ethernet autoselect (1000baseT <full-duplex,master>)
          	status: active
          port4:
          	pvid: 1
          	state=8<FORWARDING>
          	flags=0<>
          	media: Ethernet autoselect (1000baseT <full-duplex>)
          	status: active
          port5:
          	pvid: 1
          	state=8<FORWARDING>
          	flags=1<CPUPORT>
          	media: Ethernet 2500Base-KX <full-duplex>
          	status: active
          vlangroup0:
          	vlan: 1
          	members 4,5
          vlangroup1:
          	vlan: 100
          	members 1,2,3,4t,5t
          vlangroup2:
          	vlan: 10
          	members 4t,5t
          vlangroup3:
          	vlan: 20
          	members 4t,5t
          

          dead on arrival, nowhere to be found.

          1 Reply Last reply Reply Quote 0
          • F
            ffuentes
            last edited by

            @mcury Awesome find! It kinda makes sense as it was happening to me hourly. I have pfblockerng disable now. I do wonder if the corn schedule still kicks in?

            EDIT: It does. I just had to reboot the box.
            Looks like I have to find a way to disable the cron.

            B 1 Reply Last reply Reply Quote 0
            • B
              bldnightowl @ffuentes
              last edited by

              +1 also experiencing the hang described here. Disabling pfBlockerNg helped.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Are you running the dev version of pfBlocker?

                Does it 'lock up' as it's running the reload or once it's completed?

                How large are you aliases/blocklists?

                I have the non-dev package running and am not seeing that but only have limited dnsbl lists loaded.

                Steve

                B F M 3 Replies Last reply Reply Quote 0
                • B
                  bldnightowl @stephenw10
                  last edited by bldnightowl

                  @stephenw10 I am running pfBlockerNG-devel 3.0.0_10. The alias/blocklists are whatever is OOTB, I didn't customize anything.

                  By "lockup" I mean the LAN/WAN interfaces are completely unresponsive. Console access is fine.

                  P.S. I cross posted to this thread to make sure that @BBcan177 (the developer of pfBlockerNG) is aware.

                  1 Reply Last reply Reply Quote 0
                  • F
                    ffuentes @stephenw10
                    last edited by ffuentes

                    @stephenw10 pfBlockerNG net 2.1.4_24
                    I do have several lists loading.
                    My Max table Entry is: 4000000

                    Another thing I notice is that vnstad keeps crashing:

                    vnstatd 	Status Traffic Totals data collection daemon
                    
                    1 Reply Last reply Reply Quote 0
                    • M
                      mcury Rebel Alliance @stephenw10
                      last edited by

                      @stephenw10 said in 21.02 Sudden lockup:

                      Are you running the dev version of pfBlocker?

                      Does it 'lock up' as it's running the reload or once it's completed?

                      How large are you aliases/blocklists?

                      I have the non-dev package running and am not seeing that but only have limited dnsbl lists loaded.

                      Steve

                      Running the pfblockerng-devel 3.0.0_10, didn't customize anything, just ran the wizard, so it's a normal amount of aliases/blocklists.
                      No, it doesn't 'lock up' as it's running the reload or once it's completed, it finishes the process, and it's triggered if I reboot in this phase:

                      Syncing OpenVPN settings...done.
                      Configuring firewall.Segmentation fault (core dumped) <<<
                      Starting CRON... done.

                      dead on arrival, nowhere to be found.

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Ah, OK so part way through the boot after loading the alises/lists?

                        Can you give us an idea of the numbers? I have:

                        
                        ===[ Native List IP Counts ] ===================================
                        
                            8513 total
                            7311 /var/db/pfblockerng/native/Google.txt
                             968 /var/db/pfblockerng/native/Spamhaus_drop.txt
                             181 /var/db/pfblockerng/native/Facebook.txt
                              53 /var/db/pfblockerng/native/Netflix.txt
                        
                        ===[ DNSBL Domain/IP Counts ] ===================================
                        
                           21346 total
                           16998 /var/db/pfblockerng/dnsbl/Easylist_Default.txt
                            4342 /var/db/pfblockerng/dnsbl/Easylist_Privacy.txt
                               6 /var/db/pfblockerng/dnsbl/Custom_List_custom.txt
                        

                        Steve

                        M F 2 Replies Last reply Reply Quote 0
                        • M
                          mcury Rebel Alliance @stephenw10
                          last edited by

                          @stephenw10 said in 21.02 Sudden lockup:

                          Ah, OK so part way through the boot after loading the alises/lists?

                          Exactly.

                          Packages installed: Acme, NUT, pfBlockerng-devel, aws-wizard, ipsec-profile-wizard.
                          It doesn't seem to be a memory issue:

                          91785800-5e00-484e-ace8-441ba1c34f78-image.png

                          ===[ Deny List IP Counts ]===========================
                          
                             22138 total
                             15000 /var/db/pfblockerng/deny/CINS_army_v4.txt
                              4884 /var/db/pfblockerng/deny/ET_Comp_v4.txt
                              1223 /var/db/pfblockerng/deny/ET_Block_v4.txt
                               732 /var/db/pfblockerng/deny/Talos_BL_v4.txt
                               118 /var/db/pfblockerng/deny/Abuse_Feodo_C2_v4.txt
                                94 /var/db/pfblockerng/deny/Abuse_SSLBL_v4.txt
                                76 /var/db/pfblockerng/deny/Spamhaus_eDrop_v4.txt
                                 8 /var/db/pfblockerng/deny/ISC_Block_v4.txt
                                 2 /var/db/pfblockerng/deny/Spamhaus_Drop_v4.txt
                                 1 /var/db/pfblockerng/deny/Abuse_IPBL_v4.txt
                          
                          ====================[ Empty Lists w/127.1.7.7 ]==================
                          
                          Abuse_IPBL_v4.txt
                          
                          ===[ DNSBL Domain/IP Counts ] ===================================
                          
                            349821 total
                            143310 /var/db/pfblockerng/dnsbl/Maltrail_BD.txt
                            122595 /var/db/pfblockerng/dnsbl/C19_CTC.txt
                             27951 /var/db/pfblockerng/dnsbl/SFS_Toxic_BD.txt
                             13486 /var/db/pfblockerng/dnsbl/SWC.txt
                             10871 /var/db/pfblockerng/dnsbl/EasyList.txt
                              8913 /var/db/pfblockerng/dnsbl/Adaway.txt
                              6999 /var/db/pfblockerng/dnsbl/Spam404.txt
                              6671 /var/db/pfblockerng/dnsbl/MVPS.txt
                              3031 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt
                              2500 /var/db/pfblockerng/dnsbl/D_Me_ADs.txt
                              1986 /var/db/pfblockerng/dnsbl/Krisk_C19.txt
                              1478 /var/db/pfblockerng/dnsbl/Yoyo.txt
                                23 /var/db/pfblockerng/dnsbl/D_Me_Tracking.txt
                                 6 /var/db/pfblockerng/dnsbl/Juniper.txt
                                 1 /var/db/pfblockerng/dnsbl/D_Me_Malv.txt
                                 0 /var/db/pfblockerng/dnsbl/MDS_Immortal.fail
                                 0 /var/db/pfblockerng/dnsbl/MDS.fail
                                 0 /var/db/pfblockerng/dnsbl/MDL.txt
                                 0 /var/db/pfblockerng/dnsbl/D_Me_Malw.txt
                          
                          ====================[ IPv4/6 Last Updated List Summary ]==============
                          
                          Feb 11	04:49	Spamhaus_eDrop_v4
                          Feb 17	02:30	ET_Block_v4
                          Feb 17	02:30	ET_Comp_v4
                          Feb 18	03:33	Spamhaus_Drop_v4
                          Feb 18	18:05	Talos_BL_v4
                          Feb 18	18:18	CINS_army_v4
                          Feb 18	18:39	ISC_Block_v4
                          Feb 18	18:55	Abuse_Feodo_C2_v4
                          Feb 18	18:55	Abuse_SSLBL_v4
                          Feb 18	19:01	Abuse_IPBL_v4
                          
                          ====================[ DNSBL Last Updated List Summary ]==============
                          
                          Jul 31	2015	D_Me_Tracking
                          Sep 5	2018	Juniper
                          Jan 31	2020	D_Me_ADs
                          Jul 10	2020	D_Me_Malw
                          Jul 10	2020	D_Me_Malv
                          Nov 12	19:17	MDL
                          Dec 15	05:07	MVPS
                          Feb 15	02:18	Adaway
                          Feb 15	05:50	Yoyo
                          Feb 16	05:48	SWC
                          Feb 18	15:02	EasyPrivacy
                          Feb 18	15:30	Krisk_C19
                          Feb 18	17:50	C19_CTC
                          Feb 18	18:50	EasyList
                          Feb 18	19:00	SFS_Toxic_BD
                          Feb 18	19:00	Maltrail_BD
                          Feb 18	19:00	Spam404
                          ===============================================================
                          
                          Database Sanity check [  PASSED  ]
                          ------------------------
                          Masterfile/Deny folder uniq check
                          Deny folder/Masterfile uniq check
                          
                          Sync check (Pass=No IPs reported)
                          ----------
                          
                          Alias table IP Counts
                          -----------------------------
                             22138 /var/db/aliastables/pfB_PRI1_v4.txt
                          
                          pfSense Table Stats
                          -------------------
                          table-entries hard limit   400000
                          Table Usage Count         22164
                          
                           UPDATE PROCESS ENDED [ 02/18/21 19:01:58 ]
                          

                          07416874-0142-4a3c-a50e-a4b192c16ab2-image.png

                          dead on arrival, nowhere to be found.

                          F 1 Reply Last reply Reply Quote 0
                          • F
                            ffuentes @stephenw10
                            last edited by

                            @stephenw10 My SG-3100 locked up again with pfblocker disabled. no cron running.

                            Unless the cron is running even though is been told to disable.
                            You guys have a bigger issue here.

                            M B 2 Replies Last reply Reply Quote 0
                            • F
                              ffuentes @mcury
                              last edited by

                              @mcury I don't think is a pfblocker issue TBH.
                              Mine keeps locking up even though is fully disabled, I am now going to uninstall it.

                              1 Reply Last reply Reply Quote 0
                              • M
                                mcury Rebel Alliance @ffuentes
                                last edited by mcury

                                @ffuentes Try to set the pfblocker cron settings to run once a day, instead of one hour to confirm if the problem will happen to you again.
                                Or completely remove the pfblockerng package. If it happens again even without the package installed (not only disabled). We will have a better understanding about it..

                                dead on arrival, nowhere to be found.

                                F B 2 Replies Last reply Reply Quote 1
                                • F
                                  ffuentes @mcury
                                  last edited by

                                  @mcury I just uninstalled it. Let see how far the rabbit holes go. :/

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    mcury Rebel Alliance @ffuentes
                                    last edited by

                                    @ffuentes said in 21.02 Sudden lockup:

                                    @mcury I just uninstalled it. Let see how far the rabbit holes go. :/

                                    Let's hope that we all here have the same problem, one fix is faster than two or more :)

                                    dead on arrival, nowhere to be found.

                                    F 1 Reply Last reply Reply Quote 1
                                    • B
                                      bldnightowl @mcury
                                      last edited by

                                      @mcury I also disabled and got a lockup again just now. This is bad news, folks. I don’t think it’s pfBlockerNG either.

                                      How do I downgrade back to 2.4.5p1?

                                      M pttP 2 Replies Last reply Reply Quote 0
                                      • M
                                        mcury Rebel Alliance @bldnightowl
                                        last edited by

                                        @bldnightowl said in 21.02 Sudden lockup:

                                        @mcury I also disabled and got a lockup again just now. This is bad news, folks. I don’t think it’s pfBlockerNG either.

                                        How do I downgrade back to 2.4.5p1?

                                        Try to remove pfblockerng package, just to make sure tick that setting to remove the configuration too, this option is inside pfblockerng.

                                        dead on arrival, nowhere to be found.

                                        1 Reply Last reply Reply Quote 0
                                        • pttP
                                          ptt Rebel Alliance @bldnightowl
                                          last edited by

                                          How do I downgrade back to 2.4.5p1?

                                          Reinstall & restore Config Backup

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            ffuentes @mcury
                                            last edited by ffuentes

                                            @mcury said in 21.02 Sudden lockup:

                                            @ffuentes said in 21.02 Sudden lockup:

                                            @mcury I just uninstalled it. Let see how far the rabbit holes go. :/

                                            Let's hope that we all here have the same problem, one fix is faster than two or more :)

                                            Completely agree. I am trying to avoid going back to the previous version to try and help narrow this down.
                                            But is really causing a lot of pain as it seems to be very consistent. (That's good news in a technical way)

                                            I have a console up and running via USB to see if I can catch anything, but is a HARD lockup. You can't get any output at all.

                                            M 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.