Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Hotplug event causes rc.start_packages: Restarting/Starting all packages

    Scheduled Pinned Locked Moved
    General pfSense Questions
    9
    52
    6.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • 3
      32G3LiQxu8
      last edited by

      Environment

      • System: Netgate pfSense Plus
      • Version: 22.05-RELEASE (amd64)

      Issue

      I need your help to determine if this is a bug or just a setting somewhere that I am not aware of. I never experienced this in the previous version so my assumption is it is a bug but then again maybe it's a "feature" šŸ˜Š. I use a separate port on my router for my work laptop. When I boot up my laptop it causes pfSense to restart all packages which causes Wireguard and other services to go offline.

      You will see in the system logs that a Hotplug event was detected followed by an Restarting/Starting all packages

      Hotplug event detected for OPT4(opt4) static IP (192.168.58.1 )

      pfSense php-fpm[382]: /rc.start_packages: Restarting/Starting all packages.

      System Logs - I have been meaning to post for a while so that's why the log dates are from June.

      Jun 29 09:43:14 pfSense php-fpm[382]: /rc.linkup: Hotplug event detected for OPT4(opt4) static IP (192.168.58.1 )
Jun 29 09:43:14 pfSense check_reload_status[412]: Reloading filter
Jun 29 09:43:16 pfSense check_reload_status[412]: Linkup starting em5
Jun 29 09:43:16 pfSense kernel: em5: link state changed to UP
Jun 29 09:43:17 pfSense php-fpm[23931]: /rc.linkup: Hotplug event detected for OPT4(opt4) static IP (192.168.58.1 )
Jun 29 09:43:17 pfSense check_reload_status[412]: rc.newwanip starting em5
Jun 29 09:43:17 pfSense check_reload_status[412]: Reloading filter
Jun 29 09:43:18 pfSense php-fpm[23931]: /rc.newwanip: rc.newwanip: Info: starting on em5.
Jun 29 09:43:18 pfSense php-fpm[23931]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.58.1) (interface: OPT4[opt4]) (real interface: em5).
Jun 29 09:43:19 pfSense php-fpm[23931]: /rc.newwanip: Removing static route for monitor 4.2.2.2 and adding a new route through VPN_Address
Jun 29 09:43:19 pfSense php-fpm[23931]: /rc.newwanip: Gateway, NONE AVAILABLE
Jun 29 09:43:19 pfSense php-fpm[23931]: /rc.newwanip: Resyncing OpenVPN instances for interface OPT4.
Jun 29 09:43:19 pfSense php-fpm[23931]: /rc.newwanip: Creating rrd update script
Jun 29 09:43:21 pfSense php-fpm[23931]: /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 192.168.58.1 ->  192.168.58.1 - Restarting packages.
Jun 29 09:43:21 pfSense check_reload_status[412]: Starting packages
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Restarting/Starting all packages.
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Jun 29 09:43:22 pfSense check_reload_status[412]: Syncing firewall
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Jun 29 09:43:22 pfSense check_reload_status[412]: Syncing firewall
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Jun 29 09:43:22 pfSense php-fpm[382]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Jun 29 09:43:23 pfSense lighttpd_pfb[32375]: [pfBlockerNG] DNSBL Webserver stopped
Jun 29 09:43:23 pfSense tail_pfb[32702]: [pfBlockerNG] Firewall Filter Service stopped
Jun 29 09:43:23 pfSense php_pfb[33199]: [pfBlockerNG] filterlog daemon stopped
Jun 29 09:43:23 pfSense lighttpd_pfb[35611]: [pfBlockerNG] DNSBL Webserver started
Jun 29 09:43:23 pfSense tail_pfb[36705]: [pfBlockerNG] Firewall Filter Service started
Jun 29 09:43:24 pfSense php[37205]: [pfBlockerNG] filterlog daemon started
Jun 29 09:43:24 pfSense php[37674]: [pfBlockerNG] DNSBL parser daemon started

      Gateway Logs - as a result of hotplug event

      Jun 29 09:42:58 pfSense dpinger[618]: exiting on signal 15
Jun 29 09:42:58 pfSense dpinger[1441]: exiting on signal 15
Jun 29 09:42:58 pfSense dpinger[85593]: send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 500ms  loss_alarm 20%  dest_addr ISP_IPv4_GW  bind_addr ISP_IPv4_Address  identifier "WAN_DHCP "
Jun 29 09:42:58 pfSense dpinger[85891]: send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 500ms  loss_alarm 20%  dest_addr 4.2.2.2  bind_addr VPN_Address  identifier "VPN_Address "
Jun 29 09:43:19 pfSense dpinger[85593]: exiting on signal 15
Jun 29 09:43:19 pfSense dpinger[85891]: exiting on signal 15
Jun 29 09:43:19 pfSense dpinger[10941]: send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 500ms  loss_alarm 20%  dest_addr ISP_IPv4_GW  bind_addr ISP_IPv4_Address  identifier "WAN_DHCP "
Jun 29 09:43:19 pfSense dpinger[11203]: send_interval 500ms  loss_interval 2000ms  time_period 60000ms  report_interval 0ms  data_len 1  alert_interval 1000ms  latency_alarm 500ms  loss_alarm 20%  dest_addr 4.2.2.2  bind_addr VPN_Address  identifier "VPN_Address "

      Workaround

      The workaround is just attach the pfSense port to a switch so pfSense sees that port as always Up/Online.

      Let me know your thoughts

      Thank you!

      GertjanG 1 Reply Last reply Reply Quote 0
      • GertjanG
        Gertjan @32G3LiQxu8
        last edited by

        @32g3liqxu8 said in Hotplug event causes rc.start_packages: Restarting/Starting all packages:

        I never experienced this in the previous version so my assumption is it is a bug but then again maybe it's a "feature"

        pfSense is doing this since day one.
        If an interface goes down, or up, some packages (processes) are restarted.

        No "help me" PM's please. Use the forum, the community will thank you.
        Edit : and where are the logs ??

        3 1 Reply Last reply Reply Quote 0
        • 3
          32G3LiQxu8 @Gertjan
          last edited by

          @gertjan could you point me where this is mentioned in the documentation?

          If an organization is using Wireguard then the tunnel will go down because of a hotplug event regardless if that Wireguard tunnel is being used on that port or not. In my opinion it should be reworked

          GertjanG 1 Reply Last reply Reply Quote 0
          • GertjanG
            Gertjan @32G3LiQxu8
            last edited by

            @32g3liqxu8

            First things first : I get your point.

            I made it a habit not to connect or disconnect anything to my main main router, pfSense, along time ago.
            NO exceptions (only during major network re design, and that didn't happen for the last decade).
            My main LANs goes to main switches, and then all lines go off to all the wall sockets every where.

            This always makes me smile :

            pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 192.168.58.1 ->  192.168.58.1 - (so we will) Restarting packages

            So, nothing changed - 'we know it' so we 'Restarting packages' => Wtf ?
            But, it could be a WAN NIC that went down, so it lost its IP, and when it came back, it got (static or DHCP ?) the old IP back.
            I tend to say : so do nothing ...... but I'm not an IP expert, I can understand that this leave the door open to a boat load of issues.

            Wireguard gets kicked as firewall / routes / ACLs got changed ?

            If WAN or LAN goes down/up, I can imagine that unbound, dhcpd, inginx (the GUI) has to be informed about this.

            There could even be routes defined that become invalid - or became valid.

            Instead of keeping in memory a matrix of who use what resources and when and what to what when things arrive, pfSense does its own housekeeping, and then, for every package installed, fires a Restart event.
            Some packages don't have to handle this kind of event.
            Others do.
            IMHO, the Wireguard package is a bit to verbose right now - but I'm not suing it, neither do I know what interface groups are ;)

            My opinion, as 'yet another pfSense user' : it's not a big deal as physical link can go goes down, and comes up again.
            TCP is resilient to this.
            UDP : we don't care ;)

            @32g3liqxu8 said in Hotplug event causes rc.start_packages: Restarting/Starting all packages:

            I use a separate port on my router for my work laptop

            this is after all not a good idea ??!
            I'm not saying it isn't - but : looking at te logs : I won't do that.

            @32g3liqxu8 said in Hotplug event causes rc.start_packages: Restarting/Starting all packages:

            When I boot up my laptop it causes pfSense to restart all packages which causes Wireguard and other services to go offline.

            That is, they will go off-line for a couple of seconds ? More ?
            I get it, phone calls or video chat will / might get interrupted.

            No "help me" PM's please. Use the forum, the community will thank you.
            Edit : and where are the logs ??

            3 1 Reply Last reply Reply Quote 1
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              It's firing off the newwanip script so it restarts packages that might be using WAN. But this appears to be an internal interface. Do you have a gateway on that interface? Static routes?

              An alternative workaround here, to avoid using a switch, might be to create a single interface bridge on em5 and assign that as interface instead. Though I agree a workaround shouldn't be required there from what can see.

              Steve

              3 1 Reply Last reply Reply Quote 1
              • 3
                32G3LiQxu8 @stephenw10
                last edited by

                @stephenw10

                OPT4 is an internal interface and I set a static ip address. Did I do this incorrectly?

                OPT4_Interface.png

                I use policy based routing so all non OPT4 net traffic goes out WAN_DHCP. To give you a better idea of my setup, I followed this guide - WireGuard VPN Client Configuration Example to setup WireGuard as my Default Gateway. So, that is why I'm defining WAN_DHCP as my gateway in my firewall rule. I kept the rules simple since this is a work laptop and I connect using a work provided vpn.

                OPT4_Firewall_Rules.png

                Let me know if you think there is anything I should change based on what I described.

                Thank you!

                1 Reply Last reply Reply Quote 0
                • 3
                  32G3LiQxu8 @Gertjan
                  last edited by 32G3LiQxu8

                  @gertjan

                  I agree, not a huge issue for me, as I'm just a home user, who just enjoys the software and networking but I figured I would report it for anyone who might use it more for running a business, etc. Thanks again for looking over what I provided. I provided some details above of my setup. Appreciate your help!

                  1 Reply Last reply Reply Quote 1
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ok, so the OPT4 interface doesn't have gateway defined in it's config but you could still have a gateway that is in the OPT4 subnet. You would only do that though if you need static routes via that to some other subnet connected there. Is that possible?

                    3 1 Reply Last reply Reply Quote 0
                    • 3
                      32G3LiQxu8 @stephenw10
                      last edited by

                      @stephenw10

                      I only have the one work laptop in that subnet. I just wanted it separate from my main LAN (Personal). It does not need to access another subnet.

                      Would you consider what I reported a bug in the software or is that how it is supposed to behave?

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        It seems unexpected but I probably call it a missing feature. Most installs don't see an interface go up/down except in a significant network event and it's safer for those to restart packages than to leave then potentially running with the wrong IP.
                        However running rc.newwanip on something that isn't a WAN seems unnecessary and I don't see that here. For example connecting to LAN on a 4100 in 22.05:

                        Sep 16 00:32:08 	check_reload_status 	495 	Linkup starting igc0
Sep 16 00:32:08 	kernel 		igc0: link state changed to UP
Sep 16 00:32:09 	php-fpm 	38757 	/rc.linkup: DEVD Ethernet attached event for lan
Sep 16 00:32:09 	php-fpm 	38757 	/rc.linkup: HOTPLUG: Configuring interface lan
Sep 16 00:32:10 	php-fpm 	38757 	/rc.linkup: Gateway, NONE AVAILABLE
Sep 16 00:32:10 	check_reload_status 	495 	Restarting IPsec tunnels
Sep 16 00:32:14 	check_reload_status 	495 	updating dyndns lan
Sep 16 00:32:14 	check_reload_status 	495 	Reloading filter

                        Something is different about my config there though since it doesn't list the static IP even though it is configured with one. It really looks like your config has caused pfSense to think opt4 is a WAN.
                        Are you able to upload that for review? Or replicate it in a very basic config that you can share?

                        Steve

                        3 1 Reply Last reply Reply Quote 0
                        • 3
                          32G3LiQxu8 @stephenw10
                          last edited by

                          @stephenw10

                          Is there a way I can direct message you a link to download my config from my cloud provider?

                          Also, would it break the config if I pulled out my workstation information and it would also have my wireguard config in plain text. I guess, which backup areas would you need to take a look at?

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            You can upload something to me here:
                            https://nc.netgate.com/nextcloud/s/yPwaeQLsdK5rK9r
                            You can remove any details you need to.

                            If you're able to upload the status_output diagnostic file that already has passwords and cert keys etc redacted:
                            https://docs.netgate.com/pfsense/en/latest/recipes/diagnostic-data.html#copying-the-diagnostic-data-archive

                            Steve

                            3 1 Reply Last reply Reply Quote 0
                            • 3
                              32G3LiQxu8 @stephenw10
                              last edited by

                              @stephenw10

                              Files have been uploaded. Let me know if you need anything else. Thank you for your help!

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Hmm, there's really nothing significantly different in your config there. At least not currently.

                                Are you able to reproduce the issue and then upload a new status file? I don't see any connections to em5 in the logs in the last few days.

                                Steve

                                3 1 Reply Last reply Reply Quote 0
                                • 3
                                  32G3LiQxu8 @stephenw10
                                  last edited by

                                  @stephenw10

                                  Yes, however, itā€™s my work laptop so I wonā€™t be able to do it at the moment. I will test it out after work and upload a new file for your review.

                                  Currently, I have em5 attached to a switch so it doesnā€™t restart packages so thatā€™s why youā€™re not seeing any information - I think.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Ah, yes, that would do it. Ok, let me know you're able to test it again.

                                    Steve

                                    3 1 Reply Last reply Reply Quote 0
                                    • 3
                                      32G3LiQxu8 @stephenw10
                                      last edited by

                                      @stephenw10

                                      I uploaded a new status_output.tgz file for your review.

                                      Off Topic
                                      On something completely unrelated, was something recently updated and pushed out? The following occurred today:

                                      Sep 16 18:53:37 	pkg-static 	86631 	pfSense-repo upgraded: 22.05_2 -> 22.05_5
Sep 16 18:53:37 	pkg-static 	86631 	pfSense-upgrade upgraded: 1.0_26 -> 1.0_27

                                      Now, I am unable to check for any packages and it thinks I do not have any packages installed when I do.

                                      pkg-static -d update

                                      DBG(1)[69590]> pkg initialized
pkg-static: invalid url: /pfSense_plus-v22_05_amd64-core
pkg-static: Cannot parse configuration file!
                                      S 1 Reply Last reply Reply Quote 0
                                      • S
                                        SteveITS Rebel Alliance @32G3LiQxu8
                                        last edited by

                                        @32g3liqxu8 I saw that error/issue also this afternoon.

                                        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                        Upvote šŸ‘ helpful posts!

                                        1 Reply Last reply Reply Quote 1
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Yup just hit it and found the cause. Working on it now...

                                          3 1 Reply Last reply Reply Quote 1
                                          • 3
                                            32G3LiQxu8 @stephenw10
                                            last edited by

                                            @stephenw10

                                            Thank you!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post