Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense Plus

    Scheduled Pinned Locked Moved General pfSense Questions
    20 Posts 2 Posters 1.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      newUser2pfSense @bmeeks
      last edited by newUser2pfSense

      @bmeeks Hi Bill...Thank you for the response.

      I have Suricata running in Inline IPS Mode on all interfaces.
      I have edited the WAN interface by unchecking the General Settings > Enable > Checking this box enables Suricata inspection on the interface. The WAN interface is now DISABLED.
      Interesting that the network interface will go down and back up each time Suricata is restarted in Inline IPS Mode. If Suricata is causing dpinger confusion and the interface goes down and doesn't come back up, my CRON/script will restart pfSense; must be what's causing my restarts. I'm wondering if there is a pfSense development solution for this?
      I do have the Live Rule Swap on Update checkmarked.

      I have been watching the restarts since October of 2021 until now. Hoping for a solution.

      bmeeksB 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @newUser2pfSense
        last edited by bmeeks

        @newuser2pfsense:
        Cycling the physical interface is a netmap thing. Nothing pfSense can do about it. It's just the way that kernel device works. It is a side effect of putting the interface in netmap mode. When Suricata stops, it releases the netmap device and reverts the interface for normal traffic flow. When Suricata starts, it switches the interface to netmap mode. Those changes cause the "bounce".

        Enabling the Live Rule Swap option stops the rules update process from physically restarting Suricata at the end of the update. Instead, a special SIGHUP mechanism offered by the Suricata binary is used to tell it to load the new rules into RAM and then switch to them and discard the old rules. The small downside is for a brief period of time you have two copies of the ruleset in RAM at the same time.

        Because of this interface cycling, in the newest version of Suricata now out for testing in pfSense-2.7.0 DEVEL, I've made it default to using "Live Rule Swap" when Inline IPS Mode operation is selected for any interface. We plan to push the latest update to the RELEASE branch in the very near future unless some big issue is reported from the DEVEL branch.

        A benefit of removing Suricata from the WAN is that now netmap is not enabled there and so can't cycle the interface. That should keep dpinger happy.

        N 1 Reply Last reply Reply Quote 0
        • N
          newUser2pfSense @bmeeks
          last edited by

          @bmeeks Thanks Bill. I'll keep an eye out for any future restarts, if they occur at all. I was hoping you would say that dpinger wouldn't have any issues now that Suricata on the WAN interface is disabled. I would imagine at the time 2.7.0 CE is released, a new version of pfSense+ will be released as well? Or is that just my wishful thinking?

          bmeeksB 1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @newUser2pfSense
            last edited by

            @newuser2pfsense said in pfSense Plus:

            @bmeeks Thanks Bill. I'll keep an eye out for any future restarts, if they occur at all. I was hoping you would say that dpinger wouldn't have any issues now that Suricata on the WAN interface is disabled. I would imagine at the time 2.7.0 CE is released, a new version of pfSense+ will be released as well? Or is that just my wishful thinking?

            Historically, pfSense Plus updates have come more often than CE updates. I have no insider info, but I would not be surprised to see that trend continue.

            N 1 Reply Last reply Reply Quote 0
            • N
              newUser2pfSense @bmeeks
              last edited by

              @bmeeks I'll report back with any issues. Thanks Bill.

              N 1 Reply Last reply Reply Quote 0
              • N
                newUser2pfSense @newUser2pfSense
                last edited by

                @bmeeks Hey Bill. At the time of this post, there have been no drops in connectivity or restarts from my CRON/script. It's only been a single day but good so far.

                I still won't be enabling Suricata on my WAN, but for the sake of conversation, I have the Live Rule Swap on Update enabled/checkmarked. See below image.

                LiveRuleSwap.png

                You were saying that when this setting is enabled, Suricata will not restart causing the interface to cycle down and back up again. So what I think is happening is even when the Live Rule Swap on Update is checkmarked/enabled, the rules are updating causing the interface to go down. Interestingly though, I don't believe the interface is coming back up. The reason I say this is because I have the CRON/script set to restart pfSense when 2 pings to my configured DNS server every 4 minutes is not returned. So I have been having all of these restarts since I created the CRON/script from October 2021. Please see a snippet below of the most recent restarts:

                System shutdown time has arrived
                07/13/2022.04:52:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/13/2022.05:04:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/14/2022.05:08:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/14/2022.21:40:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/15/2022.04:56:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/15/2022.05:16:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/16/2022.05:04:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/16/2022.05:24:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                System shutdown time has arrived
                07/17/2022.05:08:22 All pings failed twice. Rebooting...
                Shutdown NOW!
                Shutdown NOW!

                So, would you consider my train of thought correct? If so, is there a bug? Just wondering. Thanks.

                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @newUser2pfSense
                  last edited by bmeeks

                  @newuser2pfsense said in pfSense Plus:

                  You were saying that when this setting is enabled, Suricata will not restart causing the interface to cycle down and back up again. So what I think is happening is even when the Live Rule Swap on Update is checkmarked/enabled, the rules are updating causing the interface to go down. Interestingly though, I don't believe the interface is coming back up. The reason I say this is because I have the CRON/script set to restart pfSense when 2 pings to my configured DNS server every 4 minutes is not returned. So I have been having all of these restarts since I created the CRON/script from October 2021. Please see a snippet below of the most recent restarts:
                  So, would you consider my train of thought correct? If so, is there a bug? Just wondering. Thanks.

                  Not sure I'm clear with what you are saying. Did you just check/enable that Live Rule Swap setting in the last day or two after I mentioned it, or have you had that setting always enabled all the way back to last year?

                  I'm highly confident that Suricata does NOT restart when updating rules if that box is checked. And with Suricata not restarting, then there is nothing to cause netmap to cycle the interface.

                  What time is your rules update job set to execute? You can find that on the GLOBAL SETTINGS tab. You will also see the times printed in the Rules Update log that is recorded and viewable on the UPDATES tab. The times you have listed in your log snippets don't all match up with a regularly recurring cron task. My suspicion is you have something else potentially going on causing the connectivity issue and your reboot cron task might also be a little too sensitive. Pinging a DNS server may result in dropped replies if the server gets busy. That's because ICMP is going to be considered much lower priority traffic and will be ignored if the circuit or server gets busy.

                  N 1 Reply Last reply Reply Quote 0
                  • N
                    newUser2pfSense @bmeeks
                    last edited by newUser2pfSense

                    @bmeeks I have had the Live Rule Swap on Update enabled ever since I've been using Suricata which has been several years. This has been occurring for all of that time, but of course I had Suricata set on the WAN during all this time as well. It was also enabled at the time I created this post, even before you mentioned to enable it when I had Suricata enabled on the WAN.

                    Here is the update interval:
                    updateInterval.png

                    Below is a snippet of the rules update log. You can see where it states "Updating rules configuration for...", but it also states "Live-Reload of updated rules requested for..."

                    Starting rules update... Time: 2022-07-21 12:25:15
                    Downloading Emerging Threats Open rules md5 file...
                    Checking Emerging Threats Open rules md5 file...
                    Emerging Threats Open rules are up to date.
                    Downloading Snort VRT rules md5 file...
                    Checking Snort VRT rules md5 file...
                    There is a new set of Snort rules posted.
                    Downloading file 'snortrules-snapshot-29200.tar.gz'...
                    Done downloading rules file.
                    Downloading Snort GPLv2 Community Rules md5 file...
                    Checking Snort GPLv2 Community Rules md5 file...
                    There is a new set of Snort GPLv2 Community Rules posted.
                    Downloading file 'community-rules.tar.gz'...
                    Done downloading rules file.
                    Downloading Feodo Tracker Botnet C2 IP rules file...
                    Done downloading rules file.
                    Feodo Tracker Botnet C2 IP rules are up to date.
                    Downloading ABUSE.ch SSL Blacklist rules file...
                    Done downloading rules file.
                    ABUSE.ch SSL Blacklist rules are up to date.
                    Extracting and installing Snort rules...
                    Installation of Snort rules completed.
                    Extracting and installing Snort GPLv2 Community Rules...
                    Installation of Snort GPLv2 Community Rules completed.
                    Copying new config and map files...
                    Updating rules configuration for: WAN ...
                    Updating rules configuration for: LAN ...
                    Live-Reload of updated rules is enabled...
                    Live-Reload of updated rules requested for LAN.
                    Updating rules configuration for: WLAN ...
                    Live-Reload of updated rules is enabled...
                    Live-Reload of updated rules requested for WLAN.
                    The Rules update has finished. Time: 2022-07-21 12:25:23

                    I don't know that I have any other way of detecting a connectivity drop other than the CRON reboot task.

                    1 Reply Last reply Reply Quote 0
                    • bmeeksB
                      bmeeks
                      last edited by bmeeks

                      Your reboot times from the posted log snippet do not line up as I would expect with the daily rules update jobs (every 6 hours). If Suricata was the root cause here I would expect the reboot times to be pretty much identical within 30 seconds or so. There is a random 30 seconds added to the rules update time to help balance the load on the update servers by not having all the configured Suricata installs out there on pfSense all hammer the server at the same time.

                      Notice in your original post your log shows the gateway alarm and other failures occurring at 04:55 or so, and then the reboot being commanded at 04:56.22. That is quite a long time from 4:25 when your Suricata rules job ran. Rules update jobs typically take only about a minute or two to complete. There is about 30 minutes of normal operation between those two events.

                      Now perhaps you have some Suricata rule triggering that is resulting in blocked pings. Have you investigated that avenue?

                      N 1 Reply Last reply Reply Quote 0
                      • N
                        newUser2pfSense @bmeeks
                        last edited by

                        @bmeeks I did suspect it could be a signature ID that I'm blocking but I didn't have much to go on but their names and what I could search out. The below are the only drops I have entered into SID Mgmt:

                        ET SCAN Sipvicious Scan
                        ET SCAN Sipvicious User-Agent Detected (friendly-scanner)
                        ET SCAN Suspicious inbound to MSSQL port 1433
                        ET SCAN Suspicious inbound to Oracle SQL port 1521
                        SURICATA STREAM 3way handshake SYNACK with wrong ack
                        ET SCAN Suspicious inbound to mSQL port 4333
                        ET COMPROMISED Known Compromised or Hostile Host Traffic group 3
                        ET COMPROMISED Known Compromised or Hostile Host Traffic group 4
                        ET COMPROMISED Known Compromised or Hostile Host Traffic group 5
                        ET COMPROMISED Known Compromised or Hostile Host Traffic group 6
                        ET SCAN Suspicious inbound to mySQL port 3306
                        SURICATA Applayer Detect protocol only one direction
                        SURICATA STREAM Packet with invalid timestamp
                        SURICATA STREAM bad window update
                        ET SCAN Suspicious inbound to PostgreSQL port 5432
                        SURICATA ICMPv4 invalid checksum
                        SURICATA STREAM ESTABLISHED SYNACK resend with different ACK
                        SURICATA STREAM ESTABLISHED SYNACK resend with different seq
                        SURICATA STREAM ESTABLISHED SYN resend
                        SURICATA STREAM FIN out of window
                        SURICATA STREAM TIMEWAIT ACK with wrong seq
                        SURICATA TCP header length too small
                        SURICATA TLS invalid handshake message
                        SURICATA TLS invalid record/traffic
                        SURICATA STREAM CLOSEWAIT FIN out of window

                        I'm wondering if it could be the SURICATA ICMPv4 invalid checksum. This is a head scratcher for me.

                        bmeeksB 1 Reply Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks @newUser2pfSense
                          last edited by bmeeks

                          @newuser2pfsense said in pfSense Plus:

                          @bmeeks I did suspect it could be a signature ID that I'm blocking but I didn't have much to go on but their names and what I could search out. The below are the only drops I have entered into SID Mgmt:

                          ET SCAN Sipvicious Scan
                          ET SCAN Sipvicious User-Agent Detected (friendly-scanner)
                          ET SCAN Suspicious inbound to MSSQL port 1433
                          ET SCAN Suspicious inbound to Oracle SQL port 1521
                          SURICATA STREAM 3way handshake SYNACK with wrong ack
                          ET SCAN Suspicious inbound to mSQL port 4333
                          ET COMPROMISED Known Compromised or Hostile Host Traffic group 3
                          ET COMPROMISED Known Compromised or Hostile Host Traffic group 4
                          ET COMPROMISED Known Compromised or Hostile Host Traffic group 5
                          ET COMPROMISED Known Compromised or Hostile Host Traffic group 6
                          ET SCAN Suspicious inbound to mySQL port 3306
                          SURICATA Applayer Detect protocol only one direction
                          SURICATA STREAM Packet with invalid timestamp
                          SURICATA STREAM bad window update
                          ET SCAN Suspicious inbound to PostgreSQL port 5432
                          SURICATA ICMPv4 invalid checksum
                          SURICATA STREAM ESTABLISHED SYNACK resend with different ACK
                          SURICATA STREAM ESTABLISHED SYNACK resend with different seq
                          SURICATA STREAM ESTABLISHED SYN resend
                          SURICATA STREAM FIN out of window
                          SURICATA STREAM TIMEWAIT ACK with wrong seq
                          SURICATA TCP header length too small
                          SURICATA TLS invalid handshake message
                          SURICATA TLS invalid record/traffic
                          SURICATA STREAM CLOSEWAIT FIN out of window

                          I'm wondering if it could be the SURICATA ICMPv4 invalid checksum. This is a head scratcher for me.

                          Rules that trigger a DROP of traffic will print the alert in red text on the ALERTS tab.

                          @newuser2pfsense said in pfSense Plus:

                          I have been watching the restarts since October of 2021 until now. Hoping for a solution.

                          This statement is puzzling and I can't reconcile it with your statement in the original post at the top of this thread where you say you installed Suricata two days ago and configured it. If you have been having to reboot to restore connectivity since October of 2021, yet you only installed and configured Suricata two days ago, I'm having trouble connecting Suricata directly to your issue. It seems I am misunderstanding something in your timeline of events.

                          N 1 Reply Last reply Reply Quote 0
                          • N
                            newUser2pfSense @bmeeks
                            last edited by newUser2pfSense

                            @bmeeks I'm sorry for the confusion. Yes, I've been using Suricata for several years with the issue we've been discussing - connectivity drops.

                            I wanted to update to pfSense+ from the community edition so a few days ago now I installed a fresh 2.6.0 CE and updated successfully to pfSense+ 22.05. I ran pfSense+ without Suricata for a couple of days with no issues, no connectivity drops at all and no CRON/script restarts. It was working well.

                            I then installed Suricata on pfSense+ 22.05 and configured it on all 3 of the interfaces and began seeing the same issue that I had seen in the past - daily connectivity drops causing my CRON/script to restart pfSense. I was trying to figure out what was going on and I narrowed it down to Suricata. That's when I posted. I now know not to enable Suricata on the WAN which in the last day has worked, at least so far. I hope that makes a little more sense.

                            I do see the dropped alerts in red text on the ALERTS tab.

                            Are there any other reasons why I would be getting the drops even with the Live Rule Swap on Update enabled? Just wondering. I'm now more than just a little curious.

                            bmeeksB 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks @newUser2pfSense
                              last edited by

                              @newuser2pfsense said in pfSense Plus:

                              @bmeeks I'm sorry for the confusion. Yes, I've been using Suricata for several years with the issue we've been discussing - connectivity drops.

                              I wanted to update to pfSense+ from the community edition so a few days ago now I installed a fresh 2.6.0 CE and updated successfully to pfSense+ 22.05. I ran pfSense+ without Suricata for a couple of days with no issues, no connectivity drops at all and no CRON/script restarts. It was working well.

                              I then installed Suricata on pfSense+ 22.05 and configured it on all 3 of the interfaces and began seeing the same issue that I had seen in the past - daily connectivity drops causing my CRON/script to restart pfSense. I was trying to figure out what was going on and I narrowed it down to Suricata. That's when I posted. I now know not to enable Suricata on the WAN which in the last day has worked, at least so far. I hope that makes a little more sense.

                              I do see the dropped alerts in red text on the ALERTS tab.

                              Okay, that helps me understand the timeline better.

                              Those Suricata Stream Events rules such as those beginning with SURICATA STREAM can be somewhat prolific. I would NOT set any of those rules to DROP. If you want to see that traffic, leave them at alert. But honestly, a lot of users don't use those at all as they tend to be somewhat noisy.

                              N 1 Reply Last reply Reply Quote 0
                              • N
                                newUser2pfSense @bmeeks
                                last edited by

                                @bmeeks Are there any other reasons why I would be getting the connectivity drops even with the Live Rule Swap on Update enabled? Just wondering. I'm now more than just a little curious. I'm still not going to enable Suricata on the WAN but I guess I'm just trying to understand better and what could actually be causing it.

                                bmeeksB 1 Reply Last reply Reply Quote 0
                                • bmeeksB
                                  bmeeks @newUser2pfSense
                                  last edited by bmeeks

                                  @newuser2pfsense said in pfSense Plus:

                                  @bmeeks Are there any other reasons why I would be getting the connectivity drops even with the Live Rule Swap on Update enabled? Just wondering. I'm now more than just a little curious. I'm still not going to enable Suricata on the WAN but I guess I'm just trying to understand better and what could actually be causing it.

                                  I see and respond to quite a number of different posts day to day, and I sometimes get the facts from each co-mingled in my brain.

                                  Looking again in detail at your first post, what I do not see in your log snippets is any actual interface down/up events being logged. If netmap was bringing down the interface, that would be noted in the logs. So that proves the Live Rule Swap code is not actually restarting Suricata and thus netmap is not cycling the interface.

                                  Not all hardware NICs are exactly the same. They can have different firmware versions and thus interact differently with the generic FreeBSD NIC driver for that family. It is not out of the realm of possibility that your particular NIC is not behaving well when in netmap mode. Do you have the same type of NIC on all interfaces, or do they differ? Is your WAN NIC perhaps a "notorious" Realtek device? Those can be really squirrelly on FreeBSD.

                                  N 1 Reply Last reply Reply Quote 0
                                  • N
                                    newUser2pfSense @bmeeks
                                    last edited by newUser2pfSense

                                    @bmeeks I have 2, 4 port, GbE, SuperMicro nics installed; same models - SUPERMICRO AOC-SGP-I4. I believe they are both Intel based. My WAN is on one nic by itself and the LAN and WLAN are on the other nic.

                                    bmeeksB 1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks @newUser2pfSense
                                      last edited by

                                      @newuser2pfsense said in pfSense Plus:

                                      @bmeeks I have 2, 4 port, GbE, SuperMicro nics installed; same models - SUPERMICRO AOC-SGP-I4. I believe they are both Intel based. My WAN is on one nic by itself and the LAN and WLAN are on the other nic.

                                      That card should be using em driver as best I can determine. There are at least two open FreeBSD bugs against that chipset (not Supermicro itself but the underlying Intel chip on the NIC).

                                      This one seems similar to what you are seeing, but not exactly the same: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235031.

                                      Do you have all of the hardware offloading disabled for the NICs?

                                      The only connection to Suricata I can see would be the use of netmap for Inline IPS operation. You could try swapping over to Legacy Blocking Mode for a test and enable the option for "Block DROPS Only". That would use your existing SID MGMT configuration just without netmap and Inline IPS Mode. If things are stable there, then you have your answer -- it would be a netmap compatibility issue with your NIC.

                                      N 1 Reply Last reply Reply Quote 0
                                      • N
                                        newUser2pfSense @bmeeks
                                        last edited by newUser2pfSense

                                        @bmeeks pfSense is showing me it's using igb (igb0, igb1, igb7).

                                        Here is the offloading:
                                        offloading.png

                                        Is there a specific Intel based NIC card that you would recommend that doesn't have any issues with pfSense? Just wondering.

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.