Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    ADI/NetGate firewalls become unresponsive on 2.3/2.3.1 with high ipSec Traffic

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    8 Posts 4 Posters 1.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      networkninja
      last edited by

      Hello,

      We have 2 ADI NetGate firewalls (one 8860 and one 2220) which are interconnected via an ipSec tunnel. We're seeing that both firewalls become unresponsive and require a hard reset (soft reboot hangs) when pushing close to 100Mbit of ipSec traffic. We run our nightly backups over this vpn tunnel and have been for many months, but only after updating to 2.3 did we start having this problem. There is nothing in the logs, it seems as if the firewall thinks everything is fine, but it won't respond to ping and it seems it randomly loses ethernet interfaces. At this point if you try to reboot from the console it will hang until power is physically unplugged.

      I've had to revert both firewalls to 2.2.6, but I wanted to put this out there because I'm not finding anyone else who is having this problem. If there's anything I can do to help test I will see what I can do.

      Has anyone else seen this?

      Thanks

      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        Are you Sure it is not fixed in 2.3.1?

        1 Reply Last reply Reply Quote 0
        • N
          networkninja
          last edited by

          @heper:

          Are you Sure it is not fixed in 2.3.1?

          Definitely, had the 2220 firewall crash last night running on 2.3.1

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            When a system sees the 2.3.0 SMP+IPsec issue described here https://redmine.pfsense.org/issues/6296 it will show 100% interrupt CPU on one core. This can be seen in Status > Monitoring.

            It can also be worked around by disabling all but one core as described here:

            https://forum.pfsense.org/index.php?topic=110710.msg618388#msg618388

            If neither of these conditions fit it is likely you are seeing something different.

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • N
              networkninja
              last edited by

              @Derelict:

              When a system sees the 2.3.0 SMP+IPsec issue described here https://redmine.pfsense.org/issues/6296 it will show 100% interrupt CPU on one core. This can be seen in Status > Monitoring.

              It can also be worked around by disabling all but one core as described here:

              https://forum.pfsense.org/index.php?topic=110710.msg618388#msg618388

              If neither of these conditions fit it is likely you are seeing something different.

              Thanks for the info. Our main office has a lot of users and reducing from 8 cores to one is not an option. I will keep an eye on this issue and we will try to upgrade again at 2.3.2 or when I see on here that the issue is fully resolved.

              Thanks!

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                What would be really important to verify is the 100% interrupt CPU on a node running 2.3.1 before you reboot.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • C
                  cmb
                  last edited by

                  The issue in #6296 with SMP and IPsec is 100% confirmed fixed by many different people in many circumstances. Any issues with 2.3.1 would be something different, and needs troubleshooting. No one else has reported any such issues on 2.3.1 (note that's not 2.3_1, 2.3.1 from this week). I wouldn't advise disabling cores for any reason in 2.3.1 at this point.

                  Ideally a dump from status.php if you can get to the GUI at the time would help. If not that, seeing the output of 'top -SH' might be at least somewhat telling.

                  1 Reply Last reply Reply Quote 0
                  • N
                    networkninja
                    last edited by

                    @cmb:

                    The issue in #6296 with SMP and IPsec is 100% confirmed fixed by many different people in many circumstances. Any issues with 2.3.1 would be something different, and needs troubleshooting. No one else has reported any such issues on 2.3.1 (note that's not 2.3_1, 2.3.1 from this week). I wouldn't advise disabling cores for any reason in 2.3.1 at this point.

                    Ideally a dump from status.php if you can get to the GUI at the time would help. If not that, seeing the output of 'top -SH' might be at least somewhat telling.

                    cmb, Thanks for the info. I will see what I can do but these are production systems with little tolerance for downtime. If I get a chance to I will try to test again. status.php is pretty cool, I had no idea this existed.

                    Thanks

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.