Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CARP gets corrupted when state sync is enabled

    Scheduled Pinned Locked Moved
    HA/CARP/VIPs
    2
    5
    1.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      Dule
      last edited by

      Hi all,

      Two months ago I did upgrade from 2.4.5 to 2.5.2. on my backup firewall. While I was testing functionality of all features and firewall itself, sync (state and config) between active and backup was turned off. After I was sure that my backup firewall is stable I did upgrade of my active firewall as well. Last step in my upgrade was to enable sync, and new problem escalated.

      When I turned on state sync my CARP got corrupted on active firewall. It become backup and my backup firewall CARP status was unknown. When I disable CARP on backup, active takes all the traffic back again. While I was troubleshooting this active firewall crashed and here is something interesting from my crash report

      <6>pid 37150 (snort), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 37464 (snort), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 38328 (snort), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 41361 (snort), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 53146 (snort), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 67700 (snort), jid 0, uid 0: exited on signal 11 (core dumped)
<6>carp: 3@ixl0: BACKUP -> MASTER (preempting a slower master)
<6>carp: 6@ixl1: BACKUP -> MASTER (preempting a slower master)
<6>carp: 1@ixl3: BACKUP -> MASTER (preempting a slower master)
<6>carp: 2@ixl2: BACKUP -> MASTER (preempting a slower master)
<6>carp: 5@bge0: BACKUP -> MASTER (preempting a slower master)
<6>carp: 4@bge0.1900: BACKUP -> MASTER (preempting a slower master)
<6>ovpns6: link state changed to UP
<6>ovpns7: link state changed to UP
<6>ovpns8: link state changed to UP
<6>carp: 1@ixl3: MASTER -> INIT (hardware interface up)
<6>carp: 1@ixl3: INIT -> BACKUP (initialization complete)
<6>carp: 2@ixl2: MASTER -> INIT (hardware interface up)
<6>carp: 2@ixl2: INIT -> BACKUP (initialization complete)
<6>carp: 3@ixl0: MASTER -> INIT (hardware interface up)
<6>carp: 3@ixl0: INIT -> BACKUP (initialization complete)
<6>carp: 4@bge0.1900: MASTER -> INIT (hardware interface up)
<6>carp: 4@bge0.1900: INIT -> BACKUP (initialization complete)
<6>carp: 5@bge0: MASTER -> INIT (hardware interface up)
<6>carp: 5@bge0: INIT -> BACKUP (initialization complete)
<6>carp: 6@ixl1: MASTER -> INIT (hardware interface up)
<6>carp: 6@ixl1: INIT -> BACKUP (initialization complete)
<6>carp: 1@ixl3: BACKUP -> INIT (hardware interface up)
<6>carp: 2@ixl2: BACKUP -> INIT (hardware interface up)
<6>carp: 3@ixl0: BACKUP -> INIT (hardware interface up)
<6>carp: 4@bge0.1900: BACKUP -> INIT (hardware interface up)
<6>carp: 5@bge0: BACKUP -> INIT (hardware interface up)
<6>carp: 6@ixl1: BACKUP -> INIT (hardware interface up)
<6>ovpns9: link state changed to UP
<6>ovpns9: link state changed to DOWN
<6>ovpns4: link state changed to UP
<6>ovpns4: link state changed to DOWN
<6>ovpns6: link state changed to DOWN
<6>ovpns6: link state changed to UP
<6>ovpns6: link state changed to DOWN
<6>ovpns7: link state changed to DOWN
<6>ovpns6: link state changed to UP
<6>ovpns6: link state changed to DOWN
<6>ovpns7: link state changed to UP
<6>ovpns7: link state changed to DOWN
<6>ovpns8: link state changed to DOWN
<6>ovpns7: link state changed to UP
<6>ovpns7: link state changed to DOWN
<6>ovpns6: link state changed to UP
<6>ovpns6: link state changed to DOWN
<6>ovpns7: link state changed to UP
<6>ovpns7: link state changed to DOWN
<6>ovpns8: link state changed to UP
<6>ovpns8: link state changed to DOWN
<6>ovpns8: link state changed to UP
<6>ovpns9: link state changed to UP
<6>ovpns9: link state changed to DOWN
<6>ovpns8: link state changed to DOWN
<6>ovpns3: link state changed to DOWN
<6>ovpns8: link state changed to UP
<6>ovpns9: link state changed to UP
<6>ovpns9: link state changed to DOWN
<6>ovpns8: link state changed to DOWN
<6>ovpns9: link state changed to UP
<6>ovpns4: link state changed to UP
<6>ovpns9: link state changed to DOWN
<6>ovpns4: link state changed to DOWN
<6>ovpns3: link state changed to UP
<6>ovpns4: link state changed to UP
<6>ovpns4: link state changed to DOWN
<6>ovpns6: link state changed to UP
<6>ovpns6: link state changed to DOWN
<6>ovpns4: link state changed to UP
<6>ovpns4: link state changed to DOWN
<6>ovpns7: link state changed to UP
<6>ovpns7: link state changed to DOWN
<6>ovpns5: link state changed to DOWN
<6>ovpns8: link state changed to UP
<6>ovpns8: link state changed to DOWN
<6>ovpns9: link state changed to UP
<6>ovpns9: link state changed to DOWN
<6>ovpns5: link state changed to UP
<6>ovpns4: link state changed to UP
<6>ovpns4: link state changed to DOWN
<6>carp: demoted by 240 to 240 (interface down)
<6>carp: demoted by 240 to 480 (interface down)
<6>carp: demoted by 240 to 720 (interface down)
<6>carp: demoted by 240 to 960 (interface down)
<6>carp: demoted by 240 to 1200 (interface down)
<6>carp: demoted by 240 to 1440 (interface down)
<6>carp: 6@ixl1: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1200 (interface up)
<6>carp: 5@bge0: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 960 (interface up)
<6>carp: 4@bge0.1900: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 720 (interface up)
<6>carp: 3@ixl0: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 480 (interface up)
<6>carp: 2@ixl2: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 240 (interface up)
<6>carp: 1@ixl3: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 0 (interface up)
mfi0: 886 (687668400s/0x0020/info) - Patrol Read started
mfi0: 887 (687670441s/0x0020/info) - Patrol Read complete
<6>carp: 6@ixl1: BACKUP -> MASTER (preempting a slower master)
<6>carp: 5@bge0: BACKUP -> MASTER (master timed out)
<6>carp: 4@bge0.1900: BACKUP -> MASTER (master timed out)
<6>carp: 2@ixl2: BACKUP -> MASTER (master timed out)
<6>carp: 1@ixl3: BACKUP -> MASTER (master timed out)
<6>carp: 3@ixl0: BACKUP -> MASTER (master timed out)
<6>ovpns6: link state changed to UP
<6>ovpns7: link state changed to UP
<6>ovpns8: link state changed to UP
<6>ovpns9: link state changed to UP
<6>ovpns4: link state changed to UP
<6>carp: 4@bge0.1900: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 5@bge0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 6@ixl1: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 3@ixl0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 1@ixl3: MASTER -> BACKUP (more frequent advertisement received)
<6>ovpns6: link state changed to DOWN
<6>ovpns7: link state changed to DOWN
<6>ovpns8: link state changed to DOWN
<6>ovpns9: link state changed to DOWN
<6>carp: 2@ixl2: MASTER -> BACKUP (more frequent advertisement received)
<6>ovpns4: link state changed to DOWN
<6>carp: 1@ixl3: BACKUP -> MASTER (preempting a slower master)
<6>carp: 4@bge0.1900: BACKUP -> MASTER (preempting a slower master)
<6>carp: 5@bge0: BACKUP -> MASTER (preempting a slower master)
<6>carp: 6@ixl1: BACKUP -> MASTER (preempting a slower master)
<6>carp: 3@ixl0: BACKUP -> MASTER (preempting a slower master)
<6>carp: 2@ixl2: BACKUP -> MASTER (preempting a slower master)
<6>ovpns6: link state changed to UP
<6>ovpns7: link state changed to UP
<6>ovpns8: link state changed to UP
<6>ovpns9: link state changed to UP
<6>ovpns4: link state changed to UP
<6>carp: demoted by 240 to 240 (pfsync bulk start)
<6>carp: 6@ixl1: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 3@ixl0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 5@bge0: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 4@bge0.1900: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 1@ixl3: MASTER -> BACKUP (more frequent advertisement received)
<6>carp: 2@ixl2: MASTER -> BACKUP (more frequent advertisement received)
<6>ovpns6: link state changed to DOWN
<6>ovpns7: link state changed to DOWN
<6>ovpns8: link state changed to DOWN
<6>ovpns9: link state changed to DOWN
<6>ovpns4: link state changed to DOWN
<6>carp: demoted by -240 to 0 (sysctl)
<6>carp: 3@ixl0: BACKUP -> MASTER (preempting a slower master)
<6>carp: 6@ixl1: BACKUP -> MASTER (preempting a slower master)
<6>carp: 1@ixl3: BACKUP -> MASTER (preempting a slower master)
<6>carp: 2@ixl2: BACKUP -> MASTER (preempting a slower master)
<6>carp: 5@bge0: BACKUP -> MASTER (preempting a slower master)
<6>carp: 4@bge0.1900: BACKUP -> MASTER (preempting a slower master)
<6>carp: demoted by -240 to -240 (pfsync bulk fail)
<6>ovpns6: link state changed to UP
<6>ovpns7: link state changed to UP
<6>ovpns8: link state changed to UP
<6>ovpns9: link state changed to UP
<6>ovpns4: link state changed to UP
<6>carp: demoted by 240 to 0 (sysctl)


Fatal trap 12: page fault while in kernel mode

      After reboot I start assuming that problem was caused by the Advanced config. I've disabled Hardware Checksum in Advanced > Networking tab. This was request for Snort Inline mode. I revert that config and only change is that I can sync config now but still my CARP gets corrupted instantly when I turn on state sync.

      I'm trying to figure this out for more than a week and still no success. Network cards are the same, same driver, firmware installed. Everything worked on previous version.

      Anyone have any idea what can be the cause of this?

      Thanks!

      S 1 Reply Last reply Reply Quote 0
      • S
        SteveITS Rebel Alliance @Dule
        last edited by

        @dule Are they still different versions? The config sync and state sync isn't guaranteed to work across versions, as I recall.

        Since Snort is also crashing does it work if you stop Snort?

        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
        Upvote 👍 helpful posts!

        D 1 Reply Last reply Reply Quote 0
        • D
          Dule @SteveITS
          last edited by

          @SteveITS No, they are on the same version. Yes, I know that sync should work only on the same FreeBSD versions. Snort is not working on my SYNC interface, it works on my WAN and LAN interfaces. I'm not sure how it can be a trigger in this case?

          S 1 Reply Last reply Reply Quote 0
          • S
            SteveITS Rebel Alliance @Dule
            last edited by

            @dule I wasn't necessarily saying Snort was causing the problem, just noting that it crashed.

            Do you need it on both LAN and WAN? That will inspect traffic twice, and the WAN one will inspect all packets that would have been blocked by the firewall rules anyway.

            Do you have any other packages installed?

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote 👍 helpful posts!

            D 1 Reply Last reply Reply Quote 0
            • D
              Dule @SteveITS
              last edited by

              @steveits Yes I need Snort on both interfaces, I like to know for example who from my family wants to download torrents.
              I have Zabbix and FreeRADIUS package install along with Snort.

              1 Reply Last reply Reply Quote 0
              • First post
                Last post