CARP failover kills states when OPENVPN is active

TheCyborgWeasel

pfSense CE version 2.8.1

Hi guys,

this now took me very long to find out, please if anybody could confirm this behaviour would be great (or even have a solution?)

I have 2 pfSense in an HA connection. Everything works great, failover works wonderful. Now since I've activated OPENVPN, I have the problem that states in my network between different VLANs are killed during a failover. If I disable OpenVPN, everything is fine again. I narrowed it down to the following:

Primary = Master, starting SSH between PC1 192.168.200.100 and PC2 10.0.30.20. State (including NAT, but to simplify I skip that part) is created on Port 50000 (example port) to Port 22. I checked that state on the primary and on the secondary as well: State is synchronized and available on both pfsenses. If now OpenVPN server is active and I put the primary pfSense into CARP-Maintenance Mode, I can see that the state with port 50000 still exists on the primary pfsense (which is now in backup mode) but on the secondary pfSense, which is now in master mode, the state is gone! As a consequence my ssh connection is broken. If I now reconnect my ssh connection, a new state on the secondary (master) pfsense is created, lets assume it is 192.168.200.100 Port 60000. I can verify that this state is also transferred (sync) to the primary pfsense (=still backup mode). If I now stop the maintenance mode on the primary pfSense, so that this one becomes master again, the state with port 60000 vanishes on that device. But the state on port 50000 is still available. If I now have a look at the states of the secondary (=backup now) pfsense, the state with port 60000 is still available.

In short:
primary = master, ssh state 50000 created
secondary = backup, ssh state 50000 synced

failover

primary = backup, ssh state 50000 still available
secondary = master, no state available (ssh state 50000 vanished)

new login -> new state

primary = backup, ssh state 50000 still available, ssh state 60000 synced
secondary = master, ssh state 60000 created

return to normal again

primary = master, ssh state 50000 still available ONLY (ssh state 60000 vanished)
secondary = backup, ssh state 60000 still available

In my opinion this is a bug. I asked AI for checking the source code on that behaviour and I got the following answer: When after a role transfer (master -> backup) the OpenVPN service is shut down, a pfsync-CLR packet is sent to the new master device. It has the creator-ID and the interface name of the openvpn server interface ("ovpnX"). So on the new master pfsense, all the states on the "ovpnX" interface with the creator-ID should be cleared. But upon receiving this pfsync-CLR packet the interface ID "ovpnX" is not processed. Only the creator-id is used to kill the states which also breaks my ssh connection.

Source:

1. pfSense OpenVPN link-down hook

    File:
        /usr/local/sbin/ovpn-linkdown

    Lines 25-30:

    25  if [ -n "${1}" ]; then
    26      /usr/bin/logger -p user.notice -t openvpn \
                "NOTICE [${LOG_PREFIX_OPENVPN}] Flushing states on OpenVPN interface ${1} (Link Down)"
    27      /sbin/pfctl -i "${1}" -Fs
    28  else
    29      /usr/bin/logger -p user.warning -t openvpn \
                "WARNING [${LOG_PREFIX_OPENVPN}] ovpn-linkdown called without interface"
    30  fi

2. FreeBSD PF sends a pfsync clear request with host ID and interface name

File:
    sys/netpfil/pf/pf_ioctl.c

Lines 7231-7234:

7231  if (V_pfsync_clear_states_ptr != NULL)
7232      V_pfsync_clear_states_ptr(V_pf_status.hostid, kill->psk_ifname);
7233
7234  return (killed);

3. FreeBSD pfsync receives the clear request but deletes states by creatorid only

File:
    sys/netpfil/pf/if_pfsync.c

Function:
    pfsync_in_clr()

Lines 1136-1175:

1136  pfsync_in_clr(struct mbuf *m, int offset, int count, int flags, int action)
1137  {
1138      struct pfsync_clr *clr;
1139      struct mbuf *mp;
1140      int len = sizeof(*clr) * count;
1141      int i, offp;
1142      u_int32_t creatorid;
1143
1144      mp = m_pulldown(m, offset, len, &offp);
1145      if (mp == NULL) {
1146          V_pfsyncstats.pfsyncs_badlen++;
1147          return (-1);
1148      }
1149      clr = (struct pfsync_clr *)(mp->m_data + offp);
1150
1151      for (i = 0; i < count; i++) {
1152          creatorid = clr[i].creatorid;
1153
1154          if (clr[i].ifname[0] != '\0' &&
1155              pfi_kkif_find(clr[i].ifname) == NULL)
1156              continue;
1157
1158          for (int i = 0; i <= V_pf_hashmask; i++) {
1159              struct pf_idhash *ih = &V_pf_idhash[i];
1160              struct pf_kstate *s;
1161  relock:
1162              PF_HASHROW_LOCK(ih);
1163              LIST_FOREACH(s, &ih->states, entry) {
1164                  if (s->creatorid == creatorid) {
1165                      s->state_flags |= PFSTATE_NOSYNC;
1166                      pf_remove_state(s);
1167                      goto relock;
1168                  }
1169              }
1170              PF_HASHROW_UNLOCK(ih);
1171          }
1172      }
1173
1174      return (len);
1175  }

Problem explained:

Lines 1154-1156 only verify whether the received interface name exists on the receiving firewall:

1154  if (clr[i].ifname[0] != '\0' &&
1155      pfi_kkif_find(clr[i].ifname) == NULL)
1156      continue;

There is no subsequent comparison between clr[i].ifname and the interface associated with s.
Instead, the deletion condition at line 1164 is:

1164  if (s->creatorid == creatorid) {

Every state with that creatorid is marked PFSTATE_NOSYNC, removed locally, and the hash row is scanned again:

1165  s->state_flags |= PFSTATE_NOSYNC;
1166  pf_remove_state(s);
1167  goto relock;

Therefore, the interface name carried in the PFSYNC_ACT_CLR message is used as an existence check only, not as a state-selection criterion.

I'm not able to verify this information, can someone do? Or at least reproduce my issue?
Thanks so far,

regards!

marcosm

Fixed upstream:
https://gitlab.com/FreeBSD/freebsd-src/-/commit/6a34abbe976626019d61d9d08a624f4ed14ddbd0
Redmine:
https://redmine.pfsense.org/issues/13569