pfSync Nodes list mostly empty?

SteveITS

We've been using pfSense and CARP for several years without issue. We have 5 CARP IPs.

I noticed today that under Status/CARP our "pfSync nodes" list has only 5-7 entries in it and mostly doesn't change. Am I mistaken that list should be ever changing and dependent on the number of connections/states? I could have sworn it was a much longer list in the past. Did something change or is the state sync not working?

The HA sync seems to be working just fine as I can manually run a sync by saving and no errors are logged on either router that I can see. Router1 can ping router2's pfsync interface IP.

Thanks,

Derelict

That's probably OK.

The state table should be about as full on the secondary as the primary. They never match exactly.

If that is the case and XMLRPC (config) sync is working and there are not error messages logged relating to communications between primary and secondary you are probably good to go.

SteveITS

Hi Derelict, I figured they would be similar and not quite match...have seen that in the past. My question was just that it seemed odd to see a half dozen entries instead of 50 or 100 or whatever, given there are ~25 web and mail servers behind the routers. I know I've seen longer lists in the past but possibly on different "hardware"...we used to run these as VMs and I could tell something wasn't right if the lists were dramatically different lengths.

We do have these set up in a LAGG on both sides. Could that be messing with the states in general?

Derelict

I something not working? Are the state tables on both nodes extremely off from each other?

SteveITS

The list is the same but the entire list is:

pfSync nodes:
2ce084c9
39dde4b0
7c91dd7e
df4ffcef
f11ea708

I went and looked at another HA setup and their list isn't that long either but it has about 16 nodes.

Let me ask this differently...what is a "pfSync node" and what determines how many there are?

actualrootwyrm

pfSync nodes directly correlate to the direct attachments locally and the visible pfsync(4) devices on the network and associated. If you have other pfsync(4) devices in your network which are visible to the device pfsync(4) is associated to on the pfSense (e.g. using VLAN10 for CARP+HA, plus FreeBSD or OpenBSD hosts also running pfsync(4)) then pfSense WILL report them as eligible targets. This means that generally your pfsync nodes will be equal to your non-WAN ethernet interfaces * number of hosts. So: 2 hosts, 3 non-WAN ethernet interfaces each, using multicast will generally be 7 nodes (6 + localhost).
Whereas using a directed IP interface, you generally should only see one node.

This is less a limitation and more simply the design of pf(4), pfsync(4), and carp(4). By default pfsync(4) is multicast, so it will pick up all compatible interfaces in the network. So for 1+1 configurations it is generally best to explicitly define pfsync peers. And for N-way or in networks with significant FreeBSD or OpenBSD hosts, it is best to use an interface with L2 isolation from the hosts.

SteveITS

Hmmm, interesting. I based my question on when we had these in VMs and had a VLAN set up for the sync interfaces. Sounds like there was some sort of a problem back then if the list was significantly longer (I want to say a couple dozen). Or my memory is significantly bad. :)

In hindsight, using VMs saved us money in startup costs and was cool to do, but I wish we'd gotten the SG-4860s up front...less hassles over time.

SteveITS

@actualrootwyrm said in pfSync Nodes list mostly empty?:

generally your pfsync nodes will be equal to your non-WAN ethernet interfaces * number of hosts. So: 2 hosts, 3 non-WAN ethernet interfaces each, using multicast will generally be 7 nodes (6 + localhost).

I was updating a client's routers today and ran across this again. Two SG-3100s. Four CARP IPs, LAN and 3 WAN. I upgraded the backup yesterday. Before upgrading the master today both showed about 8 pfSync nodes (I didn't count). Immediately after upgrade both show an almost identical list of about 30:

CARP Interfaces
LAN@151 192.168.1.1/24 MASTER
WAN@152 96.x.x.241/29 MASTER
WAN@153 96.x.x.242/29 MASTER
WAN@154 96.x.x.243/24 MASTER

pfSync nodes:

08f3e9d9
0a640481
10bb43ee
127140d4
14bae82b
1843a9f0
20487389
31f68ff6
4586388e
47833479
4b244e79
585ed47b
5aa116fc
80268636
862f178c
86621af0
99bf321e
9cd740e6
a4a58c60
adf52dac
c008ee1b
c3fa20b9
cac5f44f
cf73d217
d2ffd819
e42d44cd
ed167296
f15cf656
f3d9d275
f97ce9a5

Derelict

All that matters is that they are syncing and are mostly identical.