pfSync Nodes list mostly empty?

[teamits]

We've been using pfSense and CARP for several years without issue. We have 5 CARP IPs.

I noticed today that under Status/CARP our "pfSync nodes" list has only 5-7 entries in it and mostly doesn't change. Am I mistaken that list should be ever changing and dependent on the number of connections/states? I could have sworn it was a much longer list in the past. Did something change or is the state sync not working?

The HA sync seems to be working just fine as I can manually run a sync by saving and no errors are logged on either router that I can see. Router1 can ping router2's pfsync interface IP.

Thanks,

[Derelict]

That's probably OK.

The state table should be about as full on the secondary as the primary. They never match exactly.

If that is the case and XMLRPC (config) sync is working and there are not error messages logged relating to communications between primary and secondary you are probably good to go.

[teamits]

Hi Derelict, I figured they would be similar and not quite match...have seen that in the past. My question was just that it seemed odd to see a half dozen entries instead of 50 or 100 or whatever, given there are ~25 web and mail servers behind the routers. I know I've seen longer lists in the past but possibly on different "hardware"...we used to run these as VMs and I could tell something wasn't right if the lists were dramatically different lengths.

We do have these set up in a LAGG on both sides. Could that be messing with the states in general?

[Derelict]

I something not working? Are the state tables on both nodes extremely off from each other?

[teamits]

The list is the same but the entire list is:

pfSync nodes:
2ce084c9
39dde4b0
7c91dd7e
df4ffcef
f11ea708

I went and looked at another HA setup and their list isn't that long either but it has about 16 nodes.

Let me ask this differently...what is a "pfSync node" and what determines how many there are?

[actualrootwyrm]

pfSync nodes directly correlate to the direct attachments locally and the visible pfsync(4) devices on the network and associated. If you have other pfsync(4) devices in your network which are visible to the device pfsync(4) is associated to on the pfSense (e.g. using VLAN10 for CARP+HA, plus FreeBSD or OpenBSD hosts also running pfsync(4)) then pfSense WILL report them as eligible targets. This means that generally your pfsync nodes will be equal to your non-WAN ethernet interfaces * number of hosts. So: 2 hosts, 3 non-WAN ethernet interfaces each, using multicast will generally be 7 nodes (6 + localhost).
Whereas using a directed IP interface, you generally should only see one node.

This is less a limitation and more simply the design of pf(4), pfsync(4), and carp(4). By default pfsync(4) is multicast, so it will pick up all compatible interfaces in the network. So for 1+1 configurations it is generally best to explicitly define pfsync peers. And for N-way or in networks with significant FreeBSD or OpenBSD hosts, it is best to use an interface with L2 isolation from the hosts.

[teamits]

Hmmm, interesting. I based my question on when we had these in VMs and had a VLAN set up for the sync interfaces. Sounds like there was some sort of a problem back then if the list was significantly longer (I want to say a couple dozen). Or my memory is significantly bad. :)

In hindsight, using VMs saved us money in startup costs and was cool to do, but I wish we'd gotten the SG-4860s up front...less hassles over time.