Auto configuration backup shows no backups

[rahvin9999]

We have 2 pfSense machines which show no backups under Auto Configuration Backup.

Have been through the forums reading up on other posts describing the same issue.
But have not found a solution.

Basic info:

• The machines are both on version 2.5.2-RELEASE (amd64)
• Running in a HA carp setup with config sync enabled from the Primary machine to the Secondary.
• Single WAN.
• DNS and everything else works.

Actions tried:

• Cleared the Auto Config Backup settings and configured it from scratch.
• Have tried generating new SSH keys to change device key. Thinking the key might be banned/blocked for some reason.
  Regenerating SSH keys did indeed change the device key but still no backups showing after trying a manual backup.
  (reverted back to original ssh keys after that)
• Tried backup on config change instead of on a schedule but results are the same... no backups showing up.
• If I input the device key from a different pfSense machine. I can see the backups of that machine show up.

Overview of settings and logging:

• Auto Config Backup is enabled on a schedule, daily after midnight with the minute chosen at random .
• There is a device key listed under the "restore" and "backup now" tabs.
• This key is diferent for each machine.
• If I do a manual backup I get the "Backup completed successfully." message.
• Logging shows the following on the Primary machine.

Sep 23 12:34:00	php-fpm	66967	/rc.filter_synchronize: Beginning XMLRPC sync data to https://x.x.x.x:443/xmlrpc.php.
Sep 23 12:34:00	php-fpm	66967	/rc.filter_synchronize: XMLRPC versioncheck: 21.7 -- 21.7
Sep 23 12:34:00	php-fpm	66967	/rc.filter_synchronize: XMLRPC reload data success with https://x.x.x.x:443/xmlrpc.php (pfsense.host_firmware_version).
Sep 23 12:34:00	php-fpm	75926	/services_acb_backup.php: End of configuration backup to https://acb.netgate.com/save (success).
Sep 23 12:34:00	php-fpm	66967	/rc.filter_synchronize: Beginning XMLRPC sync data to https://x.x.x.x:443/xmlrpc.php.
Sep 23 12:33:58	php-fpm	75926	/services_acb_backup.php: Beginning configuration backup to https://acb.netgate.com/save
Sep 23 12:33:58	check_reload_status	381	Syncing firewall

• Logging shows the following on the Secondary machine

Sep 23 12:39:28	php-fpm	30341	/services_acb_backup.php: End of configuration backup to https://acb.netgate.com/save (success).
Sep 23 12:39:26	php-fpm	30341	/services_acb_backup.php: Beginning configuration backup to https://acb.netgate.com/save

Is this because of a size limit for the backups? Read about a simmilar issue in a forum post. But this was a post from a few years ago and it seemed the solution then was that the admins upped the backup size limit.
Our config.xml files are 16.748KB without RRD data because we have quite a few users and certificates on these pfSense machines.

Anyone that has similar issues?

[Steve_B]

The file size limits are very much greater than your config file. Tens of megabytes, and we do not block any userkeys, NDIs or users so this is mysterious.

If I understand your post correctly if you copy the userkey from the device that does not display backups and paste it into another device you see the backups appropriate to the first (failing) device?

If that is so it means that both saving and retrieving backups is working correctly on the server side.

Is the device that does display the backups (having pasted the userkey from the failing device) part of an HA pair or an isolated device?

[rahvin9999]

@steve_b

Sorry for the confusion it is the other way around.

Lets simplify and call the firewalls pfSense 1,2 and 3

• pfSense1 - This is the Primary for the HA Cluster
• pfSense2 - This is the Secondary for the HA Cluster
• pfSense3 - This is a firewall which is stand alone and servicing other clients (same WAN subnet though)

pfSense1 and pfSense2 have the issue "No backups could be located for this device."
pfSense3 shows backups.

If I enter the device key of either pfSense1 or pfSense2 into pfSense3 I get the "No backups could be located for this device." message.

If I enter the device key of pfSense3 into pfSense1 or pfSense2 it shows me the backups of pfSense3

Hope this clarifies

[Steve_B]

OK thanks. Then it sounds like the backups are not making to the server then. Would you please DM me the userkey of either firewall 1 or 2 so that I can check the logs? (Your backups are encrypted so I will NOT be able to see your config(s).

[Steve_B]

@rahvin9999
You might also try: "ping acb.netgate.com" from the command line.

[rahvin9999]

@steve_b

Resolves and succesfully pings to 208.123.73.78 from pfSense1, pfSense2 and pfSense3

64 bytes from 208.123.73.78: icmp_seq=0 ttl=45 time=122.599 ms
64 bytes from 208.123.73.78: icmp_seq=1 ttl=45 time=122.372 ms
64 bytes from 208.123.73.78: icmp_seq=2 ttl=45 time=122.598 ms
64 bytes from 208.123.73.78: icmp_seq=3 ttl=45 time=122.429 ms

[Steve_B]

@rahvin9999 The logs show a few instances of you attempting to list backups for firewall 1 (presumably when testing from another firewall) but no backups being saved.

It is possible this could be due to an error in your HA configs but that is just a guess. We are discussing internally.

[stephenw10]

Yes, a common config mistake here is to end up with HA nodes NATing their own traffic whifh can cause a problem. However I don't think that can be the problem here because each node is able to ping the acb server and can list the backups from the third firewall.

@rahvin9999 said in Auto configuration backup shows no backups:

Our config.xml files are 16.748KB

Is that nearly 17MB? That would be a really very large config, we might need to test that.

Steve

[rahvin9999]

@steve_b

From my perspective the HA config works without issues.

High Availability Sync settings on pfSense1

• pfsync is configured to sync states over a dedicated sync interfaced targeted to the IP of pfSense2
• XMLRPC Sync is configured to sync to pfSense2 on the same ip as pfsync
• XMLRPC Sync uses the admin user and everything is ticked to be synced except Synchronize admin
• we have HAProxy installed and this is set to use XMLRPC Sync via the HAProxy settings page setting "HAProxy Sync"

High Availability Sync settings on pfSense2

• pfsync is configured to sync states over a dedicated sync interfaced targeted to the IP of pfSense2
• the XMLRPC Sync section is left empty

We have 3 Carp VIP's

• WAN has one Carp VIP in a /29 subnet
• LAN has one Carp VIP in a /24 subnet
• DMZ has one Carp VIP in a /24 subnet

The config is succesfully synced from pfSense1 to pfSense2
If on pfSense1 I:

• Disable CARP (or)
• Enable CARP Persistent maintenance mode (or)
• Pull the powercable from pfSense1
  pfSense2 takes over everything. We have a lot of OpenVPN and IPSec clients. Who succesfully reconnect to pfSense2.
  If I boot/reenable pfSense1 everything goes back to pfSense1

[rahvin9999]

@stephenw10

Yes, that is nearly 17MB.

We currently have 3000+ Users, Certificates and OpenVPN Client Specific Overrides in the config.

Only issue that many users and certificates gives is, that some pages take some time to load.
That and rebooting takes some time as it is stuck at "Synchronizing users settings" for a while.
But that is due to the fact that the xeon D cpu used has horrible single thread performance. If I load this config on a server with a faster CPU it loads multiple factors faster.

<edit: Spelling and grammar>

[stephenw10]

Ok, that is very large. We are checking on that now but I suspect you are hitting a config size limit.

I would usually recommend using external authentication by the time you're approaching that many users. There are certainly parts of the gui that will struggle with that number of entries.

Steve

[Steve_B]

Would you try a backup now please?

[rahvin9999]

@steve_b

Did a test backup on both pfSense1 and pfSense2 and they now show up in the list.

Bedankt!

[Steve_B]

Glad it worked. As @stephenw10 said, an external authentication system would be worthwhile for that many users, but in the meantime I increased the max file size to 30 MiB.

[rahvin9999]

@steve_b

I am looking into moving the users and the certificates of the pfSense machines to a dedicated solution.

Thanks for the help with this issue!