Auto Configuration Backup fails after WAN IP change
-
ACB has been running fine for the last year or two, but I have noticed that since my WAN IP changed on the 20th January, backups have been failing and I am stumped as to how to fix it. If I go to the Restore tab I can see previous backups up to 19th January.
The error message received is
An error occurred while uploading the encrypted pfSense configuration to https://acb.netgate.com/save (Operation timed out after 30018 milliseconds with 0 bytes received)It doesn't appear to be a DNS error or routing issue. I can
curl -L http://acb.netgate.comfrom the console and get back some HTML:... <body> Automatic Configuration Backup system for Netgate/pfSense firewalls </body>...So, I am wondering if the backups are somehow being refused because the public IP of my router has changed? Does anyone have any ideas as to what the problem may be or how to diagnose further?
I'm running pfSense CE 2.7.2-RELEASE.
Thanks,
-
Thought I should add, I tried the following changes but all have failed to fix the issue:
- Full reboot.
- Changing the auto-backup frequency from "every configuration change" to "on a regular schedule".
- Removing old backups from the Restore tab.
- Performing a manual backup.
- Regenerating the SSH Host keys to generate a new Device ID for backup.
-
Can you resolve acb.netgate.com? Can you ping it?
I'm not aware of any blocking in front of that server but there may be some. Send me the new WAN IP in chat and I'll check it.
-
@stephenw10 Yes, Ping works. A GET of http://acb.netgate.com also works. And I can also get a list of previous successful backups from the Restore tab in the web UI....
[2.7.2-RELEASE][admin@redacted.lan]/root: ping acb.netgate.com PING acb.netgate.com (208.123.73.69): 56 data bytes 64 bytes from 208.123.73.69: icmp_seq=0 ttl=251 time=120.254 ms 64 bytes from 208.123.73.69: icmp_seq=1 ttl=251 time=120.423 ms 64 bytes from 208.123.73.69: icmp_seq=2 ttl=251 time=120.271 ms 64 bytes from 208.123.73.69: icmp_seq=3 ttl=251 time=120.336 ms --- acb.netgate.com ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 120.254/120.321/120.423/0.066 ms [2.7.2-RELEASE][admin@redacted.lan]/root: curl -L http://acb.netgate.com <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>Auto Config Backup</title> </head> <body> Automatic Configuration Backup system for Netgate/pfSense firewalls </body> </html> -
@codersaur said in Auto Configuration Backup fails after WAN IP change:
So, I am wondering if the backups are somehow being refused because the public IP of my router has changed?
Why ?
When you@codersaur said in Auto Configuration Backup fails after WAN IP change:
I can curl -L http://acb.netgate.com from
weren't you using the same WAN IP ?
Go here : Services > Dynamic DNS > Check IP Services and copy the URL.
'curl' to it. It will show your WAN IP, right ?This could be very well a probably temporary glitch as "http://acb.netgate.com" can get a bit overworked as more and more and more pfSense devices dump their daily config file.
@codersaur said in Auto Configuration Backup fails after WAN IP change:
"every configuration change"
Yeah, what about not abusing a free service (
) - I mean : ones a day or so is just fine for most needs. -
We are digging into this internally. We spent a while looking at it yesterday and so far it looks like back end problem.
-
@stephenw10 said in Auto Configuration Backup fails after WAN IP change:
We are digging into this internally. We spent a while looking at it yesterday and so far it looks like back end problem.
Thanks Stephen, I look forward to your conclusions so we can get this resolved asap - I've been having to make manual backups for the last three weeks.
-
@stephenw10 did your investigation yield any useful conclusions? My backup is still failing each morning.
Is there some way I can purge the local backup files in case there is some lingering corruption which is causing the issue when they are uploaded?
-
The file that gets uploaded is generated from the config each time. However you can check if it's still present in /tmp. The file name is {file_hash}.tmp. It should be removed from there after uploading.
-
And check the file size ?
Be ware : backup files are here : /cf/conf/backup/ -
@stephenw10 so there's no .tmp file.
Backup dir looks normal (I guess) with each xml file being 9.1MB and being created at 04:48 as per schedule (the ACME service also seems to trigger a backup at 03:16 every third day).
I've run the latest xml file through a validator and it seems fine. So I am still at a loss as to why my backup uploads to acb.netgate.com are failing...
/root: ls -lah /cf/conf/backup/ total 279376 drwxr-xr-x 2 root wheel 1.5K Feb 21 04:48 . drwxr-xr-x 5 root wheel 1.0K Feb 21 04:48 .. -rw-r--r-- 1 root wheel 4.6K Feb 21 04:48 backup.cache -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461091.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461092.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461093.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461096.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461097.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461098.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461100.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461101.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461102.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461105.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461106.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461107.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461109.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461110.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461111.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461112.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461115.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461116.xml -rw-r--r-- 1 root wheel 9.1M Feb 13 15:38 config-1739461117.xml -rw-r--r-- 1 root wheel 9.1M Feb 14 04:48 config-1739461118.xml -rw-r--r-- 1 root wheel 9.1M Feb 15 03:16 config-1739508480.xml -rw-r--r-- 1 root wheel 9.1M Feb 15 04:48 config-1739589365.xml -rw-r--r-- 1 root wheel 9.1M Feb 16 04:48 config-1739594880.xml -rw-r--r-- 1 root wheel 9.1M Feb 17 04:48 config-1739681280.xml -rw-r--r-- 1 root wheel 9.1M Feb 18 03:16 config-1739767680.xml -rw-r--r-- 1 root wheel 9.1M Feb 18 04:48 config-1739848565.xml -rw-r--r-- 1 root wheel 9.1M Feb 19 04:48 config-1739854080.xml -rw-r--r-- 1 root wheel 9.1M Feb 20 04:48 config-1739940480.xml -rw-r--r-- 1 root wheel 9.1M Feb 21 03:16 config-1740026880.xml -rw-r--r-- 1 root wheel 9.1M Feb 21 04:48 config-1740107765.xml -
Yes, it pretty much has to be the backend rejecting it. I'll poke our guys again.
-
My typical config-xxx.xml files are 650 Kbytes or so for a 24.11 on a 4100.
9+ Mbytes seems a lot to me. -
@Gertjan The xml files seem to include the rrddata. If I make a manual backup without rrddata the size is ~380 Kb.
There doesn't seem to be an option in the ACB config page to include/exclude rrd data like there is on the manual Diagnostics > Backup/Restore...
-
Since I can't thumb up ATM. Anyways.
I'm having this same issue as codersaur.
Look forward to the fix. For now disabled.
Netgate 4200 24.11
-
@williamrolison When did it start failing and can you correlate with any changes on your side? If yours also started failing around 20th Jan that might indicate a server-side change and rule out the WAN IP change a factor. Conversely, if you also had a WAN IP change that might also help track down the root cause.
-
Started for me on Feb 13. Feb 11 was when IP changed.
-
The encrypted files uploaded to acb always exclude RRD data so that file size should be no issue. But clearly something is....
-
@stephenw10 is there any progress update on this issue? Thanks.
-
Nothing yet. I'll see if our IT guys found anything.....