Unable to resolve acb.netgate.com notifications every 10 seconds for hours…

[fabrizior]

Everything seems to be working as normal after a 36-hour power failure, except the config backups to netgate.

Getting 6+ notification emails a MiNUTE!

An error occurred while uploading the encrypted  configuration to https://acb.netgate.com/save ( Unable to resolve acb.netgate.com )

DNS resolves acb.netgate.com just fine from the root shell on my pfsense box and from every client I’ve tried.
Last bit of traceroute from pfsense:

 6  ZAYO-BANDWI.ear5.Dallas1.Level3.net (4.14.49.2)  42.169 ms  41.627 ms  42.461 ms
 7  ae0.aus01-mls-dc-core-b.infr.zcolo.com (64.20.229.166)  42.337 ms
    ae0.aus01-mls-dc-core-a.infr.zcolo.com (64.20.229.158)  40.808 ms  40.573 ms
 8  net66-219-34-194.static-customer.corenap.com (66.219.34.194)  41.019 ms  41.791 ms  42.028 ms
 9  fw1-zcolo.netgate.com (208.123.73.4)  41.394 ms  42.687 ms  43.779 ms
10  208.123.73.212 (208.123.73.212)  41.866 ms  42.641 ms  41.585 ms

Any additional ideas on triaging this?
Any clues on how to disable this particular notification without disabling notifications completely?
Is there a way to prioritize and or get an hourly/daily digest rather than 1:1 instantaneous notifications for less-urgent messages?

[NollipfSense]

@fabrizior said in Unable to resolve acb.netgate.com notifications every 10 seconds for hours…:

Everything seems to be working as normal after a 36-hour power failure, except the config backups to netgate.

Did you have an UPS? Is acb,netgate.com a cloud storage for backups? Have always keep config backups locally.

[fabrizior]

@NollipfSense
I’m referring to the AutoConfigBackups functionality built-into pfSense which stores your encrypted config at netgate.com for you.

UPS, yes, but… its the same UPS for the entire rack, including a large NAS, so only a 30m runtime. just enough for a properly managed shutdown.
Not enough justification for budget to provide 36 hrs runtime for the network infra (incl. POE WiFi APs) without the NAS. All Internet providers were down too (have no cellular/sat svcs) so not much sense in keeping the router up anyway.

This issue started once I re-energized the rack after power was restored. Router, modems, switches, and APs all came up in parallel. NAS only after network was stable.

I’ve unchecked the automatic backups, hoping to disable the cause for the email flood, but that doesn’t seem to eliminate the issue. now have over 2000 emails and growing by the minute. Very surprised the retries continue.

Will schedule a reboot to see if that clears any queued backups. If that works to stop the failed backup retries and email notifications, I’ll re-enable ACB and see if they start failing or succeeding again.

Can’t find any log detail that would hint as to why these backup attempts keep failing to resolv the netgate FQDN!!!

[Gertjan]

@NollipfSense said in Unable to resolve acb.netgate.com notifications every 10 seconds for hours…:

acb,netgate.com

Check : it's acb.netgate.com

A not filtered DNS, default DNS settings, using plain resolving, gives me :

[23.05.1-RELEASE][root@pfSense.bhf.net]/root: host acb.netgate.com
acb.netgate.com has address 208.123.73.212