Azure pfSense Plus 23.09 VM backup failed

stephenw10

Hmm, that worked last time I tested it. Let me retest....

That failed after >5 hours? That seems like a very long time! Is that typical for your VM?

Steve

shift

I think it is a timeout after a long time:

Could not communicate with the VM agent for snapshot status - Snapshot VM sub task timed out

All other Linux VMs are usually backed up in a few minutes, as was previously the case with the pfSense VM.

It is really strange, the waagent log does not show errors or such. Except for this warning which is not in another (linux) vm:

2023-11-15T06:02:54.422675Z WARNING ExtHandler ExtHandler [PERIODIC] This status is being reported by the Guest Agent since no status file was reported by extension Microsoft.Azure.RecoveryServices.VMSnapshotLinux: [ExtensionStatusError] Status file /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9207.0/status/14.status does not exist

I have looked again at https://learn.microsoft.com/en-us/azure/backup/backup-azure-vms-introduction and https://learn.microsoft.com/en-us/azure/backup/backup-azure-arm-vms-prepare. All looks okay.

There is one error in the waagent log which is related to iptables which is not exiting in freebsd. Thats why the waagent script is coming from linux.

2023-11-10T07:23:18.882211Z WARNING MonitorHandler ExtHandler Error fetching the network configuration: [Errno 2] No such file or directory: 'ip'
2023-11-10T07:23:18.896576Z INFO ExtHandler ExtHandler
2023-11-10T07:23:18.896091Z INFO EnvHandler ExtHandler EnvMonitor: Detected hostname change: vm-fw -> vm-fw.westeurope.cloudapp.azure.com
2023-11-10T07:23:18.896844Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState started [incarnation_1 channel: WireServer source: Fabric activity: 11f04869-1a64-4d1b-8e33-df9fa
ca3e911 correlation b281e99e-fab0-720e-1115-7b57b256d661 created: 2023-11-10T07:22:13.854462Z]
2023-11-10T07:23:18.899744Z INFO ExtHandler ExtHandler No extension handlers found, not processing anything.
2023-11-10T07:23:18.902090Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [incarnation_1 5 ms]

2023-11-10T07:23:18.931125Z INFO ExtHandler ExtHandler Looking for existing remote access users.
2023-11-10T07:23:18.937344Z WARNING ExtHandler ExtHandler Unable to determine version of iptables: [Errno 2] No such file or directory: 'iptables'
2023-11-10T07:23:18.937565Z WARNING ExtHandler ExtHandler Unable to retrieve firewall packets droppedUnable to determine version of iptables: [Errno 2] No such file or directory:
'iptables'

Does this information help further?

stephenw10

Ok I replicated that. We are looking into it.

shift

@stephenw10 Hi, could you give us a status on the progress? We are ready to test!

stephenw10

One of our devs was looking at it but then 23.09.1 happened. Let me check....

shift

@stephenw10 Hi, since 23.09.1 is released the question arise if the backup for pfSense plus in Azure is fixed with this version? Is it?

stephenw10

It is not changed there as far as I know. It looks to be the script that is fetched by the agent that changed since it also no longer works in 23.05.1 and it did when I tested it back in August.

shift

@stephenw10 Hi and good morning. Release 24.03 is out but i do not see mentioned something about reintroducing Azure VM backup. Do you have news on that topic?

stephenw10

Yes, I believe we have found the upstream problem and have poked the appropriate people to get it addressed. That only happened yesterday though so no ETA yet.

shift

We have just rebuilt our instance with image version 24.03 - the vm backup is still not working. Can we please escalate this issue? I find this really annoying, we are spending €2,869.08 per year on this instance and the platform backup is not working? For ... a long time? Why is that? What's going on with the QA at Netgate? I am happy to assist.

stephenw10

I believe there was an upstream issue that was being worked on but let me recheck...

shift

Hi @stephenw10, could you please share your check with us? The backup feature is still not fixed.

stephenw10

I'll check again. Last I heard the code the agent pulls in has been updated to fix and we're just waiting for it to be deployed by Azure.

shift

@stephenw10 Hi, we developed a Azure custom image with Opnsense including the current Azure Agent. With that Azure backup works normally. We stay with this solution, migration from pfSense was bit of work but worthwhile! You can close this task, thank you for your effort.

ebrenton

FYI, This is still an issue in version 24.11

A fix was posted here, which includes adding missing symlinks...
https://github.com/Azure/WALinuxAgent/issues/1335

Can we please get this fixed in the production build

stephenw10

AFAIK this current failure is unrelated to that linked issue. Are you saying that still works for you?

ebrenton

Yes. I just tested this morning after applying the commands in that post (with version numbers updated) and the backup completed successfully. Before this, the backup would fail every time with the GuestAgentSnapshotTaskStatusError error.

stephenw10

Hmm, OK. Maybe something changed again then. It's painful because the Azure agent pulls in code dynamically when it loads so Azure can change it at any time.

ebrenton

@stephenw10 Please let me know if you make any progress on this.
We are in the process of moving our services to the Azure cloud and pfSense seems to be working well as an ipSec solution. But it is essential that we have a DR plan for this VM.

stephenw10

In the mean time you can still always make backups with it stopped. That has always worked. Though obviously it's far more inconvenient.

We are digging...