Azure pfSense Plus 23.09 VM backup failed

shift

I think it is a timeout after a long time:

Could not communicate with the VM agent for snapshot status - Snapshot VM sub task timed out

All other Linux VMs are usually backed up in a few minutes, as was previously the case with the pfSense VM.

It is really strange, the waagent log does not show errors or such. Except for this warning which is not in another (linux) vm:

2023-11-15T06:02:54.422675Z WARNING ExtHandler ExtHandler [PERIODIC] This status is being reported by the Guest Agent since no status file was reported by extension Microsoft.Azure.RecoveryServices.VMSnapshotLinux: [ExtensionStatusError] Status file /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9207.0/status/14.status does not exist

I have looked again at https://learn.microsoft.com/en-us/azure/backup/backup-azure-vms-introduction and https://learn.microsoft.com/en-us/azure/backup/backup-azure-arm-vms-prepare. All looks okay.

There is one error in the waagent log which is related to iptables which is not exiting in freebsd. Thats why the waagent script is coming from linux.

2023-11-10T07:23:18.882211Z WARNING MonitorHandler ExtHandler Error fetching the network configuration: [Errno 2] No such file or directory: 'ip'
2023-11-10T07:23:18.896576Z INFO ExtHandler ExtHandler
2023-11-10T07:23:18.896091Z INFO EnvHandler ExtHandler EnvMonitor: Detected hostname change: vm-fw -> vm-fw.westeurope.cloudapp.azure.com
2023-11-10T07:23:18.896844Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState started [incarnation_1 channel: WireServer source: Fabric activity: 11f04869-1a64-4d1b-8e33-df9fa
ca3e911 correlation b281e99e-fab0-720e-1115-7b57b256d661 created: 2023-11-10T07:22:13.854462Z]
2023-11-10T07:23:18.899744Z INFO ExtHandler ExtHandler No extension handlers found, not processing anything.
2023-11-10T07:23:18.902090Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [incarnation_1 5 ms]

2023-11-10T07:23:18.931125Z INFO ExtHandler ExtHandler Looking for existing remote access users.
2023-11-10T07:23:18.937344Z WARNING ExtHandler ExtHandler Unable to determine version of iptables: [Errno 2] No such file or directory: 'iptables'
2023-11-10T07:23:18.937565Z WARNING ExtHandler ExtHandler Unable to retrieve firewall packets droppedUnable to determine version of iptables: [Errno 2] No such file or directory:
'iptables'

Does this information help further?

stephenw10

Ok I replicated that. We are looking into it.

shift

@stephenw10 Hi, could you give us a status on the progress? We are ready to test!

stephenw10

One of our devs was looking at it but then 23.09.1 happened. Let me check....

shift

@stephenw10 Hi, since 23.09.1 is released the question arise if the backup for pfSense plus in Azure is fixed with this version? Is it?

stephenw10

It is not changed there as far as I know. It looks to be the script that is fetched by the agent that changed since it also no longer works in 23.05.1 and it did when I tested it back in August.

shift

@stephenw10 Hi and good morning. Release 24.03 is out but i do not see mentioned something about reintroducing Azure VM backup. Do you have news on that topic?

stephenw10

Yes, I believe we have found the upstream problem and have poked the appropriate people to get it addressed. That only happened yesterday though so no ETA yet.

shift

We have just rebuilt our instance with image version 24.03 - the vm backup is still not working. Can we please escalate this issue? I find this really annoying, we are spending €2,869.08 per year on this instance and the platform backup is not working? For ... a long time? Why is that? What's going on with the QA at Netgate? I am happy to assist.

stephenw10

I believe there was an upstream issue that was being worked on but let me recheck...

shift

Hi @stephenw10, could you please share your check with us? The backup feature is still not fixed.

stephenw10

I'll check again. Last I heard the code the agent pulls in has been updated to fix and we're just waiting for it to be deployed by Azure.

shift

@stephenw10 Hi, we developed a Azure custom image with Opnsense including the current Azure Agent. With that Azure backup works normally. We stay with this solution, migration from pfSense was bit of work but worthwhile! You can close this task, thank you for your effort.