Azure pfSense Plus 23.09 VM backup failed

shift

We upgraded these days our pfSense Plus Firewall instance from the Azure Marketplace to 23.09. Also the image changed (without documentation…). We manage the instance with terraform and a restore of the latest config. We did this not the first time, everything worked. We saw that in 23.09 the waagent is instantly working without fixing paths like it was necessary before.

We make an Azure backup of the OS disk and the VM. This worked before with adjusted waagent before. With the "native" waagent from the version 23.09 the disk snapshot is still working but the VM backup does not work anymore:

We are a bit tired of this kind of problems since we spend quite a lot time to get the waagent from previous version running. We were happy that this is intended to work in the new version "out of the box" but it doesnt.

Could someone from Netgate who is in charge for the integration into Azure could please respond to this issue? Thank you!

stephenw10

Hmm, that worked last time I tested it. Let me retest....

That failed after >5 hours? That seems like a very long time! Is that typical for your VM?

Steve

shift

@stephenw10 Thank you!

I think it is a timeout after a long time:

Could not communicate with the VM agent for snapshot status - Snapshot VM sub task timed out

All other Linux VMs are usually backed up in a few minutes, as was previously the case with the pfSense VM.

It is really strange, the waagent log does not show errors or such. Except for this warning which is not in another (linux) vm:

2023-11-15T06:02:54.422675Z WARNING ExtHandler ExtHandler [PERIODIC] This status is being reported by the Guest Agent since no status file was reported by extension Microsoft.Azure.RecoveryServices.VMSnapshotLinux: [ExtensionStatusError] Status file /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9207.0/status/14.status does not exist

I have looked again at https://learn.microsoft.com/en-us/azure/backup/backup-azure-vms-introduction and https://learn.microsoft.com/en-us/azure/backup/backup-azure-arm-vms-prepare. All looks okay.

There is one error in the waagent log which is related to iptables which is not exiting in freebsd. Thats why the waagent script is coming from linux.

2023-11-10T07:23:18.882211Z WARNING MonitorHandler ExtHandler Error fetching the network configuration: [Errno 2] No such file or directory: 'ip'
2023-11-10T07:23:18.896576Z INFO ExtHandler ExtHandler
2023-11-10T07:23:18.896091Z INFO EnvHandler ExtHandler EnvMonitor: Detected hostname change: vm-fw -> vm-fw.westeurope.cloudapp.azure.com
2023-11-10T07:23:18.896844Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState started [incarnation_1 channel: WireServer source: Fabric activity: 11f04869-1a64-4d1b-8e33-df9fa
ca3e911 correlation b281e99e-fab0-720e-1115-7b57b256d661 created: 2023-11-10T07:22:13.854462Z]
2023-11-10T07:23:18.899744Z INFO ExtHandler ExtHandler No extension handlers found, not processing anything.
2023-11-10T07:23:18.902090Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [incarnation_1 5 ms]

2023-11-10T07:23:18.931125Z INFO ExtHandler ExtHandler Looking for existing remote access users.
2023-11-10T07:23:18.937344Z WARNING ExtHandler ExtHandler Unable to determine version of iptables: [Errno 2] No such file or directory: 'iptables'
2023-11-10T07:23:18.937565Z WARNING ExtHandler ExtHandler Unable to retrieve firewall packets droppedUnable to determine version of iptables: [Errno 2] No such file or directory:
'iptables'

Does this information help further?

stephenw10

Ok I replicated that. We are looking into it.

shift

@stephenw10 Hi, could you give us a status on the progress? We are ready to test!

stephenw10

One of our devs was looking at it but then 23.09.1 happened. Let me check....

shift

@stephenw10 Hi, since 23.09.1 is released the question arise if the backup for pfSense plus in Azure is fixed with this version? Is it?

stephenw10

It is not changed there as far as I know. It looks to be the script that is fetched by the agent that changed since it also no longer works in 23.05.1 and it did when I tested it back in August.

shift

@stephenw10 Hi and good morning. Release 24.03 is out but i do not see mentioned something about reintroducing Azure VM backup. Do you have news on that topic?

stephenw10

Yes, I believe we have found the upstream problem and have poked the appropriate people to get it addressed. That only happened yesterday though so no ETA yet.

shift

We have just rebuilt our instance with image version 24.03 - the vm backup is still not working. Can we please escalate this issue? I find this really annoying, we are spending €2,869.08 per year on this instance and the platform backup is not working? For ... a long time? Why is that? What's going on with the QA at Netgate? I am happy to assist.

stephenw10

I believe there was an upstream issue that was being worked on but let me recheck...

shift

Hi @stephenw10, could you please share your check with us? The backup feature is still not fixed.

stephenw10

I'll check again. Last I heard the code the agent pulls in has been updated to fix and we're just waiting for it to be deployed by Azure.

shift

@stephenw10 Hi, we developed a Azure custom image with Opnsense including the current Azure Agent. With that Azure backup works normally. We stay with this solution, migration from pfSense was bit of work but worthwhile! You can close this task, thank you for your effort.