Got pfSense on Azure working but pfSense update breaks

doiiido

Re: Pfsense Azure - Internet by WAN and not by Azure

I'd spent over 180 hours dealing with this problem and got it working:

Check if your NICs have ip fowarding enabled;

You cannot set a static ip or gateway on pfSense and LAN clients (somehow the routes are not applied correctly), set fixed ip on your NICs config on azure;

I've set only one route: 0.0.0.0/0 to my "virtual appliance" IP on route config on azure;

DHCP on LAN from pfSense will not work for azure VMs (it works for VPN TAP clients if ip not in conflict);
Check if all ports and protocols are allowed on the network security group applied on LAN;
WALinuxAgent can be installed through this:
https://forum.netgate.com/post/1007499

My setup is based on 2 Windows servers VMs with routes through the pfsense firewall and multiple site-to-site VPNs connected to the firewall, allowing external clients to access services inside the servers.

I got into another problem now:

I've mirrored the environment locally and pfSense seems to detect it's running on azure. The update configuration got messed up only on the Azure VM:

And everything under "/usr/local/etc/pfSense/pkg/repos" gets wiped out and pkg breaks.
Logs fom "/conf/upgrade_log*" :

ERROR: It was not possible to determine pkg remote version
ERROR: It was not possible to determine pkg remote version
>>> Updating repositories metadata... failed.
ERROR: It was not possible to determine pfSense remote version
ERROR: It was not possible to determine pfSense-base remote version
ERROR: It was not possible to determine pfSense-kernel-pfSense remote version
Your system is up to date

Best Regards,
Lincoln.

NightlyShark

First of all, (because I know nothing of Azure) do you have the option to pass the VNICs as PCIe devices and not VIRTIO?

Second, did you apply a rate limit on the NICs (from the Azure GUI) that matches that which Azure can manage? (Virtual NICs have issues with bandwidth, especially if the arrangement is virtual to bridge to firewall to bridge to virtual (which I suspect is the case for Azure VMs, could/must be wrong, though), I found, even in my Proxmox VIRTIO vtnet bridges it is best to manually set 1GBit and a 1500 MTU, otherwise I get packet drops).

Third, because running a VM on the cloud means that it accesses the internet from a Datacenter and most (big) DNS providers use different server clusters geographically with the same IPs globally (see 8.8.8.8 and 1.1.1.1) and have ISPs modify their routing tables, could it be that you use a DNS server for PfSense that is meant to be used by the public and not by an Azure Datacenter? Those errors at the end of the log could also mean that the IP of the update server is not accessible from the Azure "location".

Four, did you disable all offloading (most probably, yes, I imagine)?

Five, does the Azure VM stack allow you to accelerate AES-NI workloads? If not, 2-3 IPsec or OpenVPN site-to-site--s would bring the VM to it's knees.

SteveITS

@doiiido FWIW Netgate sells this as a service…https://www.netgate.com/pfsense-plus-software/how-to-buy#Azure

https://docs.netgate.com/pfsense/en/latest/solutions/azure-appliance/faq.html#does-the-appliance-support-a-live-update-of-the-software

https://www.netgate.com/pfsense-plus-azure-cloud

I thought they had an Arm Azure version too but not finding it.

doiiido

@SteveITS
Yep, but I've created a Community Edition version (pfSense CE) from scratch on hyper-v and uploaded it.
Everything works but it can't update on GUI because of this weird bug....

doiiido

@NightlyShark
Hi,
Azure doesn't have much options on the VMs hardware intrinsics and everything is shown to the VM the same way as it was on hyper-v, I haven't changed the rate limit but don't think it's an issue because the internal VMs and the sites both have flawless conectivity.
Offloading is disabled and AES-NI is supported but doesn't work (openssl shows errors related to it on logs if enabled).

About the routes, I'll be checking the outbound packets to discover the IPs used by pfsense for update, but it seems weird as every other service from global ips routes and works (DNS, NTP, Microsoft, etc...)

I run 4 site-to-site tunnels, the cpu usage is about 10% and memory is about 50% on a B1S size machine (1 vcpu, 1GB RAM).

Best Regards,
Lincoln.

NightlyShark

@doiiido The last think that comes to my mind is to check if DNS resolution is OK, and then if there is any firewall from the Azure side

markes20754

Did you ever figure this out? I've spent hours uploading different configs of 2.7.x CE to Azure and they can all take updates on-prem but break after being moved to Azure. 2.6.0 doesn't have the problem but if you update 2.6.0 to 2.7.0 in Azure it breaks all updates with the same symptoms you've described.

SteveITS

@markes20754 There was another thread I guess it was around the same time as this one, where Netgate said they put some changes/fixes/drivers/whatever into Plus to let it work on Azure. Info on their solution is above.

markes20754

@SteveITS Thanks -- odd because PFSense has been working in Azure for quite a bit and upgrades worked. Now no packages can be downloaded or anything as of 2.7.0

doiiido

@markes20754 You can restore the dpkg files and folder from the local image and update through dpkg, but the GUI updates no longer work.

markes20754

@doiiido Thanks -- as much as I held off I just went with OPNSense for my Azure deployments. Hopefully Netgate addresses the issue in CE but I suspect they've blocked Azure Hardware IDs from getting updates if they're not paying and CE got included in that.