How to install the Azure Linux Agent (waagent) in the new versions of PfSense

[mark.dayton]

@jamiegb Got this working with the below

ln -sf /usr/local/sbin/waagent /usr/sbin/waagent
ln -sf /usr/local/sbin/waagent2.0 /usr/sbin/waagent2.0
echo '#! /bin/sh' >> /usr/local/etc/rc.d/waagent.sh
echo '/usr/local/sbin/waagent --daemon' >> /usr/local/etc/rc.d/waagent.sh
chmod +x /usr/local/etc/rc.d/waagent.sh
echo "y" | /usr/local/sbin/waagent -deprovision+user
echo 'waagent_enable="YES"' >> /etc/rc.conf.local

[JamieGB]

@mark-dayton said in How to install the Azure Linux Agent (waagent) in the new versions of PfSense:

@jamiegb Got this working with the below

ln -sf /usr/local/sbin/waagent /usr/sbin/waagent
ln -sf /usr/local/sbin/waagent2.0 /usr/sbin/waagent2.0
echo '#! /bin/sh' >> /usr/local/etc/rc.d/waagent.sh
echo '/usr/local/sbin/waagent --daemon' >> /usr/local/etc/rc.d/waagent.sh
chmod +x /usr/local/etc/rc.d/waagent.sh
echo "y" | /usr/local/sbin/waagent -deprovision+user
echo 'waagent_enable="YES"' >> /etc/rc.conf.local

Mark that is some fantastic work. I have no doubt your solution is going to help many people! Can't thank you enough!!!

[mark.dayton]

@jamiegb I can't take credit for this unfortunately, found this in another post on this forum! Tweaked slightly for rc.conf.local but it works :)
It takes a little time to start in Azure but afterwards im able to access the backend nic just fine.

[JamieGB]

@mark-dayton said in How to install the Azure Linux Agent (waagent) in the new versions of PfSense:

waagent

So I tried this out. Unfortunately no success. But I see the commands are creating aliases for both waagent 1 and 2, yet the script being called with the --daemon parameter seems to be version 1. BSD needs version 2 of waagent so I'll persevere and report back my findings.

I wish I could just access the console, SSH or web interface to see what;;'s going on, but it's totally offline. When I made the image I set both interfaces to DHCP but deployed the VM with 2 NICs, one with a public IP and both with static internal IPs. Maybe this is stopping the network stack loading up....

[mark.dayton]

@jamiegb Okay so I loosely followed this article here https://www.christofvg.be/2019/01/12/pfSense-on-Azure-Part-1-Create-pfSense-Virtual-Machine/

But when getting to the point of installing the agent i used the following commands, note python 3.8 for latest pfsense version 2.5.2.

pkg upgrade
pkg install -y sudo bash git
ln -s /usr/local/bin/python3.8 /usr/local/bin/python

Clone the Git repository

git clone https://github.com/Azure/WALinuxAgent.git

Enter the WALinuxAgent directory

cd WALinuxAgent

List all available versions

git tag

Checkout the latest (stable) version of the agent

git checkout v2.4.0.2

Install the agent

python setup.py install
ln -sf /usr/local/sbin/waagent /usr/sbin/waagent
ln -sf /usr/local/sbin/waagent2.0 /usr/sbin/waagent2.0
echo '#! /bin/sh' >> /usr/local/etc/rc.d/waagent.sh
echo '/usr/local/sbin/waagent --daemon' >> /usr/local/etc/rc.d/waagent.sh
chmod +x /usr/local/etc/rc.d/waagent.sh
echo "y" | /usr/local/sbin/waagent -deprovision+user
echo 'waagent_enable="YES"' >> /etc/rc.conf.local

You can check whether it's running before deprovision by running service waagent status.

I deployed the image with both NICs DHCP and also left them DHCP in Azure. You may have to wait a few minutes after booting the VM to access the LAN NIC on either http or https whichever you have configured.

Below are the commands i used for creating the VM

Initialize variables

$storageType = "Standard_LRS"
$location = "UK South"

$storageAccountId = "xxxxxxxxxxxxxxxxxxxxx"
$sourceVhdUri = "https://xxxxxxxxxxxxxx.blob.core.windows.net/temp/AZ-PFSENSE01.vhd"

Create the disk configuration

$diskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $location -CreateOption Import -StorageAccountId $storageAccountId -SourceUri $sourceVhdUri

Create the Managed Disk

New-AzureRmDisk -Disk $diskConfig -ResourceGroupName <resource group> -DiskName az-pfsense_disk1

Get the object of the existing Managed Disk

$disk = Get-AzureRmDisk -DiskName az-pfsense_disk1 -ResourceGroupName <resource group>

Get the object for the existing Virtual Network

$VirtualNetwork = Get-AzureRmVirtualNetwork -Name <vnet> -ResourceGroupName <resource group>

Create a new Virtual Machine object

$virtualMachine = New-AzureRmVMConfig -VMName az-pfsense01 -VMSize Standard_B2s

Attach the existing Managed Disk to the Virtual Machine

$virtualMachine = Set-AzureRmVMOSDisk -VM $virtualMachine -ManagedDiskId $disk.Id -CreateOption Attach -Linux

#Create pub IP
$pubip = New-AzureRmPublicIpAddress -Name "az-pfsense01-pip1" -ResourceGroupName $pfresourcegroup -Location $location -AllocationMethod Dynamic

Create the NIC's for the frontend and the backend, note 7 is the subnet ID yours maybe different

$frontEndNic = New-AzureRmNetworkInterface -Name az-pfsense01-frontend-nic -ResourceGroupName <resource group> -Location 'UK South' -SubnetId $VirtualNetwork.Subnets[7].Id

$backEndNic = New-AzureRmNetworkInterface -Name az-pfsense01-backend-nic -ResourceGroupName <resource group> -Location 'UK South' -SubnetId $VirtualNetwork.Subnets[7].Id -PublicIpAddressId $az-pfsense01-pip1.Id

Add the NIC's to the Virtual Machine

$virtualMachine = Add-AzureRmVMNetworkInterface -VM $virtualMachine -Id $frontEndNic.Id -Primary
$virtualMachine = Add-AzureRmVMNetworkInterface -VM $virtualMachine -Id $backEndNic.Id

Create the Virtual Machine

New-AzureRmVM -VM $virtualMachine -ResourceGroupName <resource group> -Location 'UK South'

[tomih]

Deployed pfsense community edition on azure and tried to get this waagent up with above guide.

WALinuxAgent-2.2.39 running on freebsd 12.2
Python: 3.8.10
Goal state agent: 2.2.39

Agent seems to start ok, but there is something wrong with configuration as it is not able to communicatate properly to azure. It is trying to communicate on azure 168.63.129.16 but it fails. Any idea how to troubleshoot or fix?

log:

[2.5.2-RELEASE][admin@x]/root: cat /var/log/waagent.log
2022/01/20 12:39:08.073418 INFO Daemon Azure Linux Agent Version:2.2.39
2022/01/20 12:39:08.074016 INFO Daemon OS: freebsd 12.2
2022/01/20 12:39:08.074137 INFO Daemon Python: 3.8.10
2022/01/20 12:39:08.074266 VERBOSE Daemon Command: [ps -p 69164]
2022/01/20 12:39:08.081280 INFO Daemon CGroups: not supported by platform
2022/01/20 12:39:08.082525 WARNING Daemon Too many files under: /var/lib/waagent/events, removing oldest
2022/01/20 12:39:08.083576 WARNING Daemon Too many files under: /var/lib/waagent/events, removing oldest
2022/01/20 12:39:08.083847 INFO Daemon Run daemon
2022/01/20 12:39:08.084029 INFO Daemon No RDMA handler exists for distro='freebsd' version='12.2'
2022/01/20 12:39:08.084164 INFO Daemon Activate resource disk
2022/01/20 12:39:08.084255 VERBOSE Daemon Command: [gpart list]
2022/01/20 12:39:08.088782 VERBOSE Daemon Command: [sysctl dev.storvsc | grep pnpinfo | grep deviceid=]
2022/01/20 12:39:08.093447 VERBOSE Daemon Command: [sysctl dev.storvsc | grep pnpinfo | grep deviceid=00000000-0001]
2022/01/20 12:39:08.097994 VERBOSE Daemon Command: [sysctl dev.storvsc | grep pnpinfo | grep deviceid=00000000-0001|awk -F . '{print $3}']
2022/01/20 12:39:08.103197 VERBOSE Daemon Command: [camcontrol devlist -b | grep blkvsc1 | awk '{print $1}']
2022/01/20 12:39:08.107349 VERBOSE Daemon Command: [camcontrol devlist | grep scbus3 | awk -F \( '{print $2}'|sed -e 's/.*(//'| sed -e 's/).*//']
2022/01/20 12:39:08.113020 INFO Daemon Resource disk device da1 found.
2022/01/20 12:39:08.113227 VERBOSE Daemon Command: [gpart show -p da1]
2022/01/20 12:39:08.116966 INFO Daemon Resource disk partition /dev/da1s1 found.
2022/01/20 12:39:08.117168 VERBOSE Daemon Command: [mount]
2022/01/20 12:39:08.119764 INFO Daemon Resource disk /dev/da1s1 is already mounted
2022/01/20 12:39:08.120058 INFO Daemon Clean protocol
2022/01/20 12:39:08.120283 INFO Daemon Running default provisioning handler
2022/01/20 12:39:08.120421 INFO Daemon Copying ovf-env.xml
2022/01/20 12:39:08.121037 ERROR Daemon Provisioning failed: [ProtocolError] [CopyOvfEnv] Error mounting dvd: [OSUtilError] Failed to get dvd device from /dev
Inner error: The following devices were found, but none matched the pattern [(sr[0-9]|hd[c-z]|cdrom[0-9]|cd[0-9])]: ['dumpdev', 'reroot', 'devctl', 'devctl2', 'console', 'sndstat', 'hv_tsc', 'random', 'urandom', 'geom.ctl', 'pci', 'fd', 'stdin', 'stdout', 'stderr', 'uinput', 'nfslock', 'midistat', 'sequencer0', 'music0', 'tcp_log', 'auditpipe', 'audit', 'klog', 'ctty', 'consolectl', 'sysmouse', 'input', 'fido', 'io', 'kbdmux0', 'kbd1', 'bpf', 'bpf0', 'speaker', 'mem', 'kmem', 'netmap', 'full', 'null', 'zero', 'acpi', 'apmctl', 'apm', 'atkbd0', 'kbd0', 'psm0', 'bpsm0', 'ttyu0', 'ttyu0.init', 'ttyu0.lock', 'cuau0', 'cuau0.init', 'cuau0.lock', 'ttyu1', 'ttyu1.init', 'ttyu1.lock', 'cuau1', 'cuau1.init', 'cuau1.lock', 'ufssuspend', 'zfs', 'netdump', 'pf', 'usbctl', 'xpt0', 'mdctl', 'devstat', 'hv_kvp_dev', 'ttyv0', 'ttyv1', 'ttyv2', 'ttyv3', 'ttyv4', 'ttyv5', 'ttyv6', 'ttyv7', 'ttyv8', 'ttyv9', 'ttyva', 'ttyvb', 'mlx5ctl', 'fd0', 'pass0', 'pass1', 'da0', 'da1', 'da0p1', 'da0p2', 'da0p3', 'da1s1', 'md0', 'gpt', 'cpuctl0', 'tun1', 'ovpns1', 'pts']
 (0s)
2022/01/20 12:39:08.121266 INFO Daemon Detect protocol endpoints
2022/01/20 12:39:08.121370 INFO Daemon Clean protocol
2022/01/20 12:39:08.121472 INFO Daemon WireServer endpoint is not found. Rerun dhcp handler
2022/01/20 12:39:08.121560 INFO Daemon Test for route to 168.63.129.16
2022/01/20 12:39:08.121644 WARNING Daemon No route exists to 168.63.129.16
2022/01/20 12:39:08.121720 INFO Daemon Checking for dhcp lease cache
2022/01/20 12:39:08.121803 INFO Daemon Cache exists [False]
2022/01/20 12:39:08.121874 INFO Daemon Send dhcp request
2022/01/20 12:39:08.121958 VERBOSE Daemon Command: [ifconfig -l ether]
2022/01/20 12:39:08.124464 VERBOSE Daemon Command: [ifconfig hn0]
2022/01/20 12:39:08.128119 VERBOSE Daemon Interface info: (hn0,10.0.4.5,00:22:48:9e:5c:a6)
2022/01/20 12:39:08.128332 VERBOSE Daemon Command: [ifconfig -l ether]
2022/01/20 12:39:08.130849 VERBOSE Daemon Command: [ifconfig hn0]
2022/01/20 12:39:08.134468 VERBOSE Daemon Interface info: (hn0,10.0.4.5,00:22:48:9e:5c:a6)
2022/01/20 12:39:08.134785 VERBOSE Daemon BuildDhcpRequest: transactionId:A84FB5EC,A84FB5EC
2022/01/20 12:39:08.134926 VERBOSE Daemon Command: [ifconfig -l ether]
2022/01/20 12:39:08.137423 VERBOSE Daemon Command: [ifconfig hn0]
2022/01/20 12:39:08.141030 VERBOSE Daemon Interface info: (hn0,10.0.4.5,00:22:48:9e:5c:a6)
2022/01/20 12:39:08.141252 VERBOSE Daemon Command: [route add 255.255.255.255 -iface hn0]
2022/01/20 12:39:08.143829 VERBOSE Daemon Send DHCP request: Setting socket.timeout=10, entering recv
2022/01/20 12:39:18.187653 WARNING Daemon Failed to send DHCP request: [DhcpError] timed out
2022/01/20 12:39:18.188396 VERBOSE Daemon Send DHCP request: Setting socket.timeout=10, entering recv
2022/01/20 12:39:28.287475 WARNING Daemon Failed to send DHCP request: [DhcpError] timed out
2022/01/20 12:39:38.387588 VERBOSE Daemon Send DHCP request: Setting socket.timeout=10, entering recv
2022/01/20 12:39:48.482816 WARNING Daemon Failed to send DHCP request: [DhcpError] timed out

Routing table has these entries related to that ip:
168.63.129.16 00:22:48:9e:5c:a6 UHS hn0
168.63.129.16/32 10.0.4.1 UGS hn0

[alejjime]

@tomih I have the same configuration, I did the same thing and the waagent still can't communicate with Azure, so the backups give error because the agent is not active. Has anyone here setup it correctly?

[shift]

@alejjime We did it with above method for the current agent version:

waagent --version
WALinuxAgent-2.9.0.4 running on freebsd 14.0
Python: 3.9.16
Goal state agent: 2.9.0.4

The Azure backup seems to work, but in /var/log/waagent.log some errors appear. That does not feel good. waagent tries to exec iproute and base64 which is not freebsd. We need a specific configuration for Freebsd/pfsense.

We have pfsense+ expecting to get a well integrated pfSense from netgate.

/etc/waagent.conf

#
# Microsoft Azure Linux Agent Configuration
#

# Enable extension handling. Do not disable this unless you do not need password reset,
# backup, monitoring, or any extension handling whatsoever.
Extensions.Enabled=y

# Which provisioning agent to use. Supported values are "auto" (default), "waagent",
# "cloud-init", or "disabled".
# CHANGED
#Provisioning.Agent=auto
Provisioning.Agent=waagent

# Password authentication for root account will be unavailable.
# CHANGED
#Provisioning.DeleteRootPassword=y
Provisioning.DeleteRootPassword=n

# Generate fresh host key pair.
# CHANGED
#Provisioning.RegenerateSshHostKeyPair=y
Provisioning.RegenerateSshHostKeyPair=n

# Supported values are "rsa", "dsa", "ecdsa", "ed25519", and "auto".
# The "auto" option is supported on OpenSSH 5.9 (2011) and later.
Provisioning.SshHostKeyPairType=rsa

# Monitor host name changes and publish changes via DHCP requests.
# CHANGED
#Provisioning.MonitorHostName=y
Provisioning.MonitorHostName=n

# Decode CustomData from Base64.
Provisioning.DecodeCustomData=n

# Execute CustomData after provisioning.
Provisioning.ExecuteCustomData=n

# Algorithm used by crypt when generating password hash.
#Provisioning.PasswordCryptId=6

# Length of random salt used when generating password hash.
#Provisioning.PasswordCryptSaltLength=10

# Format if unformatted. If 'n', resource disk will not be mounted.
ResourceDisk.Format=y

# File system on the resource disk
# Typically ext3 or ext4. FreeBSD images should use 'ufs' here.
ResourceDisk.Filesystem=ufs

# Mount point for the resource disk
ResourceDisk.MountPoint=/mnt/resource

# Create and use swapfile on resource disk.
# CHANGED
#ResourceDisk.EnableSwap=y
ResourceDisk.EnableSwap=n

# Size of the swapfile.
ResourceDisk.SwapSizeMB=16384

# Comma-separated list of mount options. See mount(8) for valid options.
ResourceDisk.MountOptions=None

# Enable verbose logging (y|n)
# CHANGED
#Logs.Verbose=y
Logs.Verbose=n

# Enable Console logging, default is y
# Logs.Console=y

# Is FIPS enabled
OS.EnableFIPS=n

# Root device timeout in seconds.
OS.RootDeviceScsiTimeout=300

# If "None", the system default version is used.
OS.OpensslPath=None

# Set the path to SSH keys and configuration files
OS.SshDir=/etc/ssh

OS.PasswordPath=/etc/master.passwd

OS.SudoersDir=/usr/local/etc/sudoers.d

# If set, agent will use proxy server to access internet
#HttpProxy.Host=None
#HttpProxy.Port=None

# Detect Scvmm environment, default is n
# DetectScvmmEnv=n

#
# Lib.Dir=/var/lib/waagent

#
# DVD.MountPoint=/mnt/cdrom/secure

#

#
# Pid.File=/var/run/waagent.pid

#
# Extension.LogDir=/var/log/azure

#
# Home.Dir=/home

# Enable RDMA management and set up, should only be used in HPC images
# OS.EnableRDMA=y

# Enable or disable goal state processing auto-update, default is enabled
# AutoUpdate.Enabled=y

# Determine the update family, this should not be changed
# AutoUpdate.GAFamily=Prod

# Determine if the overprovisioning feature is enabled. If yes, hold extension
# handling until inVMArtifactsProfile.OnHold is false.
# Default is enabled
# EnableOverProvisioning=y

# Allow fallback to HTTP if HTTPS is unavailable
# Note: Allowing HTTP (vs. HTTPS) may cause security risks
# OS.AllowHTTP=n

# Add firewall rules to protect access to Azure host node services
# SHIFT
#OS.EnableFirewall=y
OS.EnableFirewall=n

/var/log/waagent.log

[stderr]
No Python interpreter found, which is an OMS extension dependency. Please install either Python 2 or 3.
, duration=0
2023-04-23T19:41:16.615044Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Target handler state: enabled [etag_8616106893584706626]
2023-04-23T19:41:16.615663Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] [Enable] current handler state is: enabled
2023-04-23T19:41:16.615834Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Update settings file: 32.settings
2023-04-23T19:41:16.616118Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Requested extension state: enabled
2023-04-23T19:41:16.616521Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Enable extension: [main/handle.sh enable]
2023-04-23T19:41:16.617006Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Executing command: /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0/main/handle.sh enable with environment variables: {"AZURE_GUEST_AGENT_UNINSTALL_CMD_EXIT_CODE": "NOT_RUN", "AZURE_GUEST_AGENT_EXTENSION_PATH": "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0", "AZURE_GUEST_AGENT_EXTENSION_VERSION": "1.0.9197.0", "AZURE_GUEST_AGENT_WIRE_PROTOCOL_ADDRESS": "168.63.129.16", "ConfigSequenceNumber": "32", "AZURE_GUEST_AGENT_EXTENSION_SUPPORTED_FEATURES": "[{\"Key\": \"ExtensionTelemetryPipeline\", \"Value\": \"1.0\"}]"}
2023-04-23T19:41:22.734896Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Command: main/handle.sh enable
[stdout]
2023/04/23 21:41:16 ERROR:CalledProcessError.  Error Code is 2
2023/04/23 21:41:16 ERROR:CalledProcessError.  Command string was /usr/local/bin/base64 -d /tmp/tmpnq7t2kkp | /usr/bin/openssl smime  -inform DER -decrypt -recip /var/lib/waagent/1152F8ACD5DB9098C67A1CF132A8A9EEAEF8F326.crt  -inkey /var/lib/waagent/1152F8ACD5DB9098C67A1CF132A8A9EEAEF8F326.prv
2023/04/23 21:41:16 ERROR:CalledProcessError.  Command result was /bin/sh: /usr/local/bin/base64: not found
2023/04/23 21:41:16 ERROR:Error reading S/MIME message
2023/04/23 21:41:16 ERROR:13937147924480:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/crypto/asn1/asn1_lib.c:101:
2023/04/23 21:41:16 ERROR:[Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]JSON exception decoding /bin/sh: /usr/local/bin/base64: not found
2023/04/23 21:41:16 ERROR:Error reading S/MIME message
2023/04/23 21:41:16 ERROR:13937147924480:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/crypto/asn1/asn1_lib.c:101:
2023/04/23 21:41:16 ERROR:
[stderr]
2023-04-23T19:41:22.737482Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [etag_8616106893584706626 10411 ms]

2023-04-23T19:41:22.758160Z INFO ExtHandler ExtHandler Extension status: [('Microsoft.Azure.RecoveryServices.VMSnapshotLinux', 'success'), ('Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux', 'NotReady')]
2023-04-23T19:41:22.758641Z INFO ExtHandler ExtHandler All extensions in the goal state have reached a terminal state: [('Microsoft.Azure.RecoveryServices.VMSnapshotLinux', 'success'), ('Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux', 'NotReady')]

[TheBigBear]

@shift How did you get the backup to work?
Routing to 168.63.129.16 on LAN is in place and works for dns.
But the Azure metadata connections only work from 1st interface on the vm.
And backup, and azure virtual machine momnitor etc needs that working on WAN (1st interface) and does not work to LAN (2nd interface) .

[shift]

@thebigbear We did not work on routing. This should work out of the box. Our backup is working now. But the waagent is not in a perfect condition for freebsd as mentioned...