How to install the Azure Linux Agent (waagent) in the new versions of PfSense
-
@jamiegb I'm also struggling with this issue, tried editing rc.conf.local with waagent_enable=yes and that doesn't work.
Also tried sysrc waagent_enable=yes
That doesn't work either.
If anyone has any other suggestions would be welcome
-
@jamiegb Got this working with the below
ln -sf /usr/local/sbin/waagent /usr/sbin/waagent
ln -sf /usr/local/sbin/waagent2.0 /usr/sbin/waagent2.0
echo '#! /bin/sh' >> /usr/local/etc/rc.d/waagent.sh
echo '/usr/local/sbin/waagent --daemon' >> /usr/local/etc/rc.d/waagent.sh
chmod +x /usr/local/etc/rc.d/waagent.sh
echo "y" | /usr/local/sbin/waagent -deprovision+user
echo 'waagent_enable="YES"' >> /etc/rc.conf.local -
@mark-dayton said in How to install the Azure Linux Agent (waagent) in the new versions of PfSense:
@jamiegb Got this working with the below
ln -sf /usr/local/sbin/waagent /usr/sbin/waagent
ln -sf /usr/local/sbin/waagent2.0 /usr/sbin/waagent2.0
echo '#! /bin/sh' >> /usr/local/etc/rc.d/waagent.sh
echo '/usr/local/sbin/waagent --daemon' >> /usr/local/etc/rc.d/waagent.sh
chmod +x /usr/local/etc/rc.d/waagent.sh
echo "y" | /usr/local/sbin/waagent -deprovision+user
echo 'waagent_enable="YES"' >> /etc/rc.conf.localMark that is some fantastic work. I have no doubt your solution is going to help many people! Can't thank you enough!!!
-
@jamiegb I can't take credit for this unfortunately, found this in another post on this forum! Tweaked slightly for rc.conf.local but it works :)
It takes a little time to start in Azure but afterwards im able to access the backend nic just fine. -
@mark-dayton said in How to install the Azure Linux Agent (waagent) in the new versions of PfSense:
waagent
So I tried this out. Unfortunately no success. But I see the commands are creating aliases for both waagent 1 and 2, yet the script being called with the --daemon parameter seems to be version 1. BSD needs version 2 of waagent so I'll persevere and report back my findings.
I wish I could just access the console, SSH or web interface to see what;;'s going on, but it's totally offline. When I made the image I set both interfaces to DHCP but deployed the VM with 2 NICs, one with a public IP and both with static internal IPs. Maybe this is stopping the network stack loading up....
-
@jamiegb Okay so I loosely followed this article here https://www.christofvg.be/2019/01/12/pfSense-on-Azure-Part-1-Create-pfSense-Virtual-Machine/
But when getting to the point of installing the agent i used the following commands, note python 3.8 for latest pfsense version 2.5.2.
pkg upgrade
pkg install -y sudo bash git
ln -s /usr/local/bin/python3.8 /usr/local/bin/pythonClone the Git repository
git clone https://github.com/Azure/WALinuxAgent.git
Enter the WALinuxAgent directory
cd WALinuxAgent
List all available versions
git tag
Checkout the latest (stable) version of the agent
git checkout v2.4.0.2
Install the agent
python setup.py install
ln -sf /usr/local/sbin/waagent /usr/sbin/waagent
ln -sf /usr/local/sbin/waagent2.0 /usr/sbin/waagent2.0
echo '#! /bin/sh' >> /usr/local/etc/rc.d/waagent.sh
echo '/usr/local/sbin/waagent --daemon' >> /usr/local/etc/rc.d/waagent.sh
chmod +x /usr/local/etc/rc.d/waagent.sh
echo "y" | /usr/local/sbin/waagent -deprovision+user
echo 'waagent_enable="YES"' >> /etc/rc.conf.localYou can check whether it's running before deprovision by running service waagent status.
I deployed the image with both NICs DHCP and also left them DHCP in Azure. You may have to wait a few minutes after booting the VM to access the LAN NIC on either http or https whichever you have configured.
Below are the commands i used for creating the VM
Initialize variables
$storageType = "Standard_LRS"
$location = "UK South"$storageAccountId = "xxxxxxxxxxxxxxxxxxxxx"
$sourceVhdUri = "https://xxxxxxxxxxxxxx.blob.core.windows.net/temp/AZ-PFSENSE01.vhd"Create the disk configuration
$diskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $location -CreateOption Import -StorageAccountId $storageAccountId -SourceUri $sourceVhdUri
Create the Managed Disk
New-AzureRmDisk -Disk $diskConfig -ResourceGroupName <resource group> -DiskName az-pfsense_disk1
Get the object of the existing Managed Disk
$disk = Get-AzureRmDisk -DiskName az-pfsense_disk1 -ResourceGroupName <resource group>
Get the object for the existing Virtual Network
$VirtualNetwork = Get-AzureRmVirtualNetwork -Name <vnet> -ResourceGroupName <resource group>
Create a new Virtual Machine object
$virtualMachine = New-AzureRmVMConfig -VMName az-pfsense01 -VMSize Standard_B2s
Attach the existing Managed Disk to the Virtual Machine
$virtualMachine = Set-AzureRmVMOSDisk -VM $virtualMachine -ManagedDiskId $disk.Id -CreateOption Attach -Linux
#Create pub IP
$pubip = New-AzureRmPublicIpAddress -Name "az-pfsense01-pip1" -ResourceGroupName $pfresourcegroup -Location $location -AllocationMethod DynamicCreate the NIC's for the frontend and the backend, note 7 is the subnet ID yours maybe different
$frontEndNic = New-AzureRmNetworkInterface -Name az-pfsense01-frontend-nic -ResourceGroupName <resource group> -Location 'UK South' -SubnetId $VirtualNetwork.Subnets[7].Id
$backEndNic = New-AzureRmNetworkInterface -Name az-pfsense01-backend-nic -ResourceGroupName <resource group> -Location 'UK South' -SubnetId $VirtualNetwork.Subnets[7].Id -PublicIpAddressId $az-pfsense01-pip1.Id
Add the NIC's to the Virtual Machine
$virtualMachine = Add-AzureRmVMNetworkInterface -VM $virtualMachine -Id $frontEndNic.Id -Primary
$virtualMachine = Add-AzureRmVMNetworkInterface -VM $virtualMachine -Id $backEndNic.IdCreate the Virtual Machine
New-AzureRmVM -VM $virtualMachine -ResourceGroupName <resource group> -Location 'UK South'
-
Deployed pfsense community edition on azure and tried to get this waagent up with above guide.
WALinuxAgent-2.2.39 running on freebsd 12.2
Python: 3.8.10
Goal state agent: 2.2.39Agent seems to start ok, but there is something wrong with configuration as it is not able to communicatate properly to azure. It is trying to communicate on azure 168.63.129.16 but it fails. Any idea how to troubleshoot or fix?
log:
[2.5.2-RELEASE][admin@x]/root: cat /var/log/waagent.log 2022/01/20 12:39:08.073418 INFO Daemon Azure Linux Agent Version:2.2.39 2022/01/20 12:39:08.074016 INFO Daemon OS: freebsd 12.2 2022/01/20 12:39:08.074137 INFO Daemon Python: 3.8.10 2022/01/20 12:39:08.074266 VERBOSE Daemon Command: [ps -p 69164] 2022/01/20 12:39:08.081280 INFO Daemon CGroups: not supported by platform 2022/01/20 12:39:08.082525 WARNING Daemon Too many files under: /var/lib/waagent/events, removing oldest 2022/01/20 12:39:08.083576 WARNING Daemon Too many files under: /var/lib/waagent/events, removing oldest 2022/01/20 12:39:08.083847 INFO Daemon Run daemon 2022/01/20 12:39:08.084029 INFO Daemon No RDMA handler exists for distro='freebsd' version='12.2' 2022/01/20 12:39:08.084164 INFO Daemon Activate resource disk 2022/01/20 12:39:08.084255 VERBOSE Daemon Command: [gpart list] 2022/01/20 12:39:08.088782 VERBOSE Daemon Command: [sysctl dev.storvsc | grep pnpinfo | grep deviceid=] 2022/01/20 12:39:08.093447 VERBOSE Daemon Command: [sysctl dev.storvsc | grep pnpinfo | grep deviceid=00000000-0001] 2022/01/20 12:39:08.097994 VERBOSE Daemon Command: [sysctl dev.storvsc | grep pnpinfo | grep deviceid=00000000-0001|awk -F . '{print $3}'] 2022/01/20 12:39:08.103197 VERBOSE Daemon Command: [camcontrol devlist -b | grep blkvsc1 | awk '{print $1}'] 2022/01/20 12:39:08.107349 VERBOSE Daemon Command: [camcontrol devlist | grep scbus3 | awk -F \( '{print $2}'|sed -e 's/.*(//'| sed -e 's/).*//'] 2022/01/20 12:39:08.113020 INFO Daemon Resource disk device da1 found. 2022/01/20 12:39:08.113227 VERBOSE Daemon Command: [gpart show -p da1] 2022/01/20 12:39:08.116966 INFO Daemon Resource disk partition /dev/da1s1 found. 2022/01/20 12:39:08.117168 VERBOSE Daemon Command: [mount] 2022/01/20 12:39:08.119764 INFO Daemon Resource disk /dev/da1s1 is already mounted 2022/01/20 12:39:08.120058 INFO Daemon Clean protocol 2022/01/20 12:39:08.120283 INFO Daemon Running default provisioning handler 2022/01/20 12:39:08.120421 INFO Daemon Copying ovf-env.xml 2022/01/20 12:39:08.121037 ERROR Daemon Provisioning failed: [ProtocolError] [CopyOvfEnv] Error mounting dvd: [OSUtilError] Failed to get dvd device from /dev Inner error: The following devices were found, but none matched the pattern [(sr[0-9]|hd[c-z]|cdrom[0-9]|cd[0-9])]: ['dumpdev', 'reroot', 'devctl', 'devctl2', 'console', 'sndstat', 'hv_tsc', 'random', 'urandom', 'geom.ctl', 'pci', 'fd', 'stdin', 'stdout', 'stderr', 'uinput', 'nfslock', 'midistat', 'sequencer0', 'music0', 'tcp_log', 'auditpipe', 'audit', 'klog', 'ctty', 'consolectl', 'sysmouse', 'input', 'fido', 'io', 'kbdmux0', 'kbd1', 'bpf', 'bpf0', 'speaker', 'mem', 'kmem', 'netmap', 'full', 'null', 'zero', 'acpi', 'apmctl', 'apm', 'atkbd0', 'kbd0', 'psm0', 'bpsm0', 'ttyu0', 'ttyu0.init', 'ttyu0.lock', 'cuau0', 'cuau0.init', 'cuau0.lock', 'ttyu1', 'ttyu1.init', 'ttyu1.lock', 'cuau1', 'cuau1.init', 'cuau1.lock', 'ufssuspend', 'zfs', 'netdump', 'pf', 'usbctl', 'xpt0', 'mdctl', 'devstat', 'hv_kvp_dev', 'ttyv0', 'ttyv1', 'ttyv2', 'ttyv3', 'ttyv4', 'ttyv5', 'ttyv6', 'ttyv7', 'ttyv8', 'ttyv9', 'ttyva', 'ttyvb', 'mlx5ctl', 'fd0', 'pass0', 'pass1', 'da0', 'da1', 'da0p1', 'da0p2', 'da0p3', 'da1s1', 'md0', 'gpt', 'cpuctl0', 'tun1', 'ovpns1', 'pts'] (0s) 2022/01/20 12:39:08.121266 INFO Daemon Detect protocol endpoints 2022/01/20 12:39:08.121370 INFO Daemon Clean protocol 2022/01/20 12:39:08.121472 INFO Daemon WireServer endpoint is not found. Rerun dhcp handler 2022/01/20 12:39:08.121560 INFO Daemon Test for route to 168.63.129.16 2022/01/20 12:39:08.121644 WARNING Daemon No route exists to 168.63.129.16 2022/01/20 12:39:08.121720 INFO Daemon Checking for dhcp lease cache 2022/01/20 12:39:08.121803 INFO Daemon Cache exists [False] 2022/01/20 12:39:08.121874 INFO Daemon Send dhcp request 2022/01/20 12:39:08.121958 VERBOSE Daemon Command: [ifconfig -l ether] 2022/01/20 12:39:08.124464 VERBOSE Daemon Command: [ifconfig hn0] 2022/01/20 12:39:08.128119 VERBOSE Daemon Interface info: (hn0,10.0.4.5,00:22:48:9e:5c:a6) 2022/01/20 12:39:08.128332 VERBOSE Daemon Command: [ifconfig -l ether] 2022/01/20 12:39:08.130849 VERBOSE Daemon Command: [ifconfig hn0] 2022/01/20 12:39:08.134468 VERBOSE Daemon Interface info: (hn0,10.0.4.5,00:22:48:9e:5c:a6) 2022/01/20 12:39:08.134785 VERBOSE Daemon BuildDhcpRequest: transactionId:A84FB5EC,A84FB5EC 2022/01/20 12:39:08.134926 VERBOSE Daemon Command: [ifconfig -l ether] 2022/01/20 12:39:08.137423 VERBOSE Daemon Command: [ifconfig hn0] 2022/01/20 12:39:08.141030 VERBOSE Daemon Interface info: (hn0,10.0.4.5,00:22:48:9e:5c:a6) 2022/01/20 12:39:08.141252 VERBOSE Daemon Command: [route add 255.255.255.255 -iface hn0] 2022/01/20 12:39:08.143829 VERBOSE Daemon Send DHCP request: Setting socket.timeout=10, entering recv 2022/01/20 12:39:18.187653 WARNING Daemon Failed to send DHCP request: [DhcpError] timed out 2022/01/20 12:39:18.188396 VERBOSE Daemon Send DHCP request: Setting socket.timeout=10, entering recv 2022/01/20 12:39:28.287475 WARNING Daemon Failed to send DHCP request: [DhcpError] timed out 2022/01/20 12:39:38.387588 VERBOSE Daemon Send DHCP request: Setting socket.timeout=10, entering recv 2022/01/20 12:39:48.482816 WARNING Daemon Failed to send DHCP request: [DhcpError] timed out
Routing table has these entries related to that ip:
168.63.129.16 00:22:48:9e:5c:a6 UHS hn0
168.63.129.16/32 10.0.4.1 UGS hn0 -
@tomih I have the same configuration, I did the same thing and the waagent still can't communicate with Azure, so the backups give error because the agent is not active. Has anyone here setup it correctly?
-
@alejjime We did it with above method for the current agent version:
waagent --version WALinuxAgent-2.9.0.4 running on freebsd 14.0 Python: 3.9.16 Goal state agent: 2.9.0.4
The Azure backup seems to work, but in /var/log/waagent.log some errors appear. That does not feel good. waagent tries to exec iproute and base64 which is not freebsd. We need a specific configuration for Freebsd/pfsense.
We have pfsense+ expecting to get a well integrated pfSense from netgate.
/etc/waagent.conf
# # Microsoft Azure Linux Agent Configuration # # Enable extension handling. Do not disable this unless you do not need password reset, # backup, monitoring, or any extension handling whatsoever. Extensions.Enabled=y # Which provisioning agent to use. Supported values are "auto" (default), "waagent", # "cloud-init", or "disabled". # CHANGED #Provisioning.Agent=auto Provisioning.Agent=waagent # Password authentication for root account will be unavailable. # CHANGED #Provisioning.DeleteRootPassword=y Provisioning.DeleteRootPassword=n # Generate fresh host key pair. # CHANGED #Provisioning.RegenerateSshHostKeyPair=y Provisioning.RegenerateSshHostKeyPair=n # Supported values are "rsa", "dsa", "ecdsa", "ed25519", and "auto". # The "auto" option is supported on OpenSSH 5.9 (2011) and later. Provisioning.SshHostKeyPairType=rsa # Monitor host name changes and publish changes via DHCP requests. # CHANGED #Provisioning.MonitorHostName=y Provisioning.MonitorHostName=n # Decode CustomData from Base64. Provisioning.DecodeCustomData=n # Execute CustomData after provisioning. Provisioning.ExecuteCustomData=n # Algorithm used by crypt when generating password hash. #Provisioning.PasswordCryptId=6 # Length of random salt used when generating password hash. #Provisioning.PasswordCryptSaltLength=10 # Format if unformatted. If 'n', resource disk will not be mounted. ResourceDisk.Format=y # File system on the resource disk # Typically ext3 or ext4. FreeBSD images should use 'ufs' here. ResourceDisk.Filesystem=ufs # Mount point for the resource disk ResourceDisk.MountPoint=/mnt/resource # Create and use swapfile on resource disk. # CHANGED #ResourceDisk.EnableSwap=y ResourceDisk.EnableSwap=n # Size of the swapfile. ResourceDisk.SwapSizeMB=16384 # Comma-separated list of mount options. See mount(8) for valid options. ResourceDisk.MountOptions=None # Enable verbose logging (y|n) # CHANGED #Logs.Verbose=y Logs.Verbose=n # Enable Console logging, default is y # Logs.Console=y # Is FIPS enabled OS.EnableFIPS=n # Root device timeout in seconds. OS.RootDeviceScsiTimeout=300 # If "None", the system default version is used. OS.OpensslPath=None # Set the path to SSH keys and configuration files OS.SshDir=/etc/ssh OS.PasswordPath=/etc/master.passwd OS.SudoersDir=/usr/local/etc/sudoers.d # If set, agent will use proxy server to access internet #HttpProxy.Host=None #HttpProxy.Port=None # Detect Scvmm environment, default is n # DetectScvmmEnv=n # # Lib.Dir=/var/lib/waagent # # DVD.MountPoint=/mnt/cdrom/secure # # # Pid.File=/var/run/waagent.pid # # Extension.LogDir=/var/log/azure # # Home.Dir=/home # Enable RDMA management and set up, should only be used in HPC images # OS.EnableRDMA=y # Enable or disable goal state processing auto-update, default is enabled # AutoUpdate.Enabled=y # Determine the update family, this should not be changed # AutoUpdate.GAFamily=Prod # Determine if the overprovisioning feature is enabled. If yes, hold extension # handling until inVMArtifactsProfile.OnHold is false. # Default is enabled # EnableOverProvisioning=y # Allow fallback to HTTP if HTTPS is unavailable # Note: Allowing HTTP (vs. HTTPS) may cause security risks # OS.AllowHTTP=n # Add firewall rules to protect access to Azure host node services # SHIFT #OS.EnableFirewall=y OS.EnableFirewall=n
/var/log/waagent.log
[stderr] No Python interpreter found, which is an OMS extension dependency. Please install either Python 2 or 3. , duration=0 2023-04-23T19:41:16.615044Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Target handler state: enabled [etag_8616106893584706626] 2023-04-23T19:41:16.615663Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] [Enable] current handler state is: enabled 2023-04-23T19:41:16.615834Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Update settings file: 32.settings 2023-04-23T19:41:16.616118Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Requested extension state: enabled 2023-04-23T19:41:16.616521Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Enable extension: [main/handle.sh enable] 2023-04-23T19:41:16.617006Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Executing command: /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0/main/handle.sh enable with environment variables: {"AZURE_GUEST_AGENT_UNINSTALL_CMD_EXIT_CODE": "NOT_RUN", "AZURE_GUEST_AGENT_EXTENSION_PATH": "/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0", "AZURE_GUEST_AGENT_EXTENSION_VERSION": "1.0.9197.0", "AZURE_GUEST_AGENT_WIRE_PROTOCOL_ADDRESS": "168.63.129.16", "ConfigSequenceNumber": "32", "AZURE_GUEST_AGENT_EXTENSION_SUPPORTED_FEATURES": "[{\"Key\": \"ExtensionTelemetryPipeline\", \"Value\": \"1.0\"}]"} 2023-04-23T19:41:22.734896Z INFO ExtHandler [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9197.0] Command: main/handle.sh enable [stdout] 2023/04/23 21:41:16 ERROR:CalledProcessError. Error Code is 2 2023/04/23 21:41:16 ERROR:CalledProcessError. Command string was /usr/local/bin/base64 -d /tmp/tmpnq7t2kkp | /usr/bin/openssl smime -inform DER -decrypt -recip /var/lib/waagent/1152F8ACD5DB9098C67A1CF132A8A9EEAEF8F326.crt -inkey /var/lib/waagent/1152F8ACD5DB9098C67A1CF132A8A9EEAEF8F326.prv 2023/04/23 21:41:16 ERROR:CalledProcessError. Command result was /bin/sh: /usr/local/bin/base64: not found 2023/04/23 21:41:16 ERROR:Error reading S/MIME message 2023/04/23 21:41:16 ERROR:13937147924480:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/crypto/asn1/asn1_lib.c:101: 2023/04/23 21:41:16 ERROR:[Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]JSON exception decoding /bin/sh: /usr/local/bin/base64: not found 2023/04/23 21:41:16 ERROR:Error reading S/MIME message 2023/04/23 21:41:16 ERROR:13937147924480:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/crypto/asn1/asn1_lib.c:101: 2023/04/23 21:41:16 ERROR: [stderr] 2023-04-23T19:41:22.737482Z INFO ExtHandler ExtHandler ProcessExtensionsGoalState completed [etag_8616106893584706626 10411 ms] 2023-04-23T19:41:22.758160Z INFO ExtHandler ExtHandler Extension status: [('Microsoft.Azure.RecoveryServices.VMSnapshotLinux', 'success'), ('Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux', 'NotReady')] 2023-04-23T19:41:22.758641Z INFO ExtHandler ExtHandler All extensions in the goal state have reached a terminal state: [('Microsoft.Azure.RecoveryServices.VMSnapshotLinux', 'success'), ('Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux', 'NotReady')]
-
@shift How did you get the backup to work?
Routing to 168.63.129.16 on LAN is in place and works for dns.
But the Azure metadata connections only work from 1st interface on the vm.
And backup, and azure virtual machine momnitor etc needs that working on WAN (1st interface) and does not work to LAN (2nd interface) . -
@thebigbear We did not work on routing. This should work out of the box. Our backup is working now. But the waagent is not in a perfect condition for freebsd as mentioned...