OSProvisioningTimedOut error when trying to deploy pfsense from Azure marketplace

ebrenton

I am trying to create a new Azure VM using "pfSense Plus Public Cloud Firewall/VPN/Router" from the Azure marketplace.

I've tried 3 times now and received the same error every time...

The virtual machine is in a failed state.
OS Provisioning for VM 'StackpolePfSenseFWRouter' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. For details on how to check current provisioning state of Windows VMs, refer to https://aka.ms/WindowsVMLifecycle and Linux VMs, refer to https://aka.ms/LinuxVMLifecycle.Learn more about common virtual machine error codes.

Details:
Error code: OSProvisioningTimedOut
Provisioning state: Failed
Provisioning state error code: ProvisioningState/failed/OSProvisioningTimedOut
Guest agent status: VM status blob is found but not yet populated.

I've waited over 2 hours and it is still unresponsive.
Azure portal shows that the VM is running, but I cannot connect via SSH or HTTP(s).
If I try to restart the VM it just hangs, then errors out.

Thank you.

stephenw10

What do you see on the virtual serial console? Or on a screenshot of the VM?

What machine size are you using?

Steve

stephenw10

Just tested using B1ms and it deployed fine:

Microsoft Azure - Netgate Device ID: 6a0453bc947ff4xxxxxx

*** Welcome to Netgate pfSense Plus 24.03-RELEASE (amd64) on pfSense-24.03-test-SW1 ***

 WAN (wan) -> hn0 -> v4/DHCP4: 10.1.7.17/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart GUI
 3) Reset admin account and password  12) PHP shell + Netgate pfSense Plus tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell                             

Enter an option:

It does show a bunch of output from the Azure agent at the console when you initially connect.

ebrenton

I cannot connect via SSH, it simply times out.
Port 22 IS open and I tried the private and public IP.
The size I'm using is DS1 v2. I don't think I even had B1ms as an option when I ran the deployment.

stephenw10

If you look at the serial console or diagnostic screenshot in Azure what does it show?

ebrenton

Here is the output from the serial console...

Welcome to Netgate pfSense Plus 24.03-RELEASE...

...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.36/mach/CORE
32-bit compatibility ldconfig path:
done.
3014
Config.xml is corrupted and is 0 bytes. Could not restore a previous backup.Launching the init system...Updating CPU Microcode...
CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz (2593.91-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x50657 Family=0x6 Model=0x55 Stepping=7
Features=0xf8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,SS>
Features2=0xfeda3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0xd09f2fb9<FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,AVX512CD,AVX512BW,AVX512VL>
Structured Extended Features3=0x400<MD_CLEAR>
XSAVE Features=0xb<XSAVEOPT,XSAVEC,XSAVES>
Hypervisor: Origin = "Microsoft Hv"
Done.
Config.xml is corrupted and is 0 bytes. Could not restore a previous backup.Starting CRON... done.
Config.xml is corrupted and is 0 bytes. Could not restore a previous backup.Netgate pfSense Plus 24.03-RELEASE amd64 20240418-2354
Bootup complete

FreeBSD/amd64 (Amnesiac) (ttyu0)

login: hn0: got notify, nvs type 128
pcib0: <Hyper-V PCI Express Pass Through> on vmbus0
pcib0: PCI VMBus using version 0x10004
pci0: <PCI bus> on pcib0
mlx5_core0: <mlx5_core> at device 2.0 on pci0
mlx5: Mellanox Core driver 3.7.1 (November 2021)mlx5_core0: WARN: mlx5_init_once:994:(pid 0): Unable to find vendor specific capabilities
mce0: Ethernet address: 60:45:bd:37:4b:11
mce0: link state changed to DOWN
hn0: link state changed to DOWN
mlx5_core0: WARN: mlx5_fwdump_prep:91:(pid 0): Unable to find vendor-specific capability, error 2
hvkvp0: detached
hvkvp0: <Hyper-V KVP> on vmbus0

stephenw10

Hmm, the mce NIC should not appear there. You are just launching the image in DS1 v2? Any specific zone? Anthing custom in the options?

ebrenton

This is in the US central zone, and no, no custom options.

Also, it will not let me log in with the credentials I entered during setup.
I AM able to get in with root, and when I list /etc/passwd, the username I specified during setup is not listed, nor is admin.

ebrenton

Here is the initial output from startup...

Loading /boot/loader.conf.local
Loading kernel...
/boot/kernel/kernel text=0x19eec0 text=0xff4c38 text=0x17e3db4 data=0x180 data=0x22d718+0x3d18e8 0x8+0x1cb0f0+0x8+0x1da290/
Loading configured modules...
can't find '/boot/entropy'
can't find '/etc/hostid'

Booting [/boot/kernel/kernel]...               
staging 0x36c00000-0x3aa7d000 (not copying) tramp 0x3aa7d000 PT4 0x3aa7e000
Start @ 0xffffffff8039f000 ...
EFI framebuffer information:
addr, size     0x40000000, 0x800000
dimensions     1024 x 768
stride         1024
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2024 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 15.0-CURRENT #0 plus-RELENG_24_03-n256311-e71f834dd81: Fri Apr 19 00:28:14 UTC 2024
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_03-main/obj/amd64/Y4MAEJ2R/var/jenkins/workspace/pfSense-Plus-snapshots-24_03-main/sources/FreeBSD-src-plus-RELENG_24_03/amd64.amd64/sys/pfSense amd64
FreeBSD clang version 17.0.6 (https://github.com/llvm/llvm-project.git llvmorg-17.0.6-0-g6009708b4367)
SRAT: Ignoring memory at addr 0x1a0200000
SRAT: Ignoring memory at addr 0x1000000000
SRAT: Ignoring memory at addr 0x10000200000
SRAT: Ignoring memory at addr 0x20000200000
SRAT: Ignoring memory at addr 0x40000200000
SRAT: Ignoring memory at addr 0x80000200000
VT(efifb): resolution 1024x768
Hyper-V Version: 10.0.20348 [SP1]
  Features0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz (2593.90-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x50657  Family=0x6  Model=0x55  Stepping=7
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfeda3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xd09f2fb9<FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,AVX512CD,AVX512BW,AVX512VL>
  Structured Extended Features3=0x400<MD_CLEAR>
  XSAVE Features=0xb<XSAVEOPT,XSAVEC,XSAVES>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 3758096384 (3584 MB)
avail memory = 3504218112 (3341 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VRTUAL MICROSFT>
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
ioapic0 <Version 1.1> irqs 0-23
TCP_ratelimit: Is now initialized
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80750310, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff807503c0, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80750470, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80770010, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff807700c0, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80770170, 0) error 1
ranom: entropy devicl policy registered
Tiounter "Hyper-V-T00000 Hz quality 3000
kbd0 at kbdmux0
WARNING: Device "spkr"  Giant locked and may be deleted e FreeBSD 15.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
netgate0: <Microsoft Azure>
smbios0: <System Management BIOS> at iomem 0x3ff83000-0x3ff83017
smbios0: Version: 3.1
acpi0: <VRTUAL MICROSFT>
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_syscontainer0: <System Container> on acpi0
vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
vmgenc0: <VM Generation Counter> on acpi0
vmbus_res0: <Hyper-V Vmbus Resource> irq 5 on acpi0
Timecounter "TSC-low" frequency 1296953099 Hz quality 800
Timecounters tick every 10.000 msec
vmbus0: version 4.0
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
hvkbd0: <Hyper-V KBD> on vmbus0
kbd1 at hvkbd0
hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
hvkvp0: <Hyper-V KVP> on vmbus0
hvshutdown0: <Hyper-V Shutdown> on vmbus0
hvtimesync0: <Hyper-V Timesync> on vmbus0
hvtimesync0: RTT
hn0: <Hyper-V Netrandom: unblocking device.
hn0: Ethernet address: 60:45:bd:37:4b:11
storvsc0: <Hyper-V SCSI> on vmbus0
hn0: link state cstorvsc1: <Hyper-V SCSI> on vmbus0
Trying to mount root from ufs:/dev/gpt/pfSense [rw]...
da0 at storvsc0 bus 0 scbus0 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 10241MB (20973568 512 byte sectors)
da1 at storvsc0 bus 0 scbus0 target 0 lun 1
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 7168MB (14680064 512 byte sectors)
WARNING: / was not properly dismounted
Configuring crash dumps...
No suitable dump device was found.
Growing root partition to fill device
da0 recovering is not needed
da0p3 resized
growfs: requested size 9.8GB is equal to the current filesystem size 9.8GB
** SU+J Recovering /dev/gpt/pfSense
** Reading 82247680 byte journal from inode 4.
** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.
** 365 journal records in 17920 bytes for 65.18% utilization
** Freed 20 inodes (4 dirs) 6 blocks, and 41 frags.
/dev/gpt/pfSense: 
**** FILE SYSTEM MARKED CLEAN ****
Filesystems are clean, continuing...
Mounting filesystems...

        __
 _ __  / _|___  ___ _ __  ___  ___      _
| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \   _| |_
| |_) |  _\__ \  __/ | | \__ \  __/  |_   _|
| .__/|_| |___/\___|_| |_|___/\___|    |_|
|_|


Welcome to Netgate pfSense Plus 24.03-RELEASE...

stephenw10

Hmm, I can't replicate that.

Try deploying it without 'accelerated networking' enabled if it was enabled previously. Though in testing here it deploys fine with or without it.

ebrenton

I was able to deploy the old fashioned way... download the iso, create a virtual disk and vm in Hyper-v Manager, and upload to Azure. The VM is up and running and I am configuring it.

I'm going to abandon the marketplace option for now, but thank you very much for all your help.

stephenw10

Hmm, what image from the Marketplace exactly were you testing?

ebrenton

"pfSense Plus Public Cloud Firewall/VPN/Router"