2.4.5 New Install Slow to boot on Hyper-V 2019

IsaacFL

I did a new install of 2.4.5 on Hyper-V 2019 Core and loaded my config.

It seems to work but during boot it hits a place where it says "Mounting filesystems. . ." and it hangs for about 30 seconds before continuing. Total Boot time is about 2.5 minutes.

An upgraded 2.4.4 to 2.4.5 system does not have this delay.

I rebooted with a file check and took a picture.

It is right before the "random: unblocking device" that there is a 30 second hang.

Mats

are you using UFS or ZFS ?

Just wanna ask so that I might be able to try to repro the issue

IsaacFL

@Mats UFS

I think I just did the defaults during install.

XBH

I'm observing the same or actually worse.
New installation HyperV 2019, with NVMe SSD, quad core Intel CPU.
VM generation 2, 8 vCPU, 8GB RAM, Default Auto config (I understand UFS file system), version 2.4.5
Installation is generally slow, reboot is slow.

But the worst is, when I save any configuration changes through the web UI, the CPU jumps >90% and stays this way sometimes for few minutes.
During this time web UI times out and the traffic is not going through.
This is on clean, just installed system, no packages.

I've downloaded 2.4.4 from some mirror and installed as above. Few configuration saves, with no issues.
Upgraded to 2.4.5, first reboot took 13 minutes most of the time >70% CPU before it was functional.

Problem with saving changes is back!

Installed 2.4.5 with ZFS (default settings), same issues.

Com DAC

I've got a basic install with no packages and noticed this too. It takes much longer to boot than 2.4.4 and configuration changes take quite a while to save. The last change I made was just to switch to dark theme and even that caused the box to freeze up for a bit stopping traffic flow.

It appears to only be my VM on Server 2019 as my other VM on Windows 10 (1909) doesn't seem to have this issue (or it isn't noticeable). Both systems have lots of resources and are on SSD's with pfSense setup using UFS.

Gertjan

@Com-DAC said in 2.4.5 New Install Slow to boot on Hyper-V 2019:

Windows 10 (1909)

Thanks for detailing the OS used.
I'm using 1909 Windows 10 Pro (only Pro has the Hyper thing I guess) using SSD.
I allocated it 500 Mbytes space and just 1 core. UFS also.

pfSense enables Internet access through my AP even before I can login into Windows This AP hooked up to one of the NIC's reserved strictly for the VM/pfSense - and that is in less ten 15 seconds (thanks SSD !) after a system (device !) boot.

edit : https://forum.netgate.com/topic/149595/2-4-5-a-20200110-1421-and-earlier-high-cpu-usage-from-pfctl/18 - and one more for one core !

Com DAC

I've got 2 cores assigned to each pfSense VM. The Server 2019 instance has 1GB ram and the Windows 10 instance has 512MB ram.

I'm also using the classic scheduler on Server 2019 as I trust all the VM's I'm running and I got better performance from pfSense using that scheduler.
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-scheduler-types

I'm wondering if this is a scheduler issue as Server 2019 does default to a different scheduler than previous server versions and Windows 10 uses the root scheduler which is quite different than the server defaults.

XBH

@Com-DAC Possible, I've just tried 2.4.5 on 2019 with Classic scheduler. Seems to be better. I'll test more next week.

Mats

i just did a fresh install on hyper-v server 2019. (I5 2405s 32 gb ram and an ssd
Standard install with ufs

takes 30-35s to mount filesystems

IsaacFL

@Mats said in 2.4.5 New Install Slow to boot on Hyper-V 2019:

i just did a fresh install on hyper-v server 2019. (I5 2405s 32 gb ram and an ssd
Standard install with ufs

takes 30-35s to mount filesystems

that was about the time I was seeing.

Com DAC

I just did another test and had two browser windows open. In one window I was watching the Diganostics / System Activity page and in the other window I just went to System / General Setup changed nothing and clicked save. In system activity I saw this:

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
31257 root        90    0  8828K  4832K CPU1    1   0:10  63.96% /sbin/pfctl -o basic -f /tmp/rules.debug
98023 root        52    0 10288K  5724K select  1   0:09  58.98% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid

During this time my pings went way up until the save completed (at least no packet loss happened). It only lasted about 4 seconds but I don't recall this happening when making changes in 2.4.4.

cuco

@Com-DAC Your problem sounds like this issue: #10414

Workaround: #901871

IsaacFL

I did a test VM install of just vanilla FreeBSD 11.3 with the same parameters as my pfsense install, and the FreeBSD VMdoes not seem to have any of the issues at boot that the pfsense has.

So not sure what to do after this?

mor

1 core work-around in https://forum.netgate.com/topic/149595/2-4-5-a-20200110-1421-and-earlier-high-cpu-usage-from-pfctl/17# resolved here;
On-top of slow very slow boot, I also found dropping back to 1 CPU also resolved an issue of system dropping offline post any change of network settings or FW rules.

XBH

HyperV scheduler change does seem to help much. It behaves inconsistently. I have 2 VMs on 2 hosts both changed to Classic scheduler. One is fine. Second one has the issue.

carl2187

I see this issue too (lockups, no network for long duration's randomly, overall slowness) on my server 2016 datacenter hyper-v pfsense vms, ~~but not on my windows 10 1909 x64 hyper-v pfsense vms with 2.4.5.~~

doing more tests, reloading the firewall filter does cause the issue on windows 10 1909 hyper-v as well. vm config: gen 2, zfs, 6GB ram, 4x cpu cores assigned.

gonna try the same vm in virtual box and see if the issue exists there too. at this point i've seen the issue on 2016 server hyper-v and windows 10 1909.