2.2.2 dead slow disk operations on esxi 5.5U2 (HP proliant DL 380e Gen8)
-
Hello.
After two upgrades, 2.2.0 -> 2.2.1 -> 2.2.2 I noticed that the upgrade from 2.2.1 to 2.2.2 took LOTS of hours to end. The cause was the "rm" command, used to delete the current packages for updating later, tooking a really huge amount of time to end.the "rm" commands stucked everytime on the BIOWR state.
After the upgrade finished the system continues with same problem. I have run the following tests that confirms some big performance problem!
Please advice!
ESXI has a local 4 disk RAID 10 configuration with 3TB of space available.
/root: diskinfo -c /dev/da0
/dev/da0
512 # sectorsize
8589934592 # mediasize in bytes (8.0G)
16777216 # mediasize in sectors
0 # stripesize
0 # stripeoffset
1044 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
# Disk ident.I/O command overhead:
time to read 10MB block 0.953807 sec = 0.047 msec/sector
time to read 20480 sectors 238.720495 sec = 11.656 msec/sector
calculated command overhead = 11.610 msec/sector/root: diskinfo -t /dev/da0
/dev/da0
512 # sectorsize
8589934592 # mediasize in bytes (8.0G)
16777216 # mediasize in sectors
0 # stripesize
0 # stripeoffset
1044 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
# Disk ident.Seek times:
Full stroke: 250 iter in 1.456670 sec = 5.827 msec
Half stroke: 250 iter in 1.771193 sec = 7.085 msec
Quarter stroke: 500 iter in 5.854107 sec = 11.708 msec
Short forward: 400 iter in 3.205245 sec = 8.013 msec
Short backward: 400 iter in 4.655065 sec = 11.638 msec
Seq outer: 2048 iter in 23.831172 sec = 11.636 msec
Seq inner: 2048 iter in 0.044004 sec = 0.021 msec
Transfer rates:
outside: 102400 kbytes in 9.538774 sec = 10735 kbytes/sec
middle: 102400 kbytes in 10.635576 sec = 9628 kbytes/sec
inside: 102400 kbytes in 0.068144 sec = 1502700 kbytes/sec -
Is Vm Tools loaded and running??
Run "kldstat" from the CLI and report.
-
Hi,
I dont know why but kldstat never loads the vmtools modules/root: kldstat
Id Refs Address Size Name
1 1 0xffffffff80200000 22d0f80 kernelI have the most recent form the packages menu
-
Yes and that could be your problem.
uninstall the tools sine they dont work and disable your serial port and reboot.
Report back.
-
The tools packages has been broken of late
For comparisons here is those tests on my esxi box.. Nothing fancy just single disk
[2.2.2-RELEASE][root@pfSense.local.lan]/root: diskinfo -c /dev/da0
/dev/da0
512 # sectorsize
8589934592 # mediasize in bytes (8.0G)
16777216 # mediasize in sectors
0 # stripesize
0 # stripeoffset
1044 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
# Disk ident.I/O command overhead:
time to read 10MB block 0.104814 sec = 0.005 msec/sector
time to read 20480 sectors 10.542941 sec = 0.515 msec/sector
calculated command overhead = 0.510 msec/sector[2.2.2-RELEASE][root@pfSense.local.lan]/root: diskinfo -t /dev/da0
/dev/da0
512 # sectorsize
8589934592 # mediasize in bytes (8.0G)
16777216 # mediasize in sectors
0 # stripesize
0 # stripeoffset
1044 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
# Disk ident.Seek times:
Full stroke: 250 iter in 0.258565 sec = 1.034 msec
Half stroke: 250 iter in 0.134784 sec = 0.539 msec
Quarter stroke: 500 iter in 0.164177 sec = 0.328 msec
Short forward: 400 iter in 0.162630 sec = 0.407 msec
Short backward: 400 iter in 0.144883 sec = 0.362 msec
Seq outer: 2048 iter in 1.035187 sec = 0.505 msec
Seq inner: 2048 iter in 0.710852 sec = 0.347 msec
Transfer rates:
outside: 102400 kbytes in 1.325136 sec = 77275 kbytes/sec
middle: 102400 kbytes in 0.681892 sec = 150170 kbytes/sec
inside: 102400 kbytes in 0.464460 sec = 220471 kbytes/sec[2.2.2-RELEASE][root@pfSense.local.lan]/root: kldstat
Id Refs Address Size Name
1 10 0xffffffff80200000 22d0f80 kernel
2 1 0xffffffff824d1000 8160 vmblock.ko
3 1 0xffffffff824da000 13680 vmhgfs.ko
4 1 0xffffffff824ee000 5630 vmxnet.ko
5 1 0xffffffff82611000 22f7 vmmemctl.ko
[2.2.2-RELEASE][root@pfSense.local.lan]/root:Mine is on esxi 6 on a HP N40L
-
I will do that right now.
Brian what is that "serial port" you are asking for removal? -
same problem….. :( now without vmtools
johnpoz nice values... I wish the same. -
johnpoz you used the packages menu to install the vmtools? My get installed ok but the modules never get loaded. ESXI detects a thirdparty vmtools installed
-
shoul I try this?
install OpenVM Tools from FreeBSD ports via pfsense console itself:pkg install open-vm-tools-nox11
-
In a VM a serial console could be attached to the VM
Pls. remove if you have it attached and reboot.
I will do that right now.
Brian what is that "serial port" you are asking for removal? -
I think John reported that it was still _9 installing despite claiming *_11
shoul I try this?
install OpenVM Tools from FreeBSD ports via pfsense console itself:pkg install open-vm-tools-nox11
-
Yeah its still installing _9 even though it should _12 from the latest package.
I would not install the ports from bsd, it installs a HUGE amount of other packages. But on my clean 2.2.2 64 bit test pfsense vm the package installed fine - before it was hanging and never finishing. And scripts seem to work, I was able to shutdown guest and pfsense did a clean shutdown, etc.
-
already tried vmtools via "nox" and via ESXI install CD and the results are always the same.
The firewall is working ok minus this disk problem…. I think I will install an 2.2.0 version and see what values I will get on disk access. If its ok I no longer will upgrade pfsense until something or someone finds this and fixes... :(
But first will try a clean 2.2.2 install, maibe I am luck with that
Thank you everybody
