2.2.2 dead slow disk operations on esxi 5.5U2 (HP proliant DL 380e Gen8)



  • Hello.
    After two upgrades, 2.2.0 -> 2.2.1 -> 2.2.2  I noticed that the upgrade from 2.2.1 to 2.2.2 took LOTS of hours to end. The cause was the "rm" command, used to delete the current packages for updating later, tooking a really huge amount of time to end.

    the "rm" commands stucked everytime on the BIOWR state.

    After the upgrade finished the system continues with same problem. I have run the following tests that confirms some big performance problem!

    Please advice!

    ESXI has a local 4 disk RAID 10 configuration with 3TB of space available.

    /root: diskinfo -c /dev/da0
    /dev/da0
    512        # sectorsize
    8589934592  # mediasize in bytes (8.0G)
    16777216    # mediasize in sectors
    0          # stripesize
    0          # stripeoffset
    1044        # Cylinders according to firmware.
    255        # Heads according to firmware.
    63          # Sectors according to firmware.
                # Disk ident.

    I/O command overhead:
    time to read 10MB block      0.953807 sec =    0.047 msec/sector
    time to read 20480 sectors 238.720495 sec =  11.656 msec/sector
    calculated command overhead =  11.610 msec/sector

    /root: diskinfo -t /dev/da0
    /dev/da0
    512        # sectorsize
    8589934592  # mediasize in bytes (8.0G)
    16777216    # mediasize in sectors
    0          # stripesize
    0          # stripeoffset
    1044        # Cylinders according to firmware.
    255        # Heads according to firmware.
    63          # Sectors according to firmware.
                # Disk ident.

    Seek times:
    Full stroke:   250 iter in  1.456670 sec =    5.827 msec
    Half stroke:   250 iter in  1.771193 sec =    7.085 msec
    Quarter stroke:   500 iter in  5.854107 sec =  11.708 msec
    Short forward:   400 iter in  3.205245 sec =    8.013 msec
    Short backward:   400 iter in  4.655065 sec =  11.638 msec
    Seq outer: 2048 iter in  23.831172 sec =  11.636 msec
    Seq inner: 2048 iter in  0.044004 sec =    0.021 msec
    Transfer rates:
    outside:      102400 kbytes in  9.538774 sec =    10735 kbytes/sec
    middle:        102400 kbytes in  10.635576 sec =    9628 kbytes/sec
    inside:        102400 kbytes in  0.068144 sec =  1502700 kbytes/sec


  • Banned

    Is Vm Tools loaded and running??

    Run "kldstat" from the CLI and report.



  • Hi,
    I dont know why but kldstat never loads the vmtools modules

    /root: kldstat
    Id Refs Address            Size    Name
    1    1 0xffffffff80200000 22d0f80  kernel

    I have the most recent form the packages menu


  • Banned

    Yes and that could be your problem.

    uninstall the tools sine they dont work and disable your serial port and reboot.

    Report back.


  • LAYER 8 Global Moderator

    The tools packages has been broken of late

    For comparisons here is those tests on my esxi box.. Nothing fancy just single disk

    [2.2.2-RELEASE][root@pfSense.local.lan]/root: diskinfo -c /dev/da0
    /dev/da0
            512            # sectorsize
            8589934592      # mediasize in bytes (8.0G)
            16777216        # mediasize in sectors
            0              # stripesize
            0              # stripeoffset
            1044            # Cylinders according to firmware.
            255            # Heads according to firmware.
            63              # Sectors according to firmware.
                            # Disk ident.

    I/O command overhead:
            time to read 10MB block      0.104814 sec      =    0.005 msec/sector
            time to read 20480 sectors  10.542941 sec      =    0.515 msec/sector
            calculated command overhead                    =    0.510 msec/sector

    [2.2.2-RELEASE][root@pfSense.local.lan]/root: diskinfo -t /dev/da0
    /dev/da0
            512            # sectorsize
            8589934592      # mediasize in bytes (8.0G)
            16777216        # mediasize in sectors
            0              # stripesize
            0              # stripeoffset
            1044            # Cylinders according to firmware.
            255            # Heads according to firmware.
            63              # Sectors according to firmware.
                            # Disk ident.

    Seek times:
            Full stroke:      250 iter in  0.258565 sec =    1.034 msec
            Half stroke:      250 iter in  0.134784 sec =    0.539 msec
            Quarter stroke:  500 iter in  0.164177 sec =    0.328 msec
            Short forward:    400 iter in  0.162630 sec =    0.407 msec
            Short backward:  400 iter in  0.144883 sec =    0.362 msec
            Seq outer:      2048 iter in  1.035187 sec =    0.505 msec
            Seq inner:      2048 iter in  0.710852 sec =    0.347 msec
    Transfer rates:
            outside:      102400 kbytes in  1.325136 sec =    77275 kbytes/sec
            middle:        102400 kbytes in  0.681892 sec =  150170 kbytes/sec
            inside:        102400 kbytes in  0.464460 sec =  220471 kbytes/sec

    [2.2.2-RELEASE][root@pfSense.local.lan]/root: kldstat
    Id Refs Address            Size    Name
    1  10 0xffffffff80200000 22d0f80  kernel
    2    1 0xffffffff824d1000 8160    vmblock.ko
    3    1 0xffffffff824da000 13680    vmhgfs.ko
    4    1 0xffffffff824ee000 5630    vmxnet.ko
    5    1 0xffffffff82611000 22f7    vmmemctl.ko
    [2.2.2-RELEASE][root@pfSense.local.lan]/root:

    Mine is on esxi 6 on a HP N40L



  • I will do that right now.
    Brian what is that "serial port" you are asking for removal?



  • same problem….. :( now without vmtools
    johnpoz nice values... I wish the same.



  • johnpoz you used the packages menu to install the vmtools? My get installed ok but the modules never get loaded. ESXI detects a thirdparty vmtools installed



  • shoul I try this?
    install OpenVM Tools from FreeBSD ports via pfsense console itself:

    pkg install open-vm-tools-nox11


  • Banned

    In a VM a serial console could be attached to the VM

    Pls. remove if you have it attached and reboot.

    @luisneves:

    I will do that right now.
    Brian what is that "serial port" you are asking for removal?


  • Banned

    I think John reported that it was still _9 installing despite claiming *_11

    @luisneves:

    shoul I try this?
    install OpenVM Tools from FreeBSD ports via pfsense console itself:

    pkg install open-vm-tools-nox11


  • LAYER 8 Global Moderator

    Yeah its still installing _9 even though it should _12 from the latest package.

    I would not install the ports from bsd, it installs a HUGE amount of other packages.  But on my clean 2.2.2 64 bit test pfsense vm the package installed fine - before it was hanging and never finishing.  And scripts seem to work, I was able to shutdown guest and pfsense did a clean shutdown, etc.



  • already tried vmtools via "nox" and via ESXI install CD and the results are always the same.

    The firewall is working ok minus this disk problem…. I think I will install an 2.2.0 version and see what values I will get on disk access. If its ok I no longer will upgrade pfsense until something or someone finds this and fixes... :(

    But first will try a clean 2.2.2 install, maibe I am luck with that

    Thank you everybody


Log in to reply