Virtio cannot allocate

  • So I'm trying out the latest snapshot pfSense-Full-Update-2.1-BETA1-i386-20130112-0035.
    I cloned my existing 1.2.3 VM on my kvm host, and then did a firmware update, and it updated fine without issue, I was using e1000 for nics.

    After modifying /boot/loader.conf.local, and fstab, I rebooted and reenter my nic information on the nic setup and it continues booting and then hangs at configuring wan.

    SO I did it again, and again and saw this during the nic setup.

    vtnet0 cannot allocate mbufs for Rx virtqueue.

    I have no idea what that means.

    if I set nics back to e1000 but leave disk-drive to virtio, it works fine.

    I'm using centos 6.3 for the host and running pfsense in kvm.

  • @Mad:

    vtnet0 cannot allocate mbufs for Rx virtqueue.

    I have no idea what that means.

    My guess is that the vtnet driver can't allocate a mbuf for receive. (mbuf is a network buffer. Pretty much all network drivers allocate mbufs to hold received frames. ) That you apparently do have enough mbuffs in pfSense when your interfaces are e1000 (em in pfSense/FreeBSD) and don't have enough mbufs when you change one (or more?) e1000 interface to vtnet suggests the vtnet driver attempts to allocate rather more receive mbufs that the em driver.

    How many vtnet interfaces are you using?  How many mbufs does the FreeBSD vtnet driver attempt to allocate per interface? You almost certainly need to increase the maximum number of mbufs allowed (but by how much?).Depending on the current size, it may be sufficient to increase the RAM size of the VM in which pfSense runs.

    The FreeBSD man page collection at doesn't seem to include man pages for vtnet or virtio.

    If you can answer at least some of the questions, it could give some ideas for further suggestions.

  • 4 nics

    1 wan
    2 lans
    1 dmz

    They are bridged to vlan interfaces on the physical ethernet, example: eth1.1 eth1.2 and so on. Makes it easier to setup in pfsense.

    I was running the VM on 256M of ram, how much does vtnet mbufs need?

    I was messing around with it, and it seems to work fine if I enable virtio on one of the interfaces.

    I tried a fresh install and it started screaming at me that em0, em1, vtnet0, and vtnet1  "cannot allocate receive buffer" and "cannot allocate mbufs for Rx virtqueue." It continues to boot but I think dhcp was off because it said static on LAN interface. When I tried to set the lan ip address it hung at disabling dhcpd.

    After that I decided to nuke the experiment and go back to my 1.2.3 setup. I've been pondering for the last few hours, I was thinking maybe I didn't have enough ram to do it.

    I'll clone and update it again and add more ram and see if that fixes the problem, but I have my doubts, it could be I'm not running the latest kvm since centos likes to trail redhat in updates.

  • well it worked, I started off with 1024, but that was a no go. so I threw 2048 at it, and it booted. It still throws the errors but it works.

    Can someone find out what the memory buffers are for the vtnet drivers?

  • Have a look at:

    It's more about Broadcom and Intel NICs but upping kern.ipc.nmbclusters might already helping a little.
    In fact FreeNAS recommends even twice the size and Intel recommends an even higher limit when using a 10GE NIC using ixgbe.
    (which also requires more memory I guess)

    From my short tests with virtio I haven't been happy with the storage driver, it was less stable than plain IDE, but vtnet seemed reasonably solid though.

Log in to reply