More than two virtio causes pfsense to hang during boot
-
Hello all I seem to be having issues with the new 2.1.
After my old pfsense 1.2.3 box died a horrible death, I decided to rebuild with 2.1.
It's a virtualized instance, the host is centos 6.5 using KVM version:qemu-kvm-0.12.1.2-2.415.el6_5.4.
Now initially I started out with four e1000 nics bridged to vlans on eth1 of the host, I had it setup that way before, because captive portal vlans on the same nic in 1.2.3 didn't play nice.Anyways I noticed an issue with e1000, I had to restart pfsense twice before it would actually work, meaning the first boot it couldn't configure WAN (waiting a long time before it would move on) and when trying to access pfsense via the browser or pinging it, the host was unreachable. After telling pfsense to reboot, the second boot up it was able to configure WAN quickly and then I was able to ping and access the pfsense web interface. But I have to do this every time I start pfsense or if I have to reboot again, I have to do it once more. Bit of a pain if you ask me.
So I did some testing and noticed that if I use the realtek or one of the other device models, pfsense would work properly. But I need a gigabit links for my network and e1000 and virtio are the only ones that does gigabit.
Well long story short, switching the four e1000 to virtio causes problems, pfsense hangs during boot at "Configuring (insert random network name)" or hangs at "Generating RRD Graphs."
I played around a bit and and got two virtio working, but if you add a third one, that one complains about something, I can't remember exactly but it was something about "Cannot allocate mbufs." If you add another on on top of that one they both complain about it.
Now I managed to get it to boot with two virtio and two realteks and, in pfsense web interface my mbuf usage is extremely high. (17702/25600)
So I'm wondering if mbuf is the issue on why I can't use more than two virtio devices?
-
I had a similar issue with hardware interfaces. The machine booted up and pfSense started, however, the mbuf was very high and sometimes the machine crashed.
I could get it work for me by adding this 2 lines to /boot/loader.conf.local:
kern.ipc.nmbclusters="131072" hw.igb.num_queues=1"igb" in the second line is the name of my if driver. For virtio it will be something else like vr. I don't know. Take a look at your interface table in pfSense (eg. vr1 or so, than take "vr").
If /boot/loader.conf.local doesn't exist you may create it at first.
This pumps up the network memory buffer and set the interface to use 1 queue only. -
I ended up fixing it, by changing the mbuf value in the /boot/loader.conf to 64k it's using about 32k right now.
kern.ipc.nmbclusters="65536" -
Keep an eye on the mbuf. I've read in an other thread that it flowed over after a couple of days.
-
I also read that too.
Hasn't budge much.
Uptime 8 Days 18 Hours 55 Minutes 40 SecondsMBUF Usage 52% (34310/65536) -
Thank you for that hint.
Now I have tried activating multiple queuing also. It seems to be stable.