Pfsense running under ubuntu 11.04 with device pass through

Pootle · Jul 20, 2011, 1:50 PM

Just a quick note to say I have got pfsense running under KVM on an ubuntu 11.04 server with the NICs passed through to the VM. Only issue was a known freebsd problem with kernel 2.6.38 which is easily fixed by upgrading the kernel to 2.6.39. I'm using pfsense 2.0 RC3.

The performance with device pass-through is dramatically better than using host networking (given that freebsd can't do virtio without rebuilding to get virtio drivers).

This way it's a pretty much an out of the box solution, apart from the kernel upgrade, which is pretty easy.

I'm using an MSI motherboard - 890FXA GD70 and Athlon 235e CPU (there's a couple of other VMs on there…)

bootlog · Sep 3, 2011, 11:22 PM

What is actually the known problem you are talking about?
I have the same setup and my problem is, that the (pass through) interfaces get no link.
Is that the issue that is fixed by a kernel upgrade?

Pootle · Sep 4, 2011, 11:28 AM

@bootlog:

What is actually the known problem you are talking about?
I have the same setup and my problem is, that the (pass through) interfaces get no link.
Is that the issue that is fixed by a kernel upgrade?

dont have access to details just now, but it was a problem that gave some nasty dmesg errors on the host (about the first nic), and the pfsense vm couldn't actually use the device. The kernel upgrade totally fixed the problem.

bootlog · Sep 5, 2011, 10:06 AM

I tried 2.6.39 now and it does not fix the problem.
When i start a linux VM with the same interface passed through, it works perfectly. Only with pfsense it doesn't get connection (though it says em0 is up).
Ideas anyone?

Pootle · Sep 5, 2011, 2:45 PM

@bootlog:

I tried 2.6.39 now and it does not fix the problem.
When i start a linux VM with the same interface passed through, it works perfectly. Only with pfsense it doesn't get connection (though it says em0 is up).
Ideas anyone?

Flying blind here bootlog. What type of nic does pfsense think it has got? Does this match the actual nic type?

Can you post your vm xmlconfig, and the lines from your host and pfsense vm dmesg logs where the nic's get hooked up?

Pootle · Sep 6, 2011, 8:54 AM

This might be helpful to anyone wanting to setup pass through devices for pfsense, it's the shell script I used
to create a pfsense vm.

Remember that you must also ensure that the host doesn't grab the devices (by blacklisting the drivers or other
means).

sudo virt-install -n pfsense -r 392 -d \
--vcpus=1 \
--os-variant=freebsd8 \
--disk path=pfsensedisk.img,size=30,sparse=false,format=raw \
--disk path=pfSense-2.0-RC3-amd64-20110621-2308.iso,perms=ro,device=cdrom \
-k local \
--vnc \
--nonetworks \
--host-device=pci_0000_06_00_0 \
--host-device=pci_0000_07_00_0

bootlog · Sep 7, 2011, 10:35 AM

I'm back to 2.6.38, using IPCop (Linux) for now (works without problems, using the same config).
I will have another try with pfsense some later, when i have the time, because to me it seems MUCH better than IPCop.

abiscula · Sep 9, 2011, 6:15 AM

@Pootle:

Just a quick note to say I have got pfsense running under KVM on an ubuntu 11.04 server with the NICs passed through to the VM. Only issue was a known freebsd problem with kernel 2.6.38 which is easily fixed by upgrading the kernel to 2.6.39. I'm using pfsense 2.0 RC3.

The performance with device pass-through is dramatically better than using host networking (given that freebsd can't do virtio without rebuilding to get virtio drivers).

This way it's a pretty much an out of the box solution, apart from the kernel upgrade, which is pretty easy.

I'm using an MSI motherboard - 890FXA GD70 and Athlon 235e CPU (there's a couple of other VMs on there…)

Regarding pass through, have you used any unsupported card? Have you ever tested it with a fast ethernet card?

I'm really interested in this setup but before replacing my processor with a vt-d one I need to make sure fast ethernet old PCI cards are supported by VmDirectPath (Pass through)

Thanks

Pootle · Sep 9, 2011, 12:40 PM

@abiscula:

Regarding pass through, have you used any unsupported card? Have you ever tested it with a fast ethernet card?

Not sure what you mean by fast - I'm using a couple of realtek gigabit cards passed through, it's not easy to find a pci-x NIC that isn't supported in ubuntu…

I'm really interested in this setup but before replacing my processor with a vt-d one I need to make sure fast ethernet old PCI cards are supported by VmDirectPath (Pass through)

Given the IO is passed through, I don't think that the card is particularly relevant. I'm running with pci-x cards. You could potentially use pci cards, but the pass through capability will pass through all the pci cards (or none of them). I've never tried pci pass through - only pci-x.

Remember that the motherboard has to be IOMMU capable as well…

abiscula · Sep 9, 2011, 5:43 PM

@Pootle:

@abiscula:

Regarding pass through, have you used any unsupported card? Have you ever tested it with a fast ethernet card?

Not sure what you mean by fast - I'm using a couple of realtek gigabit cards passed through, it's not easy to find a pci-x NIC that isn't supported in ubuntu…

I'm really interested in this setup but before replacing my processor with a vt-d one I need to make sure fast ethernet old PCI cards are supported by VmDirectPath (Pass through)

Given the IO is passed through, I don't think that the card is particularly relevant. I'm running with pci-x cards. You could potentially use pci cards, but the pass through capability will pass through all the pci cards (or none of them). I've never tried pci pass through - only pci-x.

Remember that the motherboard has to be IOMMU capable as well…

Working with an intel DP67BA just need to get a new processors since I did not know core i3s do not support VT-D.