PfSense on kvm w/ Intel 10GbE NIC and Virtual Functions - one-way transmission
-
I have pfSense 2.1.1 installed on a kvm VM that sits on Ubuntu 14, installed on an Intel Grizzly Pass OEM server with an Intel X520-DA2 10GbE NIC. We have SR-IOV and IOMMU installed and virtual functions working perfectly with Ubuntu and CentOS. However, FreeBSD seems to have an issue with the VFs. Packets check out, but they don't check in - like a backwards Bates motel.
[pfsense/pci_0000_81_10_2 @ 192.168.100.2] <–-- 192.168.100.0/24 ----> [centos/pci_0000_81_10_6 @ 192.168.100.101]
If I run a tcpdump on pfsense_ix0 and centos_eth1, I can see the ARP request leave pfsense and reach centos, and centos respond, but pfsense does not receive the response. I can see an increase in received packets via 'netstat.'
There is no vSwitch between kvm and the Guest OS - all LAN access is via the VF on VLAN 100.
I found a similar complaint here: https://forums.freebsd.org/viewtopic.php?t=10248
I'm working with the vendor who has this system hosted (sadly, it's not local) to see if that particular BIOS open is available and enabled on our system so I can have it shut down, but haven't gotten to that point yet. I'm trolling for ideas/suggestions at this point because I'm rather stuck.
I'm also in the process of compiling the newer 10GbE drivers for FreeBSD from Intel.
[2.1.4-RELEASE][admin@pfSense]/root(1): uname -a FreeBSD pfSense 8.3-RELEASE-p16 FreeBSD 8.3-RELEASE-p16 #0: Fri Jun 20 13:19:29 EDT 2014 root@pf2_1_1_amd64.pfsense.org:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
[2.1.4-RELEASE][admin@pfSense]/root(3): dmesg | grep Virtual CPU: QEMU Virtual CPU version 2.0.0 (2793.28-MHz K8-class CPU) ix0: <intel(r) pro="" 10gbe="" virtual="" function="" network="" driver,="" version="" -="" 1.1.4=""> mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0</intel(r)>
[2.1.4-RELEASE][admin@pfSense]/root(4): ifconfig ix0 ix0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso>ether 52:54:00:9f:19:2d inet6 fe80::5054:ff:fe9f:192d%ix0 prefixlen 64 scopeid 0x2 inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 nd6 options=1 <performnud>media: Ethernet autoselect status: active</performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>
I'm curious if anyone has any input or experience with an issue similar to this.
-
We tried the 2.2 alpha on FreeBSD 10 with the same results. The same driver version (1.1.4) is used in both releases. So far we have not been successful in upgrading the Intel VF drivers on the 2.1/8.3, either.
Unfortunately, this is a deal-killer that will force us to dump pfSense as a production firewall. Any direction or suggestion would be greatly appreciated.
-
Same result(s) on native FreeBSD 8.3-RELEASE w/ the Intel VF v1.1.2 drivers.
freebsd1# tcpdump -i ix0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes 21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 ^C 13 packets captured 13 packets received by filter 0 packets dropped by kernel [root@centos1 ~]# tcpdump -i eth1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 ^C 24 packets captured 24 packets received by filter 0 packets dropped by kernel
-
May be you should try virtio drivers instead of sr-iov?