PfSense on kvm w/ Intel 10GbE NIC and Virtual Functions - one-way transmission



  • I have pfSense 2.1.1 installed on a kvm VM that sits on Ubuntu 14, installed on an Intel Grizzly Pass OEM server with an Intel X520-DA2 10GbE NIC. We have SR-IOV and IOMMU installed and virtual functions working perfectly with Ubuntu and CentOS. However, FreeBSD seems to have an issue with the VFs. Packets check out, but they don't check in - like a backwards Bates motel.

    [pfsense/pci_0000_81_10_2 @ 192.168.100.2] <–-- 192.168.100.0/24 ----> [centos/pci_0000_81_10_6 @ 192.168.100.101]

    If I run a tcpdump on pfsense_ix0 and centos_eth1, I can see the ARP request leave pfsense and reach centos, and centos respond, but pfsense does not receive the response. I can see an increase in received packets via 'netstat.'

    There is no vSwitch between kvm and the Guest OS - all LAN access is via the VF on VLAN 100.

    I found a similar complaint here: https://forums.freebsd.org/viewtopic.php?t=10248

    I'm working with the vendor who has this system hosted (sadly, it's not local) to see if that particular BIOS open is available and enabled on our system so I can have it shut down, but haven't gotten to that point yet. I'm trolling for ideas/suggestions at this point because I'm rather stuck.

    I'm also in the process of compiling the newer 10GbE drivers for FreeBSD from Intel.

    [2.1.4-RELEASE][admin@pfSense]/root(1): uname -a
    FreeBSD pfSense 8.3-RELEASE-p16 FreeBSD 8.3-RELEASE-p16 #0: Fri Jun 20 13:19:29 EDT 2014     root@pf2_1_1_amd64.pfsense.org:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64
    
    [2.1.4-RELEASE][admin@pfSense]/root(3): dmesg | grep Virtual
    CPU: QEMU Virtual CPU version 2.0.0 (2793.28-MHz K8-class CPU)
    ix0: <intel(r) pro="" 10gbe="" virtual="" function="" network="" driver,="" version="" -="" 1.1.4=""> mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0</intel(r)>
    
    [2.1.4-RELEASE][admin@pfSense]/root(4): ifconfig ix0
    ix0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso>ether 52:54:00:9f:19:2d
            inet6 fe80::5054:ff:fe9f:192d%ix0 prefixlen 64 scopeid 0x2
            inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
            nd6 options=1 <performnud>media: Ethernet autoselect
            status: active</performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>
    

    I'm curious if anyone has any input or experience with an issue similar to this.



  • We tried the 2.2 alpha on FreeBSD 10 with the same results. The same driver version (1.1.4) is used in both releases. So far we have not been successful in upgrading the Intel VF drivers on the 2.1/8.3, either.

    Unfortunately, this is a deal-killer that will force us to dump pfSense as a production firewall. Any direction or suggestion would be greatly appreciated.



  • Same result(s) on native FreeBSD 8.3-RELEASE w/ the Intel VF v1.1.2 drivers.

    freebsd1# tcpdump -i ix0
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes
    21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    ^C
    13 packets captured
    13 packets received by filter
    0 packets dropped by kernel
    
    [root@centos1 ~]# tcpdump -i eth1
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
    17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
    17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
    ^C
    24 packets captured
    24 packets received by filter
    0 packets dropped by kernel
    


  • May be you should try virtio drivers instead of sr-iov?