Unstable connection on 2nd Xen VMs, PFSense on 1st Xen VM via physical port

  • I'm having this issue with packets being dropped/total communications loss in my setup, which is a little more complex then usual but shouldn't present any major issues.

    I've got 2 machines, both Xen hosts, with dual gigabit ports each.
    Both hosts are bridged physically via gigabit ethernet.

    I have a PFSense VM on Host 1, with 2 physical ports and 1 virtual port assigned.
    I've turned off tx checksum offloading for all 3 ports on Host 1 AND the 2 physical ports on Host 2.

    Virtual port is for the VMs on Host 1.

    PFSense is setup as a transparent firewall.

    I've setup a bridge with both physical ports and the 1 virtual port.

    Now, VMs on Host 1 have no issue and are able to communicate perfectly internally and externally.

    However, VMs on Host 2 experience unstable connectivity to all services (to Host 1/LAN/WAN).
    Sometimes we get <10kbps out, and other times full linespeed. But it seems to get saturated < 1-200MB and then we see drops on all protocols - TCP/UDP/ICMP.

    Connecting Host 2 to the network directly works perfectly. However, we do want the PFSense VM to function as the firewall for both Xen Hosts and VMs.

    Anybody run into this sort of issue before?

    Machines are Supermicros with Intel 82563EB ethernet controllers, running on PRO/1000 e1000e drivers.

  • Anybody have any pointers or tips? Seems like an impasse at this point…