Data throughput when using VIPs seems to be limited, Please help!

[alltester]

Hello everyone!

First I'd like to thank everyone here for a wonderful community, I've learned a lot from everyone that has contributed and I hope at one point to be able to do the same!  This knowledge has led me to my current setup, but now, I am in need of assistance.

The setup is as follows:

2 pfsense boxes running CARP for HA in Master-Backup config, each pfsense box is equipped with dual embedded 1Gig Intel Pro 1000 EB and 2 x Quad port Intel Pro 1000 PCIe cards (10 ports total). WAN is a 100Mbps connection.

Ports are configured in the following order for each box:
1 x WAN
2 x LAN - LAGG configuration, each port is going to a different switch for redundancy
2 x Bridged - LAGG configuration bridged to WAN interface (public IP address space)
1 x OPT2 - single port configured to accept traffic from a dedicated site to site line

Defined CARP VIPs for WAN and LAN for fail-over; I modified the devd.conf script to perform ifconfig up/down depending on the CARP state on the bridged interfaces.
–-----------------------------------------------------------------------------------

CARP notify hooks. This will call carpup/carpdown with the

interface (carp0, carp1) as the first parameter.

notify 100 {
    match "system"          "IFNET";
    match "type"            "LINK_UP";
    match "subsystem" "[a-zA-Z0-9_]+_vip[0-9]+";
    action "/etc/rc.carpmaster $subsystem";
    action "ifconfig lagg1 up";
    action "ifconfig bridge0 up";
};

notify 100 {
    match "system"          "IFNET";
    match "type"            "LINK_DOWN";
    match "subsystem" "[a-zA-Z0-9_]+_vip[0-9]+";
    action "/etc/rc.carpbackup $subsystem";
    action "ifconfig bridge0 down";
    action "ifconfig lagg1 down";
};
–-----------------------------------------------------------------------------------

My problem comes with data transfer speed and VIPs.  It looks like the VIP is limiting the incoming data throughput, and seems to be capped at 10 Mbps!  I found out when I followed: https://forum.pfsense.org/index.php?topic=41138.0 to allow me to display the WAN VIP instead of the interface's assigned address, and then performed a speed test; the connection speeds dropped from 90+ Mbps to ~4Mbps steady, and data transfer from WAN to LAN also dropped to ~9Mbps.  Removing the rule brings everything back up to normal.

After reformatting and re-doing the configuration from scratch several times, I managed to constantly repeat this scenario.  Another test I performed was creating a simple VIP (if alias) and doing a port forward for FTP.  FTP transfers from a test system on the WAN side to the LAN via the VIP are being limited to +-10Mbps!  If I use the WAN address on the interface and just forward the port to the system on the LAN side, it works at full tilt.

Is there a configuration that is limiting the interface's (VIP) speed? such as manually forcing 1000/full duplex for the VIP somewhere?

Any help or insight on this matter would be appreciated.

Thanks

Note: I forgot to include, I am running the latest 2.1.5 build.  I've tried both 32 and 64 bit compilations with the same result.

[alltester]

I just finished reinstalling 2.1.5 32bit from scratch again, created the carp and vips, tested, and it worked!!
Sadly that was short lived, as soon as I failed over to the backup system and did a speed test there, it slowed down again… speed tests are consistant 4 Mbps and data transfers such as FTP are down to ~9 Mbps... this is frustrating!

Anyone have any suggestions?

[cmb]

It's not that the VIP is being limited, you have a layer 2 or 3 issue with the IP in question. Something outside the firewall, like an IP conflict, MAC conflict, something along those lines. Packet capture filtered on that VIP on both firewalls, see what the traffic looks like.

[alltester]

cmb, Thank you for the nudge!  :D

In my frustration I forgot to troubleshoot the obvious, so far I bypassed the switch the boxes were connected to and hooked up straight to the edge router, so far for yesterday's evening and today's testing, it's going very well.  Speeds are being maintained and data transfers (FTP, HTTP, etc) are coming in at max rate.  Seems that intermediate switch was not playing nice when assigning the VIPs to the CARP interface.