Sorry Guys I'm having a real problem understanding why the rule for OPT2 is wrong and causing the inbound speed issue (especially as the active rule in my previous post is one created by PFSense itself). I have tried source: * and destination: * as shown in the above attachment but that doesn't help with the speed either.
Can you explain? or even suggest what the firewall rule for this simple test should be? (Note: with no rules for OPT2 the iperf test obviously fails as all traffic defaults to blocked).
As requested attached is a diagram. I'm testing on the OPT2/igb2 interfaces on both boxes as a simple case but the same speed issue is present on the LAN and FAILOVER interfaces (not tested the WAN side but I'd be amazed if that didn't have the same issue).
Also as requested here's the interface config from the PFSense booted box:
igb0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 0c:c4:7a:32:5c:30
inet6 fe80::ec4:7aff:fe32:5c30%igb0 prefixlen 64 scopeid 0x1
inet 192.168.1.247 netmask 0xffffff00 broadcast 192.168.1.255
inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255 vhid 2
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
carp: MASTER vhid 2 advbase 1 advskew 0
igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 0c:c4:7a:32:5c:31
inet6 fe80::ec4:7aff:fe32:5c31%igb1 prefixlen 64 scopeid 0x2
inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
status: no carrier
igb2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 0c:c4:7a:32:5c:32
inet6 fe80::ec4:7aff:fe32:5c32%igb2 prefixlen 64 scopeid 0x3
inet 10.9.8.1 netmask 0xffffff00 broadcast 10.9.8.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 0c:c4:7a:32:5c:33
inet6 fe80::ec4:7aff:fe32:5c33%igb3 prefixlen 64 scopeid 0x4
inet X.X.X.106 netmask 0xffffffe0 broadcast X.X.X.127
inet X.X.X.108 netmask 0xffffffe0 broadcast X.X.X.127 vhid 1
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
carp: MASTER vhid 1 advbase 1 advskew 0
pflog0: flags=100 <promisc>metric 0 mtu 33144
pfsync0: flags=41 <up,running>metric 0 mtu 1500
pfsync: syncdev: igb1 syncpeer: 10.10.1.2 maxupd: 128 defer: on
syncok: 1
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
nd6 options=21 <performnud,auto_linklocal>enc0: flags=0<> metric 0 mtu 1536
nd6 options=21 <performnud,auto_linklocal>ovpns2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
options=80000 <linkstate>inet6 fe80::ec4:7aff:fe32:5c30%ovpns2 prefixlen 64 scopeid 0x9
inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffffff
nd6 options=21 <performnud,auto_linklocal>Opened by PID 85860</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></up,running></promisc></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

