WAN + two OptWAN, loadbalancing fails with weird MAC-addresses behaviour

Eugene

Hello,
probably missing something simple here…. nevertheless here is a problem that just kills me.
pfSense-1.2.3
LAN=192.168.0.1/24
WAN=192.168.3.2/24 gateway 192.168.3.1
WAN1(OPT1 em1)=10.0.0.2/24 gateway 10.0.0.1
WAN2(OPT2 em0)=10.0.1.2/24 gateway 10.0.1.1
All interfaces are on different nics, static addressing is everywhere, no external switching between interfaces.
I have a loadbalancing pool from WAN1 and WAN2 with ip to monitor=respective gateways
On LAN I have a rule routing all traffic through this loadbalancing pool.
Test scenario: from PC connected to LAN I periodically go to the same web-site 97.107.134.79 waiting for all states to disappear between attempts.
Now dumps. This is my first attempt that goes through WAN2:

# tcpdump -ni em0 -e host 97.107.134.79
20:39:48.143093 00:1b:21:7c:a1:6c > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.1.2.2263 > 97.107.134.79.80: S 3695044687:3695044687(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:39:48.144004 00:1b:21:7c:a1:6c > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.1.2.30502 > 97.107.134.79.80: S 4101759538:4101759538(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:39:48.237129 00:26:f2:56:49:bc > 00:1b:21:7c:a1:6c, ethertype IPv4 (0x0800), length 66: 97.107.134.79.80 > 10.0.1.2.2263: S 2213084595:2213084595(0) ack 3695044688 win 5840 <mss 6="" 1420,nop,nop,sackok,nop,wscale="">20:39:48.237379 00:26:f2:56:49:bc > 00:1b:21:7c:a1:6c, ethertype IPv4 (0x0800), length 66: 97.107.134.79.80 > 10.0.1.2.30502: S 2203729880:2203729880(0) ack 4101759539 win 5840</mss></mss></mss>

After some time the same connection request goes through WAN1, please pay attention to destination MAC-addresses:

# tcpdump -ni em1 -e host 97.107.134.79
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
20:41:44.681552 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.37567 > 97.107.134.79.80: S 238567 8252:2385678252(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:44.686356 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.28692 > 97.107.134.79.80: S 425659 6844:4256596844(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:44.934133 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.56267 > 97.107.134.79.80: S 150322 3532:1503223532(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:47.680118 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.37567 > 97.107.134.79.80: S 238567 8252:2385678252(0) win 8192</mss></mss></mss>

So, in the second case it uses different interface but destination-mac of WAN2 gateway!

# arp -an
? (10.0.0.1) at 00:26:f2:56:4b:86 on em1 [ethernet]
? (10.0.1.1) at 00:26:f2:56:49:bc on em0 [ethernet]
...

The problem is intermittent - approximately 90% of connection over new interface fail. But if an attempt is successfull then previous interface starts using 'wrong' destination mac.
More weird stuff: everything works as expected if I forward traffic through loadbalancers WAN-WAN1 or WAN-WAN2.

Could anybody please give me a hint on what is going on here -(((
Thanks!

Eugene

Ok, let me rephrase the question.
Is anybody using outbound loadbalancing across two or more interfaces when the pool does not include main WAN interface?
Thanks.

databeestje

yes

Eugene

Unbelievable but it was hardware issue.