WAN + two OptWAN, loadbalancing fails with weird MAC-addresses behaviour

[Eugene]

Hello,
probably missing something simple here…. nevertheless here is a problem that just kills me.
pfSense-1.2.3
LAN=192.168.0.1/24
WAN=192.168.3.2/24 gateway 192.168.3.1
WAN1(OPT1 em1)=10.0.0.2/24 gateway 10.0.0.1
WAN2(OPT2 em0)=10.0.1.2/24 gateway 10.0.1.1
All interfaces are on different nics, static addressing is everywhere, no external switching between interfaces.
I have a loadbalancing pool from WAN1 and WAN2 with ip to monitor=respective gateways
On LAN I have a rule routing all traffic through this loadbalancing pool.
Test scenario: from PC connected to LAN I periodically go to the same web-site 97.107.134.79 waiting for all states to disappear between attempts.
Now dumps. This is my first attempt that goes through WAN2:

# tcpdump -ni em0 -e host 97.107.134.79
20:39:48.143093 00:1b:21:7c:a1:6c > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.1.2.2263 > 97.107.134.79.80: S 3695044687:3695044687(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:39:48.144004 00:1b:21:7c:a1:6c > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.1.2.30502 > 97.107.134.79.80: S 4101759538:4101759538(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:39:48.237129 00:26:f2:56:49:bc > 00:1b:21:7c:a1:6c, ethertype IPv4 (0x0800), length 66: 97.107.134.79.80 > 10.0.1.2.2263: S 2213084595:2213084595(0) ack 3695044688 win 5840 <mss 6="" 1420,nop,nop,sackok,nop,wscale="">20:39:48.237379 00:26:f2:56:49:bc > 00:1b:21:7c:a1:6c, ethertype IPv4 (0x0800), length 66: 97.107.134.79.80 > 10.0.1.2.30502: S 2203729880:2203729880(0) ack 4101759539 win 5840</mss></mss></mss> 

After some time the same connection request goes through WAN1, please pay attention to destination MAC-addresses:

# tcpdump -ni em1 -e host 97.107.134.79
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
20:41:44.681552 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.37567 > 97.107.134.79.80: S 238567 8252:2385678252(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:44.686356 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.28692 > 97.107.134.79.80: S 425659 6844:4256596844(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:44.934133 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.56267 > 97.107.134.79.80: S 150322 3532:1503223532(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:47.680118 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.37567 > 97.107.134.79.80: S 238567 8252:2385678252(0) win 8192</mss></mss></mss> 

So, in the second case it uses different interface but destination-mac of WAN2 gateway!

# arp -an
? (10.0.0.1) at 00:26:f2:56:4b:86 on em1 [ethernet]
? (10.0.1.1) at 00:26:f2:56:49:bc on em0 [ethernet]
...

The problem is intermittent - approximately 90% of connection over new interface fail. But if an attempt is successfull then previous interface starts using 'wrong' destination mac.
More weird stuff: everything works as expected if I forward traffic through loadbalancers WAN-WAN1 or WAN-WAN2.

Could anybody please give me a hint on what is going on here -(((
Thanks!

[Eugene]

Ok, let me rephrase the question.
Is anybody using outbound loadbalancing across two or more interfaces when the pool does not include main WAN interface?
Thanks.

[databeestje]

yes

[Eugene]

Unbelievable but it was hardware issue.