Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    WAN + two OptWAN, loadbalancing fails with weird MAC-addresses behaviour

    Scheduled Pinned Locked Moved Routing and Multi WAN
    4 Posts 2 Posters 3.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      Eugene
      last edited by

      Hello,
      probably missing something simple here…. nevertheless here is a problem that just kills me.
      pfSense-1.2.3
      LAN=192.168.0.1/24
      WAN=192.168.3.2/24 gateway 192.168.3.1
      WAN1(OPT1 em1)=10.0.0.2/24 gateway 10.0.0.1
      WAN2(OPT2 em0)=10.0.1.2/24 gateway 10.0.1.1
      All interfaces are on different nics, static addressing is everywhere, no external switching between interfaces.
      I have a loadbalancing pool from WAN1 and WAN2 with ip to monitor=respective gateways
      On LAN I have a rule routing all traffic through this loadbalancing pool.
      Test scenario: from PC connected to LAN I periodically go to the same web-site 97.107.134.79 waiting for all states to disappear between attempts.
      Now dumps. This is my first attempt that goes through WAN2:

      # tcpdump -ni em0 -e host 97.107.134.79
      20:39:48.143093 00:1b:21:7c:a1:6c > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.1.2.2263 > 97.107.134.79.80: S 3695044687:3695044687(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:39:48.144004 00:1b:21:7c:a1:6c > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.1.2.30502 > 97.107.134.79.80: S 4101759538:4101759538(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:39:48.237129 00:26:f2:56:49:bc > 00:1b:21:7c:a1:6c, ethertype IPv4 (0x0800), length 66: 97.107.134.79.80 > 10.0.1.2.2263: S 2213084595:2213084595(0) ack 3695044688 win 5840 <mss 6="" 1420,nop,nop,sackok,nop,wscale="">20:39:48.237379 00:26:f2:56:49:bc > 00:1b:21:7c:a1:6c, ethertype IPv4 (0x0800), length 66: 97.107.134.79.80 > 10.0.1.2.30502: S 2203729880:2203729880(0) ack 4101759539 win 5840</mss></mss></mss> 
      

      After some time the same connection request goes through WAN1, please pay attention to destination MAC-addresses:

      # tcpdump -ni em1 -e host 97.107.134.79
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
      20:41:44.681552 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.37567 > 97.107.134.79.80: S 238567 8252:2385678252(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:44.686356 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.28692 > 97.107.134.79.80: S 425659 6844:4256596844(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:44.934133 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.56267 > 97.107.134.79.80: S 150322 3532:1503223532(0) win 8192 <mss 1460,nop,wscale="" 8,nop,nop,sackok="">20:41:47.680118 00:1b:21:7c:a4:57 > 00:26:f2:56:49:bc, ethertype IPv4 (0x0800), length 66: 10.0.0.2.37567 > 97.107.134.79.80: S 238567 8252:2385678252(0) win 8192</mss></mss></mss> 
      

      So, in the second case it uses different interface but destination-mac of WAN2 gateway!

      # arp -an
      ? (10.0.0.1) at 00:26:f2:56:4b:86 on em1 [ethernet]
      ? (10.0.1.1) at 00:26:f2:56:49:bc on em0 [ethernet]
      ...
      
      

      The problem is intermittent - approximately 90% of connection over new interface fail. But if an attempt is successfull then previous interface starts using 'wrong' destination mac.
      More weird stuff: everything works as expected if I forward traffic through loadbalancers WAN-WAN1 or WAN-WAN2.

      Could anybody please give me a hint on what is going on here -(((
      Thanks!

      http://ru.doc.pfsense.org

      1 Reply Last reply Reply Quote 0
      • E
        Eugene
        last edited by

        Ok, let me rephrase the question.
        Is anybody using outbound loadbalancing across two or more interfaces when the pool does not include main WAN interface?
        Thanks.

        http://ru.doc.pfsense.org

        1 Reply Last reply Reply Quote 0
        • D
          databeestje
          last edited by

          yes

          1 Reply Last reply Reply Quote 0
          • E
            Eugene
            last edited by

            Unbelievable but it was hardware issue.

            http://ru.doc.pfsense.org

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.