Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    OPTWAN BLOCKS RETURN PACKET SAME WAN SUBNET

    Scheduled Pinned Locked Moved Routing and Multi WAN
    6 Posts 2 Posters 3.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      Cafecomleiteee
      last edited by

      Hello Guys,

      I hope some one is able to light things up here.

      I have a WAN (cable, dynamic IP) and OPTWAN (symetric line) that is used to publish my web server which sits in the DMZ.
      I do have Port Forward rules from OPTWAN port 80 -> DMZ host port 80 and corresponding inbound rule on the interface.
      For almost 99% of my web visitors all goes well as expected and website is accessed without any trouble.

      The issue here is with the other 1% that uses the same internet cable provider as I do with dynamic IP and that has been leased an IP within the same subnet as my own WAN IP.
      Strangely for these cases it seems that Pfsense is routing the return packet to these guys using the WAN interface instead of the original incoming OPTWAN.

      Technical example:
      WAN IP: 182.61.228.71/19
      OPTWAN: 189.72.184.85/27

      So hosts on the same network as I am (182.61.224.0/19), ranging from 182.61.224.1 - 182.61.255.254, when they try to access my website the return packet attempts to return using WAN interface instead of the incoming OPTWAN.

      Any intellegent thoughts on how to make this work for 100% of the cases?

      Many thanks in advance,
      Aleksander

      1 Reply Last reply Reply Quote 0
      • E
        Eugene
        last edited by

        Can you give us tcpdump for this case?
        These hosts might experience problems though.
        When you configure an interface with default gateway x.x.x.x then all rules for inbound traffic for this interface appear with "reply-to (… x.x.x.x)" and all packets coming from your firewall back to provider have MAC address of your x.x.x.x gateway. If your provider's gateway does not reroute these packets to proper hosts then this traffic is lost.
        But again packets do leave the same interface they come from.

        http://ru.doc.pfsense.org

        1 Reply Last reply Reply Quote 0
        • C
          Cafecomleiteee
          last edited by

          I'll have some trouble capturing the TCPDUMP, since I have to find one of those guys that have IPs in the same subnet.

          What do you mean by "These hosts might experience problems though."?

          I understand that by default packets must leave from the same interface they come from, but it just seems they jump to the routing table entry where the subnet x.x.x.x is routed via WAN interface.

          Anyway, thanks a lot! I'll grab the tcpdump info somehow and get back here.

          1 Reply Last reply Reply Quote 0
          • C
            Cafecomleiteee
            last edited by

            Confirmed!
            Traffic is returning trying to return through the wrong interface.

            BELOW TCPDUMP of em1 (WAN) and em2(OPTWAN)

            tcpdump em1 (WAN): There should be no traffic here!
            14:24:56.137233 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435380149="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:24:57.259102 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435380430="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:25:20.134428 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435386149="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:25:21.258854 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435386430="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:25:35.229581 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435389923="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:38.227242 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435390672="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:39.256627 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435390930="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:44.224415 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435392172="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:45.456986 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435392480="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:56.224687 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435395172="" 1460,sackok,timestamp="" 132869521,nop,wscale="">### OPTWAN ###
            tcpdump -i em2 host 182.61.228.80
            tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
            listening on em2, link-type EN10MB (Ethernet), capture size 96 bytes
            14:24:26.722546 IP 182.61.228.80.47443 > xxxxxxxxxxxxx.com.br.http: S 3745816249:3745816249(0) win 5840 <mss 6="" 132852391="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:29.717661 IP 182.61.228.80.47443 > xxxxxxxxxxxxx.com.br.http: S 3745816249:3745816249(0) win 5840 <mss 6="" 132853141="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:35.149338 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132854497="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:38.143475 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132855247="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:44.141701 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132856747="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:56.137047 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132859747="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:20.134258 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132865747="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:35.229364 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132869521="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:38.227060 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132870271="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:44.224222 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132871771="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:56.224472 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132874771="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:20.214908 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132880771="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:35.740235 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132884652="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:38.739544 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132885402="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:44.739785 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132886902="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:56.733062 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132889902="" 1460,sackok,timestamp="" 0,nop,wscale="">14:27:20.730739 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132895902="" 1460,sackok,timestamp="" 0,nop,wscale="">Since I'm have ever been an iptables guy, I had to study PF a little bit in the weekend. Shouldn't there be a route-to command or should the reply-to should have the same effect? The thing is that somehow the packet is using the routing table!

            Any thoughts?

            Attached I'm sending my wide open rules.

            rule_dmz.JPG
            rule_dmz.JPG_thumb
            rule_optwan.JPG
            rule_optwan.JPG_thumb
            rule_wan.JPG
            rule_wan.JPG_thumb</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss>

            1 Reply Last reply Reply Quote 0
            • E
              Eugene
              last edited by

              You should see reply-to option if you run
              pfctl -sr | grep em2
              apparently it does not work in your case. your WAN subnet is 'directly connected' to your WAN interface, so it seems to be overriding what 'reply-to' tells.

              http://ru.doc.pfsense.org

              1 Reply Last reply Reply Quote 0
              • C
                Cafecomleiteee
                last edited by

                Eugene thanks for the hand!

                SOLUTION ADOPTED. Just to have documented in case other people also needs this.

                Since I'm a new to PF (pfsense) I didn't manage to correctly solve this subnet routing problem. So as I'm using Virtual machines for PFsense, I just created a PFSENSE02 machine, and now my current PFSENSE01 has its WAN with a static DMZ ip address. This way I could isolate the subnets avoiding routing problems.
                I'm far from proud of the solution adopted, but business comes first, so what matters is making the clients happy!

                INTERNET (cable) –- WAN-PFSENSE02-LAN ---- WAN-PFSENSE01-OPTWAN --- INTERNET (symetric line)
                                                                                                    |
                                                                                          LAN / DMZ / OTHER NETs

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.