OPTWAN BLOCKS RETURN PACKET SAME WAN SUBNET



  • Hello Guys,

    I hope some one is able to light things up here.

    I have a WAN (cable, dynamic IP) and OPTWAN (symetric line) that is used to publish my web server which sits in the DMZ.
    I do have Port Forward rules from OPTWAN port 80 -> DMZ host port 80 and corresponding inbound rule on the interface.
    For almost 99% of my web visitors all goes well as expected and website is accessed without any trouble.

    The issue here is with the other 1% that uses the same internet cable provider as I do with dynamic IP and that has been leased an IP within the same subnet as my own WAN IP.
    Strangely for these cases it seems that Pfsense is routing the return packet to these guys using the WAN interface instead of the original incoming OPTWAN.

    Technical example:
    WAN IP: 182.61.228.71/19
    OPTWAN: 189.72.184.85/27

    So hosts on the same network as I am (182.61.224.0/19), ranging from 182.61.224.1 - 182.61.255.254, when they try to access my website the return packet attempts to return using WAN interface instead of the incoming OPTWAN.

    Any intellegent thoughts on how to make this work for 100% of the cases?

    Many thanks in advance,
    Aleksander



  • Can you give us tcpdump for this case?
    These hosts might experience problems though.
    When you configure an interface with default gateway x.x.x.x then all rules for inbound traffic for this interface appear with "reply-to (… x.x.x.x)" and all packets coming from your firewall back to provider have MAC address of your x.x.x.x gateway. If your provider's gateway does not reroute these packets to proper hosts then this traffic is lost.
    But again packets do leave the same interface they come from.



  • I'll have some trouble capturing the TCPDUMP, since I have to find one of those guys that have IPs in the same subnet.

    What do you mean by "These hosts might experience problems though."?

    I understand that by default packets must leave from the same interface they come from, but it just seems they jump to the routing table entry where the subnet x.x.x.x is routed via WAN interface.

    Anyway, thanks a lot! I'll grab the tcpdump info somehow and get back here.



  • Confirmed!
    Traffic is returning trying to return through the wrong interface.

    BELOW TCPDUMP of em1 (WAN) and em2(OPTWAN)

    tcpdump em1 (WAN): There should be no traffic here!
    14:24:56.137233 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435380149="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:24:57.259102 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435380430="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:25:20.134428 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435386149="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:25:21.258854 IP 172.16.10.3.http > 182.61.228.80.58053: S 756132053:756132053(0) ack 3884652054 win 5792 <mss 2="" 435386430="" 1460,sackok,timestamp="" 132854497,nop,wscale="">14:25:35.229581 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435389923="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:38.227242 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435390672="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:39.256627 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435390930="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:44.224415 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435392172="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:45.456986 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435392480="" 1460,sackok,timestamp="" 132869521,nop,wscale="">14:25:56.224687 IP 172.16.10.3.http > 182.61.228.80.58522: S 841869203:841869203(0) ack 526229802 win 5792 <mss 2="" 435395172="" 1460,sackok,timestamp="" 132869521,nop,wscale="">### OPTWAN ###
    tcpdump -i em2 host 182.61.228.80
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on em2, link-type EN10MB (Ethernet), capture size 96 bytes
    14:24:26.722546 IP 182.61.228.80.47443 > xxxxxxxxxxxxx.com.br.http: S 3745816249:3745816249(0) win 5840 <mss 6="" 132852391="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:29.717661 IP 182.61.228.80.47443 > xxxxxxxxxxxxx.com.br.http: S 3745816249:3745816249(0) win 5840 <mss 6="" 132853141="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:35.149338 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132854497="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:38.143475 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132855247="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:44.141701 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132856747="" 1460,sackok,timestamp="" 0,nop,wscale="">14:24:56.137047 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132859747="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:20.134258 IP 182.61.228.80.58053 > xxxxxxxxxxxxx.com.br.http: S 3884652053:3884652053(0) win 5840 <mss 6="" 132865747="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:35.229364 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132869521="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:38.227060 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132870271="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:44.224222 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132871771="" 1460,sackok,timestamp="" 0,nop,wscale="">14:25:56.224472 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132874771="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:20.214908 IP 182.61.228.80.58522 > xxxxxxxxxxxxx.com.br.http: S 526229801:526229801(0) win 5840 <mss 6="" 132880771="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:35.740235 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132884652="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:38.739544 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132885402="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:44.739785 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132886902="" 1460,sackok,timestamp="" 0,nop,wscale="">14:26:56.733062 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132889902="" 1460,sackok,timestamp="" 0,nop,wscale="">14:27:20.730739 IP 182.61.228.80.50800 > xxxxxxxxxxxxx.com.br.http: S 1470738648:1470738648(0) win 5840 <mss 6="" 132895902="" 1460,sackok,timestamp="" 0,nop,wscale="">Since I'm have ever been an iptables guy, I had to study PF a little bit in the weekend. Shouldn't there be a route-to command or should the reply-to should have the same effect? The thing is that somehow the packet is using the routing table!

    Any thoughts?

    Attached I'm sending my wide open rules.






    </mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss>



  • You should see reply-to option if you run
    pfctl -sr | grep em2
    apparently it does not work in your case. your WAN subnet is 'directly connected' to your WAN interface, so it seems to be overriding what 'reply-to' tells.



  • Eugene thanks for the hand!

    SOLUTION ADOPTED. Just to have documented in case other people also needs this.

    Since I'm a new to PF (pfsense) I didn't manage to correctly solve this subnet routing problem. So as I'm using Virtual machines for PFsense, I just created a PFSENSE02 machine, and now my current PFSENSE01 has its WAN with a static DMZ ip address. This way I could isolate the subnets avoiding routing problems.
    I'm far from proud of the solution adopted, but business comes first, so what matters is making the clients happy!

    INTERNET (cable) –- WAN-PFSENSE02-LAN ---- WAN-PFSENSE01-OPTWAN --- INTERNET (symetric line)
                                                                                        |
                                                                              LAN / DMZ / OTHER NETs


Log in to reply