Port Forwarding not working on a SDSL line from some sources

  • Hello,

    In our enviroment we have two external WAN links, one over ADSL line and the second over a SDSL link with fix IP-Addr. The SDSL link is used for external access to our web and mail server and as failover for the ADSL link.
    I have created a forward rule for port 80 and 443 to the internal webserver in the DMZ zone. Everything looks fine but from some sources the webserver is not reachable. I have checked the traffic with tcpdump and saw that the traffic is going through the router to the webserver but than the TCP ack is not routed back to the source. The only issue I saw is the low source port 1025, normally the source port is random and higher.

    dump from pfSense

    19:30:17.586550 IP > S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:20.491681 IP > S 1831246874:1831246874(0) win 65535</mss> 

    dump from webserver

    19:30:17.624478 IP > S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:17.624535 IP > S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:20.528978 IP > S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:20.529018 IP > S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:21.824150 IP > S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:27.822980 IP > S 235272043:235272043(0) ack 1831246875 win 5840</mss></mss></mss></mss></mss> 

    dump from a working request to the webserver:

    19:38:52.130146 IP > S 2435969140:2435969140(0) win 5840 <mss 5="" 3500830918="" 1452,sackok,timestamp="" 0,nop,wscale="">19:38:52.130243 IP > S 787344165:787344165(0) ack 2435969141 win 5792 <mss 7="" 1936312902="" 1460,sackok,timestamp="" 3500830918,nop,wscale="">19:38:52.155235 IP > . ack 1 win 183 <nop,nop,timestamp 1936312902="" 3500830920="">19:38:52.175476 IP > P 1:359(358) ack 1 win 183 <nop,nop,timestamp 1936312902="" 3500830922=""></nop,nop,timestamp></nop,nop,timestamp></mss></mss> 

    If I access the internal webserver over the ADSL link from the same sorce addr it is also working, also if I switch back to the old IP-Cop router.

    Has somebody any ideas?



  • If the pfSense box the default gateway on the web server? If not, you'll need to add a static route in the web server. Actually no, if the port forward is not done in the default gateway then it probably just won't work.

    Another Edit: What's probably happening here is that the return traffic is going out on the ADSL line instead of the SDSL line. You'll probably have to use Advanced Outbound NAT so that the web server uses the SDSL line as its connection while your other traffic can use the ADSL line (I'm assuming this is the way you want it set up).

  • Something sounds wrong here.  In general, if both interfaces are on the pfsense, return traffic should use the same interface as the inbound traffic (this is what reply-to pf option does, and I thought it was the default?)  It would be nice to see more info about your config (rules and NAT and such.)

  • Hi all,

    That's what I think too, the incomming TCP connection should be take the same way back. In 95% of all connections this is working fine. That means it is not a general bug. Only two of your customers located in the US have the problem to access our internal web-server.
    I don't know if it is important, but the source port of the not working clients is 1024 or 1025 (see my tcp dump). Working clients have random ports higher than 10000. Another issue is the tcp window size, it is everytime set to 65535 on the not workign clients. Can this cause any trouble?

    Here are two screenshots of the pfSense

    Port forward:

    The default gateway is set to the pfSense, but it dosen't matter in this case. I also have thought the return traffic is routed to the ADSL line, but than I should see the traffic in the pfSense box

    the routing table of the web server:

    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface    *        U     0      0        0 eth0   *        U     0      0        0 virbr0     *          U     0      0        0 eth0
    default         UG    0      0        0 eth0

    192.168.10.X is DMZ net is the pfSense is the www server

    If you need further information please let me know.

    Thanks Bozan

  • You could try and disabling Checksum Offloading. IIRC it can help if some user can't access a web server behind pfSense. http://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear

  • Could you draw us a network diagram?  e.g. where the two WAN links are wrt the pfsense and the LAN?

  • Here it is .. just a simple drawing …


    I have disabled the Checksum Offloading, but no response from the customers jet.
    First I made a mistake and unchecked "Disable NAT Reflection". Don't do that never !  :o

  • What does pfsense routing table look like?

  • The IPv4 table:

    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default        UGS         0   100689    ng0          UH          0        0    lo0
    141.16.150.XX/28   link#1             UC          0        0   sis0
    141.16.150.XX      00:09:43:ac:c7:80  UHLW        1      750   sis0   1042    link#4             UC          0        1    re1     00:30:48:8a:8e:24  UHLW        1     6726    re1   1179     00:e0:81:45:ff:08  UHLW        1    34736    re1    702       UGS         0        0   tun1       UH          1        0   tun1       UGS         0        0   tun0       UH          1        0   tun0    link#2             UC          0        1    re0
    ... clients in LAN net ...     00:e0:81:45:ff:30  UHLW        1    20930    re0    984       lo0                UHS         0        0    lo0

    Do you want me to post IPv6 too?

    Just another info I have updated to version 2.0 release today.
    Currently I have no reply from our customers if it is now working.

  • No, IPv4 is fine.  It looks like the adsl line is the default gateway.  I freely admit not knowing a lot about multiwan, so you might want to have this thread moved to that board.  That said, I have looked at 2.0 multiwan a bit, so I know it is supposed to track states based on the interface the state started on.  Like this:

    # pfctl -s rules | grep reply
    pass in quick on em0 reply-to (em0 WANIP) inet proto tcp from any to port = imap flags S/SA keep state allow-opts label "USER_RULE: Special rule to test disablereplyto"

    Do you see that in your rules?

  • Thanks for your help!

    The problem seems to be solved after I have applied the hint from Perry. The SDSL line is a SIS NIC, but than the internal net to the webservers is a Realtek 1GE NIC and it seems it has exactly the checksum problems. After disabling Checksum Offloading is seems to work fine.
    I suppose there is a incompatibility between CISCO routers and Realtek NIC's, because I have figured out there are CISCO routers at the endpoint from the customers.

    Thanks a lot!

    I saw a lot of the rules.

    pass in quick on sis0 reply-to (sis0 141.16.150.XX) inet proto tcp from any to port = http flags S/SA keep state (source-track rule, max-src-states 10000, max-src-nodes 100) label "USER_RULE: NAT http Website"

    Yes the ADSL is the default gateway. I think today I will switch it to the SDSL line, because I have another problem binding openVPN to the SDSL interface. (I'll start another thread soon :-))