Port Forwarding not working on a SDSL line from some sources



  • Hello,

    In our enviroment we have two external WAN links, one over ADSL line and the second over a SDSL link with fix IP-Addr. The SDSL link is used for external access to our web and mail server and as failover for the ADSL link.
    I have created a forward rule for port 80 and 443 to the internal webserver in the DMZ zone. Everything looks fine but from some sources the webserver is not reachable. I have checked the traffic with tcpdump and saw that the traffic is going through the router to the webserver but than the TCP ack is not routed back to the source. The only issue I saw is the low source port 1025, normally the source port is random and higher.

    dump from pfSense

    
    19:30:17.586550 IP 211.120.232.220.1025 > 141.16.150.11.80: S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:20.491681 IP 211.120.232.220.1025 > 141.16.150.11.80: S 1831246874:1831246874(0) win 65535</mss> 
    

    dump from webserver

    
    19:30:17.624478 IP 211.120.232.220.1025 > 192.168.10.250.http: S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:17.624535 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:20.528978 IP 211.120.232.220.1025 > 192.168.10.250.http: S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:20.529018 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:21.824150 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:27.822980 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840</mss></mss></mss></mss></mss> 
    

    dump from a working request to the webserver:

    
    19:38:52.130146 IP 81.95.0.246.48222 > 192.168.10.250.http: S 2435969140:2435969140(0) win 5840 <mss 5="" 3500830918="" 1452,sackok,timestamp="" 0,nop,wscale="">19:38:52.130243 IP 192.168.10.250.http > 81.95.0.246.48222: S 787344165:787344165(0) ack 2435969141 win 5792 <mss 7="" 1936312902="" 1460,sackok,timestamp="" 3500830918,nop,wscale="">19:38:52.155235 IP 81.95.0.246.48222 > 192.168.10.250.http: . ack 1 win 183 <nop,nop,timestamp 1936312902="" 3500830920="">19:38:52.175476 IP 81.95.0.246.48222 > 192.168.10.250.http: P 1:359(358) ack 1 win 183 <nop,nop,timestamp 1936312902="" 3500830922=""></nop,nop,timestamp></nop,nop,timestamp></mss></mss> 
    

    If I access the internal webserver over the ADSL link from the same sorce addr it is also working, also if I switch back to the old IP-Cop router.

    Has somebody any ideas?

    Thanks,

    Bozan



  • If the pfSense box the default gateway on the web server? If not, you'll need to add a static route in the web server. Actually no, if the port forward is not done in the default gateway then it probably just won't work.

    Another Edit: What's probably happening here is that the return traffic is going out on the ADSL line instead of the SDSL line. You'll probably have to use Advanced Outbound NAT so that the web server uses the SDSL line as its connection while your other traffic can use the ADSL line (I'm assuming this is the way you want it set up).



  • Something sounds wrong here.  In general, if both interfaces are on the pfsense, return traffic should use the same interface as the inbound traffic (this is what reply-to pf option does, and I thought it was the default?)  It would be nice to see more info about your config (rules and NAT and such.)



  • Hi all,

    @danswartz
    That's what I think too, the incomming TCP connection should be take the same way back. In 95% of all connections this is working fine. That means it is not a general bug. Only two of your customers located in the US have the problem to access our internal web-server.
    I don't know if it is important, but the source port of the not working clients is 1024 or 1025 (see my tcp dump). Working clients have random ports higher than 10000. Another issue is the tcp window size, it is everytime set to 65535 on the not workign clients. Can this cause any trouble?

    Here are two screenshots of the pfSense
    Firewall:

    Port forward:

    @Briantist
    The default gateway is set to the pfSense, but it dosen't matter in this case. I also have thought the return traffic is routed to the ADSL line, but than I should see the traffic in the pfSense box

    the routing table of the web server:

    
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.10.0    *               255.255.255.0   U     0      0        0 eth0
    192.168.122.0   *               255.255.255.0   U     0      0        0 virbr0
    169.254.0.0     *               255.255.0.0     U     0      0        0 eth0
    default         192.168.10.1    0.0.0.0         UG    0      0        0 eth0
    
    

    192.168.10.X is DMZ net
    192.168.10.1 is the pfSense
    192.168.10.250 is the www server

    If you need further information please let me know.

    Thanks Bozan



  • You could try and disabling Checksum Offloading. IIRC it can help if some user can't access a web server behind pfSense. http://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear



  • Could you draw us a network diagram?  e.g. where the two WAN links are wrt the pfsense and the LAN?



  • Here it is .. just a simple drawing …

    Thanks

    @Perry
    I have disabled the Checksum Offloading, but no response from the customers jet.
    First I made a mistake and unchecked "Disable NAT Reflection". Don't do that never !  :o



  • What does pfsense routing table look like?



  • The IPv4 table:

    
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            217.0.116.180      UGS         0   100689    ng0
    127.0.0.1          127.0.0.1          UH          0        0    lo0
    141.16.150.XX/28   link#1             UC          0        0   sis0
    141.16.150.XX      00:09:43:ac:c7:80  UHLW        1      750   sis0   1042
    192.168.10.0/24    link#4             UC          0        1    re1
    192.168.10.250     00:30:48:8a:8e:24  UHLW        1     6726    re1   1179
    192.168.10.251     00:e0:81:45:ff:08  UHLW        1    34736    re1    702
    192.168.12.0/24    192.168.12.2       UGS         0        0   tun1
    192.168.12.2       192.168.12.1       UH          1        0   tun1
    192.168.13.0/24    192.168.13.2       UGS         0        0   tun0
    192.168.13.2       192.168.13.1       UH          1        0   tun0
    192.168.15.0/24    link#2             UC          0        1    re0
    
    ... clients in LAN net ...
    
    192.168.15.251     00:e0:81:45:ff:30  UHLW        1    20930    re0    984
    217.83.9.235       lo0                UHS         0        0    lo0
    
    

    Do you want me to post IPv6 too?

    Just another info I have updated to version 2.0 release today.
    Currently I have no reply from our customers if it is now working.



  • No, IPv4 is fine.  It looks like the adsl line is the default gateway.  I freely admit not knowing a lot about multiwan, so you might want to have this thread moved to that board.  That said, I have looked at 2.0 multiwan a bit, so I know it is supposed to track states based on the interface the state started on.  Like this:

    # pfctl -s rules | grep reply
    pass in quick on em0 reply-to (em0 WANIP) inet proto tcp from any to 192.168.56.1 port = imap flags S/SA keep state allow-opts label "USER_RULE: Special rule to test disablereplyto"
    
    

    Do you see that in your rules?



  • Thanks for your help!

    The problem seems to be solved after I have applied the hint from Perry. The SDSL line is a SIS NIC, but than the internal net to the webservers is a Realtek 1GE NIC and it seems it has exactly the checksum problems. After disabling Checksum Offloading is seems to work fine.
    I suppose there is a incompatibility between CISCO routers and Realtek NIC's, because I have figured out there are CISCO routers at the endpoint from the customers.

    Thanks a lot!

    @danswartz
    I saw a lot of the rules.

    
    pass in quick on sis0 reply-to (sis0 141.16.150.XX) inet proto tcp from any to 192.168.10.250 port = http flags S/SA keep state (source-track rule, max-src-states 10000, max-src-nodes 100) label "USER_RULE: NAT http Website"
    
    

    Yes the ADSL is the default gateway. I think today I will switch it to the SDSL line, because I have another problem binding openVPN to the SDSL interface. (I'll start another thread soon :-))


Locked