Port Forwarding not working on a SDSL line from some sources
-
Hello,
In our enviroment we have two external WAN links, one over ADSL line and the second over a SDSL link with fix IP-Addr. The SDSL link is used for external access to our web and mail server and as failover for the ADSL link.
I have created a forward rule for port 80 and 443 to the internal webserver in the DMZ zone. Everything looks fine but from some sources the webserver is not reachable. I have checked the traffic with tcpdump and saw that the traffic is going through the router to the webserver but than the TCP ack is not routed back to the source. The only issue I saw is the low source port 1025, normally the source port is random and higher.dump from pfSense
19:30:17.586550 IP 211.120.232.220.1025 > 141.16.150.11.80: S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:20.491681 IP 211.120.232.220.1025 > 141.16.150.11.80: S 1831246874:1831246874(0) win 65535</mss>dump from webserver
19:30:17.624478 IP 211.120.232.220.1025 > 192.168.10.250.http: S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:17.624535 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:20.528978 IP 211.120.232.220.1025 > 192.168.10.250.http: S 1831246874:1831246874(0) win 65535 <mss 1380,nop,nop,sackok="">19:30:20.529018 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:21.824150 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840 <mss 1460,nop,nop,sackok="">19:30:27.822980 IP 192.168.10.250.http > 211.120.232.220.1025: S 235272043:235272043(0) ack 1831246875 win 5840</mss></mss></mss></mss></mss>dump from a working request to the webserver:
19:38:52.130146 IP 81.95.0.246.48222 > 192.168.10.250.http: S 2435969140:2435969140(0) win 5840 <mss 5="" 3500830918="" 1452,sackok,timestamp="" 0,nop,wscale="">19:38:52.130243 IP 192.168.10.250.http > 81.95.0.246.48222: S 787344165:787344165(0) ack 2435969141 win 5792 <mss 7="" 1936312902="" 1460,sackok,timestamp="" 3500830918,nop,wscale="">19:38:52.155235 IP 81.95.0.246.48222 > 192.168.10.250.http: . ack 1 win 183 <nop,nop,timestamp 1936312902="" 3500830920="">19:38:52.175476 IP 81.95.0.246.48222 > 192.168.10.250.http: P 1:359(358) ack 1 win 183 <nop,nop,timestamp 1936312902="" 3500830922=""></nop,nop,timestamp></nop,nop,timestamp></mss></mss>If I access the internal webserver over the ADSL link from the same sorce addr it is also working, also if I switch back to the old IP-Cop router.
Has somebody any ideas?
Thanks,
Bozan
-
If the pfSense box the default gateway on the web server?
If not, you'll need to add a static route in the web server.Actually no, if the port forward is not done in the default gateway then it probably just won't work.Another Edit: What's probably happening here is that the return traffic is going out on the ADSL line instead of the SDSL line. You'll probably have to use Advanced Outbound NAT so that the web server uses the SDSL line as its connection while your other traffic can use the ADSL line (I'm assuming this is the way you want it set up).
-
Something sounds wrong here. In general, if both interfaces are on the pfsense, return traffic should use the same interface as the inbound traffic (this is what reply-to pf option does, and I thought it was the default?) It would be nice to see more info about your config (rules and NAT and such.)
-
Hi all,
@danswartz
That's what I think too, the incomming TCP connection should be take the same way back. In 95% of all connections this is working fine. That means it is not a general bug. Only two of your customers located in the US have the problem to access our internal web-server.
I don't know if it is important, but the source port of the not working clients is 1024 or 1025 (see my tcp dump). Working clients have random ports higher than 10000. Another issue is the tcp window size, it is everytime set to 65535 on the not workign clients. Can this cause any trouble?Here are two screenshots of the pfSense
Firewall:
Port forward:
@Briantist
The default gateway is set to the pfSense, but it dosen't matter in this case. I also have thought the return traffic is routed to the ADSL line, but than I should see the traffic in the pfSense boxthe routing table of the web server:
Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.10.0 * 255.255.255.0 U 0 0 0 eth0 192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0 169.254.0.0 * 255.255.0.0 U 0 0 0 eth0 default 192.168.10.1 0.0.0.0 UG 0 0 0 eth0192.168.10.X is DMZ net
192.168.10.1 is the pfSense
192.168.10.250 is the www serverIf you need further information please let me know.
Thanks Bozan
-
You could try and disabling Checksum Offloading. IIRC it can help if some user can't access a web server behind pfSense. http://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
-
Could you draw us a network diagram? e.g. where the two WAN links are wrt the pfsense and the LAN?
-
Here it is .. just a simple drawing …
Thanks
@Perry
I have disabled the Checksum Offloading, but no response from the customers jet.
First I made a mistake and unchecked "Disable NAT Reflection". Don't do that never ! :o -
What does pfsense routing table look like?
-
The IPv4 table:
Destination Gateway Flags Refs Use Netif Expire default 217.0.116.180 UGS 0 100689 ng0 127.0.0.1 127.0.0.1 UH 0 0 lo0 141.16.150.XX/28 link#1 UC 0 0 sis0 141.16.150.XX 00:09:43:ac:c7:80 UHLW 1 750 sis0 1042 192.168.10.0/24 link#4 UC 0 1 re1 192.168.10.250 00:30:48:8a:8e:24 UHLW 1 6726 re1 1179 192.168.10.251 00:e0:81:45:ff:08 UHLW 1 34736 re1 702 192.168.12.0/24 192.168.12.2 UGS 0 0 tun1 192.168.12.2 192.168.12.1 UH 1 0 tun1 192.168.13.0/24 192.168.13.2 UGS 0 0 tun0 192.168.13.2 192.168.13.1 UH 1 0 tun0 192.168.15.0/24 link#2 UC 0 1 re0 ... clients in LAN net ... 192.168.15.251 00:e0:81:45:ff:30 UHLW 1 20930 re0 984 217.83.9.235 lo0 UHS 0 0 lo0Do you want me to post IPv6 too?
Just another info I have updated to version 2.0 release today.
Currently I have no reply from our customers if it is now working. -
No, IPv4 is fine. It looks like the adsl line is the default gateway. I freely admit not knowing a lot about multiwan, so you might want to have this thread moved to that board. That said, I have looked at 2.0 multiwan a bit, so I know it is supposed to track states based on the interface the state started on. Like this:
# pfctl -s rules | grep reply pass in quick on em0 reply-to (em0 WANIP) inet proto tcp from any to 192.168.56.1 port = imap flags S/SA keep state allow-opts label "USER_RULE: Special rule to test disablereplyto"Do you see that in your rules?
-
Thanks for your help!
The problem seems to be solved after I have applied the hint from Perry. The SDSL line is a SIS NIC, but than the internal net to the webservers is a Realtek 1GE NIC and it seems it has exactly the checksum problems. After disabling Checksum Offloading is seems to work fine.
I suppose there is a incompatibility between CISCO routers and Realtek NIC's, because I have figured out there are CISCO routers at the endpoint from the customers.Thanks a lot!
@danswartz
I saw a lot of the rules.pass in quick on sis0 reply-to (sis0 141.16.150.XX) inet proto tcp from any to 192.168.10.250 port = http flags S/SA keep state (source-track rule, max-src-states 10000, max-src-nodes 100) label "USER_RULE: NAT http Website"Yes the ADSL is the default gateway. I think today I will switch it to the SDSL line, because I have another problem binding openVPN to the SDSL interface. (I'll start another thread soon :-))