Port forwards from secondary double NAT gateway not working.



  • I had posted this in the wrong section before but am unable to modify that post. It's more of a NAT state issue.

    Local interfaces:
    LAN1 & LAN2

    Default gateway:
    WAN

    Additional gateway:
    LAN2GW (a NAT mode ADSL router on LAN2's subnet).

    I'm trying to access pfSense from the Internet using the ADSL router as a backup remote access line. The ADSL router has to be in NAT mode since its used as a gateway for other clients on LAN2 which don't use pfSense as their gateway.

    In the ADSL router I have forwarded port 80 to pfSense's LAN2 address. I created the following rule on LAN2:

    However it looks like the pfSense web server is still sending response packets through the default gateway WAN instead of LAN2GW.
    The port forward only works if I change the default gateway to LAN2GW.
    The "statefulness" doesn't seem to apply in this case and thus LAN2GW isn't being used for response packets on the same TCP connection.

    I have the same problem if I create a NAT port forward rule from LAN2 to a server on LAN1. The same ports are forwarded from the LAN2GW modem. It works when I switch the default gateway on pfSense to LAN2GW.



  • You should only admin your pfsense via SSH or perhaps VPN.  Forwarding your HTTP or even HTTPS interface to the public IP is bad.



  • @kejianshi:

    You should only admin your pfsense via SSH or perhaps VPN.  Forwarding your HTTP or even HTTPS interface to the public IP is bad.

    It's just for testing the port forward. I intend to allow LAN1 servers to be accessed over the backup LAN2GW gateway. They work when I directly have a public IP address on an interface like WAN2 but not on the double NAT interface LAN2 via LAN2GW.



  • Its funny you should ask…

    http://forum.pfsense.org/index.php/topic,60537.0/topicseen.html

    They talk about NAT forwarding off two WANs and gateway address requirements.



  • @kejianshi:

    Its funny you should ask…

    http://forum.pfsense.org/index.php/topic,60537.0/topicseen.html

    They talk about NAT forwarding off two WANs and gateway address requirements.

    I have already set the gateway on LAN2 to LAN2GW and configured NAT rules for LAN2 as follows:
    Source: LAN1 subnet or 127.0.0.1
    Destination: !LAN2 subnet

    I am able to browse the Internet using LAN2GW from LAN1 clients using policy based routing. It's the incoming port forwards thats the problem. If I change the default gateway to LAN2GW then port forwards work. It's seems like a "reply-to" problem.

    Everything works fine with direct public interfaces WAN1 and WAN2.



  • I'd imagine when double NATed for the second wan, you would get an IP like 192.168.1.2 and have to stipulate a gateway address of something like 192.168.1.1 (not the gateway address the ISP gave you because you want to use the router as the gateway).  Thats my guess.



  • @kejianshi:

    I'd imagine when double NATed for the second wan, you would get an IP like 192.168.1.2 and have to stipulate a gateway address of something like 192.168.1.1 (not the gateway address the ISP gave you because you want to use the router as the gateway).  Thats my guess.

    Yes thats exactly what I've done. LAN1 is 10.0.0.0/16 and LAN2 is 10.1.0.0/16. LAN1 has no gateway since by defauly it uses WAN. LAN2 uses LAN2GW (the NAT modem 10.1.0.2).

    Like I said I am able to browse the Internet through LAN2GW from LAN1 using policy based routing.



  • Yes - But on your two WAN interfaces, do those have gateways explicitly specified?

    "Make sure your WAN and WAN2 interfaces have a gateway selected on Interfaces > WAN/WAN2, not having a gateway selected on the Interface page will also make the system omit reply-to on the rules."



  • I also think this is not quite right:

    "LAN2 uses LAN2GW"

    I don't think you should define a gateway on LAN2.
    I think you should handle where LAN2 packets exit to the web using which WAN in Firewall > NAT > Manual Outbound NAT



  • @kejianshi:

    Yes - But on your two WAN interfaces, do those have gateways explicitly specified?

    "Make sure your WAN and WAN2 interfaces have a gateway selected on Interfaces > WAN/WAN2, not having a gateway selected on the Interface page will also make the system omit reply-to on the rules."

    WAN is PPPoE and uses a dynamic gateway. There is no option to select a gateway.
    LAN2 is an Ethernet interface with a static IP 10.1.0.1 and I have configured the gateway LAN2GW (10.1.0.2).

    My internet interfaces are WAN and LAN2. LAN2 services clients and has a backup NAT modem as well.



  • @kejianshi:

    I also think this is not quite right:

    "LAN2 uses LAN2GW"

    I don't think you should define a gateway on LAN2.
    I think you should handle where LAN2 packets exit to the web using which WAN in Firewall > NAT > Manual Outbound NAT

    I do not have a WAN2. I only have WAN and LAN1 & LAN2. I have already configured Manual Outbound NAT as follows:
    Interface: LAN2
    Source: LAN1 or 127.0.0.1
    Destination: !LAN2
    Translation Address: LAN2 address

    This way LAN1 to LAN2 is routed normally and LAN1 to Internet (via LAN2GW if any such policy based routing rules exist) get double NATed. This requires that I set LAN2GW as the gateway for the LAN2 interface.



  • Actually, I think first, try to get outbound NAT sorted.

    I'm sure there are automatic rules that were generated telling everything to exit via WAN.

    Why not make a rule that tells LAN to use WAN
    and LAN2 to use your second WAN

    I really don't think you assign gateways to static IPs for LAN interfaces.



  • OK - Then I am definitely confused.  You only have one internet connection.  Is that correct?



  • @kejianshi:

    Actually, I think first, try to get outbound NAT sorted.

    I'm sure there are automatic rules that were generated telling everything to exit via WAN.

    Why not make a rule that tells LAN to use WAN
    and LAN2 to use your second WAN

    I really don't think you assign gateways to static IPs for LAN interfaces.

    Outbound NAT works fine for LAN1 via WAN, LAN1 via LAN2GW, LAN2 via WAN and LAN2 via LAN2GW. I don't have any automatic rules; they're all manual.



  • @kejianshi:

    OK - Then I am definitely confused.  You only have one internet connection.  Is that correct?

    No I have two. WAN is PPPoE. There is another backup connection on a NAT modem connected to the LAN2 switch. This modem is what I'm calling LAN2GW (10.1.0.2).



  • OK - I'm going to stop here then and just say that I think you have a broken topology.

    I would try to get you to create 2 WAN interfaces on the pfsense box and let pfsense do all your routing for you for both networks and that way it could easily handle what you are trying to do, but I don't think I'd have much luck convincing you.



  • @kejianshi:

    OK - I'm going to stop here then and just say that I think you have a broken topology.

    I would try to get you to create 2 WAN interfaces on the pfsense box and let pfsense do all your routing for you for both networks and that way it could easily handle what you are trying to do, but I don't think I'd have much luck convincing you.

    It works fine this way. Any router does this out of the box.

    My setup is basically single-WAN (WAN, LAN1 & LAN2) and all Internet connections go through WAN.

    I have a second NAT modem on the LAN2 switch so that LAN2 clients can optionally use the NAT modem directly without pfSense.
    As a backup I want to forward some ports from the NAT modem to pfSense. That's all.



  • I do this sort of thing in 1 place - use a network as a LAN and WAN at the same time. You can call it LAN2, WAN2, LWAN2, whatever you like - I will call it LWAN2. It is a subnet that has a gateway defined, so pfSense considers it a possible way out to other places (e.g. the internet) and you can policy-route whatever you like to the gateway. Clients can also be on that subnet, which is private behind the gateway device that NATs out to the internet. For me, I give those clients DHCP from pfSense with a default route to pfSense, then I NAT them in pfSense back out to the LWAN2 gateway. This makes all the traffic originated from LWAN2 clients be NATd as the LWAN2 pfSense IP. Then LWAN2 gateway will always send the return packets to pfSense, which will unNAT them and deliver to the LWAN2 clients. This ensures that pfSense sees the flows in both directions and there is no confusion about maintaining states. Enough of that - it is a valid and working setup for clients to initiate outbound states.
    On the inbound, I have port forwards on the LWAN2 gateway to pfSense LWAN2 address. These are to an OpenVPN server on pfSense itself, so they are not on-forwarded inside LAN. That much works, so be encouraged :)
    I haven't needed to forward again into a server on LAN, so not sure what the extra trick will be to get that to reply back on the correct route.



  • OK - Good.  I'm glad it works for you, but I can't understand why it would work.
    I understand 2 WAN or more > pfsense and lots of options.
    I don't understand 1 WAN > pfsense > 2 LANS and 1 of those LANs into a switch attached to a switch thats attached to the LAN side of another router/modem.

    I'll watch to see what some of the more knowledgeable people come up with.  I couldn't make that work.



  • @phil.davis:

    I do this sort of thing in 1 place - use a network as a LAN and WAN at the same time. You can call it LAN2, WAN2, LWAN2, whatever you like - I will call it LWAN2. It is a subnet that has a gateway defined, so pfSense considers it a possible way out to other places (e.g. the internet) and you can policy-route whatever you like to the gateway. Clients can also be on that subnet, which is private behind the gateway device that NATs out to the internet. For me, I give those clients DHCP from pfSense with a default route to pfSense, then I NAT them in pfSense back out to the LWAN2 gateway. This makes all the traffic originated from LWAN2 clients be NATd as the LWAN2 pfSense IP. Then LWAN2 gateway will always send the return packets to pfSense, which will unNAT them and deliver to the LWAN2 clients. This ensures that pfSense sees the flows in both directions and there is no confusion about maintaining states. Enough of that - it is a valid and working setup for clients to initiate outbound states.
    On the inbound, I have port forwards on the LWAN2 gateway to pfSense LWAN2 address. These are to an OpenVPN server on pfSense itself, so they are not on-forwarded inside LAN. That much works, so be encouraged :)
    I haven't needed to forward again into a server on LAN, so not sure what the extra trick will be to get that to reply back on the correct route.

    This is exactly what I do. I am able to policy route through LAN2GW to the Internet. All combinations of outbound work fine including non-NATed LAN1 to LAN2.

    However in my case I am not even able to access pfSense's local services from the Internet when port forwarded from LAN2GW. I'm not able to access the web UI or SSH from LAN2GW port forwards. If I change the default gateway to LAN2GW the port forwards start working.



  • @kejianshi:

    OK - Good.  I'm glad it works for you, but I can't understand why it would work.
    I understand 2 WAN or more > pfsense and lots of options.
    I don't understand 1 WAN > pfsense > 2 LANS and 1 of those LANs into a switch attached to a switch thats attached to the LAN side of another modem.

    I'll watch to see what some of the more knowledgeable people come up with.  I couldn't make that work.

    It won't work unless you manually define outbound NAT rules. pfSense will NOT create outbound NAT rules if you simply set a gateway on LAN2. The rule should also exclude LAN1 to LAN2 traffic from being NATed.



  • I'm not arguing - I'm watching.
    Ill be shocked and amazed if it works, but if it does I'll have learned something new.
    Looks like phil.davis has done this before, so you are in luck.  I'll just watch and learn.



  • It won't work unless you manually define outbound NAT rules. pfSense will NOT create outbound NAT rules if you simply set a gateway on LAN2.

    Correct - basically when you add a gateway to an interface, pfSense treats that as a WAN for the purpose of generating automatic goodies - so if you give (what happens to be called) LAN2 a gateway, then I expect the automatic outbound NAT will make outbound NAT rules from LAN to WAN and LAN to LAN2 (treating LAN2 as another WAN). But it won't be able to 2nd-guess you and make outbound NAT rules from LAN2 to anywhere.
    As KurianOfBorg says, once you get the necessary manual outbound NAT rules defined, all the outbound client connections work fine.
    But I am struggling to think what might be happening to the incoming port forwards. If I get a chance I'll try it out on my Alix at home, I have 2 ISPs, 3 physical ports and can make ordinary LAN, primary wired ISP on WAN and a "LWAN2" that has other clients and a way out to the internet via a 2nd ISP on a NATd 3G device.



  • @kejianshi:

    I'm not arguing - I'm watching.
    Ill be shocked and amazed if it works, but if it does I'll have learned something new.
    Looks like phil.davis has done this before, so you are in luck.  I'll just watch and learn.

    Another interesting fact is at the remote location, if I don't use a NAT router and use a remote PC with a public IP address instead, I am able to connect to the port forwards on LAN2GW and the response packets are coming from WAN's public IP address! You'd think the socket implementation on the OS would see the tuple is mismatched but the return packets are still arriving via a different route. A NAT router at the remote location would discard these packets, but Windows with a public IP address is not.



  • This is interesting to me, assuming you get it to work, because someone was before trying to do something else I didn't think was very probable.  Trying to run a Openvpn server at end 1.  Send a vpn client config to end 2.  Then have pfsense on end 1 be able to grab an IP at end 2 and NAT that public IP to all computers attached to pfsense at end 1 such that those computers at end 1 were surfing the web using end 2's public IP.  (VPN in reverse)

    As I said, I don't understand how this will work not having done it, but as with your scenario my first thought is "that shouldn't work".
    But, I'm wrong abit, so if it does work thats cool.  I'll have learned something.



  • @phil.davis:

    As KurianOfBorg says, once you get the necessary manual outbound NAT rules defined, all the outbound client connections work fine.
    But I am struggling to think what might be happening to the incoming port forwards.

    Actually, servers do not even need any kind of outbound firewall rules. Simply forwarding a port from any WAN* interface to a LAN server will allow the server to communicate with the remote client. This works fine if I configure two interfaces WAN1 and WAN2 with public IP addresses. I am able to connect remotely through a WAN2 port forward even though WAN1 is the default gateway and even when there are no outbound firewall rules at all. Only the outbound NAT rule for WAN2 is required.

    This fails when WAN2 is a double NAT (in my case the LAN2 interface with LAN2GW manually set).

    Can you show me your port forward rule and the associated firewall rule and maybe the outbound NAT rule as well? Did you have to set the gateway explicitly on the associated firewall rule? I am not able to get this working even for local services on pfSense's LAN2 address where the only port forward is on the NAT modem to pfSense's LAN2 address.



  • I found the problem. Even though the inbound rules were defined on the LAN2 interface, the responses were using the policy based routing rule on my LAN interface group rule for "*** to * through WAN gateway**". The associated firewall rules on LAN2 from the NAT port forward were not being used at all. I changed the LAN interface group rule to "LAN1/LAN2 to * through WAN gateway" so that it doesn't match the packets being forwarded by the NAT modem.

    Now I am able to port forward to both pfSense as well as to LAN1 servers from the NAT modem on LAN2.