Deploying pfsense behind ISP router with double nat



  • Hi,

    Just been trying to setup a pfSense router today and not made much progress with my attempts, hence it's time for a forum post.

    For background I've deployed my own custom Debian based routers before, but I am new to pfSense.

    My ISP router does not support static routes, hence I require a double-nat'ted configuration.

    I had my Debian based router setup at another address. I recently moved to a new address with a new ISP. The Debian based equipment all works fine here. But I can't get the pfSense box to work. I've probably overlooked something simple, but I can't figure it out.

    The setup is quite simple. I have an ISP router which does NAT from the ISP assigned WAN IP address to the internal network address space of 192.168.0.X.

    I assigned a static IP to the pfSense router of 192.168.0.200. Although this shouldn't be necessary. I also tried without. Neither config worked. This is for interface "WAN".

    The pfSense box has another interface "LAN". It has static IP 192.168.100.254. It is connected to an unmanaged switch, and this is connected to another linux PC. This machine has the IP 192.168.100.1. This was assigned by pfSense DHCP.

    If I try to ping 192.168.0.1 (ISP router) from the linux PC (other side of pfSense router) I get 100 % packet loss.

    I'm not familiar with freebsd, so I don't know how to start going about diagnosing what is wrong. I've watched a couple of hours of youtube videos on how to set this up for "lab / home networks" and read some forum posts but not made any further progress. Perhaps the information is wrong, so I will not re-post it here.

    To summarize, we have

    [ISP] (192.168.0.1) <-> (192.168.0.200) [pfSense] (192.168.100.254) <-> (192.168.100.1)

    And I cannot ping across the pfSense device.

    What are the first steps I should take to debug this? Thanks.

    Currently Firewall->NAT->Outbound is set ot "Automatic" and there are 4 rules...

    2 for LAN, 2 for WAN. The source IPs are 127.0.0.0/8 ::1/128, the source ports are *, the destination IPs are *, and the dst ports are either * or 500. Description says "Auto Created Rule"

    Unfortunately I can't paste what I can see easily because I can't access the pfSense device and the internet at the same time. If this information is important I can perhaps take a photo of it.



  • @hypernova said in Deploying pfsense behind ISP router with double nat:

    My ISP router does not support static routes, hence I require a double-nat'ted configuration.

    What does one have to do with the other?



  • Without NAT it doesn't work.



  • @hypernova

    Again, what does double NAT have to do with static routes. They are completely unrelated. If the first router works with a static route, then you should be able to replace it with a similarly configured pfSense.



  • Ok I've been trying to figure this out, but mostly just got myself confused. I was pretty convinced double NAT was required, as nothing worked on my previous setup without it.

    Now I am currently not so sure about it.



  • @JKnott I'm sorry I don't understand your point about static route.

    My ISP router does not support static routes. I thought this was why double NATting was required - but having written some stuff down on paper I'm now not so sure.



  • Perhaps it is helpful to start from a simpler point.

    I disabled NAT on the pfSense box.

    I am trying to ping 192.168.0.1 from my PC. I cannot get a response. However I can ping the pfSense box.

    So I cannot ping something on the other side of the pfSense box. Why is this is so, or what should I do to diagnose this issue?



  • As another test, if I use a laptop connected to the 192.168.0.X network to ping 192.168.0.1, it works. However I also cannot ping 192.168.100.254.

    This is because my ISP router does not know where 192.168.100.X is.

    NAT does not help in this case of course, but this is why I concluded NAT was required on the pfSense box. So that the network address range 192.168.100.X would be translated via nat into a 192.168.0.200:<port> address, which my ISP router does understand, because 192.168.0.200 is on the 192.168.0.X network...



  • @hypernova said in Deploying pfsense behind ISP router with double nat:

    I'm sorry I don't understand your point about static route.

    You were the one that first mentioned static routes. Those are not normally used for consumer level connections. I have absolutely no idea why you even mentioned that in the first place.



  • @JKnott I mentioned it, as I explained above, because I thought NAT was required due to the fact that my ISP does not support static routes.

    I am not sure if I am mistaken about that. I've spent hours trying to get the pfsense box to work - or at least do something.

    So far I've not had any success with it. I have no idea what diagnostics should be done.

    If you have any suggestions about what I should do next I will be glad to hear them.

    Essentially allow me to ask the most basic question.

    I have an ISP router. I attach a pfsense box to it. How should I configure the pfsense box to get internet access to devices on the other side of the pfsense box.



  • @hypernova

    Do you even know there's something with that 192.168.100.254 address? While any address within the local address block, other than .0 or .255, can be used for the router, typically .1 is used.

    Since you want to use pfSense as a router, you should set your modem to be in bridge mode, not gateway. This will get rid of double NAT. PfSense will then receive the needed connection info via DHCP, so you have nothing to configure on the WAN side. Also, by using bridge mode, you may also get IPv6, assuming your ISP is providing it.



  • @JKnott said in Deploying pfsense behind ISP router with double nat:

    @hypernova

    Do you even know there's something with that 192.168.100.254 address? While any address within the local address block, other than .0 or .255, can be used for the router, typically .1 is used.

    The pfSense box has the address 192.168.100.254. The attached desktop on the LAN side has address 192.168.100.1.

    Did you actually read what I posted?

    Since you want to use pfSense as a router, you should set your modem to be in bridge mode, not gateway. This will get rid of double NAT. PfSense will then receive the needed connection info via DHCP, so you have nothing to configure on the WAN side. Also, by using bridge mode, you may also get IPv6, assuming your ISP is providing it.

    My ISP router does not have a bridge mode. It can receive an IP via DHCP. I have now set a reserved address. I don't know why you bring this up, I can't see the relevance of it.



  • I tried starting again with a fresh install of pfsense, keeping all the default settings.

    I can now ping the ISP router, but I cannot ping anything further, such as 8.8.8.8.

    Any suggestions?



  • @hypernova said in Deploying pfsense behind ISP router with double nat:

    Any suggestions?

    No, that really should work out of the box on LAN.



  • @Bob-Dig said in Deploying pfsense behind ISP router with double nat:

    @hypernova said in Deploying pfsense behind ISP router with double nat:

    Any suggestions?

    No, that really should work out of the box on LAN.

    I would have thought so too... Here's some traceroute info. I don't know if this is helpful?

    Through pfsense router:

    traceroute 192.168.0.1
    traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets
    1 pfSense.localdomain (192.168.1.1) 0.268 ms 0.266 ms 0.273 ms
    2 * * *
    3 * * *
    4 * * *
    5 * * *
    6 * * *
    7 * * *
    8 * * *
    9 * * *
    10 * * *
    11 * * *
    12 * * *
    13 * * *
    14 * * *
    15 * * *
    16 * * *
    17 * * *
    18 * * *
    19 * * *
    20 * * *
    21 * * *
    22 * * *
    23 * * *
    24 * * *
    25 * * *
    26 * * *
    27 * * *
    28 * * *
    29 * * *
    30 * * *
    (end of output)

    Through my debian based router:

    traceroute 192.168.0.1
    traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets
    1 pigrey (192.168.2.254) 0.214 ms 0.242 ms 0.283 ms
    2 192.168.0.1 (192.168.0.1) 3.293 ms 4.517 ms 5.505 ms

    The second output looks sensible. The first does not look at all sensible.

    After a reboot I was able to ping 8.8.8.8, but the response was slow.

    I was not able to ping www.google.com. So this suggests perhaps there is something wrong in the configuration which is interfering with the ability for DNS to resolve.

    In the logs I am seeing a lot of instances of a particular error:

    "wan dhcp sendto error (error 65)"

    This might be related?



  • @hypernova I hope you don't Block private networks on WAN?



  • @Bob-Dig said in Deploying pfsense behind ISP router with double nat:

    @hypernova I hope you don't Block private networks on WAN?

    Interfaces->WAN/LAN->Reserved Networks

    both checkboxes unchecked - is this what you refer to?



  • Well this is strange... I managed to get something working, and I think I'm now connected via the pfsense router...

    I added a new USB interface - a gigabit one, connected via USB 2.0 (so it won't actually be gigabit.)

    I was using a USB 2.0 to 100Mbs interface. That is still attached as WAN, and the other one is now attached at OPT.

    Why is this other USB interface working when the other one did not? Is this a known issue, some form of compatibility problem with certain USB interfaces?



  • Having thought about this for a while, I believe I remember what got me down the path of implementing double nat some months ago.

    I think I am correct in stating that this is required for external access, such as to ssh ports.

    The reason being that with most ISP routers (at least all the ones I have come across) there is no way to open a port to anything other than the immediate local network.

    For me this is 192.168.0.X.

    However I wish to direct ssh traffic to another machine, on another network.

    Hence why double nat is required?



  • 1:
    Without the pfSense box doing NAT on the WAN , your ISP router needs a static route (for the linux lan), in order to send the ping reply packages back to (via) the pfsense box.
    You might be "bitten" by RFC1918 default blocking of inbound wan packets too.

    2:
    If you let pfSense NAT on the wan port , you won't need any routes in the ISP Router, as all apears to come from the pfSense , that is on a known Lan (The ISP inside Lan)

    3:
    You might want to look at your ISP routers "Portforwarding possibilities".
    I had such a ISP setup , where the ISP outer did NAT , and i needed to run a Linux FTP/WEB server behind it.

    I had an option to portforward "everything" to one specific inside ip address (easy setup).

    Just portforward everything on your ISP router to the pfSense , and then portforward the interesting ports in the pfSense to the correct pfSense inside ip's.

    /Bingo

    PS:
    If you let pfSense NAT , and does not block RFC1918 on WAN (Your ISP router uses RFC1918, on the inside Lan).

    Then your ping/access of your ISP router from teh Linux PC , should work flawlessly.



  • @hypernova said in Deploying pfsense behind ISP router with double nat:

    The reason being that with most ISP routers (at least all the ones I have come across) there is no way to open a port to anything other than the immediate local network.
    For me this is 192.168.0.X.
    However I wish to direct ssh traffic to another machine, on another network.
    Hence why double nat is required?

    ????

    If ssh is blocked by the first router, how can double NAT possibly fix that?

    The solution is to put the modem in bridge mode and use pfsense as your only router.



  • @bingo600 said in Deploying pfsense behind ISP router with double nat:

    1:
    Without the pfSense box doing NAT on the WAN , your ISP router needs a static route (for the linux lan), in order to send the ping reply packages back to (via) the pfsense box.
    You might be "bitten" by RFC1918 default blocking of inbound wan packets too.

    Yes - this is my problem. I cannot assign static routes on my ISP router. There is no such functionality. This is presumably because I am not a business customer, and they require a business plan for such things.

    2:
    If you let pfSense NAT on the wan port , you won't need any routes in the ISP Router, as all apears to come from the pfSense , that is on a known Lan (The ISP inside Lan)

    3:
    You might want to look at your ISP routers "Portforwarding possibilities".
    I had such a ISP setup , where the ISP outer did NAT , and i needed to run a Linux FTP/WEB server behind it.

    I had an option to portforward "everything" to one specific inside ip address (easy setup).

    Just portforward everything on your ISP router to the pfSense , and then portforward the interesting ports in the pfSense to the correct pfSense inside ip's.

    This might be possible. My ISP router has port forwarding abilities.

    The options are as such;

    Local IP: (has to be 192.168.0.X, aka same network)
    Local Start Port:
    Local End Port:
    External Start Port:
    External End Port:
    Protocol: UDP, TCP or BOTH

    What options should I be choosing here?

    Surely if I port forward everything, including things like Port 80, this will break access for other users on net 192.168.0.X ?

    /Bingo

    PS:
    If you let pfSense NAT , and does not block RFC1918 on WAN (Your ISP router uses RFC1918, on the inside Lan).

    Then your ping/access of your ISP router from teh Linux PC , should work flawlessly.

    This is why I think the only option is double NAT, but then this breaks communication between 192.168.0.X and the networks behind pfSense.



  • @JKnott said in Deploying pfsense behind ISP router with double nat:

    @hypernova said in Deploying pfsense behind ISP router with double nat:

    The reason being that with most ISP routers (at least all the ones I have come across) there is no way to open a port to anything other than the immediate local network.
    For me this is 192.168.0.X.
    However I wish to direct ssh traffic to another machine, on another network.
    Hence why double nat is required?

    ????

    If ssh is blocked by the first router, how can double NAT possibly fix that?

    Where did I say this?

    The solution is to put the modem in bridge mode and use pfsense as your only router.

    Possibly, however I have read about problems with my particular router and using modem mode. This also would break traffic for other users on 192.168.0.X.

    I can fix that by adding yet another router, but then I need to go and build such a thing. I'm currently working on that as a solution, I just haven't got round to it yet.



  • @hypernova said in Deploying pfsense behind ISP router with double nat:

    Just portforward everything on your ISP router to the pfSense , and then portforward the interesting ports in the pfSense to the correct pfSense inside ip's.

    This might be possible. My ISP router has port forwarding abilities.

    The options are as such;

    Local IP: (has to be 192.168.0.X, aka same network)
    Local Start Port:
    Local End Port:
    External Start Port:
    External End Port:
    Protocol: UDP, TCP or BOTH

    What options should I be choosing here?

    Surely if I port forward everything, including things like Port 80, this will break access for other users on net 192.168.0.X ?

    Not necessarily a problem.
    I would expect the portforwarding rule to only be on inbound traffic , hitting the ISP router.

    Meaning if you portforward ie. port 80 , it will still allow users on the "inside lan" to browse to the outside internet. It's highly unlikely that they would get a source port of 80 or 443 assigned as outbound port on the ISP router.

    You would only need to portforward ports that you have a dedicated service for in the "Inside" ie 80 if you have a web server.

    If the server is behind pfsense you'd have to portforward 80 from the isp router to the pfsense outside (on the inside lan) , and then once again on the pfsense from pfsense outside to the real server ip on pfsense "inside"

    All the normal traffic (ie ssh to the ISP outer) will be handled (and natted) by the pfsense "wan/outside nat"

    /Bingo

    Edit:

    Watch out for the default rule , that would block RFC1918 sourced ip packages from entering from the wan side.

    If you do the standard pfSense WAN IP NAT ,
    "Everything" would seem work (you can access all of internet)

    But that rule would block any comms sourced from any unit on the ISP inside, including ssh or ping from the isp router. As those reply packets would have an RFC1918 ip address as source (& dest aka. pfsense ouside).

    If you don't have any servers (services) that has to be reached from inet , then just enable standard "Wan NAT" , and don't do default blocking of RFC1918 on the WAN.

    I'd remove the default blocking of RFC1918 .. BUT
    As the first rule on my WAN interface i'd permit the ISP inside /24 , and then i'd block the FULL RFC1918.



  • Thanks for the detailed reply. I haven't quite followed everything said, so I want to re-wind a bit.

    It is probably more helpful to phrase the problem like this:

    My ISP router does not have static route functions.

    So if a user is connected to the ISP router network (192.168.0.X), they (presumably) won't be able to connect to any networks beyond any other routers that I connect to my ISP router combo box.

    For example, say I have some services, like even something as simple as a NAS box, behind a pfSense router...

    Let's say that the pfSense router has two interfaces with addresses 192.168.0.200 and 192.168.1.1.

    Then there are a bunch of things, along with a NAS box, connected on network 192.168.1.X.

    Problem / Solution A:

    There's no way for a user to connect to the NAS from the 192.168.0.X side? (Or is there? Can I forward a port using pfSense, and then from the point of view of the 192.168.0.X network, any services behind the pfSense router just appear to have the same IP 192.168.0.200, with different port numbers. Either way this solution isn't ideal because it requires my novice users to remember port numbers and know how to use them. Still I'm not sure if this will work or not?)

    Problem / Solution B:

    This was my original idea. Have all users connect to a new AP on the 192.168.1.X network. Then if they ask for the IP address 192.168.N.X, the pfSense router has static routes set to direct traffic to the relevant routing point. Problem solved? No requirement to know port numbers. Note that I say 192.168.N.X here because I am assuming I have at least one other network with address 192.168.2.X. The reason for this is just to have some level of network segmentation for security and traffic management reasons. I have two machines which send a lot of data to each other and I don't want them locking up the network for everyone else.

    However, what confused me was whether I would need an additional layer of NAT across the pfSense router. To be honest with you, I don't fully remember at this point why I thought this would be required. Having read again through this thread I guess it isn't required, assuming I go with Solution B. It isn't required in this case, because I can connect a machine on network 192.168.1.X and access the web - so obviously that works.

    Summary:

    Sorry for the long post. I hope it is clear. Any comments on either the suggestions A or B?



  • Not necessarily a problem.
    I would expect the portforwarding rule to only be on inbound traffic , hitting the ISP router.

    Meaning if you portforward ie. port 80 , it will still allow users on the "inside lan" to browse to the outside internet. It's highly unlikely that they would get a source port of 80 or 443 assigned as outbound port on the ISP router.

    You are correct here - but I should add the following caveat.

    I believe most things (like web browsers) assign random ports in the range of 1024 to 65536 or whatever the maximum port number is. (16 bit integer, I don't remember exactly?)

    So, what I meant to say was this...

    Surely just directing all traffic with destination port in the range of > 1024 would break other users connections?

    Consider this example:

    A computer with address 192.168.0.35 connects to a webserver with return port of 1024.

    If port 1024 is always forwarded to IP 192.168.0.200 (for example) then the response from the webserver requested from IP 192.168.0.35 will never reach that address... because it will be forwarded to 192.168.0.200.

    Unless there's a caveat I don't understand here?


Log in to reply