NAT Reflection (Pure NAT) not working for same subnet (v2.2.2)



  • I am trying to get NAT Reflection (Pure NAT) completely working on pfSense 2.2.2. I cannot use Split DNS (some NATs change the destination port, and there are access restrictions between internal subnets). "NAT + Proxy" didn't work either, and I don't want that anyhow.

    The short summary is that I've tried both to enable "Pure NAT" at a single NAT rule and at the system level (System::Advanced::Firewall/NAT::Network Address Translation). The best I can get is to have it work on every internal subnet except the target one. E.g. I can't be on a machine and have it reflect back to itself via the public IP:port.

    I know the technical reason why it's not working. The rules that pfSense is setting up are only half the necessary solution.  ( http://www.openbsd.org/faq/pf/rdr.html#reflect ) pfSense is not creating the "pass out on … nat-to" rule to change the source IP. I have confirmed this is exactly what's happening. (See Edit 1.a down below)

    At first I used netstat -n on each machine to watch the connections as I tried to hit a web site via reflection. Finally, I used "Diagnostics::Execute  Command" to run pfctl -s nat and used BeyondCompare to look at how things changed. The output from pfctl confirms that when I turn on Pure NAT (either at a NAT rule or globally), pfSense doesn't create all of the needed rules to make it completely work.

    Does anyone know what's going on and how to fix it?

    For a more concrete example: Here's one of my NAT rules to my "betaweb" alias, from the WAN ("re0"), as reported by pfctl. I masked the IP address.

    
    rdr pass on re0 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin</betaweb> 
    

    When I turn on Pure NAT, I get some added rules. "betaweb" is in the subnet served by "re2_vlan2" (But I've confirmed that the behavior is identical from "re1", so please don't waste time on the Vlan idea).

    
    rdr pass on re0 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    rdr pass on re1 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    rdr pass on re2_vlan2 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    rdr pass on openvpn inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin</betaweb></betaweb></betaweb></betaweb> 
    

    Considering what I've read on the Internet about OpenBSD's ipfilter (See Edit 1.b down below), I would have expected another rule for "re2_vlan2" that rewrites the source-address on same-subnet redirected packets so that the target machine would send them back to the gateway instead of trying to directly contact the source machine. But there's nothing added, as reported by pfctl.

    Edit 1.a: As of FreeBSD 10.1, its implementation of pf does not yet support "nat-to" and "rdr-to". ( https://www.freebsd.org/cgi/man.cgi?query=pf.conf&sektion=5 )
    Edit 1.b: pfSense is based on FreeBSD, not OpenBSD. Additionally, it uses "pf", not "ipfilter".


  • LAYER 8 Global Moderator

    "I cannot use Split DNS"

    And why is that exactly??  You have something hard coded to use a public IP and doesn't use dns?  Fix the something! ;)



  • I can't use it for the 2 reasons mentioned in my post. Split DNS is a niche solution.

    [1] The external port is different than the internal one; I can't change this requirement. Externally it's "https://some.domain/…" but internally, the URL would have to be "https://some.domain:1234/..."  and DNS doesn't address this situation.

    [2] Some subnets are not allowed access to others. I could work around this, but it would require adding (essentially) duplicate firewall rules on most of the interfaces pfSense controls. I do not want this extra complication as it makes maintenance more difficult.

    NAT Reflection is a much more elegant solution across the board for configuration and maintenance, both within pfSense and the various applications within the networks it supports.


  • Banned

    This NAT Reflection packet ping-pong is a nasty hack, not a solution to anything.



  • So… you have no idea.

    Should I file a bug report then?

    As for the "nasty hack" comment, that's your opinion. I'm of the opinion that "Split DNS" is its own "nasty hack". Both for the above-mentioned port-related shortcoming as well as the fact that you can't serve different IPs based on the source subnet. NAT Reflection is standard, basic stuff for a NAT'ing router. It's just an IP-rewrite operation that rewrites both IPs instead of one. It relies on the same stateful packet inspection (SPI) functionality that outbound NAT'ed connections rely on.


  • LAYER 8 Global Moderator

    So your using nat to change dest ports – that is the reason you can not use split dns??  Resolving the ADDRESS closest to you and on your own local network, to use that address is not a niche anything.. That would be the correct thing to do..

    NAT has been and always be an abomination!! ;)  Be it your reflecting it or not..



  • @johnpoz:

    So your using nat to change dest ports – that is the reason you can not use split dns??  ...

    Yes. That was answered twice already.



  • Since no one has anything constructive to offer here, I'll proceed with the bug report. Hopefully, the process won't take so long that I forget to post an update here for other Googlers that pass by.


  • LAYER 8 Global Moderator

    So your problem is your design then.. And still haven't answered the question with a valid response..

    I have a server listening on port X..  Why would I want to hit pubic IP on port Y to be reflected back to port X.. Why not just hit the private network address that my box is on already with port X..

    You seem stuck on wanting to NAT more than is required.. Yes you have public IP and you want to get to private IP this must be natted.  And hey you want public IP port X to go to private IP Y.. Ok that is fine..  But why does another machine same private network have to hit public IP on X just to be sent back to private Y??

    Why not just go to private Y directly??

    You saying you are doing port redirection is not an answer to WHY you would need or want to do such a nonsense thing..


  • LAYER 8 Netgate

    I know you are dead set in your opinion that NAT reflection should work like you want it to but I think you are going to be disappointed.

    Your real problem is you expect something to perform NAT but the client and servers are on the same subnet.  Anything that will make that happen will be firmly in the "ugly hack" category.

    Put your servers behind an OPTX interface and NAT to them from WAN

    Then since your ports have to be translated, use split DNS to point your LAN hosts directly at the inside (real) IP address of the servers and translate the ports (not the addresses) with rules on LAN similar to those on WAN.

    Set it up that way and it'll just work.



  • @Derelict:

    Then since your ports have to be translated, use split DNS to point your LAN hosts directly at the inside (real) IP address of the servers and translate the ports (not the addresses) with rules on LAN similar to those on WAN.

    Set it up that way and it'll just work.

    I already have this functionality. And it appears that you don't realize that your proposal requires the functionality of the "ugly hack" you eschew. Translating the ports is the same functionality as translating the IP: it's all Network Address Translation.

    The problem explained in the initial post is that the pfSense-supported feature of NAT Reflection is not completely working, because the necessary pf rule is not being set. It's a single checkbox. I'm asking for people to offer help as to why the functionality offered by this checkbox wouldn't be working. Perhaps there's a related setting or feature-set that is documented to interfere with it? I don't know; that's why I'm asking. If there was a supported way for me to manually add in a persistent pf rule as a workaround, that would be a possibility.

    It's like I'm asking about why my backup-camera isn't working on my new car. But instead of trying to help, people decide that time is better spent pontificating about how stupid backup-cameras are, and that I just shouldn't use them.



  • You need to enable "Enable automatic outbound NAT for Reflection", which adds the "nat on …" rules to accommodate routing traffic back out the same interface it came in on.



  • Sounds like you have that box checked. You're not looking for nat-to on a pass out, first make sure you're actually looking for the right thing.

    Example for a port forward to WAN IP 192.0.2.127 port 1234 to internal host 192.168.111.100, where vmx0 is WAN, vmx1 is LAN.

    # grep "rdr on" /tmp/rules.debug
    rdr on vmx0 proto tcp from any to 192.0.2.127 port 1234 -> 192.168.111.100
    rdr on vmx1 proto tcp from any to 192.0.2.127 port 1234 -> 192.168.111.100
    # grep "nat on" /tmp/rules.debug
    no nat on vmx1 proto tcp from vmx1 to 192.168.111.100 port 1234
    nat on vmx1 proto tcp from 192.168.111.0/24 to 192.168.111.100 port 1234 -> 192.168.111.1 port 1024:65535
    
    

    If you don't have such rules, most likely it's because you have a gateway chosen on the interface in question under Interfaces>LAN or whatever the internal interface is. That's strictly for Internet connections, and outbound NAT for reflection is only for non-Internet connections. If that's not the case, we'll need some more details about your config. This definitely works in general.



  • @cmb:

    Sounds like you have that box checked. You're not looking for nat-to on a pass out, first make sure you're actually looking for the right thing.

    I see my mistake: I conflated OpenBSD with FreeBSD. FreeBSD doesn't currently implement the "nat-to" stuff, so yes, I looked for the wrong thing in filter.inc.

    @cmb:

    Example for a port forward to WAN IP 192.0.2.127 port 1234 to internal host 192.168.111.100, where vmx0 is WAN, vmx1 is LAN.

    # grep "rdr on" /tmp/rules.debug
    rdr on vmx0 proto tcp from any to 192.0.2.127 port 1234 -> 192.168.111.100
    rdr on vmx1 proto tcp from any to 192.0.2.127 port 1234 -> 192.168.111.100
    # grep "nat on" /tmp/rules.debug
    no nat on vmx1 proto tcp from vmx1 to 192.168.111.100 port 1234
    nat on vmx1 proto tcp from 192.168.111.0/24 to 192.168.111.100 port 1234 -> 192.168.111.1 port 1024:65535
    
    

    If you don't have such rules, most likely it's because you have a gateway chosen on the interface in question under Interfaces>LAN or whatever the internal interface is. That's strictly for Internet connections, and outbound NAT for reflection is only for non-Internet connections. If that's not the case, we'll need some more details about your config. This definitely works in general.

    I do not have a gateway set on my (internal) "BETA" interface.  The config I have in /tmp/rules.debug does not have anything like the 2 "nat on" rules you listed.

    However, I manually recreated the necessary "nat on" rule on the "Firewall: NAT: Outbound" page, and now NAT Reflection is completely working as expected. So, at least I have a workaround for now. Thanks for the example. When I first attempted a workaround by manually creating an outbound NAT rule, I didn't know that "rdr" applies before "nat", I mistakenly put the WAN IP:port for the destination.

    If ever I get time, I expect that I'll have to rebuild my config on a test platform to discover what's interfering with this functionality. In the mean time, this is the relevant part of /tmp/rules.debug now. ( betaweb resolves to 10.121.12.131, and re2_vlan2 serves 10.121.12.0/24 ):

    
    ...
    BETA = "{ re2_vlan2 }"
    ...
    betaweb = "<betaweb>"
    ...
    # Outbound NAT rules (manual)
    ...
    nat on $BETA  proto tcp from 10.121.12.0/24 to 10.121.12.131/32 port 443 -> 10.121.12.1/32 port 1024:65535
    ...
    rdr pass on re0 proto tcp from any to x.x.x.198 port 443 -> $betaweb
    # Reflection redirect
    rdr pass on { re1 re2_vlan2 openvpn } proto tcp from any to x.x.x.198 port 443 -> $betaweb
    ...</betaweb> 
    

    …and the same thing, as displayed by pfctl:

    
    ...
    nat on re2_vlan2 inet proto tcp from 10.121.12.0/24 to 10.121.12.131 port = https -> 10.121.12.1 port 1024:65535
    ...
    rdr pass on re0 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    rdr pass on re1 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    rdr pass on re2_vlan2 inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    rdr pass on openvpn inet proto tcp from any to x.x.x.198 port = https -> <betaweb>round-robin
    ...</betaweb></betaweb></betaweb></betaweb> 
    


  • I suspect it's something to do with the round robin you're doing. Would you mind sharing a copy of your config with me? If you copy/paste it from status.php it'll sanitize it to remove certs, passwords, etc. Can email to cmb at pfsense dot org or attach to a PM here and I'll delete it after testing.



  • I'm interested in a fully integrated solution to this as well.  I've been using pfsense as an enterprise firewall for a long time, and freebsd for even longer; roughly 20 years now.  There are use cases (like mine) where split DNS is simply not an option.

    For example, a current project of mine has me using pfsense in a multi-wan environment protecting a corporate internal LAN as well as separate LAN subnets for machines (VMs) the company is hosting for customers.  These segments are all in their own VLAN and intentionally disallowed from accessing other customer subnets as well as the corporate LAN subnet.

    For example, imagine the following five networks:
    WAN1 : 1.0.0.0/28 (VLAN 1)
    WAN2 : 2.0.0.0/28 (VLAN 2)
    LAN: 3.0.0.0/24 (VLAN 10)
    CUST1 : 4.0.0.0/24 (VLAN 100)
    CUST2 : 4.0.1.0/24 (VLAN 101)

    This is very similar to a setup I'm currently working with.  All of the public IPs are assigned as virtual IPs on a single pfsense install.  LAN and CUST networks are allowed nearly unrestricted outbound traffic to the internet.  LAN traffic is allowed into each CUST subnet for management.  CUST networks are not allowed to initiate traffic to the LAN or to other CUST networks.

    Desired operation looks basically like this:
    WANx -> LAN : Restricted.  Some explicit port forwards.
    WANx -> CUST* : Restricted.  Some explicit port forwards.
    LAN -> CUST* : Unrestricted.
    CUST1 -> CUST1 : Unrestricted
    CUST2 -> CUST2 : Unrestricted
    CUST -> LAN : Rejected
    CUST1 -> CUST2 : Rejected
    CUST2 -> CUST1 : Rejected

    There are other ways to do it, but "NAT reflection" is the simplest, requiring the fewest changes to existing infrastructure.



  • Not working for me too.

    Version 2.2.4 (amd64)

    Old instalation 2.1.5 (amd64) is work.
    Update 2.1.5 to 2.2.4 too work

    New instalation 2.2.4 not work



  • Still not working  :o



  • Pure NAT for same subnet works fine. You also need to enable the source NAT option, "Enable automatic outbound NAT for Reflection".



  • I know this is a bit of thread necromancy on my part, but I had the same issue as the OP, and was able to use the discussion here to put together a fix for the bug. (Yes, I really do think it is an actual bug).

    I figured I would put my findings here for future people googling the issue.

    HOWEVER.. when I try to submit, I am being told that I am spam. "Post content was flagged as spam by Akismet.com" Grr.

    So here is my analysis:
    https://docs.google.com/document/d/1DCtqI2q3RlaK6HkTgp_xFw6poxWg6wiJlzw0V7lDtGU/edit?usp=sharing


  • LAYER 8 Global Moderator

    And do you have this checked?
    nat.jpg



  • Yes.

    Screen Shot 2020-05-18 at 8.32.37 AM.png


  • LAYER 8 Global Moderator

    Well works fine here..

    I just created a port forward to 192.168.9.10 port 80..

    portforward80.jpg

    I enabled nat reflection, pure nat... Hit my wan IP 64.53 - and there you go it WAD..
    purenat.jpg

    You can see it natted my traffic from 192.168.9.100 to 64.53.x.x to souce IP of my lan interface 192.168.9.253

    capture.jpg

    traffic.jpg



  • Ok. I believe that it works for you. But, It does not work for me.
    Whatever this bug is, it is clear that it is not happening to everyone.

    I don't know what would be peculiar about my setup.
    It is relatively fresh, only a few weeks old, and I have only a few rules set up (so far). Nothing complex at all. This NAT reflection was the first "interesting" thing I tried.

    I don't know what I could do to convince you that I really am experiencing this issue, but I'm happy to try if you have suggestions.


  • LAYER 8 Global Moderator

    @rossc719 said in NAT Reflection (Pure NAT) not working for same subnet (v2.2.2):

    Whatever this bug is

    A "bug" can be duplicated and normally easy to replicated...

    Whatever is going on with your specific scenario is not a bug, unless you can show other people having the issues that is not from 5 years ago - which I just tried and works as designed..

    When create the port forward and you have that checked, it will auto create the rule that does the source natting... Did you check that after the fact? After you have created your port forward?

    To be honest they really could just rip out and remove all the nat reflection shit to be honest.. Its NOT how things should be done in the first place... I would never in a million years setup such a hack ;) If I wanted to get to my webserver next to me, then I would resolve its fully qualified domain to that IP, or use that IP... Makes ZERO sense to hit my wan IP in the first place ;)

    But as you can see, even someone that doesn't use nat reflection, and hates it at professional level - it clearly WAD.. It was couple clicks to get that nonsense working. Simple 1 click setting up host override easier and cleaner solution ;)

    BTW - what mask are you running on your network?

    I've got a web server running @ 192.168.5.1.
    I am testing using my phone @ 192.168.10.1.
    My LAN interface of the pfsense is @ 192.168.0.1.
    
    

    To any sane person that would be 3 different networks.. Are you using a /16 or something? To allow for the 65K devices you have on it? ;)



  • When create the port forward and you have that checked, it will auto create the rule that does the source natting... Did you check that after the fact? After you have created your port forward?

    Yes. It was not there.
    Specifically, there were no "nat on" lines that related to port 80 at all.

    In fact, there still aren't:

    [2.4.5-RELEASE][admin@juno.wombat.net]/root: grep "nat on" /tmp/rules.debug
    nat on $LAN inet from 192.168.0.0/18 to 192.168.0.0/18 -> 192.168.0.1/32 port 1024:65535
    nat on $WAN inet from $tonatsubnets to any port 500 -> WAN_ADDRESS/32  static-port
    nat on $WAN inet6 from $tonatsubnets to any port 500 -> (igb0)  static-port
    nat on $WAN inet from $tonatsubnets to any -> WAN_ADDRESS/32 port 1024:65535
    nat on $WAN inet6 from $tonatsubnets to any -> (igb0) port 1024:65535
    no nat on igb1 proto udp from (igb1) to 192.168.0.2 port 53
    nat on igb1 proto udp from 192.168.0.0/18 to 192.168.0.2 port 53 -> 192.168.0.1 port 1024:65535
    

    But as you can see, ... It was couple clicks to get that nonsense working.

    Indeed. For you.
    Again... I really do believe that it works for you.
    It does not work for me.

    BTW - what mask are you running on your network?

    192.168.0.0/18


  • LAYER 8 Global Moderator

    @rossc719 said in NAT Reflection (Pure NAT) not working for same subnet (v2.2.2):

    192.168.0.0/18

    Use something sane! ;)



  • @johnpoz said in NAT Reflection (Pure NAT) not working for same subnet (v2.2.2):

    Use something sane! ;)

    I don't see that this is related to the issue at hand.


  • LAYER 8 Global Moderator

    Maybe its a bug for when people use insane masks and try and do nat reflection ;)

    2.4.5-RELEASE][admin@sg4860.local.lan]/root: grep "nat on" /tmp/rules.debug
    no nat on igb0 proto tcp from (igb0) to 192.168.9.10 port 6666
    nat on igb0 proto tcp from 192.168.9.0/24 to 192.168.9.10 port 6666 -> 192.168.9.253 port 1024:65535
    [2.4.5-RELEASE][admin@sg4860.local.lan]/root:
    

    These get created when I do nat reflection, just did another one for 6666 port



  • @johnpoz said in NAT Reflection (Pure NAT) not working for same subnet (v2.2.2):

    Maybe its a bug for when people use insane masks and try and do nat reflection ;)

    Perhaps, but seems unlikely. Especially given that the OP seems to have used 10.121.12.0/24.


  • LAYER 8 Global Moderator

    The OP was using version 2.2 of pfsense.. So whatever has ZERO to do with today... Unless your using version 2.2 of pfsense?

    Thought you said pfsense IP was 0.1 here there are nat rules to 0.2

    no nat on igb1 proto udp from (igb1) to 192.168.0.2 port 53



  • @johnpoz said in NAT Reflection (Pure NAT) not working for same subnet (v2.2.2):

    The OP was using version 2.2 of pfsense.. So whatever has ZERO to do with today... Unless your using version 2.2 of pfsense?

    Nope, I am using 2.4.5-RELEASE.

    These get created when I do nat reflection, just did another one for 6666 port

    Again, I believe that it works for you.
    But, it does not work for me.

    Thought you said pfsense IP was 0.1 here there are nat rules to 0.2

    This is unrelated to the issue at hand.
    0.2 is my DNS server.
    0.1 is pfsense.


  • LAYER 8 Global Moderator

    Have no idea what else is unique to your setup... Someone using a /18 mask prob has all sorts of nonsense setup ;)



  • If anyone is interested in debugging this at a later date, please let me know.
    I am very interested in working with anyone who has any constructive ideas for how to move forward.

    For now, the hack I described above, (The manually created outgoing NAT rule) seems to patch over the bug, so I will go ahead and use that.

    Cheers


  • LAYER 8 Global Moderator

    @rossc719 said in NAT Reflection (Pure NAT) not working for same subnet (v2.2.2):

    seems to patch over the bug

    There is no BUG.. The problem your seeing is something unique to your setup.. More than happy to help you work through it.. But as I have shown, I can not reproduce your issue. The problem is something unique to your setup.



  • More than happy to help you work through it..

    Excellent. How would you like to proceed?



  • FWIW, I'm still interested in trying to work through it with anyone who is up for it.

    I guess I'll stop updating this thread, but if someone is reading this in 2028, and is interested in helping, please assume I am still interested.


Log in to reply