pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs



  • Trying to setup a simple NAT for a single TCP port 8080,
    to be able to access a simple web server on raspberry rPI (yes on 8080) anywhere from Internet.

    Yet I'm unable to get it working - but it is more strange issue than one would expect.
    I would say that I has to be making a configuration issue, BUT ... ! let us see:

    Setup goes like this: ISP (ubnt client as gw) -> (WAN port, assigned 192.168.1.99) SG-1100 box (LAN port) -> raspberry (web server 192.168.2.2:8080).

    My ISP gives me ethernet cable right from their UBNT client (subnet 192.168.1.1/24), and the ubnt happily gives anything connected to it an IP address, so including the SG-1100 pfSense box.

    So obviously the primary test is to see if the ISP is /or is not/ blocking the incoming wan traffic.
    (it is not blocked, see below).

    If I replace the SG-1100 with a Mikrotic Hex router, or, connect the rPI directly to ISP, in both cases I can access the web server from Internet (via LTE mobile phone etc.) and also via https://www.yougetsignal.com/tools/open-ports/ .
    So I know it works and it gets through all the way, and stops on the pfSense.
    Also, the rPI is able to get to internet, and its IP address is correctly assigned via the pfSense DHCP server.

    I have also replaced (just for a test) the pfSense with Mikrotik and set up the DST-NAT on the Mikrotik.
    And it works OK, no issues, on the first try. So I know the ISP could be DST-NAT'd, and so I do expect pfSense box to do so as well.

    My SG-1100 box also has the pfBlockerNG installed and active.
    Is there anything I have to do with a regard to this (in order to get the NAT working)?

    I have also done a packet capture on the ISP outlet - it IS coming from ISP (this one is a test with yougetsignal tool):
    (FYI) this capture is done by Mikrotik device:
    Image1.png

    I have read all of this and verified every single step:
    http://doc.pfsense.org/index.php/Port_Forward_Troubleshooting
    and even so the pfSense box is not doing the NAT thing.

    Troubleshooting Points

    1. rules are correct, cleared and repeated multiple times... no avail
    2. testing from outside (NAT does not work regardless on what I set within the NAT reflect setup)
    3. done, nothing shows up in a log !!!! yet the ISP sends the packet !.
    4. nothing shows up in there (states)
    5. using packet capture, including promiscuous mode within the pfSense box, and the packet is not captured
      (contrary to a swap where pfSense box is replaced with Mikrotik router or even my notebook and I do get the TCP8080 packet !)

    Common Problems

    1. correct
    2. not applicable, it works with pfSense box removed from the network chain
    3. server does that and so it gets to internet
    4. server does that
    5. no blocking on ISP side, confirmed (f.ex. by removing pfSense box)
    6. not doing that (instead testing from outside)
    7. no such setup (unless there is some hidden trickery by pfBlockerNG ? can anyone help here?)
    8. not applicable, see way above
    9. no captive portal used anywhere
    10. single wan
    11. no floating rules present
    12. no "disable reply-to" checked
    13. not checked
    14. no gw set
    15. upnp disabled

    Anyone, any ideas ?

    The MOST INTERESTING part is that there is NO packet in the capture !
    ... yet it has to be on wire, as indirectly repeatedly proven by swapping pfSense by another box.

    what is going on in here ???

    • pfSense is set up per the manuals and etc.
    • packet is on wire, yet pfSense is not showing it (capture, logs)
    • all logs are silent regarding the port 8080 (yet other stuff pours in the logs as expected)
    • standard internet access (LAN -> WAN) works OK, and pfBlockerNG/DNSBL blocks all the domains etc. OK
    • where am I making an error ? is there some hidden configuration to take care of ? or is the pfBlockerNG ? or pfSense error?


  • @icansoft said in pfSense NAT not working, nor showing relaed incoming packet in Packet Capture (even yet it is on wire) or in logs:

    192.168.1.99

    First and foremost.. port forwarding works fine on pfsense. If it didn't then there would be 100,000 + posts about it complaining.

    So that leaves a configuration or some other issue that is in the way. You state you have a private space IP on your WAN so right there you are double NAT'ed. I would ask if your ISP is truly forwarding those packets to you.. Is your public IP address yours alone? or shared by many?? The clue that your do not see the traffic in your packet capture says that the traffic is not making it to you.



  • Hello!

    Setting for :

    Interfaces -> WAN -> Block private networks and loopback addresses

    ?

    John



  • Well, I agree with your post, pfSense most works well for many, and this is why I have bought it. I think it is a great thing !

    Yet - why can't I get the NAT to run ? what am I specifically doing wrong ? there must be something hmm
    I would typically blame myself for a wrong configuration (and fix it), but this one goes, as it seems, much deeper, and so I have posted it to this forum.

    Specifically - Given that if I physically replace pfSense box with a Mikrotik box, and do a DST NAT on that very Mikrotik router, with a same setting as for the pfSense, then the (Mikrotik) NAT works fine and I can get to the rPI from internet.
    Once the pfSense is put back, no more NAT for me.

    @serbus - unchecked:
    495ea8e9-59ab-466d-8558-c2a2e3117ff6-obrazek.png

    One more detail, the ISP "gives" me a static Public IP just for myself (I pay for it). True, granted, it goes through their own system (and gets through their routers], yet as I wrote above, if I remove the pfSense then I can reach the local resource (my webserver) from Internet, just as I need to.
    So what to do with the pfSense, how to configure it properly to works just as well as the (for example) Mikrotik does ?
    I want to use the pfSense as my border gw and firewall.

    Or is it due the pfBlockerNG ?
    Why won't the SG-1100 box t show the dreaded packet in the capture in a first place ?



  • @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    has the pfBlockerNG installed and active.

    Turn it off and see if things start working.

    Something should show up either way in the firewall logs if the packets are making it there. with pfblocker I believe you would have to enable the logs on the pfblocker dashboard for traffic it blocks.



  • Yes, that is something I'm afraid of, that the pfBlockerNG is the culprit.
    I still keep my fingers crossed that it is not the case, but who knows - OK let me test it.

    Turning it off, how ? just by unchecking the Firewall-pfBlockerNG-Enable checkbox ?

    Anyway, would that explain why I do not see anything related to WAN TCP 8080 traffic in any of the logs (including any of the Firewall and pfBlockerNG logs / files)... ? and no trace in the packet capture (even in a promiscuos mode) ?
    I have found some posts about the TCP port 0 issue (that would be the ISP error), but judging by the packet capture (see the original post) it is not the case.... hmm

    hey, thanks for spending time with this, appreciated



  • @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    just by unchecking the Firewall-pfBlockerNG-Enable checkbox ?

    Yes. The pfblocker firewall rules on your WAN should disappear.


  • LAYER 8 Global Moderator

    @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    and no trace in the packet capture (even in a promiscuos mode) ?

    If your not seeing it with a sniff on pfsense, then it would be IMPOSSIBLE for it to forward it to anything..

    If you can not see the traffic on pfsense wan, then your issue is upstream of pfsense.



  • @johnpoz TRUE

    NAT not working is the trigger. By investigating that, I have found all the additional info presented here.

    So, why I can capture and DST-NAT the traffic on any other box I have here on hand, just not with the SG-1100 ?
    Same config, same wires, repeated tests... and the SG-1100 passes the wan-lan traffic for the users (internet browsing from a LAN side)...

    I was afraid that there is some pre-filter or such, that would remove the packets before the Packet Capture, is that possible ?

    I'm puzzled.



  • @chpalmer

    pfBlockerNG off, this is all that shows up in there now:

    4e4ce97d-80ca-4d02-b2a4-8b286984648b-obrazek.png

    82f1000b-2695-4f08-b3ff-f4a9cc4e8868-obrazek.png

    yet no activity

    Seems like I'll first need to figure out why the SG-1100 box is not getting the TCP 8080 traffic.
    I was afraid it is due to some (FW ?) rules, well is that possible ?

    Tried to re-connect WAN, even to renew to get a different IP... no difference
    1259f6cb-be7d-4030-b146-865296e0d435-obrazek.png



  • Make sure the device in front of pfsense is truly sending the traffic to the pfsense WAN address.

    You have shown at least two different WAN addresses to this poing.. .99 and .55


  • LAYER 8 Global Moderator

    @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    that would remove the packets before the Packet Capture, is that possible ?

    No... If your not seeing the packets on pfsense with packet capture.. Your either sniffing on the wrong interface or traffic is not getting there. Do you see the outbound and return traffic on this sniff?



  • @chpalmer
    the .99 and .55 are two consecutive assignments by ISP's DHCP server, as I have renewed it during testing.
    Good catch, but this unfortunately is not the issue.

    @johnpoz
    sniffing on WAN (by pfSense system), seeing all the various traffic but none of the TCP 8080 which gets generated by various test tools (like the internet's port forwarding tester).
    Unfortunately I do not own HUB for 100Mbit network to see what is on the line. So I have only the pfSense as a direct capture, or I can do indirect capture by replacing pfSense by Mikrotik or my notebook (wireshark). On both I do see the TCP 8080 traffic, once triggered of the Internet by the test web tool.

    Would that ever be possible that the UBNT client (UBNT bullet) "detects" that it has a pfSense box connected and selectively decides not to send a TCP port 8080 flow to it ? and once the pfSense box is replaced with anything else I have on hand, it suddenly works ? why, how ?

    I think I'll go for a complete fresh reinstall. Can't come up with anything else on the pfSense box.
    OK one more test, to disconnect the pfSense from ISP and to connect it to a simulated ISP (by my local standalone router). This might show some hints.

    BTW. thanks guys for helping out, appreciated, +1 karma
    yet I'm puzzled ... what is going on



  • So how would someone on the outside such as one of us get you your box? How does your ISP know to send that traffic to your box?



  • @chpalmer by going to a specific public IP (a.b.c.d), which the ISP routes through their system down to the POP where I'm connected. That is the UBNT Bullet wifi client, which provides the 192.168.1.1/24 network.
    If I connect the Mikrotik router to it (WAN side of the router), setup the textbook Mikrotik's DST-NAT for the 8080, it works like charm and by going a.b.c.d:8080 (like via my LTE phone), it shows the correct webserver's output.
    The webserver (rPI) is on the LAN side of the Mikrotik (subnet 192.168.2.1/24).

    I hoped to connect the webserver to pfsense box LAN and have the pfSense to DST-NAT the same way as the Mikrotik does (but without having the Mikrotik in there).
    Which shall be doable by just switching the Mikrotik and pfSense in place, and configuring it properly.
    And this is where I failed... it seems
    For the LAN->WAN part the pfSense works like a charm, including the pfBlockerNG, and I can browse the web.
    But the opposite direction, no joy.


  • LAYER 8 Global Moderator

    Here is what I tell every user that is having issues with pfsense forwarding.. Go to can you see me, put in the port your trying to forward on pfsense... If you don't see it on pfsense wan, then just plain impossible for pfsense to forward what it never sees.


  • LAYER 8 Moderator

    All I read up until now is "other boxes do it fine" etc etc.
    What I don't read is: How the hell is the ISPs UBNT Bullet WiFi thingy that gives out a private IP space (.1.1/24) and does DHCP getting to know it should forward anything at all to your changing DHCP address where you put pfSense at?

    I've not read anything how the upstream box is configured up until now. If I've missed it somehow I apologize but how should traffic to any external WAN IP, that gets only forwarded to your ISP endpoint POP (the UBNT box) get to a router afterwards without any settings on the UBNT side to actually forward anything to an IP? And a dynamic one nonetheless? That doesn't make sense to me.



  • @johnpoz

    Here is what I tell every user that is having issues with pfsense forwarding.. Go to can you see me, put in the port your trying to forward on pfsense... If you don't see it on pfsense wan, then just plain impossible for pfsense to forward what it never sees.

    I hear you man, this was the first thing I've done, before ever posting this. But it got so wierd I had to - see as I wrote about the other routing working fine in there and pfSense nada. Quite a mystery worth figuring out.

    So I've run a following test:

    Disconnected the pfSense box from ISP and connected it to a plain standalone router LAN (which simulated the ISP / Internet), set up just for this test, with a factory reset done.
    To the same router's LAN (another switch port) I've connected a notebook.
    The pfSense WAN got assigned 192.168.88.253.
    So, off to the browser on the notebook, type 192.168.88.253:8080, and boom voila, the web server page is served !

    So to conclude

    • the initial issue - no NAT action for my setup - brought me so far
      (thanks for keeping up with me, appreciated)
    • I definitelly have to agree, due to the final test, that pfSense NAT works OK, including the pfBlockerNG
      (confirming the moderator's posts above), and I'm glad the issue is most likely not in there specifically, ufff
    • all the other boxes are OK and are doing the NAT thing, with the way the ISP is configured, just the pfSense is not

    I'm grateful for all the comments, which helped to lead me to figuring this out. Thanks.

    I suppose this thread might get moved to a different topic than NAT then.

    @JeGr unfortunately, the UBNT client is out of my reach and I have no way to get it
    Which makes me wonder, how comes that, the Mikrotik router does the NAT fine, and the pfSense not ?
    I do not blame pfSense, just help me out with what to configure there...
    that is what I'm after.
    Factory defaulted Mikrotik with a single rule added works. No other specific config due to whatever the ISP might be doing. So if that router handles it well, I do expect the pfSense to do the same - but it seems it needs to be configured somehow more. Any ideas ?

    If I figure it out, I'll post it here.
    I need to (and want to) get the pfSense to work in the very setup.


  • LAYER 8 Moderator

    @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    Which makes me wonder, how comes that, the Mikrotik router does the NAT fine, and the pfSense not ?

    Now we're talking. Exactly!

    Factory defaulted Mikrotik with a single rule added works. No other specific config due to whatever the ISP might be doing. So if that router handles it well, I do expect the pfSense to do the same - but it seems it needs to be configured somehow more. Any ideas ?

    Somehow I smell "darkmagic(tm)". Almost sounds like some kind of upnp/natpnp at work that automagically requests the port upstream from the UBNT device, which in turn maps the port, which in turn leads to the package being delivered. As I don't play around with Mikrotik that much, I don't know if that's what they do per default, but that would be one guess (some kinda NAT punch technique).



  • @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    Factory defaulted Mikrotik with a single rule added works.

    None of us would be able to understand why this would be unless your ISP had done port forwarding to it's IP address.. which your pfsense box is obviously not obtaining. I.E. if your Microtik box got 192.168.1.60 and they set up a port forward to that box then your pfsense would also have to get .60 in order to receive that traffic. Since it does not we can only guess.



  • @JeGr said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    Almost sounds like some kind of upnp/natpnp at work that automagically requests the port upstream

    ^^ or that..


  • LAYER 8 Moderator

    @chpalmer said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    Factory defaulted Mikrotik with a single rule added works.

    None of us would be able to understand why this would be unless your ISP had done port forwarding to it's IP address.. which your pfsense box is obviously not obtaining. I.E. if your Microtik box got 192.168.1.60 and they set up a port forward to that box then your pfsense would also have to get .60 in order to receive that traffic. Since it does not we can only guess.

    @chpalmer also has a good point. Does your mikrotik router get the same IP every time you plug it in behind the UBNT device? Does it also work every time? What if you shut it down and statically configure pfSense' WAN to the IP that the mikrotik had before?

    @chpalmer said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    ^^ or that..

    that's what happens if we write at the same time, hehe 😄



  • @JeGr

    Somehow I smell "darkmagic(tm)". Almost sounds like some kind of upnp/natpnp at work that automagically requests the port upstream from the UBNT device, which in turn maps the port, which in turn leads to the package being delivered.

    Right, right, I'm going to investigate. Not that I know Mikrotik any better to be able to answer this.

    Anyway - have a look at the captured packet on top, it comes with the Source IP as original and unmodified, and target is the IP of the only device on the UBNT (ISP) subnet. If that is that Mikrotik (or my NB for that matter), the packet is captured... if it is pfSense, no luck.

    @chpalmer

    The ISP has no idea what is on my network, and I've just verified it by changing MAC on the Mikrotik, it gets assignment from UBNT (different .ip) and yet it DST-NATs OK.



  • OK by thinking this through (and thank you all for helping me out to do so),
    there definitelly must be some sort of detection mechanism on the ISP side.

    I think @chpalmer has the best answer, which makes the best of the sense, ... yet there must be something which I'm missing (and therefore it would be hard for others to figure it out as well).

    I'll write an email with a query to the ISP, on how the configuration is done.
    Perharps they somehow detect the device connected to a client of theirs, and it just happen that the pfSense is not handled properly on the ISP's side.
    No idea if I ever get a response though.

    Anyway, I would consider the case closed. There is no more that could be done on the pfSense side.

    Many thanks to all, you guys helped.
    Please eventually move this thread to a more proper topic, if required.


  • LAYER 8 Moderator

    @icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

    If that is that Mikrotik (or my NB for that matter), the packet is captured... if it is pfSense, no luck.

    Yeah we get that. But even UBNT has no automagic guessing crystal ball as to automagically guess/read, what it will hand out your router as DHCP address and again automagically forward every packet to that address. Also: if you would attach that line from your UBNT to a switch and then to both the pfSense and mikrotik router, it would hand out two addresses as that's what DHCP does. So where should it route your packets to? That's why I'm guessing strange things happening like some UPnP/NATPnP magic.

    Various home/SOHO routers like those AVM boxes have an option like "allow devices on the LAN to ask for port forwardings via UPnP". If something like this is working on the Mikrotik (as in you create a forwarding and it sends the request upstream), no wonder it "just works". But as my example above: what would happen if you'd attach multiple routers behind it? The box could never just assume "send anything to xy" with more then one DHCP client.

    So either the Mikrotik is doing some UPnP (or other magic) that the UBNT box is allowing or it gets the same IP everytime and your ISP configured the box to always route it to e.g. IP .99 or anything the like.

    I'd recommend using a static IP on the WAN side of your pfSense box anyway. I'd talk to the ISP if they could disable / rearrange the DHCP to exclude e.g. 192.168.1.2 or .1.254 from the DHCP segment and if they could please full-forward all traffic to said IP .2 or .254

    So you don't have to guess what does what and simply know it will be sent there to which device whatsoever has it :)



  • @JeGr

    Must be somewhere within these lines. There is simply no other answer.

    Quite possible that the Mikrotik has set it up somehow, and it persisted for some time, but it got (who knows) cleared when switched to pfSense. Without additional info, it is now all just guessing.

    I'll query the ISP on what are they doing there. Doubt they'll talk... but that is a different story.

    I feel bad the pfSense was the one who got the blame, I have to apologize to it for that :) but the best apology is that I yet want to keep it and use it, it is a great thing, with a great support BTW.
    But hey, without going this way, I would have not got this far. This whole discussion helped, a lot !



  • Oh, and I forgot to write down a "solution" that is right now working within my place, at the moment.
    In case anyone else would face a similar issue (with a small rural european ISP... and a unknown uppath setup):

    Originally, I had the pfSense connected right to the ISP's POP; this did not go well for the NAT (even though it was finally not the issue of the pfSense's NAT itself, but an issue of the system setup including the pfSense);
    now I have simply inserted a vanilla Mikrotik router in between (ISP and pfSense).
    The router does src-nat and dst-nat, and the pfSense nicely humms right behind that, doing the firewall job as well as the NAT.
    Of course this is not a clear way to do, nor a solution I'd ever like to keep, but it is a band-aid I have just needed.

    Thanks to all of you within this thread, your comments allowed me to figure this quick dirty fix up !


  • LAYER 8 Moderator

    I'll query the ISP on what are they doing there. Doubt they'll talk... but that is a different story.

    Just as a quick follow up: If you pay for your own public IP to get forwarded to you, they should have no trouble setting their UBNT POP the way you want. Otherwise what's the gain in paying for something you can't successfully use all the way you want? ;)