pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs

icansoft

Trying to setup a simple NAT for a single TCP port 8080,
to be able to access a simple web server on raspberry rPI (yes on 8080) anywhere from Internet.

Yet I'm unable to get it working - but it is more strange issue than one would expect.
I would say that I has to be making a configuration issue, BUT ... ! let us see:

Setup goes like this: ISP (ubnt client as gw) -> (WAN port, assigned 192.168.1.99) SG-1100 box (LAN port) -> raspberry (web server 192.168.2.2:8080).

My ISP gives me ethernet cable right from their UBNT client (subnet 192.168.1.1/24), and the ubnt happily gives anything connected to it an IP address, so including the SG-1100 pfSense box.

So obviously the primary test is to see if the ISP is /or is not/ blocking the incoming wan traffic.
(it is not blocked, see below).

If I replace the SG-1100 with a Mikrotic Hex router, or, connect the rPI directly to ISP, in both cases I can access the web server from Internet (via LTE mobile phone etc.) and also via https://www.yougetsignal.com/tools/open-ports/ .
So I know it works and it gets through all the way, and stops on the pfSense.
Also, the rPI is able to get to internet, and its IP address is correctly assigned via the pfSense DHCP server.

I have also replaced (just for a test) the pfSense with Mikrotik and set up the DST-NAT on the Mikrotik.
And it works OK, no issues, on the first try. So I know the ISP could be DST-NAT'd, and so I do expect pfSense box to do so as well.

My SG-1100 box also has the pfBlockerNG installed and active.
Is there anything I have to do with a regard to this (in order to get the NAT working)?

I have also done a packet capture on the ISP outlet - it IS coming from ISP (this one is a test with yougetsignal tool):
(FYI) this capture is done by Mikrotik device:

I have read all of this and verified every single step:
http://doc.pfsense.org/index.php/Port_Forward_Troubleshooting
and even so the pfSense box is not doing the NAT thing.

Troubleshooting Points

rules are correct, cleared and repeated multiple times... no avail
testing from outside (NAT does not work regardless on what I set within the NAT reflect setup)
done, nothing shows up in a log !!!! yet the ISP sends the packet !.
nothing shows up in there (states)
using packet capture, including promiscuous mode within the pfSense box, and the packet is not captured
(contrary to a swap where pfSense box is replaced with Mikrotik router or even my notebook and I do get the TCP8080 packet !)

Common Problems

correct
not applicable, it works with pfSense box removed from the network chain
server does that and so it gets to internet
server does that
no blocking on ISP side, confirmed (f.ex. by removing pfSense box)
not doing that (instead testing from outside)
no such setup (unless there is some hidden trickery by pfBlockerNG ? can anyone help here?)
not applicable, see way above
no captive portal used anywhere
single wan
no floating rules present
no "disable reply-to" checked
not checked
no gw set
upnp disabled

Anyone, any ideas ?

The MOST INTERESTING part is that there is NO packet in the capture !
... yet it has to be on wire, as indirectly repeatedly proven by swapping pfSense by another box.

what is going on in here ???

pfSense is set up per the manuals and etc.
packet is on wire, yet pfSense is not showing it (capture, logs)
all logs are silent regarding the port 8080 (yet other stuff pours in the logs as expected)
standard internet access (LAN -> WAN) works OK, and pfBlockerNG/DNSBL blocks all the domains etc. OK
where am I making an error ? is there some hidden configuration to take care of ? or is the pfBlockerNG ? or pfSense error?

chpalmer

@icansoft said in pfSense NAT not working, nor showing relaed incoming packet in Packet Capture (even yet it is on wire) or in logs:

192.168.1.99

First and foremost.. port forwarding works fine on pfsense. If it didn't then there would be 100,000 + posts about it complaining.

So that leaves a configuration or some other issue that is in the way. You state you have a private space IP on your WAN so right there you are double NAT'ed. I would ask if your ISP is truly forwarding those packets to you.. Is your public IP address yours alone? or shared by many?? The clue that your do not see the traffic in your packet capture says that the traffic is not making it to you.

serbus

Hello!

Setting for :

Interfaces -> WAN -> Block private networks and loopback addresses

?

John

icansoft

Well, I agree with your post, pfSense most works well for many, and this is why I have bought it. I think it is a great thing !

Yet - why can't I get the NAT to run ? what am I specifically doing wrong ? there must be something hmm
I would typically blame myself for a wrong configuration (and fix it), but this one goes, as it seems, much deeper, and so I have posted it to this forum.

Specifically - Given that if I physically replace pfSense box with a Mikrotik box, and do a DST NAT on that very Mikrotik router, with a same setting as for the pfSense, then the (Mikrotik) NAT works fine and I can get to the rPI from internet.
Once the pfSense is put back, no more NAT for me.

@serbus - unchecked:

One more detail, the ISP "gives" me a static Public IP just for myself (I pay for it). True, granted, it goes through their own system (and gets through their routers], yet as I wrote above, if I remove the pfSense then I can reach the local resource (my webserver) from Internet, just as I need to.
So what to do with the pfSense, how to configure it properly to works just as well as the (for example) Mikrotik does ?
I want to use the pfSense as my border gw and firewall.

Or is it due the pfBlockerNG ?
Why won't the SG-1100 box t show the dreaded packet in the capture in a first place ?

chpalmer

@icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

has the pfBlockerNG installed and active.

Turn it off and see if things start working.

Something should show up either way in the firewall logs if the packets are making it there. with pfblocker I believe you would have to enable the logs on the pfblocker dashboard for traffic it blocks.

icansoft

Yes, that is something I'm afraid of, that the pfBlockerNG is the culprit.
I still keep my fingers crossed that it is not the case, but who knows - OK let me test it.

Turning it off, how ? just by unchecking the Firewall-pfBlockerNG-Enable checkbox ?

Anyway, would that explain why I do not see anything related to WAN TCP 8080 traffic in any of the logs (including any of the Firewall and pfBlockerNG logs / files)... ? and no trace in the packet capture (even in a promiscuos mode) ?
I have found some posts about the TCP port 0 issue (that would be the ISP error), but judging by the packet capture (see the original post) it is not the case.... hmm

hey, thanks for spending time with this, appreciated

chpalmer

@icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

just by unchecking the Firewall-pfBlockerNG-Enable checkbox ?

Yes. The pfblocker firewall rules on your WAN should disappear.

johnpoz

@icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

and no trace in the packet capture (even in a promiscuos mode) ?

If your not seeing it with a sniff on pfsense, then it would be IMPOSSIBLE for it to forward it to anything..

If you can not see the traffic on pfsense wan, then your issue is upstream of pfsense.

icansoft

@johnpoz TRUE

NAT not working is the trigger. By investigating that, I have found all the additional info presented here.

So, why I can capture and DST-NAT the traffic on any other box I have here on hand, just not with the SG-1100 ?
Same config, same wires, repeated tests... and the SG-1100 passes the wan-lan traffic for the users (internet browsing from a LAN side)...

I was afraid that there is some pre-filter or such, that would remove the packets before the Packet Capture, is that possible ?

I'm puzzled.

icansoft

@chpalmer

pfBlockerNG off, this is all that shows up in there now:

yet no activity

Seems like I'll first need to figure out why the SG-1100 box is not getting the TCP 8080 traffic.
I was afraid it is due to some (FW ?) rules, well is that possible ?

Tried to re-connect WAN, even to renew to get a different IP... no difference

chpalmer

Make sure the device in front of pfsense is truly sending the traffic to the pfsense WAN address.

You have shown at least two different WAN addresses to this poing.. .99 and .55

johnpoz

@icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

that would remove the packets before the Packet Capture, is that possible ?

No... If your not seeing the packets on pfsense with packet capture.. Your either sniffing on the wrong interface or traffic is not getting there. Do you see the outbound and return traffic on this sniff?

icansoft

@chpalmer
the .99 and .55 are two consecutive assignments by ISP's DHCP server, as I have renewed it during testing.
Good catch, but this unfortunately is not the issue.

@johnpoz
sniffing on WAN (by pfSense system), seeing all the various traffic but none of the TCP 8080 which gets generated by various test tools (like the internet's port forwarding tester).
Unfortunately I do not own HUB for 100Mbit network to see what is on the line. So I have only the pfSense as a direct capture, or I can do indirect capture by replacing pfSense by Mikrotik or my notebook (wireshark). On both I do see the TCP 8080 traffic, once triggered of the Internet by the test web tool.

Would that ever be possible that the UBNT client (UBNT bullet) "detects" that it has a pfSense box connected and selectively decides not to send a TCP port 8080 flow to it ? and once the pfSense box is replaced with anything else I have on hand, it suddenly works ? why, how ?

I think I'll go for a complete fresh reinstall. Can't come up with anything else on the pfSense box.
OK one more test, to disconnect the pfSense from ISP and to connect it to a simulated ISP (by my local standalone router). This might show some hints.

BTW. thanks guys for helping out, appreciated, +1 karma
yet I'm puzzled ... what is going on

chpalmer

So how would someone on the outside such as one of us get you your box? How does your ISP know to send that traffic to your box?

icansoft

@chpalmer by going to a specific public IP (a.b.c.d), which the ISP routes through their system down to the POP where I'm connected. That is the UBNT Bullet wifi client, which provides the 192.168.1.1/24 network.
If I connect the Mikrotik router to it (WAN side of the router), setup the textbook Mikrotik's DST-NAT for the 8080, it works like charm and by going a.b.c.d:8080 (like via my LTE phone), it shows the correct webserver's output.
The webserver (rPI) is on the LAN side of the Mikrotik (subnet 192.168.2.1/24).

I hoped to connect the webserver to pfsense box LAN and have the pfSense to DST-NAT the same way as the Mikrotik does (but without having the Mikrotik in there).
Which shall be doable by just switching the Mikrotik and pfSense in place, and configuring it properly.
And this is where I failed... it seems
For the LAN->WAN part the pfSense works like a charm, including the pfBlockerNG, and I can browse the web.
But the opposite direction, no joy.

johnpoz

Here is what I tell every user that is having issues with pfsense forwarding.. Go to can you see me, put in the port your trying to forward on pfsense... If you don't see it on pfsense wan, then just plain impossible for pfsense to forward what it never sees.

JeGr

All I read up until now is "other boxes do it fine" etc etc.
What I don't read is: How the hell is the ISPs UBNT Bullet WiFi thingy that gives out a private IP space (.1.1/24) and does DHCP getting to know it should forward anything at all to your changing DHCP address where you put pfSense at?

I've not read anything how the upstream box is configured up until now. If I've missed it somehow I apologize but how should traffic to any external WAN IP, that gets only forwarded to your ISP endpoint POP (the UBNT box) get to a router afterwards without any settings on the UBNT side to actually forward anything to an IP? And a dynamic one nonetheless? That doesn't make sense to me.

icansoft

@johnpoz

Here is what I tell every user that is having issues with pfsense forwarding.. Go to can you see me, put in the port your trying to forward on pfsense... If you don't see it on pfsense wan, then just plain impossible for pfsense to forward what it never sees.

I hear you man, this was the first thing I've done, before ever posting this. But it got so wierd I had to - see as I wrote about the other routing working fine in there and pfSense nada. Quite a mystery worth figuring out.

So I've run a following test:

Disconnected the pfSense box from ISP and connected it to a plain standalone router LAN (which simulated the ISP / Internet), set up just for this test, with a factory reset done.
To the same router's LAN (another switch port) I've connected a notebook.
The pfSense WAN got assigned 192.168.88.253.
So, off to the browser on the notebook, type 192.168.88.253:8080, and boom voila, the web server page is served !

So to conclude

the initial issue - no NAT action for my setup - brought me so far
(thanks for keeping up with me, appreciated)
I definitelly have to agree, due to the final test, that pfSense NAT works OK, including the pfBlockerNG
(confirming the moderator's posts above), and I'm glad the issue is most likely not in there specifically, ufff
all the other boxes are OK and are doing the NAT thing, with the way the ISP is configured, just the pfSense is not

I'm grateful for all the comments, which helped to lead me to figuring this out. Thanks.

I suppose this thread might get moved to a different topic than NAT then.

@JeGr unfortunately, the UBNT client is out of my reach and I have no way to get it
Which makes me wonder, how comes that, the Mikrotik router does the NAT fine, and the pfSense not ?
I do not blame pfSense, just help me out with what to configure there...
that is what I'm after.
Factory defaulted Mikrotik with a single rule added works. No other specific config due to whatever the ISP might be doing. So if that router handles it well, I do expect the pfSense to do the same - but it seems it needs to be configured somehow more. Any ideas ?

If I figure it out, I'll post it here.
I need to (and want to) get the pfSense to work in the very setup.

JeGr

@icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

Which makes me wonder, how comes that, the Mikrotik router does the NAT fine, and the pfSense not ?

Now we're talking. Exactly!

Factory defaulted Mikrotik with a single rule added works. No other specific config due to whatever the ISP might be doing. So if that router handles it well, I do expect the pfSense to do the same - but it seems it needs to be configured somehow more. Any ideas ?

Somehow I smell "darkmagic(tm)". Almost sounds like some kind of upnp/natpnp at work that automagically requests the port upstream from the UBNT device, which in turn maps the port, which in turn leads to the package being delivered. As I don't play around with Mikrotik that much, I don't know if that's what they do per default, but that would be one guess (some kinda NAT punch technique).

chpalmer

@icansoft said in pfSense NAT not working, nor showing related incoming packet in Packet Capture (even yet it is on wire) or in logs:

Factory defaulted Mikrotik with a single rule added works.

None of us would be able to understand why this would be unless your ISP had done port forwarding to it's IP address.. which your pfsense box is obviously not obtaining. I.E. if your Microtik box got 192.168.1.60 and they set up a port forward to that box then your pfsense would also have to get .60 in order to receive that traffic. Since it does not we can only guess.