Port 80 not forwarding

johnpoz

@elmojo openvpn is simple and straight forward. I do believe at some point wireguard will give it a run for its money. Especially if they come up with the QR code to just scan ;)

There was a bit issue, of whats the term to use - quagmire prob works ;)

It was in, and then it was pulled, and some social media craziness, etc. Now there is a package again listed..

I'm a fan of openvpn. Its rock solid, used it for years and years.. It can do tcp for your tunnel, which I do not believe WG supports.. While tcp is not optimal for the vpn, it can come in quite handy when the UDP port is blocked - its a given that 443 is almost always open ;) And you can connect via proxy even.. So when your say at the office, and they run a proxy, and you want to vpn into your home network ;) Use to do that daily before covid and now just full time work from home.

Elmojo

@johnpoz That's good info, thanks.
I'll probably stick with OpenVPN, since it seems to be a bit more robust and mature.

Elmojo

Okay, this is giving me a headache. I thought we had it all sorted out. I was able to forward ports, and checking them against the canyouseeme site was reporting them as open.
However, as of today, they are suddenly closed again?!
I was attempting to forward a different set of ports, for another container.
Once I realized it wasn't working, I set everything back to the way it was, but even the original rules aren't working now. What gives?
Here are my NAT and WAN rules screens. See anything I broke that could be accounting for this behavior?

I can sniff the traffic on the WAN and see it hitting the wall, but canyouseeme reports an immediate "connection refused". Ports that aren't forwarded tend to time out after about 10 secs, so I know it's doing something, just not what I've asked it to do.
I'm too old for this... lol

johnpoz

@elmojo just out of curiosity why would you forward 1194 to the wan? Openvpn is listening on that - there is no need to forward it.

So when you sniff on the wan do you see a R go back?

Your forward for 80 is showing an active state that /1/1 there - so did your server refuse it?

https://docs.netgate.com/pfsense/en/latest/troubleshooting/nat-port-forwards.html

Elmojo

@johnpoz
The 1194 forward is part of the OpenVPN wizard setup. I don't believe I did it manually.
I don't see any explicit blocks in the main log, but when I sniff the WAN I see it hit from outside, and a response go back from my side. It doesn't say what sort of reply it is, only that it's on some random port. Looks like this... (this is 2 attempts)

23:42:49.722015 IP 52.202.215.126.36854 > 174.xx.xxx.184.80: tcp 0
23:42:49.722035 IP 174.xx.xxx.184.80 > 52.202.215.126.36854: tcp 0
23:42:51.138779 IP 52.202.215.126.36860 > 174.xx.xxx.184.80: tcp 0
23:42:51.138801 IP 174.xx.xxx.184.80 > 52.202.215.126.36860: tcp 0

johnpoz

@elmojo said in Port 80 not forwarding:

The 1194 forward is part of the OpenVPN wizard setup.

No it isn't.. You had to have done that manually. It makes zero sense.. The wizard will make your wan rule for you, but it would not do a port forward.

23:42:49.722035 IP 174.xx.xxx.184.80 > 52.202.215.126.36854: tcp 0

Your server you forwarded too sent a RST! Telling them to go away..

If you would up the verbosity of the sniff, you would see it was RST..

When you do that test do it on the lan side your sniff, and you will see your server sending it back.. Unless you had setup a specific reject rule pfsense wouldn't send a RST.

Did you put a specific rule in your floating tab?

Elmojo

@johnpoz
Beats me, I set it up a while back, and I'm not even using it right now. Ignore it for the current conversation. I can delete the rule if you think it may be having an impact?
There are no floating rules.
Nothing has changed from when you helped me the other day. I literally have had no time to work on it any more. I just sat down this evening to try to get SWAG set up, and it needed a different set of ports forwarded. I plugged in those forwards, noticed they weren't working, removed the entries, and here were are. Dunno man, it's baffling.
I'll have to tackle it more tomorrow, if you're available. It's nearly midnight and I'm beat. :)

Well son of a.... I think I just figured it out!
I distinctly recall you telling me before that I didn't need any service running on the server (LAN) side in order to check if the ports were open, so I hadn't bothered spinning up the container until I got that sorted out. However, I just went back, turned off SWAG, and fired up NginxProxyManager. This was really just to make sure I hadn't broken anything on the server side. I clicked the port test button, and wouldn't you know, they're testing as open now!
I guess not having anything running on the server to "accept" the request, for lack of a better term, it was rejecting the packets. Man, I wish I had tried this 2 hours ago. :/
At least I can get some sleep now. lol

johnpoz

@elmojo You don't need to have anything listening if your going to sniff to see if the traffic gets to psfense..

Bob.Dig

I can't open port 443 and I don't know why, but I have another router in front of pfSense, so it is hard to tell, which device is responsible for that... but pfSense is the exposed host of that first router.

johnpoz

@bob-dig said in Port 80 not forwarding:

but pfSense is the exposed host of that first router.

If pfsense does not see traffic get to it on 443 (when you sniff on pfsens), then "something" upstream didn't allow it, be it that router in front of pfsense (likely suspect) or your ISP. etc..

Bob.Dig

@johnpoz The Problem for me, the ISP is not known for doing that and also the router is not. I can see which port the router has opened to WAN and there is no 443. Do you have any tip for seeing it in pfSense?
What I did after noticing this problem was placing a reject tcp 443 floating WAN in on top and logging and doing a port test on a website, still port is stealth and no log entry...
Also sorry, I have not read this whole thread.

johnpoz

@bob-dig said in Port 80 not forwarding:

still port is stealth and no log entry..

Then it didn't get to pfsense.. Pfsense has zero control over what gets to it or not.. It can not do anything with traffic it never sees, be that ignore it, or forward it or reject it.

If you sniff on pfsense when you do a test from can you see me .org for example - and you don't see it, then it didn't get to pfsense. Something upstream prevented it, or your sending to the wrong IP..

You could be standing at the plate, all ready to hit that homerun. If the pitcher never throws you the ball.. Nothing you can do about it.

Bob.Dig

@johnpoz I don't know about American Football ;) but I was able to open the web-ui of that router to the public, so it is not an ISP thing. So I guess the router firmware is faulty, I will report to them. Still, this problem seems to "big" that nobody has found it so far... thx John.

johnpoz

@bob-dig said in Port 80 not forwarding:

Still, this problem seems to "big" that nobody has found it so far

Who says its a problem.. Could just be operator error. If you setup pfsense as dmz host, ie all ports forwarded to pfsense IP in your first router. What if your 1st router is actually using 443 for its web interface, and say you enabled remote management of this router.

How would it forward 443, if its using it, etc.

Possible your using a vpn? And when you go to can you see me, its sending the traffic to your vpn IP..

Maybe your isp is blocking 443 inbound? Do other ports work?

Don't know your setup, maybe you have UPnP enabled on it, and you have some other device along side pfsense on your 1st routers lan having 443 forwarded to it.

Lots of things that could be causing what your seeing other than "bug/problem" with your 1st router.

American football ;) hehehe Ok how about this analogy. Kind of hard to take your penalty shot if there is no ball..

Bob.Dig

@johnpoz Damn, I almost cut myself off from home and all the services, because I am not at home for some weeks and I just disabled the exposed host function in the router and I was connected to it via pfSense VPN...
I really had big luck, that the ongoing rdp-connection wasn't canceled, otherwise I would have some serous problems...

Anyways, what I found out by disabling the exposed host function and enabling only one explicit port forward on port 443 to pfSense for a short time was, that this time I got a different result on grc.
I always did testing on port 80 and 443 and this time 443 was still stealth but Port 80 was just closed.
I don't use Port 80 on pfSense WAN and I think port 80 closed is the normal behavior of the first router, because it is not in stealth mode and exposed host was not activated. So now I do think it is pfSense!

Any tips how to investigate further? Everything you explicitly mentioned to me I already had checked.

Bob.Dig

@johnpoz So I just did a packet capture of the port scan with grc, exposed host active and it looks like it is coming through? I never do packet capture, please be calm with me.
It looks like this (there is more) but I am not sure what it means, could need some help here.

13:36:42.561714 IP 172.25.0.2.39667 > 4.79.142.202.443: tcp 517
13:36:42.562998 IP 172.25.0.2.8719 > 4.79.142.192.443: tcp 517
13:36:42.734132 IP 4.79.142.202.443 > 172.25.0.2.36858: tcp 1452
13:36:42.734233 IP 4.79.142.202.443 > 172.25.0.2.36858: tcp 1452
13:36:42.734332 IP 172.25.0.2.36858 > 4.79.142.202.443: tcp 0
13:36:42.742283 IP 4.79.142.202.443 > 172.25.0.2.39667: tcp 1452
13:36:42.742430 IP 4.79.142.202.443 > 172.25.0.2.39667: tcp 1452
13:36:42.742527 IP 172.25.0.2.39667 > 4.79.142.202.443: tcp 0
13:36:42.750059 IP 4.79.142.192.443 > 172.25.0.2.8719: tcp 1452
13:36:42.750193 IP 4.79.142.192.443 > 172.25.0.2.8719: tcp 1452
13:36:42.750275 IP 172.25.0.2.8719 > 4.79.142.192.443: tcp 0
13:36:42.875387 IP 4.79.142.202.443 > 172.25.0.2.41225: tcp 117
13:36:42.886209 IP 4.79.142.206.42743 > 172.25.0.2.443: tcp 0
13:36:42.898437 IP 4.79.142.202.443 > 172.25.0.2.36858: tcp 964
13:36:42.900825 IP 172.25.0.2.36858 > 4.79.142.202.443: tcp 150
13:36:42.906914 IP 4.79.142.202.443 > 172.25.0.2.39667: tcp 964

4.79.142.202 is grc port tester
172.25.0.2 is my pfSense

But I think the port tester is also testing for other things, so is port 443 on pfSense touched or is is not?

Here is the floating rule:

Bob.Dig

After doing another port test with port 80 it looks like port 80 is recognized in the packet capture and port 443 is not.

But both rules show 0 States and there are no log entries which is weird, isn't it?

johnpoz

@bob-dig state counters don't always update instantly. You might have to refresh that page if you just created the rules, and just generated traffic.

Where exactly are you sniffing,

13:36:42.562998 IP 172.25.0.2.8719 > 4.79.142.192.443: tcp 517

That clearly shows an answer from your pfsense IP as you mentioned. But 172.25 is RFC1918, its not going to get back to where you sent it.. Unless something upstream is natting that, I take your 1st router.. Since your behind a double nat.

But yes, pfsense saw that traffic and answered..

Its also possible that rule didn't get triggered if you had a state already for that traffic..

Bob.Dig

@johnpoz said in Port 80 not forwarding:

Where exactly are you sniffing,

I sniffed on WAN.

Port 80 it looks like this:

14:36:49.624193 IP shieldsup.grc.com.43073 > 172.25.0.2.http: tcp 0
14:36:50.139169 IP shieldsup.grc.com.43073 > 172.25.0.2.http: tcp 0
14:36:50.653947 IP shieldsup.grc.com.43073 > 172.25.0.2.http: tcp 0
14:36:50.995847 IP 93.184.220.29.http > 172.25.0.2.31946: tcp 0
14:36:51.168731 IP shieldsup.grc.com.43073 > 172.25.0.2.http: tcp 0
14:36:51.288596 IP 93.184.220.29.http > 172.25.0.2.18060: tcp 0
14:36:51.291502 IP 93.184.220.29.http > 172.25.0.2.28221: tcp 0

Port 443 like this:

14:39:44.028582 IP 172.25.0.2.10728 > www.grc.com.https: tcp 837
14:39:44.196798 IP www.grc.com.https > 172.25.0.2.10728: tcp 517
14:39:44.197041 IP www.grc.com.https > 172.25.0.2.10728: tcp 1452
14:39:44.197054 IP www.grc.com.https > 172.25.0.2.10728: tcp 649

So no Port 443 on my side, searched the whole capture.

This time I was resetting the state table before and after each test.

Still only 0/0 states, but maybe this is normal because of the reject? But also no log entries for both.

I still think it is something with pfSense, but it is hard to tell for me and because I am not home I can't test everything.

johnpoz

@bob-dig said in Port 80 not forwarding:

Still only 0/0 states, but maybe this is normal because of the reject? But also no log entries for both.

No - here I created a reject for 80..

I then created some traffic to me from can you see me. Rejected, logged

If I then look at the floating rule - you can see it was evaluated and how much traffic

If you increase the verbosity of your sniff, you can see the Syns and Acks or RST right in the output. So above is viewing it in wireshark (easier to follow and see exactly)... But there is from the output right in pfsense.

08:05:22.032521 00:01:5c:b9:06:46 > 00:08:a2:0c:e6:25, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 48, id 18315, offset 0, flags [DF], proto TCP (6), length 60)
    52.202.215.126.45648 > 64.53.x.x.80: Flags [S], cksum 0x38a4 (correct), seq 3282396933, win 26883, options [mss 1460,sackOK,TS val 1682776206 ecr 0,nop,wscale 7], length 0

08:05:22.032591 00:08:a2:0c:e6:25 > 00:01:5c:b9:06:46, ethertype IPv4 (0x0800), length 54: (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    64.53.x.x.80 > 52.202.215.126.45648: Flags [R.], cksum 0x8e52 (correct), seq 0, ack 3282396934, win 0, length 0

You can clearly see pfsense sent back RST via the Flags [R]

edit:
You know when your shiffing are you letting it log more than 100 packets.. Quite possible with all your normal https traffic, your just hitting 100 before you actually generate traffic to you.