Ports, rules, NAT
-
Hi All,
first time poster - and usually I dont need help with these types of things., though at present I am stumped.
Some background;
I am not a network guy, a firewall guy, nor a Linux/BSD guy, but I do ok in finding my way around these things, I've been using PfSense for about 2 months now and I've managed to setup WAN, VLANS and NATs without issue.
Almost 2 days ago one of my NAT rules stopped working, scanning it via the public IP shows the port filtered, scanning it inside the LAN also shows it as blocked/filtered. I have managed to sort this and it now shows as open, however it still doesn't seem to work.
I have 2 NAT rules and corresponding firewall rules to boot.
Port 25 and port 443 - 25 still works, but 443 only shows incoming connections from limited number of IPs, never the ones I try it from (all external). If I enable NAT reflection I can get to my website using the external name from the internal LAN, disabling this does as expected and disables me from the LAN getting to my external domain over external IP.
Doing a port scan externally I see port 443 as open, port 443 to the internal resource also shows as open on the destination, however port scanning PfSense from internal only shows me port 444 as open, which is the default interface port.
the log does show me one IP trying and being passed by the NAT rule, but I never see my own traffic from another external public IP and isitdownrightnow.com says I am down.
Any pointers?
I've removed the NAT rule and FW rule and re-created them to no avail.
I have not rebooted the VM as many posts suggests this isnt necessary.
The NAT and FW rule exist, the ports show as open from outside if you scan them, but not internally and only traffic is passing via port 25, port 443 does nothing.
I really can't see the wood for the trees here - an extra pair of eyes/thoughts please?
Pf Version 2.4.4-p3
VM - ESXi
Directly connected from my modem, so acting as the external interface (WAN)
Everything else passes out of a second LAN connection. -
you should put some screenshot of your rules / NAT if you want us to be of any help
-
Re - first post, still not familiar with the layout of this forum, it's the first one I've used where the post box appears as half the screen at the bottom split in to two.
I hope these help.
Do bare in mind these were working and I am familiar with NAT rules, there have been no power outages, no host crashes, no VM crashes, no ISP disconnections, I did remove the rule and recreate it but this has made no difference, the port 'appears' open from an external port scan, but an internal one says it is not and no incoming connections seem to actually connect.
The 92.119.113.26 IP seems to hit it a lot and I have no idea what that is or why
First image NAT
Second Firewall
Third Firewall logState table shows TIME_WAIT:TIME_WAIT
-
ok so.. what do you have on 192.168.1.35 ? did you check if there is some kind of firewall on that machine that could block the traffic? is apache/nginx or whatever you are using working correctly? from what i can see here is that 192.168.1.35 is blocking/not accepting the traffic
https://www.abuseipdb.com/check/92.119.113.26 -
Do a packet capture on the LAN interface while testing and see if the NAT'd packets are leaving for your web server.
-
It's an IIS box, and the port is open, internally I can connect to it, I can also do a port test from the PfSense box just fine.
The traffic does not appear to get in to Pf in the first place to be NAT, this is why I am stuck.
I dont have the knowledge with networking to understand packet capture logs, however the issue is not present internally, only when traffic must come in via the PfSense.
-
The packet capture basic output display is pretty simple. You start a capture via Diagnostics - Packet Capture and you filter on the LAN IP address of your server. The only packets you should see during your test would be coming from your client trying your NAT. I would expect to see no packets, confirming that pfSense isn't forwarding them on to LAN. If you do see packets going out from LAN then the problem is something else.
However, before I do anything else I would reboot your ESXi host just to remove any doubt about hardware in a weird state or the pfSense VM acting up.
-
I really dont want to have to reboot the whole host, given that port 25 NAT is working and all other VMs are running as expected.
I do appreciate the help though, I wouldn't be able to do this on my own without a magic wand.
Here is a piece of the packet capture - hopefully this means something
16:03:10.026412 IP 192.168.1.35.443 > 95.216.36.80.32944: tcp 0
16:03:11.935289 IP 192.168.1.35.443 > 95.216.36.80.53295: tcp 0
16:03:12.081959 IP 192.168.1.35.443 > 95.216.36.80.60995: tcp 0
16:03:13.074664 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
16:03:13.074938 IP 192.168.1.35.443 > 95.216.36.80.48521: tcp 0
16:03:13.128209 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
16:03:15.484324 IP 192.168.1.35.443 > 95.216.36.80.37319: tcp 0
16:03:15.593657 IP 192.168.1.35.443 > 138.201.87.102.60223: tcp 0
16:03:15.940938 IP 192.168.1.35.443 > 138.201.87.102.45092: tcp 0 -
Addition - the IIS box is a reverse proxy, proxying Exchange, even if I update the NAT to go to Exchange directly it still does not work.
I am happy to negate the need for a NAT and setup Squid / HA as a reverse proxy, however at this point I would just like it working again, then I can tackle additional challenges later.
-
means your webserver is answering - my guess would be its sending back RST.
16:03:13.074664 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
16:03:13.074938 IP 192.168.1.35.443 > 95.216.36.80.48521: tcp 0You see that 95.x from source port 48521 source port got an answer from your web server 443 back to the 48521 port.
-
Thank you @johnpoz
That confirms what I thought, that it is working, however, even having redirected the traffic directly to exchange I get the same results.
I know the ports are open on the internal side, and I see the port as open externally, but content is never collected and returned.
I dont know how to fix this quirk, hence my post.
I have rebooted both the IIS proxy and exchange, this does not help. I have not rebooted the PfSense box, I was hoping to fix it without that, but I guess if we believe this would help I can bounce it. It's a VM and should only take moments - just an annoyance to do so.
-
Here is the initial handshake and reply from my test with a web server listening on tcp/80:
After the initial handshake, the payloads should be non-zero. As John said, the NAT itself is working so your problem must be elsewhere. Do you have any security thingies that might have been updated and are now getting in the way? Antivirus or antimalware apps, etc?
-
Best thing to do would be to open your sniff in say wireshark so you can actually see what is getting sent back, or up the verbosity of the sniff. so you can see if your seeing syn, and then syn,ack back or just RST.
-
indeed as i say you must have something on 192.168.1.35 or whatever is your web server ip blocking it/not accepting it for some reason
set this option and do a capture again -
I have nothing auto-updating, I have checked and rebooted most systems except PF at this point.
I do see 'some' logs in IIS about activesync, but nothing else gets back to the client or the web-browser in terms of webmail.
I predict it will be something silly and/or trivial, but it's not clearly obvious at this point.
I do really appreciate your help, but I dont understand wireshark or have it installed at this point.
Learning more about networking generically is on my list of to-do.
Just re-running that packet capture now...
-
Does this help anyone?
16:27:43.348163 IP (tos 0x28, ttl 67, id 58574, offset 0, flags [DF], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [S], cksum 0xae6a (correct), seq 3568136006, win 29200, length 0
16:27:43.348722 IP (tos 0x0, ttl 128, id 14187, offset 0, flags [DF], proto TCP (6), length 44)
192.168.1.35.443 > 138.201.87.102.42429: Flags [S.], cksum 0x7a39 (correct), seq 1036996916, ack 3568136007, win 65392, options [mss 1460], length 0
16:27:43.388858 IP (tos 0x0, ttl 54, id 43753, offset 0, flags [none], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [R], cksum 0xe077 (correct), seq 3568136007, win 16384, length 0
16:27:44.407074 IP (tos 0x0, ttl 128, id 14188, offset 0, flags [DF], proto TCP (6), length 44)
192.168.1.35.443 > 138.201.87.102.34989: Flags [S.], cksum 0x3ef9 (correct), seq 564559956, ack 585524840, win 65392, options [mss 1460], length 0
16:27:44.445433 IP (tos 0x0, ttl 55, id 51013, offset 0, flags [none], proto TCP (6), length 40)
138.201.87.102.34989 > 192.168.1.35.443: Flags [R], cksum 0xb82e (correct), seq 585524840, win 16384, length 0
16:27:45.025532 IP (tos 0x0, ttl 128, id 1751, offset 0, flags [DF], proto TCP (6), length 44)I will re-re-boot my guests (email ones)
-
@Rod-It said in Ports, rules, NAT:
138.201.87.102.34989 > 192.168.1.35.443: Flags [R],
Well that looks like the 138.x box sent a RST to the web server.
-
@Rod-It said in Ports, rules, NAT:
16:27:43.348163 IP (tos 0x28, ttl 67, id 58574, offset 0, flags [DF], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [S], cksum 0xae6a (correct), seq 3568136006, win 29200, length 0
16:27:43.348722 IP (tos 0x0, ttl 128, id 14187, offset 0, flags [DF], proto TCP (6), length 44)
192.168.1.35.443 > 138.201.87.102.42429: Flags [S.], cksum 0x7a39 (correct), seq 1036996916, ack 3568136007, win 65392, options [mss 1460], length 0
16:27:43.388858 IP (tos 0x0, ttl 54, id 43753, offset 0, flags [none], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [R], cksum 0xe077 (correct), seq 3568136007, win 16384, length 0S (SYN), F (FIN), P (PUSH), R (RST), U (URG), W (ECN CWR), E (ECN-Echo) or . (ACK)
Flags [R] -> reset
-
So looks to me the client sent syn S and then server sent back syn,ack S.
But then the client says no thanks with a R
-
Sorry, I am not a network person.
Are we saying this is backend server dropping/rejecting or that things look ok here?
Both IIS box and Exchange will reboot shortly for completeness.
