Ports, rules, NAT



  • Hi All,

    first time poster - and usually I dont need help with these types of things., though at present I am stumped.

    Some background;

    I am not a network guy, a firewall guy, nor a Linux/BSD guy, but I do ok in finding my way around these things, I've been using PfSense for about 2 months now and I've managed to setup WAN, VLANS and NATs without issue.

    Almost 2 days ago one of my NAT rules stopped working, scanning it via the public IP shows the port filtered, scanning it inside the LAN also shows it as blocked/filtered. I have managed to sort this and it now shows as open, however it still doesn't seem to work.

    I have 2 NAT rules and corresponding firewall rules to boot.

    Port 25 and port 443 - 25 still works, but 443 only shows incoming connections from limited number of IPs, never the ones I try it from (all external). If I enable NAT reflection I can get to my website using the external name from the internal LAN, disabling this does as expected and disables me from the LAN getting to my external domain over external IP.

    Doing a port scan externally I see port 443 as open, port 443 to the internal resource also shows as open on the destination, however port scanning PfSense from internal only shows me port 444 as open, which is the default interface port.

    the log does show me one IP trying and being passed by the NAT rule, but I never see my own traffic from another external public IP and isitdownrightnow.com says I am down.

    Any pointers?

    I've removed the NAT rule and FW rule and re-created them to no avail.

    I have not rebooted the VM as many posts suggests this isnt necessary.

    The NAT and FW rule exist, the ports show as open from outside if you scan them, but not internally and only traffic is passing via port 25, port 443 does nothing.

    I really can't see the wood for the trees here - an extra pair of eyes/thoughts please?

    Pf Version 2.4.4-p3
    VM - ESXi
    Directly connected from my modem, so acting as the external interface (WAN)
    Everything else passes out of a second LAN connection.


  • LAYER 8

    you should put some screenshot of your rules / NAT if you want us to be of any help



  • Re - first post, still not familiar with the layout of this forum, it's the first one I've used where the post box appears as half the screen at the bottom split in to two.

    acbf7eb8-6e30-4d6f-a7c3-aa51899f2c8f-image.png

    d925755a-9f5b-401c-a572-986e6b99b7fb-image.png

    9e0bc45c-cd5e-4f08-a181-4aa59893484b-image.png

    I hope these help.

    Do bare in mind these were working and I am familiar with NAT rules, there have been no power outages, no host crashes, no VM crashes, no ISP disconnections, I did remove the rule and recreate it but this has made no difference, the port 'appears' open from an external port scan, but an internal one says it is not and no incoming connections seem to actually connect.

    The 92.119.113.26 IP seems to hit it a lot and I have no idea what that is or why

    First image NAT
    Second Firewall
    Third Firewall log

    State table shows TIME_WAIT:TIME_WAIT


  • LAYER 8

    ok so.. what do you have on 192.168.1.35 ? did you check if there is some kind of firewall on that machine that could block the traffic? is apache/nginx or whatever you are using working correctly? from what i can see here is that 192.168.1.35 is blocking/not accepting the traffic
    https://www.abuseipdb.com/check/92.119.113.26



  • Do a packet capture on the LAN interface while testing and see if the NAT'd packets are leaving for your web server.



  • @kiokoman

    It's an IIS box, and the port is open, internally I can connect to it, I can also do a port test from the PfSense box just fine.

    The traffic does not appear to get in to Pf in the first place to be NAT, this is why I am stuck.

    @KOM

    I dont have the knowledge with networking to understand packet capture logs, however the issue is not present internally, only when traffic must come in via the PfSense.



  • The packet capture basic output display is pretty simple. You start a capture via Diagnostics - Packet Capture and you filter on the LAN IP address of your server. The only packets you should see during your test would be coming from your client trying your NAT. I would expect to see no packets, confirming that pfSense isn't forwarding them on to LAN. If you do see packets going out from LAN then the problem is something else.

    However, before I do anything else I would reboot your ESXi host just to remove any doubt about hardware in a weird state or the pfSense VM acting up.



  • I really dont want to have to reboot the whole host, given that port 25 NAT is working and all other VMs are running as expected.

    I do appreciate the help though, I wouldn't be able to do this on my own without a magic wand.

    Here is a piece of the packet capture - hopefully this means something

    16:03:10.026412 IP 192.168.1.35.443 > 95.216.36.80.32944: tcp 0
    16:03:11.935289 IP 192.168.1.35.443 > 95.216.36.80.53295: tcp 0
    16:03:12.081959 IP 192.168.1.35.443 > 95.216.36.80.60995: tcp 0
    16:03:13.074664 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
    16:03:13.074938 IP 192.168.1.35.443 > 95.216.36.80.48521: tcp 0
    16:03:13.128209 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
    16:03:15.484324 IP 192.168.1.35.443 > 95.216.36.80.37319: tcp 0
    16:03:15.593657 IP 192.168.1.35.443 > 138.201.87.102.60223: tcp 0
    16:03:15.940938 IP 192.168.1.35.443 > 138.201.87.102.45092: tcp 0



  • Addition - the IIS box is a reverse proxy, proxying Exchange, even if I update the NAT to go to Exchange directly it still does not work.

    I am happy to negate the need for a NAT and setup Squid / HA as a reverse proxy, however at this point I would just like it working again, then I can tackle additional challenges later.


  • LAYER 8 Global Moderator

    means your webserver is answering - my guess would be its sending back RST.

    16:03:13.074664 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
    16:03:13.074938 IP 192.168.1.35.443 > 95.216.36.80.48521: tcp 0

    You see that 95.x from source port 48521 source port got an answer from your web server 443 back to the 48521 port.



  • Thank you @johnpoz

    That confirms what I thought, that it is working, however, even having redirected the traffic directly to exchange I get the same results.

    I know the ports are open on the internal side, and I see the port as open externally, but content is never collected and returned.

    I dont know how to fix this quirk, hence my post.

    I have rebooted both the IIS proxy and exchange, this does not help. I have not rebooted the PfSense box, I was hoping to fix it without that, but I guess if we believe this would help I can bounce it. It's a VM and should only take moments - just an annoyance to do so.



  • Here is the initial handshake and reply from my test with a web server listening on tcp/80:

    VirtualBox_Ubuntu MATE 1_17_08_2019_11_19_34.png

    After the initial handshake, the payloads should be non-zero. As John said, the NAT itself is working so your problem must be elsewhere. Do you have any security thingies that might have been updated and are now getting in the way? Antivirus or antimalware apps, etc?


  • LAYER 8 Global Moderator

    Best thing to do would be to open your sniff in say wireshark so you can actually see what is getting sent back, or up the verbosity of the sniff. so you can see if your seeing syn, and then syn,ack back or just RST.


  • LAYER 8

    indeed as i say you must have something on 192.168.1.35 or whatever is your web server ip blocking it/not accepting it for some reason
    Immagine.jpg
    set this option and do a capture again



  • I have nothing auto-updating, I have checked and rebooted most systems except PF at this point.

    I do see 'some' logs in IIS about activesync, but nothing else gets back to the client or the web-browser in terms of webmail.

    I predict it will be something silly and/or trivial, but it's not clearly obvious at this point.

    I do really appreciate your help, but I dont understand wireshark or have it installed at this point.

    Learning more about networking generically is on my list of to-do.

    Just re-running that packet capture now...



  • Does this help anyone?

    16:27:43.348163 IP (tos 0x28, ttl 67, id 58574, offset 0, flags [DF], proto TCP (6), length 40)
    138.201.87.102.42429 > 192.168.1.35.443: Flags [S], cksum 0xae6a (correct), seq 3568136006, win 29200, length 0
    16:27:43.348722 IP (tos 0x0, ttl 128, id 14187, offset 0, flags [DF], proto TCP (6), length 44)
    192.168.1.35.443 > 138.201.87.102.42429: Flags [S.], cksum 0x7a39 (correct), seq 1036996916, ack 3568136007, win 65392, options [mss 1460], length 0
    16:27:43.388858 IP (tos 0x0, ttl 54, id 43753, offset 0, flags [none], proto TCP (6), length 40)
    138.201.87.102.42429 > 192.168.1.35.443: Flags [R], cksum 0xe077 (correct), seq 3568136007, win 16384, length 0
    16:27:44.407074 IP (tos 0x0, ttl 128, id 14188, offset 0, flags [DF], proto TCP (6), length 44)
    192.168.1.35.443 > 138.201.87.102.34989: Flags [S.], cksum 0x3ef9 (correct), seq 564559956, ack 585524840, win 65392, options [mss 1460], length 0
    16:27:44.445433 IP (tos 0x0, ttl 55, id 51013, offset 0, flags [none], proto TCP (6), length 40)
    138.201.87.102.34989 > 192.168.1.35.443: Flags [R], cksum 0xb82e (correct), seq 585524840, win 16384, length 0
    16:27:45.025532 IP (tos 0x0, ttl 128, id 1751, offset 0, flags [DF], proto TCP (6), length 44)

    I will re-re-boot my guests (email ones)


  • LAYER 8 Global Moderator

    @Rod-It said in Ports, rules, NAT:

    138.201.87.102.34989 > 192.168.1.35.443: Flags [R],

    Well that looks like the 138.x box sent a RST to the web server.


  • LAYER 8

    @Rod-It said in Ports, rules, NAT:

    16:27:43.348163 IP (tos 0x28, ttl 67, id 58574, offset 0, flags [DF], proto TCP (6), length 40)
    138.201.87.102.42429 > 192.168.1.35.443: Flags [S], cksum 0xae6a (correct), seq 3568136006, win 29200, length 0
    16:27:43.348722 IP (tos 0x0, ttl 128, id 14187, offset 0, flags [DF], proto TCP (6), length 44)
    192.168.1.35.443 > 138.201.87.102.42429: Flags [S.], cksum 0x7a39 (correct), seq 1036996916, ack 3568136007, win 65392, options [mss 1460], length 0
    16:27:43.388858 IP (tos 0x0, ttl 54, id 43753, offset 0, flags [none], proto TCP (6), length 40)
    138.201.87.102.42429 > 192.168.1.35.443: Flags [R], cksum 0xe077 (correct), seq 3568136007, win 16384, length 0

    S (SYN), F (FIN), P (PUSH), R (RST), U (URG), W (ECN CWR), E (ECN-Echo) or . (ACK)

    Flags [R] -> reset


  • LAYER 8 Global Moderator

    So looks to me the client sent syn S and then server sent back syn,ack S.

    But then the client says no thanks with a R



  • Sorry, I am not a network person.

    Are we saying this is backend server dropping/rejecting or that things look ok here?

    Both IIS box and Exchange will reboot shortly for completeness.



  • Apologies, didn't see your latter reply

    Ok so it's IIS still saying it's not happy - at least I have a focus area.


  • LAYER 8 Global Moderator

    Im not sure who the 138.x box is, but he is the client as he sends the syn... The server at 192.x box answers with syn,ack - this is normal start of a conversation.. But then the 138.x box sends R (RST) which is basically a F Off sort of thing..



  • Strange that the client is sending the reset. There are very limited circumstances for that to happen, and I don't see any that I know of applying here.



  • Just to be clear, this happens if I send the NAT to the IIS relay OR the Exchange box directly.

    This does NOT affect clients on the LAN or any other VLAN, only requests that come in through the PfSense box.

    The IIS box has it's firewall disabled for a moment, but this should not be it as internal clients work.

    Both IIS and Exchange have been rebooted.

    While I want to rule out AV because internal clients are working, I am happy to remove/disable it for now.

    As an FYI - this is my home lab, not a production system so this is not massively critical, but a PITA nonetheless.

    Something else that may or may not be relevant (I dont think it is, but i'l include it anyway).

    About 2 days ago (probably some 6-8 hours before this started) I moved my domains DNS from DynDNS to CloudFlare - I have tried with and without proxied DNS, neither matter, however because the port shows as open on my public IP directly, I dont believe this to be related, though I am more than happy to be proven wrong.



  • If you have time, I'd love to see one more test run with the Level of Detail set to high so I can see the packet timestamp values.



  • 17:16:46.428834 IP 192.168.1.35.443 > 138.201.87.102.48424: tcp 0
    17:16:46.469427 IP 138.201.87.102.48424 > 192.168.1.35.443: tcp 0
    17:16:46.522427 IP 192.168.1.35.443 > 138.201.87.102.61203: tcp 0
    17:16:47.487447 IP 192.168.1.35.443 > 138.201.87.102.61529: tcp 0
    17:16:47.886778 IP 192.168.1.35.443 > 138.201.87.102.61068: tcp 0
    17:16:48.538018 IP 95.216.36.80.35970 > 192.168.1.35.443: tcp 0
    17:16:48.538399 IP 192.168.1.35.443 > 95.216.36.80.35970: tcp 0
    17:16:48.642150 IP 82.132.247.36.52308 > 192.168.1.35.443: tcp 55
    17:16:48.645073 IP 192.168.1.35.443 > 82.132.247.36.52308: tcp 109
    17:16:48.649273 IP 82.132.247.36.52308 > 192.168.1.35.443: tcp 46

    Guys - thank you for the help, really, it means a lot to know there are people out there willing to spend their weekends trying to help others.

    I do believe 'CloudFlare' are to blame, I removed all of the DNS proxy connections once more and it's working again.

    Does the above negate you wanting a further run?


  • LAYER 8

    if it is working for you now, i'm happy enought



  • If you're satisfied that it's working and you've found the cause then no, you don't need to do that other test run.

    I'm still curious as to what was really going on. I'm not sure how a DNS proxy issue would cause your client to send a RST. Where were these DNS proxy connections configured? Via some Cloudflare GUI or is this something you're doing on the pfSense box?

    If this was indeed a remote issue then that would explain why it was working fine until one day when it wasn't.



  • You realise I will most likely be sticking around finding other guides and helping where I can, and no doubt I will have more questions.

    So far though I really do like PfSense.

    I am doing more with it within 2 months than I ever thought I would do with any networking product, I even sold my router - which by the way I really liked too (Netgear Nighthawk R7000)

    CloudFlare can proxy DNS requests, so assuming my website (or in this case, webmail) points to 81.x.x.x, CloudFlare makes it their own IP, so it looks like 101.x.x.x, while my IP had the ports open and CloudFlare did too, it didnt seem to be relaying HTTPS connections to me, but instead to itself - I found an article on their site about this, something to do with how SSL works with their certificates - this would also explain why I was not seeing the traffic hit the box - which by the way I now see when someone connects to 443.

    Phew... nightmare.

    Apologies for the wild chase there, but just having someone else there helped me to find the issue.



  • Glad we could help.



  • I want to complete this topic by also stating I now know why my DNS proxy was re-enabled (sometimes we overthink things).

    In PfSense DynamicDNS is an option to enable Proxying DNS - this turned it back on after I disabled it on CloudFlare directly. Makes sense why it was working then stopped. Using this option is great, but when you need HTTP/S traffic to be directed, directly and a given IP you must disable proxying (I can't find their guide right now, I've lost it already).

    Here is the article linked from within PfSense that I obviously missed

    https://blog.cloudflare.com/announcing-virtual-dns-ddos-mitigation-and-global-distribution-for-dns-traffic/



  • I just wanna know why your IP is scanning me like crazy... :P Or are you behind some kind of carrier NAT?
    alt text



  • Those IPs are nothing to do with me, they are also hitting me.

    They are based in Germany and Finland, I am not, nor do I have any services from Germany or Finland.

    Using the link above posted by @kiokoman they are on an abuse list, therefore they are likely bots or malicious servers.


  • LAYER 8 Global Moderator

    @Rod-It said in Ports, rules, NAT:

    138.201.87.102

    That is the IP of the device you were saying wasn't working ;)

    In your sniffs.
    17:16:46.469427 IP 138.201.87.102.48424 > 192.168.1.35.443: tcp 0
    17:16:46.522427 IP 192.168.1.35.443 > 138.201.87.102.61203: tcp 0



  • I never said that IP wasn't working, you have a snip of a packet capture as asked for. Any external IPs within are purely coincidental.

    I didnt validate any external IPs to the forum of my own.

    I know my topic was a little confusing at times, but I never stated any specific public IP I was using.

    If anything I stated no external IPs were getting in, but some were frequently trying - and those are likely what you see from the logs.


  • LAYER 8 Global Moderator

    Ah well that makes more sense why its sent a R then ;)

    I would block those IPs from talking to your services. Especially if they are on block lists.



  • I have, the poster above, like yourself believed that IP belonged to me - but none of them do :)

    Side question based on the new addition to the topic though, without SquidGuard or any other type of filtering, can PF block known malicious or bad IPs?

    Also, if there is a way I can mark this as resolved, please let me know and I will do so.


  • LAYER 8

    you can use suricata/snort for example


  • LAYER 8 Global Moderator

    pfblocker can be used to block IPs from hitting your port forwards. Be it from country xyz, or only allowing IPs from country A, etc.. Or specific lists for known bad.

    Who exactly is using this - you mentioned something about exchange? And if your using cloudflare - you can limit it to only cloudflare IPs, etc.

    And yeah ips can be used - but its a bit more complicated when your doing https



  • I have looked at Snort / Suricata, but not in great details, I will add pfblocker to my list of to look at.

    As to who is using this, a handful of friends and family, as mentioned above this is purely my home lab, I need some genuine traffic and use from these products so I can fault-find and fix as needed, as I am sure you are all aware, the best way to learn is to do - I'm a hands-on learner, I do better with actual problems in front of me and systems to maintain than any video or guide can give me, though I do use these to fill in the gaps and top-up what I dont know or am missing, including forums like this to get me out of a situation.

    CloudFlare is purely looking after my domain DNS, I dont need it to do anything fancy at this point (I'm only 3 days in with this anyway).

    Some background on me;

    I have 23 years in IT, mostly Windows server, Exchange/Email and VMware. While I understand principals and basics of networking and firewalls, I am starting to venture more in to learning them in more detail, including my Linux knowledge, I dont need this for work as we have a network team that picks up all network issues, at home it is useful to know - why it's taken me this long to be eager I have no idea, likely because I've never really needed to know, but one can never know too much so Linux, networking and firewalls are currently my learning goal.

    I hope this helps to understand and gives you all a little on me.


Log in to reply