Ports, rules, NAT
-
It's an IIS box, and the port is open, internally I can connect to it, I can also do a port test from the PfSense box just fine.
The traffic does not appear to get in to Pf in the first place to be NAT, this is why I am stuck.
I dont have the knowledge with networking to understand packet capture logs, however the issue is not present internally, only when traffic must come in via the PfSense.
-
The packet capture basic output display is pretty simple. You start a capture via Diagnostics - Packet Capture and you filter on the LAN IP address of your server. The only packets you should see during your test would be coming from your client trying your NAT. I would expect to see no packets, confirming that pfSense isn't forwarding them on to LAN. If you do see packets going out from LAN then the problem is something else.
However, before I do anything else I would reboot your ESXi host just to remove any doubt about hardware in a weird state or the pfSense VM acting up.
-
I really dont want to have to reboot the whole host, given that port 25 NAT is working and all other VMs are running as expected.
I do appreciate the help though, I wouldn't be able to do this on my own without a magic wand.
Here is a piece of the packet capture - hopefully this means something
16:03:10.026412 IP 192.168.1.35.443 > 95.216.36.80.32944: tcp 0
16:03:11.935289 IP 192.168.1.35.443 > 95.216.36.80.53295: tcp 0
16:03:12.081959 IP 192.168.1.35.443 > 95.216.36.80.60995: tcp 0
16:03:13.074664 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
16:03:13.074938 IP 192.168.1.35.443 > 95.216.36.80.48521: tcp 0
16:03:13.128209 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
16:03:15.484324 IP 192.168.1.35.443 > 95.216.36.80.37319: tcp 0
16:03:15.593657 IP 192.168.1.35.443 > 138.201.87.102.60223: tcp 0
16:03:15.940938 IP 192.168.1.35.443 > 138.201.87.102.45092: tcp 0 -
Addition - the IIS box is a reverse proxy, proxying Exchange, even if I update the NAT to go to Exchange directly it still does not work.
I am happy to negate the need for a NAT and setup Squid / HA as a reverse proxy, however at this point I would just like it working again, then I can tackle additional challenges later.
-
means your webserver is answering - my guess would be its sending back RST.
16:03:13.074664 IP 95.216.36.80.48521 > 192.168.1.35.443: tcp 0
16:03:13.074938 IP 192.168.1.35.443 > 95.216.36.80.48521: tcp 0You see that 95.x from source port 48521 source port got an answer from your web server 443 back to the 48521 port.
-
Thank you @johnpoz
That confirms what I thought, that it is working, however, even having redirected the traffic directly to exchange I get the same results.
I know the ports are open on the internal side, and I see the port as open externally, but content is never collected and returned.
I dont know how to fix this quirk, hence my post.
I have rebooted both the IIS proxy and exchange, this does not help. I have not rebooted the PfSense box, I was hoping to fix it without that, but I guess if we believe this would help I can bounce it. It's a VM and should only take moments - just an annoyance to do so.
-
Here is the initial handshake and reply from my test with a web server listening on tcp/80:
After the initial handshake, the payloads should be non-zero. As John said, the NAT itself is working so your problem must be elsewhere. Do you have any security thingies that might have been updated and are now getting in the way? Antivirus or antimalware apps, etc?
-
Best thing to do would be to open your sniff in say wireshark so you can actually see what is getting sent back, or up the verbosity of the sniff. so you can see if your seeing syn, and then syn,ack back or just RST.
-
indeed as i say you must have something on 192.168.1.35 or whatever is your web server ip blocking it/not accepting it for some reason
set this option and do a capture again -
I have nothing auto-updating, I have checked and rebooted most systems except PF at this point.
I do see 'some' logs in IIS about activesync, but nothing else gets back to the client or the web-browser in terms of webmail.
I predict it will be something silly and/or trivial, but it's not clearly obvious at this point.
I do really appreciate your help, but I dont understand wireshark or have it installed at this point.
Learning more about networking generically is on my list of to-do.
Just re-running that packet capture now...
-
Does this help anyone?
16:27:43.348163 IP (tos 0x28, ttl 67, id 58574, offset 0, flags [DF], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [S], cksum 0xae6a (correct), seq 3568136006, win 29200, length 0
16:27:43.348722 IP (tos 0x0, ttl 128, id 14187, offset 0, flags [DF], proto TCP (6), length 44)
192.168.1.35.443 > 138.201.87.102.42429: Flags [S.], cksum 0x7a39 (correct), seq 1036996916, ack 3568136007, win 65392, options [mss 1460], length 0
16:27:43.388858 IP (tos 0x0, ttl 54, id 43753, offset 0, flags [none], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [R], cksum 0xe077 (correct), seq 3568136007, win 16384, length 0
16:27:44.407074 IP (tos 0x0, ttl 128, id 14188, offset 0, flags [DF], proto TCP (6), length 44)
192.168.1.35.443 > 138.201.87.102.34989: Flags [S.], cksum 0x3ef9 (correct), seq 564559956, ack 585524840, win 65392, options [mss 1460], length 0
16:27:44.445433 IP (tos 0x0, ttl 55, id 51013, offset 0, flags [none], proto TCP (6), length 40)
138.201.87.102.34989 > 192.168.1.35.443: Flags [R], cksum 0xb82e (correct), seq 585524840, win 16384, length 0
16:27:45.025532 IP (tos 0x0, ttl 128, id 1751, offset 0, flags [DF], proto TCP (6), length 44)I will re-re-boot my guests (email ones)
-
@Rod-It said in Ports, rules, NAT:
138.201.87.102.34989 > 192.168.1.35.443: Flags [R],
Well that looks like the 138.x box sent a RST to the web server.
-
@Rod-It said in Ports, rules, NAT:
16:27:43.348163 IP (tos 0x28, ttl 67, id 58574, offset 0, flags [DF], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [S], cksum 0xae6a (correct), seq 3568136006, win 29200, length 0
16:27:43.348722 IP (tos 0x0, ttl 128, id 14187, offset 0, flags [DF], proto TCP (6), length 44)
192.168.1.35.443 > 138.201.87.102.42429: Flags [S.], cksum 0x7a39 (correct), seq 1036996916, ack 3568136007, win 65392, options [mss 1460], length 0
16:27:43.388858 IP (tos 0x0, ttl 54, id 43753, offset 0, flags [none], proto TCP (6), length 40)
138.201.87.102.42429 > 192.168.1.35.443: Flags [R], cksum 0xe077 (correct), seq 3568136007, win 16384, length 0S (SYN), F (FIN), P (PUSH), R (RST), U (URG), W (ECN CWR), E (ECN-Echo) or . (ACK)
Flags [R] -> reset
-
So looks to me the client sent syn S and then server sent back syn,ack S.
But then the client says no thanks with a R
-
Sorry, I am not a network person.
Are we saying this is backend server dropping/rejecting or that things look ok here?
Both IIS box and Exchange will reboot shortly for completeness.
-
Apologies, didn't see your latter reply
Ok so it's IIS still saying it's not happy - at least I have a focus area.
-
Im not sure who the 138.x box is, but he is the client as he sends the syn... The server at 192.x box answers with syn,ack - this is normal start of a conversation.. But then the 138.x box sends R (RST) which is basically a F Off sort of thing..
-
Strange that the client is sending the reset. There are very limited circumstances for that to happen, and I don't see any that I know of applying here.
-
Just to be clear, this happens if I send the NAT to the IIS relay OR the Exchange box directly.
This does NOT affect clients on the LAN or any other VLAN, only requests that come in through the PfSense box.
The IIS box has it's firewall disabled for a moment, but this should not be it as internal clients work.
Both IIS and Exchange have been rebooted.
While I want to rule out AV because internal clients are working, I am happy to remove/disable it for now.
As an FYI - this is my home lab, not a production system so this is not massively critical, but a PITA nonetheless.
Something else that may or may not be relevant (I dont think it is, but i'l include it anyway).
About 2 days ago (probably some 6-8 hours before this started) I moved my domains DNS from DynDNS to CloudFlare - I have tried with and without proxied DNS, neither matter, however because the port shows as open on my public IP directly, I dont believe this to be related, though I am more than happy to be proven wrong.
-
If you have time, I'd love to see one more test run with the Level of Detail set to high so I can see the packet timestamp values.
-
17:16:46.428834 IP 192.168.1.35.443 > 138.201.87.102.48424: tcp 0
17:16:46.469427 IP 138.201.87.102.48424 > 192.168.1.35.443: tcp 0
17:16:46.522427 IP 192.168.1.35.443 > 138.201.87.102.61203: tcp 0
17:16:47.487447 IP 192.168.1.35.443 > 138.201.87.102.61529: tcp 0
17:16:47.886778 IP 192.168.1.35.443 > 138.201.87.102.61068: tcp 0
17:16:48.538018 IP 95.216.36.80.35970 > 192.168.1.35.443: tcp 0
17:16:48.538399 IP 192.168.1.35.443 > 95.216.36.80.35970: tcp 0
17:16:48.642150 IP 82.132.247.36.52308 > 192.168.1.35.443: tcp 55
17:16:48.645073 IP 192.168.1.35.443 > 82.132.247.36.52308: tcp 109
17:16:48.649273 IP 82.132.247.36.52308 > 192.168.1.35.443: tcp 46Guys - thank you for the help, really, it means a lot to know there are people out there willing to spend their weekends trying to help others.
I do believe 'CloudFlare' are to blame, I removed all of the DNS proxy connections once more and it's working again.
Does the above negate you wanting a further run?
-
if it is working for you now, i'm happy enought
-
If you're satisfied that it's working and you've found the cause then no, you don't need to do that other test run.
I'm still curious as to what was really going on. I'm not sure how a DNS proxy issue would cause your client to send a RST. Where were these DNS proxy connections configured? Via some Cloudflare GUI or is this something you're doing on the pfSense box?
If this was indeed a remote issue then that would explain why it was working fine until one day when it wasn't.
-
You realise I will most likely be sticking around finding other guides and helping where I can, and no doubt I will have more questions.
So far though I really do like PfSense.
I am doing more with it within 2 months than I ever thought I would do with any networking product, I even sold my router - which by the way I really liked too (Netgear Nighthawk R7000)
CloudFlare can proxy DNS requests, so assuming my website (or in this case, webmail) points to 81.x.x.x, CloudFlare makes it their own IP, so it looks like 101.x.x.x, while my IP had the ports open and CloudFlare did too, it didnt seem to be relaying HTTPS connections to me, but instead to itself - I found an article on their site about this, something to do with how SSL works with their certificates - this would also explain why I was not seeing the traffic hit the box - which by the way I now see when someone connects to 443.
Phew... nightmare.
Apologies for the wild chase there, but just having someone else there helped me to find the issue.
-
Glad we could help.
-
I want to complete this topic by also stating I now know why my DNS proxy was re-enabled (sometimes we overthink things).
In PfSense DynamicDNS is an option to enable Proxying DNS - this turned it back on after I disabled it on CloudFlare directly. Makes sense why it was working then stopped. Using this option is great, but when you need HTTP/S traffic to be directed, directly and a given IP you must disable proxying (I can't find their guide right now, I've lost it already).
Here is the article linked from within PfSense that I obviously missed
https://blog.cloudflare.com/announcing-virtual-dns-ddos-mitigation-and-global-distribution-for-dns-traffic/
-
I just wanna know why your IP is scanning me like crazy... :P Or are you behind some kind of carrier NAT?
-
Those IPs are nothing to do with me, they are also hitting me.
They are based in Germany and Finland, I am not, nor do I have any services from Germany or Finland.
Using the link above posted by @kiokoman they are on an abuse list, therefore they are likely bots or malicious servers.
-
@Rod-It said in Ports, rules, NAT:
138.201.87.102
That is the IP of the device you were saying wasn't working ;)
In your sniffs.
17:16:46.469427 IP 138.201.87.102.48424 > 192.168.1.35.443: tcp 0
17:16:46.522427 IP 192.168.1.35.443 > 138.201.87.102.61203: tcp 0 -
I never said that IP wasn't working, you have a snip of a packet capture as asked for. Any external IPs within are purely coincidental.
I didnt validate any external IPs to the forum of my own.
I know my topic was a little confusing at times, but I never stated any specific public IP I was using.
If anything I stated no external IPs were getting in, but some were frequently trying - and those are likely what you see from the logs.
-
Ah well that makes more sense why its sent a R then ;)
I would block those IPs from talking to your services. Especially if they are on block lists.
-
I have, the poster above, like yourself believed that IP belonged to me - but none of them do :)
Side question based on the new addition to the topic though, without SquidGuard or any other type of filtering, can PF block known malicious or bad IPs?
Also, if there is a way I can mark this as resolved, please let me know and I will do so.
-
you can use suricata/snort for example
-
pfblocker can be used to block IPs from hitting your port forwards. Be it from country xyz, or only allowing IPs from country A, etc.. Or specific lists for known bad.
Who exactly is using this - you mentioned something about exchange? And if your using cloudflare - you can limit it to only cloudflare IPs, etc.
And yeah ips can be used - but its a bit more complicated when your doing https
-
I have looked at Snort / Suricata, but not in great details, I will add pfblocker to my list of to look at.
As to who is using this, a handful of friends and family, as mentioned above this is purely my home lab, I need some genuine traffic and use from these products so I can fault-find and fix as needed, as I am sure you are all aware, the best way to learn is to do - I'm a hands-on learner, I do better with actual problems in front of me and systems to maintain than any video or guide can give me, though I do use these to fill in the gaps and top-up what I dont know or am missing, including forums like this to get me out of a situation.
CloudFlare is purely looking after my domain DNS, I dont need it to do anything fancy at this point (I'm only 3 days in with this anyway).
Some background on me;
I have 23 years in IT, mostly Windows server, Exchange/Email and VMware. While I understand principals and basics of networking and firewalls, I am starting to venture more in to learning them in more detail, including my Linux knowledge, I dont need this for work as we have a network team that picks up all network issues, at home it is useful to know - why it's taken me this long to be eager I have no idea, likely because I've never really needed to know, but one can never know too much so Linux, networking and firewalls are currently my learning goal.
I hope this helps to understand and gives you all a little on me.
-
Running an email server out of home connection, while can be a useful learning experience - not going to be very useful if your actually wanting to send mail from your "home" ip..
Most consumer IP blocks are listed and major players block them for accepting mail. Also unless you can edit your PTR for the IP in question - again blocked by most major players.
So sending would need to be through a relay if you want to actually send mail to major domains.
Have fun! And you have any questions, while the general section doesn't get all that much traffic... Some of us like to answer just general networking and IT questions not always related to pfsense, etc. ;)
-
i have a mail server myself at home for fun
fortunately I have never encountered any problems, my ip is not in any blacklist and even if i can't change my ipv4 ptr, i have a ipv6 tunnel with he.net and they let you change it. i just need to be sure that my email server use ipv6 to send emails -
There are a shitton of major players email that is not on ipv6.. And pretty much every isp consumer IPs are listed as such.. But sure you might be lucky there.. But no ptr is can be a killer..
Other than a toy/learning experience running mail servers off a home connection is pretty pointless.
-
Apologies, didnt realise I would have to cover that, was purely providing a little background.
I have a VPS I relay through, that has correct rDNS, SPF, DKIM, DMARC all setup and working, including TLS 1.2
I've been running exchange for almost 10 years this way, have my own domain, they are secure and get A+, I do not suport any less than TLS 1.2 etc. I run Nessus (community edition) and OpenVAS against my network regularly.
Almost everything I do in my lab is to mimic work, its the first place I apply security patches and PCI fixes, if I break my lab no one gets hurt, if I take a business down, it's my problem and I get it in the neck, so I practise at home first (where possible)
Do apricate the additional feedback though..
P.S. while I dont generate crazy amounts of traffic, I am all for securing my system more effectively- those tools mentioned above will be looked at better when I have more dedicated time.
And just in case anyone asks, exchange is covered, as are all my Windows licenses under an MDSN subscription by work for the sole purpose of learning. :)
Side note, if I can do the same thing as work, but for free, I often take this option. Certificates = Let's Encrypt (currently I have a paid one), email filtering - also free, firewall (from home router to PfSense), VPN - OpenVPN.
I like free and opensource, so use it as a 'similar' to works paid enterprise products.
P.P.S (expect me asking for help), my next plan is to remove the IIS proxy and move my HTTPS traffic to either Squid or HA via PfSense to be my reverse proxy. If I can do it for free and consolidate, meaning I need to manage and maintain less, I'm all for it.
-
FYI, work are moving to O365 in the near future, so my requirement to run and learn exchange may also go away, at which point I will likely start looking at putting email on my VPS server, using something else hosted at home that is free or just leaving it as is until it becomes EOL - I'm not at that stage yet though.