Packages blocked to one mailserver host only (LAN => WAN)
-
Hello all,
our office's pfSense 2.2.4 (virtualized Proxmox kvm vm) is blocking legitimate outgoing connections from one local client to our external mail server. I know the topic itself has been discussed a lot and is in the wiki, yet until know I have not found a solution to this and don't know where to look for.
pfSense is blocking TCP:A, TCP:R, TCP:FA, TCP:PA to the mail server's ports (993, 443, 25, 465) - a Kerio Connect self-hosted VM in our cluster in our colocation. The problem disappears by itself after some minutes again, but the server is not reachable from the client:
Request timeout for icmp_seq 465 36 bytes from xxxxx.biz (x.x.x.x): Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 37fd 0 0000 3f 01 377f 192.168.10.8 123.x.x.x Request timeout for icmp_seq 466 36 bytes from b2b-92-50-115-2.unitymedia.biz (92.50.115.2): Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 fcfe 0 0000 3f 01 727d 192.168.10.8 123.x.x.x
On the firewall the logs look like attached.
I know that these looks like out of state / timed out packets and read about 50+ posts and discussions on this topic. Still I don't get where to start looking. Where is the culprit here? Is it the remote side or why is pfSense blocking this?
Usually this should not be causing any problems, but in our case this blocks using the mail server while this happens, which keeps me from doing my work ;-)
Would be happy to get a hint where to look.
Best
Sebastian
-
Have you already tried to increase the state timeout? Maybe it helps.
Edit the firewall rule which allows this connection, go down to "Advanced Options" and click on Advanced, in the undermost input field enter a higher timeout in seconds. -
Thanks for taking the time to answer - and for the good idea.
The timeout seconds field is empty. What would be a reasonable value here?
-
AFAIR the default value is 60 seconds.
If you have large mails and a slow connection maybe this is not enough.
Try out what fits your purpose. To increase this value enlarges the state table and need more RAM.
For use with Exchange OMA I had set this value to 15 minutes on an ALIX board. Though there are no RAM bottleneck, however there are just about a dozen connections. -
Set to 300s and so far no problems in the last hour. Thanks for the help, really appreciated.
-
Unfortunately this did not help in the long run. The mail server is still beeing blocked from time to time for some duration (about 5-10 minutes, then it starts working again all by itself):
PING my.mailserver.com (xxx.xxx.xxx.xxx): 56 data bytes 36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 1762 0 0000 3f 01 581a 192.168.10.8 217.76.104.48 Request timeout for icmp_seq 0 36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 6c42 0 0000 3f 01 033a 192.168.10.8 xxx.xxx.xxx.xxx Request timeout for icmp_seq 1 36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 6f6a 0 0000 3f 01 0012 192.168.10.8 xxx.xxx.xxx.xxx
That's just ICMP, but all other protocols fail as well.
Any more ideas where to look?
-
We still fight with this problem. I tried everything suggested in this thread, but still the pfSense blocks connections to that server only from time to time. Any ideas? I would be willing to pay for solutions/help actually, but don't know of any pfSense support in our region (Cologne/Bonn, Germany).
Update: Firewall log reports this as blocking rule: @5(1000000103) block drop in log inet all label "Default deny rule IPv4" for the LAN if. I understand that this is actually always the case and that this is out of state traffic - but how does it happen and why does this lead to a full block for the mail server?
Best,
Sebastian -
Yeah you still fight the problem and we STILL have not seen the firewall rules configured.
-
Sorry, no one requested the rules config ;-)
Rules config attached to this post.Thanks for the reply and help. I appreciate it.
-
I thought you would post a screenshot… not this. There is not a single mention of port 25, 143, 465 or port 993 in those rules. No idea how you expect the traffic to get passed.
-
There is only one rule for LAN: Allow all (see screenshot attached).
I also attached the WAN rules.From my understanding the allow LAN to all rule should be enough, shouldn't it?
![Screenshot 2015-11-25 11.56.13.png](/public/imported_attachments/1/Screenshot 2015-11-25 11.56.13.png)
![Screenshot 2015-11-25 11.56.13.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 11.56.13.png_thumb)
-
What's that advanced stuff in there in the LAN rule? (Certainly not default).
What's UM interface (shown on all of your screenshots with the traffic blocked?!) -
That's actually strange as there has not been done any advanced modification.
UM = UnityMedia = WAN.
![Screenshot 2015-11-25 12.24.49.png](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.49.png)
![Screenshot 2015-11-25 12.24.49.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.49.png_thumb)
![Screenshot 2015-11-25 12.24.59.png](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.59.png)
![Screenshot 2015-11-25 12.24.59.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.59.png_thumb)
![Screenshot 2015-11-25 12.25.04.png](/public/imported_attachments/1/Screenshot 2015-11-25 12.25.04.png)
![Screenshot 2015-11-25 12.25.04.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 12.25.04.png_thumb) -
That's actually strange as there has not been done any advanced modification.
Clearly was. That "Advanced Options - This allows packets with IP options to pass." certainly ain't ticked by default and I think you completely misunderstood this suggestion. You shouldn't tick anything there, you should put a custom timeout to "State Timeout in seconds (TCP only)"
Where's the mailserver located? On WAN or behind pfSense?!
Where are you testing this from? -
Disabled the option "allow packets" to default. Yes, I clearly misunderstood the suggestion.
I had the custom state timeout enabled with 300 (seconds) for some time, but it did not help.
Mailserver is located at a remote colocation behind another pfSense.
Testing from MacOS Apple Mail, terminal on MacOS clients and several vm's (Debian based) in our local network.
-
Mailserver is located at a remote colocation behind another pfSense.
Great. So, perhaps move your debugging efforts there.
-
I already did. Logs there do not mention any blocks at all. Nothing, nada. Our local public IP is not beeing blocked anywhere at that remote location and other servers can actually access the mail server while it is beeing blocked at the office.
I will go on digging through the logs, but IMHO this is happening at our local pfSense.
-
OK, so I tried all things recommended here and this still happens.
Remote firewall does not log anything and other clients can connect to the server. Only our network can't.
Any other ideas what to check?
-
I filtered the log for source UM (WAN) and the destination mail server and vice versa. It seems completely random to me.
-
Hi @taenzerme,
Just my opinion but maybe it could be ISP problem? If you didn't change any config before the issue, maybe it could be rerouting from your ISP? I once encountered a problem where I have problem connecting to our mail server due to ISP reroute.
-
Hi @Ojisang,
thanks. It just happened again and I checked traceroute to our mail server on the firewall directly:
[2.2.5-RELEASE][root@firewall.office.xxxxxxxxxxx.xxx]/root: traceroute my.mailserver.com traceroute to my.mailserver.com (xxx.xxx.xxx.48), 64 hops max, 52 byte packets 1 firewall (192.168.10.253) 0.261 ms 0.227 ms 0.284 ms 2 firewall (192.168.10.253) 0.602 ms 0.884 ms 0.543 ms 3 firewall (192.168.10.253) 0.754 ms 0.691 ms 0.663 ms 4 firewall (192.168.10.253) 0.808 ms 0.850 ms 0.825 ms 5 firewall (192.168.10.253) 1.120 ms 1.122 ms 1.479 ms 6 firewall (192.168.10.253) 1.268 ms 1.207 ms 1.104 ms 7 firewall (192.168.10.253) 1.417 ms 1.421 ms 1.346 ms 8 firewall (192.168.10.253) 1.525 ms 1.600 ms 1.495 ms 9 firewall (192.168.10.253) 1.781 ms 1.843 ms 1.800 ms 10 firewall (192.168.10.253) 2.093 ms 2.010 ms 2.003 ms 11 firewall (192.168.10.253) 2.168 ms 2.211 ms 2.252 ms 12 firewall (192.168.10.253) 2.347 ms 2.430 ms 2.498 ms 13 firewall (192.168.10.253) 2.830 ms 2.611 ms 2.509 ms 14 firewall (192.168.10.253) 2.782 ms 2.887 ms 2.885 ms 15 firewall (192.168.10.253) 2.953 ms 3.058 ms 3.445 ms 16 firewall (192.168.10.253) 3.646 ms 4.801 ms 3.203 ms 17 firewall (192.168.10.253) 3.896 ms 3.469 ms 3.399 ms 18 firewall (192.168.10.253) 4.041 ms 4.201 ms 3.944 ms 19 firewall (192.168.10.253) 3.912 ms 3.796 ms 4.495 ms 20 firewall (192.168.10.253) 3.882 ms 3.985 ms 4.003 ms 21 firewall (192.168.10.253) 4.303 ms 4.193 ms 4.332 ms 22 firewall (192.168.10.253) 4.639 ms 4.320 ms 4.595 ms 23 firewall (192.168.10.253) 4.571 ms 4.765 ms 4.742 ms 24 firewall (192.168.10.253) 4.568 ms 5.112 ms 3.946 ms 25 firewall (192.168.10.253) 4.422 ms 4.144 ms 13.145 ms 26 firewall (192.168.10.253) 5.019 ms 5.222 ms 5.074 ms 27 firewall (192.168.10.253) 5.339 ms 5.377 ms 5.451 ms 28 firewall (192.168.10.253) 5.545 ms 5.846 ms 5.626 ms 29 firewall (192.168.10.253) 5.791 ms 5.780 ms 5.856 ms 30 firewall (192.168.10.253) 6.212 ms 6.636 ms 6.098 ms 31 firewall (192.168.10.253) 6.258 ms 6.014 ms 6.440 ms 32 firewall (192.168.10.253) 6.355 ms 6.512 ms 6.676 ms 33 firewall (192.168.10.253) 6.618 ms 6.605 ms 6.692 ms 34 firewall (192.168.10.253) 6.458 ms 6.836 ms 6.798 ms 35 firewall (192.168.10.253) 7.249 ms 7.280 ms 6.963 ms 36 firewall (192.168.10.253) 7.442 ms 7.836 ms 7.169 ms 37 firewall (192.168.10.253) 7.234 ms 7.284 ms 7.819 ms 38 firewall (192.168.10.253) 7.534 ms 7.646 ms 7.665 ms 39 firewall (192.168.10.253) 7.725 ms 10.499 ms 7.702 ms 40 firewall (192.168.10.253) 7.878 ms 8.240 ms 8.000 ms 41 firewall (192.168.10.253) 8.207 ms 8.306 ms 9.364 ms 42 firewall (192.168.10.253) 8.752 ms 8.749 ms 8.409 ms 43 firewall (192.168.10.253) 8.591 ms 8.576 ms 9.108 ms 44 firewall (192.168.10.253) 9.119 ms 8.698 ms 8.927 ms 45 firewall (192.168.10.253) 8.906 ms 9.308 ms 10.098 ms 46 firewall (192.168.10.253) 9.643 ms 9.299 ms 9.971 ms 47 firewall (192.168.10.253) 9.331 ms 9.299 ms 9.776 ms 48 firewall (192.168.10.253) 9.662 ms 9.608 ms 9.670 ms 49 firewall (192.168.10.253) 10.352 ms 9.835 ms 9.935 ms 50 firewall (192.168.10.253) 9.766 ms 10.049 ms 10.542 ms 51 firewall (192.168.10.253) 10.807 ms 10.577 ms 10.820 ms 52 firewall (192.168.10.253) 10.282 ms 10.765 ms 10.631 ms 53 firewall (192.168.10.253) 11.281 ms 10.871 ms 10.529 ms 54 firewall (192.168.10.253) 10.925 ms 10.899 ms 10.074 ms 55 firewall (192.168.10.253) 11.268 ms 11.171 ms 11.254 ms 56 firewall (192.168.10.253) 11.288 ms 11.433 ms 11.684 ms 57 firewall (192.168.10.253) 12.210 ms 11.275 ms 11.124 ms 58 firewall (192.168.10.253) 12.057 ms 11.766 ms 14.988 ms 59 firewall (192.168.10.253) 12.532 ms 12.135 ms 11.912 ms 60 firewall (192.168.10.253) 11.902 ms 12.742 ms 12.017 ms 61 firewall (192.168.10.253) 12.558 ms 12.342 ms 12.777 ms 62 firewall (192.168.10.253) 12.777 ms 12.545 ms 10.054 ms 63 firewall (192.168.10.253) 20.855 ms 13.525 ms 14.310 ms 64 firewall (192.168.10.253) 12.572 ms 12.898 ms 13.967 ms
A traceroute to google looks like that:
[2.2.5-RELEASE][root@firewall.office.xxxxxxxxxxx.xxx]/root: traceroute www.google.de traceroute: Warning: www.google.de has multiple addresses; using 173.194.116.127 traceroute to www.google.de (173.194.116.127), 64 hops max, 52 byte packets 1 xxxxxxx (xxxxxx) 0.375 ms 0.318 ms 0.339 ms 2 0.0.0.0 (0.0.0.0) 5.196 ms 5.845 ms 6.048 ms 3 xxxxxxxxxx (80.69.105.49) 11.142 ms 12.488 ms 10.850 ms 4 xxxxxx (xxxxxx) 52.380 ms 20.000 ms 10.509 ms 5 de-fra03b-ri1-ae5-0.aorta.net (84.116.133.118) 9.858 ms de-fra03b-ri1-ae12-0.aorta.net (84.116.133.97) 11.882 ms de-fra03b-ri1-ae5-0.aorta.net (84.116.133.118) 11.355 ms 6 72.14.219.9 (72.14.219.9) 10.885 ms 10.995 ms 13.836 ms 7 216.239.56.20 (216.239.56.20) 14.416 ms 11.722 ms 13.738 ms 8 64.233.175.143 (64.233.175.143) 12.426 ms 11.914 ms 11.362 ms 9 fra02s27-in-f31.1e100.net (173.194.116.127) 11.707 ms 12.732 ms 13.010 ms
- as expected.
We're using Unitymedia Germany Cable services.
Doesn't that look like a loop?