Packages blocked to one mailserver host only (LAN => WAN)

[taenzerme]

Hello all,

our office's pfSense 2.2.4 (virtualized Proxmox kvm vm) is blocking legitimate outgoing connections from one local client to our external mail server. I know the topic itself has been discussed a lot and is in the wiki, yet until know I have not found a solution to this and don't know where to look for.

pfSense is blocking TCP:A, TCP:R, TCP:FA, TCP:PA to the mail server's ports (993, 443, 25, 465) - a Kerio Connect self-hosted VM in our cluster in our colocation. The problem disappears by itself after some minutes again, but the server is not reachable from the client:

Request timeout for icmp_seq 465
36 bytes from xxxxx.biz (x.x.x.x): Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 37fd   0 0000  3f  01 377f 192.168.10.8  123.x.x.x

Request timeout for icmp_seq 466
36 bytes from b2b-92-50-115-2.unitymedia.biz (92.50.115.2): Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 fcfe   0 0000  3f  01 727d 192.168.10.8  123.x.x.x 

On the firewall the logs look like attached.

I know that these looks like out of state / timed out packets and read about 50+ posts and discussions on this topic. Still I don't get where to start looking. Where is the culprit here? Is it the remote side or why is pfSense blocking this?

Usually this should not be causing any problems, but in our case this blocks using the mail server while this happens, which keeps me from doing my work ;-)

Would be happy to get a hint where to look.

Best
Sebastian

[viragomann]

Have you already tried to increase the state timeout? Maybe it helps.
Edit the firewall rule which allows this connection, go down to "Advanced Options" and click on Advanced, in the undermost input field enter a higher timeout in seconds.

[taenzerme]

Thanks for taking the time to answer - and for the good idea.

The timeout seconds field is empty. What would be a reasonable value here?

[viragomann]

AFAIR the default value is 60 seconds.
If you have large mails and a slow connection maybe this is not enough.
Try out what fits your purpose. To increase this value enlarges the state table and need more RAM.
For use with Exchange OMA I had set this value to 15 minutes on an ALIX board. Though there are no RAM bottleneck, however there are just about a dozen connections.

[taenzerme]

Set to 300s and so far no problems in the last hour. Thanks for the help, really appreciated.

[taenzerme]

Unfortunately this did not help in the long run. The mail server is still beeing blocked from time to time for some duration (about 5-10 minutes, then it starts working again all by itself):

PING my.mailserver.com (xxx.xxx.xxx.xxx): 56 data bytes
36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 1762   0 0000  3f  01 581a 192.168.10.8  217.76.104.48 

Request timeout for icmp_seq 0
36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 6c42   0 0000  3f  01 033a 192.168.10.8  xxx.xxx.xxx.xxx 

Request timeout for icmp_seq 1
36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 6f6a   0 0000  3f  01 0012 192.168.10.8  xxx.xxx.xxx.xxx 

That's just ICMP, but all other protocols fail as well.

Any more ideas where to look?

[taenzerme]

We still fight with this problem. I tried everything suggested in this thread, but still the pfSense blocks connections to that server only from time to time. Any ideas? I would be willing to pay for solutions/help actually, but don't know of any pfSense support in our region (Cologne/Bonn, Germany).

Update: Firewall log reports this as blocking rule: @5(1000000103) block drop in log inet all label "Default deny rule IPv4" for the LAN if. I understand that this is actually always the case and that this is out of state traffic - but how does it happen and why does this lead to a full block for the mail server?

Best,
Sebastian

[doktornotor]

Yeah you still fight the problem and we STILL have not seen the firewall rules configured.

[taenzerme]

Sorry, no one requested the rules config ;-)
Rules config attached to this post.

Thanks for the reply and help. I appreciate it.

filter-config-20151125114043.txt

[doktornotor]

I thought you would post a screenshot… not this. There is not a single mention of port 25, 143, 465 or port 993 in those rules. No idea how you expect the traffic to get passed.

[taenzerme]

There is only one rule for LAN: Allow all (see screenshot attached).
I also attached the WAN rules.

From my understanding the allow LAN to all rule should be enough, shouldn't it?


[doktornotor]

What's that advanced stuff in there in the LAN rule? (Certainly not default).
What's UM interface (shown on all of your screenshots with the traffic blocked?!)

[taenzerme]

That's actually strange as there has not been done any advanced modification.

UM = UnityMedia = WAN.


[doktornotor]

@taenzerme:

That's actually strange as there has not been done any advanced modification.

Clearly was. That "Advanced Options - This allows packets with IP options to pass." certainly ain't ticked by default and I think you completely misunderstood this suggestion. You shouldn't tick anything there, you should put a custom timeout to "State Timeout in seconds (TCP only)"

Where's the mailserver located? On WAN or behind pfSense?!
Where are you testing this from?

[taenzerme]

Disabled the option "allow packets" to default. Yes, I clearly misunderstood the suggestion.

I had the custom state timeout enabled with 300 (seconds) for some time, but it did not help.

Mailserver is located at a remote colocation behind another pfSense.

Testing from MacOS Apple Mail, terminal on MacOS clients and several vm's (Debian based) in our local network.

[doktornotor]

@taenzerme:

Mailserver is located at a remote colocation behind another pfSense.

Great. So, perhaps move your debugging efforts there.

[taenzerme]

I already did. Logs there do not mention any blocks at all. Nothing, nada. Our local public IP is not beeing blocked anywhere at that remote location and other servers can actually access the mail server while it is beeing blocked at the office.

I will go on digging through the logs, but IMHO this is happening at our local pfSense.

[taenzerme]

OK, so I tried all things recommended here and this still happens.

Remote firewall does not log anything and other clients can connect to the server. Only our network can't.

Any other ideas what to check?

[taenzerme]

I filtered the log for source UM (WAN) and the destination mail server and vice versa. It seems completely random to me.

[Ojisang]

Hi @taenzerme,

Just my opinion but maybe it could be ISP problem? If you didn't change any config before the issue, maybe it could be rerouting from your ISP? I once encountered a problem where I have problem connecting to our mail server due to ISP reroute.