Packages blocked to one mailserver host only (LAN => WAN)



  • Hello all,

    our office's pfSense 2.2.4 (virtualized Proxmox kvm vm) is blocking legitimate outgoing connections from one local client to our external mail server. I know the topic itself has been discussed a lot and is in the wiki, yet until know I have not found a solution to this and don't know where to look for.

    pfSense is blocking TCP:A, TCP:R, TCP:FA, TCP:PA to the mail server's ports (993, 443, 25, 465) - a Kerio Connect self-hosted VM in our cluster in our colocation. The problem disappears by itself after some minutes again, but the server is not reachable from the client:

    
    Request timeout for icmp_seq 465
    36 bytes from xxxxx.biz (x.x.x.x): Destination Host Unreachable
    Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
     4  5  00 5400 37fd   0 0000  3f  01 377f 192.168.10.8  123.x.x.x
    
    Request timeout for icmp_seq 466
    36 bytes from b2b-92-50-115-2.unitymedia.biz (92.50.115.2): Destination Host Unreachable
    Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
     4  5  00 5400 fcfe   0 0000  3f  01 727d 192.168.10.8  123.x.x.x 
    
    

    On the firewall the logs look like attached.

    I know that these looks like out of state / timed out packets and read about 50+ posts and discussions on this topic. Still I don't get where to start looking. Where is the culprit here? Is it the remote side or why is pfSense blocking this?

    Usually this should not be causing any problems, but in our case this blocks using the mail server while this happens, which keeps me from doing my work ;-)

    Would be happy to get a hint where to look.

    Best
    Sebastian



  • Have you already tried to increase the state timeout? Maybe it helps.
    Edit the firewall rule which allows this connection, go down to "Advanced Options" and click on Advanced, in the undermost input field enter a higher timeout in seconds.



  • Thanks for taking the time to answer - and for the good idea.

    The timeout seconds field is empty. What would be a reasonable value here?



  • AFAIR the default value is 60 seconds.
    If you have large mails and a slow connection maybe this is not enough.
    Try out what fits your purpose. To increase this value enlarges the state table and need more RAM.
    For use with Exchange OMA I had set this value to 15 minutes on an ALIX board. Though there are no RAM bottleneck, however there are just about a dozen connections.



  • Set to 300s and so far no problems in the last hour. Thanks for the help, really appreciated.



  • Unfortunately this did not help in the long run. The mail server is still beeing blocked from time to time for some duration (about 5-10 minutes, then it starts working again all by itself):

    
    PING my.mailserver.com (xxx.xxx.xxx.xxx): 56 data bytes
    36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable
    Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
     4  5  00 5400 1762   0 0000  3f  01 581a 192.168.10.8  217.76.104.48 
    
    Request timeout for icmp_seq 0
    36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable
    Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
     4  5  00 5400 6c42   0 0000  3f  01 033a 192.168.10.8  xxx.xxx.xxx.xxx 
    
    Request timeout for icmp_seq 1
    36 bytes from firewall.office.xxxx.xxx (192.168.10.253): Destination Host Unreachable
    Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
     4  5  00 5400 6f6a   0 0000  3f  01 0012 192.168.10.8  xxx.xxx.xxx.xxx 
    
    

    That's just ICMP, but all other protocols fail as well.

    Any more ideas where to look?




  • We still fight with this problem. I tried everything suggested in this thread, but still the pfSense blocks connections to that server only from time to time. Any ideas? I would be willing to pay for solutions/help actually, but don't know of any pfSense support in our region (Cologne/Bonn, Germany).

    Update: Firewall log reports this as blocking rule: @5(1000000103) block drop in log inet all label "Default deny rule IPv4" for the LAN if. I understand that this is actually always the case and that this is out of state traffic - but how does it happen and why does this lead to a full block for the mail server?

    Best,
    Sebastian


  • Banned

    Yeah you still fight the problem and we STILL have not seen the firewall rules configured.



  • Sorry, no one requested the rules config ;-)
    Rules config attached to this post.

    Thanks for the reply and help. I appreciate it.

    filter-config-20151125114043.txt


  • Banned

    I thought you would post a screenshot… not this. There is not a single mention of port 25, 143, 465 or port 993 in those rules. No idea how you expect the traffic to get passed.



  • There is only one rule for LAN: Allow all (see screenshot attached).
    I also attached the WAN rules.

    From my understanding the allow LAN to all rule should be enough, shouldn't it?

    ![Screenshot 2015-11-25 11.56.13.png](/public/imported_attachments/1/Screenshot 2015-11-25 11.56.13.png)
    ![Screenshot 2015-11-25 11.56.13.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 11.56.13.png_thumb)


  • Banned

    What's that advanced stuff in there in the LAN rule? (Certainly not default).
    What's UM interface (shown on all of your screenshots with the traffic blocked?!)



  • That's actually strange as there has not been done any advanced modification.

    UM = UnityMedia = WAN.

    ![Screenshot 2015-11-25 12.24.49.png](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.49.png)
    ![Screenshot 2015-11-25 12.24.49.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.49.png_thumb)
    ![Screenshot 2015-11-25 12.24.59.png](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.59.png)
    ![Screenshot 2015-11-25 12.24.59.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 12.24.59.png_thumb)
    ![Screenshot 2015-11-25 12.25.04.png](/public/imported_attachments/1/Screenshot 2015-11-25 12.25.04.png)
    ![Screenshot 2015-11-25 12.25.04.png_thumb](/public/imported_attachments/1/Screenshot 2015-11-25 12.25.04.png_thumb)


  • Banned

    @taenzerme:

    That's actually strange as there has not been done any advanced modification.

    Clearly was. That "Advanced Options - This allows packets with IP options to pass." certainly ain't ticked by default and I think you completely misunderstood this suggestion. You shouldn't tick anything there, you should put a custom timeout to "State Timeout in seconds (TCP only)"

    Where's the mailserver located? On WAN or behind pfSense?!
    Where are you testing this from?



  • Disabled the option "allow packets" to default. Yes, I clearly misunderstood the suggestion.

    I had the custom state timeout enabled with 300 (seconds) for some time, but it did not help.

    Mailserver is located at a remote colocation behind another pfSense.

    Testing from MacOS Apple Mail, terminal on MacOS clients and several vm's (Debian based) in our local network.


  • Banned

    @taenzerme:

    Mailserver is located at a remote colocation behind another pfSense.

    Great. So, perhaps move your debugging efforts there.



  • I already did. Logs there do not mention any blocks at all. Nothing, nada. Our local public IP is not beeing blocked anywhere at that remote location and other servers can actually access the mail server while it is beeing blocked at the office.

    I will go on digging through the logs, but IMHO this is happening at our local pfSense.



  • OK, so I tried all things recommended here and this still happens.

    Remote firewall does not log anything and other clients can connect to the server. Only our network can't.

    Any other ideas what to check?
































  • I filtered the log for source UM (WAN) and the destination mail server and vice versa. It seems completely random to me.










  • Hi @taenzerme,

    Just my opinion but maybe it could be ISP problem? If you didn't change any config before the issue, maybe it could be rerouting from your ISP? I once encountered a problem where I have problem connecting to our mail server due to ISP reroute.



  • Hi @Ojisang,

    thanks. It just happened again and I checked traceroute to our mail server on the firewall directly:

    
    [2.2.5-RELEASE][root@firewall.office.xxxxxxxxxxx.xxx]/root: traceroute my.mailserver.com
    traceroute to my.mailserver.com (xxx.xxx.xxx.48), 64 hops max, 52 byte packets
     1  firewall (192.168.10.253)  0.261 ms  0.227 ms  0.284 ms
     2  firewall (192.168.10.253)  0.602 ms  0.884 ms  0.543 ms
     3  firewall (192.168.10.253)  0.754 ms  0.691 ms  0.663 ms
     4  firewall (192.168.10.253)  0.808 ms  0.850 ms  0.825 ms
     5  firewall (192.168.10.253)  1.120 ms  1.122 ms  1.479 ms
     6  firewall (192.168.10.253)  1.268 ms  1.207 ms  1.104 ms
     7  firewall (192.168.10.253)  1.417 ms  1.421 ms  1.346 ms
     8  firewall (192.168.10.253)  1.525 ms  1.600 ms  1.495 ms
     9  firewall (192.168.10.253)  1.781 ms  1.843 ms  1.800 ms
    10  firewall (192.168.10.253)  2.093 ms  2.010 ms  2.003 ms
    11  firewall (192.168.10.253)  2.168 ms  2.211 ms  2.252 ms
    12  firewall (192.168.10.253)  2.347 ms  2.430 ms  2.498 ms
    13  firewall (192.168.10.253)  2.830 ms  2.611 ms  2.509 ms
    14  firewall (192.168.10.253)  2.782 ms  2.887 ms  2.885 ms
    15  firewall (192.168.10.253)  2.953 ms  3.058 ms  3.445 ms
    16  firewall (192.168.10.253)  3.646 ms  4.801 ms  3.203 ms
    17  firewall (192.168.10.253)  3.896 ms  3.469 ms  3.399 ms
    18  firewall (192.168.10.253)  4.041 ms  4.201 ms  3.944 ms
    19  firewall (192.168.10.253)  3.912 ms  3.796 ms  4.495 ms
    20  firewall (192.168.10.253)  3.882 ms  3.985 ms  4.003 ms
    21  firewall (192.168.10.253)  4.303 ms  4.193 ms  4.332 ms
    22  firewall (192.168.10.253)  4.639 ms  4.320 ms  4.595 ms
    23  firewall (192.168.10.253)  4.571 ms  4.765 ms  4.742 ms
    24  firewall (192.168.10.253)  4.568 ms  5.112 ms  3.946 ms
    25  firewall (192.168.10.253)  4.422 ms  4.144 ms  13.145 ms
    26  firewall (192.168.10.253)  5.019 ms  5.222 ms  5.074 ms
    27  firewall (192.168.10.253)  5.339 ms  5.377 ms  5.451 ms
    28  firewall (192.168.10.253)  5.545 ms  5.846 ms  5.626 ms
    29  firewall (192.168.10.253)  5.791 ms  5.780 ms  5.856 ms
    30  firewall (192.168.10.253)  6.212 ms  6.636 ms  6.098 ms
    31  firewall (192.168.10.253)  6.258 ms  6.014 ms  6.440 ms
    32  firewall (192.168.10.253)  6.355 ms  6.512 ms  6.676 ms
    33  firewall (192.168.10.253)  6.618 ms  6.605 ms  6.692 ms
    34  firewall (192.168.10.253)  6.458 ms  6.836 ms  6.798 ms
    35  firewall (192.168.10.253)  7.249 ms  7.280 ms  6.963 ms
    36  firewall (192.168.10.253)  7.442 ms  7.836 ms  7.169 ms
    37  firewall (192.168.10.253)  7.234 ms  7.284 ms  7.819 ms
    38  firewall (192.168.10.253)  7.534 ms  7.646 ms  7.665 ms
    39  firewall (192.168.10.253)  7.725 ms  10.499 ms  7.702 ms
    40  firewall (192.168.10.253)  7.878 ms  8.240 ms  8.000 ms
    41  firewall (192.168.10.253)  8.207 ms  8.306 ms  9.364 ms
    42  firewall (192.168.10.253)  8.752 ms  8.749 ms  8.409 ms
    43  firewall (192.168.10.253)  8.591 ms  8.576 ms  9.108 ms
    44  firewall (192.168.10.253)  9.119 ms  8.698 ms  8.927 ms
    45  firewall (192.168.10.253)  8.906 ms  9.308 ms  10.098 ms
    46  firewall (192.168.10.253)  9.643 ms  9.299 ms  9.971 ms
    47  firewall (192.168.10.253)  9.331 ms  9.299 ms  9.776 ms
    48  firewall (192.168.10.253)  9.662 ms  9.608 ms  9.670 ms
    49  firewall (192.168.10.253)  10.352 ms  9.835 ms  9.935 ms
    50  firewall (192.168.10.253)  9.766 ms  10.049 ms  10.542 ms
    51  firewall (192.168.10.253)  10.807 ms  10.577 ms  10.820 ms
    52  firewall (192.168.10.253)  10.282 ms  10.765 ms  10.631 ms
    53  firewall (192.168.10.253)  11.281 ms  10.871 ms  10.529 ms
    54  firewall (192.168.10.253)  10.925 ms  10.899 ms  10.074 ms
    55  firewall (192.168.10.253)  11.268 ms  11.171 ms  11.254 ms
    56  firewall (192.168.10.253)  11.288 ms  11.433 ms  11.684 ms
    57  firewall (192.168.10.253)  12.210 ms  11.275 ms  11.124 ms
    58  firewall (192.168.10.253)  12.057 ms  11.766 ms  14.988 ms
    59  firewall (192.168.10.253)  12.532 ms  12.135 ms  11.912 ms
    60  firewall (192.168.10.253)  11.902 ms  12.742 ms  12.017 ms
    61  firewall (192.168.10.253)  12.558 ms  12.342 ms  12.777 ms
    62  firewall (192.168.10.253)  12.777 ms  12.545 ms  10.054 ms
    63  firewall (192.168.10.253)  20.855 ms  13.525 ms  14.310 ms
    64  firewall (192.168.10.253)  12.572 ms  12.898 ms  13.967 ms
    
    

    A traceroute to google looks like that:

    
    [2.2.5-RELEASE][root@firewall.office.xxxxxxxxxxx.xxx]/root: traceroute www.google.de
    traceroute: Warning: www.google.de has multiple addresses; using 173.194.116.127
    traceroute to www.google.de (173.194.116.127), 64 hops max, 52 byte packets
     1  xxxxxxx (xxxxxx)  0.375 ms  0.318 ms  0.339 ms
     2  0.0.0.0 (0.0.0.0)  5.196 ms  5.845 ms  6.048 ms
     3  xxxxxxxxxx (80.69.105.49)  11.142 ms  12.488 ms  10.850 ms
     4  xxxxxx (xxxxxx)  52.380 ms  20.000 ms  10.509 ms
     5  de-fra03b-ri1-ae5-0.aorta.net (84.116.133.118)  9.858 ms
        de-fra03b-ri1-ae12-0.aorta.net (84.116.133.97)  11.882 ms
        de-fra03b-ri1-ae5-0.aorta.net (84.116.133.118)  11.355 ms
     6  72.14.219.9 (72.14.219.9)  10.885 ms  10.995 ms  13.836 ms
     7  216.239.56.20 (216.239.56.20)  14.416 ms  11.722 ms  13.738 ms
     8  64.233.175.143 (64.233.175.143)  12.426 ms  11.914 ms  11.362 ms
     9  fra02s27-in-f31.1e100.net (173.194.116.127)  11.707 ms  12.732 ms  13.010 ms
    
    
    • as expected.

    We're using Unitymedia Germany Cable services.

    Doesn't that look like a loop?


Log in to reply