Postfix behind pfSense can't send anything but very small emails



  • Hi,

    I am facing an issue with one of the 6 servers behind pfSense, where small emails like "test" in the body get delivered, but anything larger gets stuck with

    (conversation with alt1.gmail-smtp-in.l.google.com[74.125.205.26] timed out while sending message body)
    

    The same happens with emails send to other emails, not only gmail.

    All other servers seem to work fine, but they are identically configured, just standard OpenSUSE config.

    I've checked hostnames, local firewalls on the servers (turned off), postfix configuration etc. and everything is identical.
    This server used to send emails just fine before I replaced the old routers with pfSense box.
    There is nothing in the firewall log about the IP of the server in question.

    We are on dual WAN with a LAN rule to use WAN1 for all port 25 traffic as well as the email providers we use.
    Disabling this rules does not help.

    Could it be the MTU? Or what else could be causing this?

    packets captured on the LAN side when an email is sent:

    18:07:02.859982 IP (tos 0x0, ttl 64, id 41224, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61436 > 64.233.166.27.25: Flags [.], cksum 0xcb17 (correct), ack 1467295376, win 237, options [nop,nop,TS val 7420544 ecr 145432344], length 1418
    18:07:03.163932 IP (tos 0x0, ttl 64, id 14938, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.15325 > 74.125.205.26.25: Flags [.], cksum 0x0983 (correct), ack 3117250333, win 237, options [nop,nop,TS val 7420620 ecr 747195245], length 1418
    18:07:04.183215 IP (tos 0x0, ttl 64, id 54508, offset 0, flags [DF], proto TCP (6), length 60)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [s], cksum 0xfa7e (correct), seq 1345711910, win 29200, options [mss 1460,sackOK,TS val 7420874 ecr 0,nop,wscale 7], length 0
    18:07:04.196227 IP (tos 0x0, ttl 48, id 11759, offset 0, flags [none], proto TCP (6), length 60)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [S.], cksum 0x49c7 (correct), seq 2454828461, ack 1345711911, win 42540, options [mss 1430,sackOK,TS val 102117011 ecr 7420874,nop,wscale 7], length 0
    18:07:04.196457 IP (tos 0x0, ttl 64, id 54509, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x1db9 (correct), ack 1, win 229, options [nop,nop,TS val 7420878 ecr 102117011], length 0
    18:07:04.221116 IP (tos 0x0, ttl 48, id 11760, offset 0, flags [none], proto TCP (6), length 103)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [P.], cksum 0x41ae (correct), ack 1, win 333, options [nop,nop,TS val 102117036 ecr 7420878], length 51
    18:07:04.221319 IP (tos 0x0, ttl 64, id 54510, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x1d67 (correct), ack 52, win 229, options [nop,nop,TS val 7420884 ecr 102117036], length 0
    18:07:04.221333 IP (tos 0x0, ttl 64, id 54511, offset 0, flags [DF], proto TCP (6), length 78)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [P.], cksum 0xfafb (correct), ack 52, win 229, options [nop,nop,TS val 7420884 ecr 102117036], length 26
    18:07:04.234371 IP (tos 0x0, ttl 48, id 11761, offset 0, flags [none], proto TCP (6), length 52)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [.], cksum 0x1cd8 (correct), ack 27, win 333, options [nop,nop,TS val 102117049 ecr 7420884], length 0
    18:07:04.238698 IP (tos 0x0, ttl 48, id 11762, offset 0, flags [none], proto TCP (6), length 220)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [P.], cksum 0x9a17 (correct), ack 27, win 333, options [nop,nop,TS val 102117053 ecr 7420884], length 168
    18:07:04.238941 IP (tos 0x0, ttl 64, id 54512, offset 0, flags [DF], proto TCP (6), length 140)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [P.], cksum 0x6fb1 (correct), ack 220, win 237, options [nop,nop,TS val 7420888 ecr 102117053], length 88
    18:07:04.252417 IP (tos 0x0, ttl 48, id 11763, offset 0, flags [none], proto TCP (6), length 92)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [P.], cksum 0xfb86 (correct), ack 115, win 333, options [nop,nop,TS val 102117067 ecr 7420888], length 40
    18:07:04.291903 IP (tos 0x0, ttl 64, id 54513, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x1bec (correct), ack 260, win 237, options [nop,nop,TS val 7420902 ecr 102117067], length 0
    18:07:04.667117 IP (tos 0x0, ttl 48, id 11764, offset 0, flags [none], proto TCP (6), length 92)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [P.], cksum 0xf4b1 (correct), ack 115, win 333, options [nop,nop,TS val 102117482 ecr 7420902], length 40
    18:07:04.667253 IP (tos 0x0, ttl 48, id 11765, offset 0, flags [none], proto TCP (6), length 93)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [P.], cksum 0x0a32 (correct), ack 115, win 333, options [nop,nop,TS val 102117482 ecr 7420902], length 41
    18:07:04.667310 IP (tos 0x0, ttl 64, id 54514, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x19c8 (correct), ack 300, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 0
    18:07:04.667315 IP (tos 0x0, ttl 64, id 54515, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x199f (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 0
    18:07:04.667560 IP (tos 0x0, ttl 64, id 54516, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xc2b7 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.667579 IP (tos 0x0, ttl 64, id 54517, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x46f3 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.667591 IP (tos 0x0, ttl 64, id 54518, offset 0, flags [DF], proto TCP (6), length 1312)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [P.], cksum 0xe046 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1260
    18:07:04.667598 IP (tos 0x0, ttl 64, id 54519, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xbd9d (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.667615 IP (tos 0x0, ttl 64, id 54520, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x8679 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.667621 IP (tos 0x0, ttl 64, id 54521, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x39f4 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.667627 IP (tos 0x0, ttl 64, id 54522, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xbee3 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.667633 IP (tos 0x0, ttl 64, id 54523, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x5751 (correct), ack 341, win 237, options [nop,nop,TS val 7420995 ecr 102117482], length 1418
    18:07:04.681058 IP (tos 0x0, ttl 48, id 11766, offset 0, flags [none], proto TCP (6), length 64)
        64.233.166.27.25 > 192.168.200.20.61438: Flags [.], cksum 0x4853 (correct), ack 115, win 353, options [nop,nop,TS val 102117496 ecr 7420995,nop,nop,sack 1 {2951:4211}], length 0
    18:07:04.879938 IP (tos 0x0, ttl 64, id 54524, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xc273 (correct), ack 341, win 237, options [nop,nop,TS val 7421049 ecr 102117496], length 1418
    18:07:05.311938 IP (tos 0x0, ttl 64, id 54525, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xc207 (correct), ack 341, win 237, options [nop,nop,TS val 7421157 ecr 102117496], length 1418
    18:07:06.175935 IP (tos 0x0, ttl 64, id 54526, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xc12f (correct), ack 341, win 237, options [nop,nop,TS val 7421373 ecr 102117496], length 1418
    18:07:07.907932 IP (tos 0x0, ttl 64, id 54527, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xbf7e (correct), ack 341, win 237, options [nop,nop,TS val 7421806 ecr 102117496], length 1418
    18:07:11.115947 IP (tos 0x0, ttl 64, id 14939, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.15325 > 74.125.205.26.25: Flags [.], cksum 0x01bf (correct), ack 1, win 237, options [nop,nop,TS val 7422608 ecr 747195245], length 1418
    18:07:11.371923 IP (tos 0x0, ttl 64, id 54528, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xbc1c (correct), ack 341, win 237, options [nop,nop,TS val 7422672 ecr 102117496], length 1418
    18:07:18.299906 IP (tos 0x0, ttl 64, id 54529, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xb558 (correct), ack 341, win 237, options [nop,nop,TS val 7424404 ecr 102117496], length 1418
    18:07:27.019975 IP (tos 0x0, ttl 64, id 14940, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.15325 > 74.125.205.26.25: Flags [.], cksum 0xf236 (correct), ack 1, win 237, options [nop,nop,TS val 7426584 ecr 747195245], length 1418
    18:07:32.171990 IP (tos 0x0, ttl 64, id 54530, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0xa7cc (correct), ack 341, win 237, options [nop,nop,TS val 7427872 ecr 102117496], length 1418
    18:07:57.259944 IP (tos 0x0, ttl 64, id 41225, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61436 > 64.233.166.27.25: Flags [.], cksum 0x95f7 (correct), ack 1, win 237, options [nop,nop,TS val 7434144 ecr 145432344], length 1418
    18:07:58.795926 IP (tos 0x0, ttl 64, id 14941, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.15325 > 74.125.205.26.25: Flags [.], cksum 0xd32e (correct), ack 1, win 237, options [nop,nop,TS val 7434528 ecr 747195245], length 1418
    18:07:59.883921 IP (tos 0x0, ttl 64, id 54531, offset 0, flags [DF], proto TCP (6), length 1470)
        192.168.200.20.61438 > 64.233.166.27.25: Flags [.], cksum 0x8cbc (correct), ack 341, win 237, options [nop,nop,TS val 7434800 ecr 102117496], length 1418
    
    [/s]
    


  • I guess it could be MTU - if the small email body fits entirely in 1 packet that is smaller than the MTU anyway. When the email gets bigger there are multiple packets with some/all at the full MTU.

    Or it could be asymmetric routing - if pfSense is not seeing the traffic in both directions then the newly-establish state will timeout after a few seconds = small interactions will succeed but states that have to have traffic flowing for a while will dies in the middle. That could happen if you have some tricky multi-gateway and/or multi-router topology.



  • Thanks.

    There is no weird or complicated setup, just 2 WANS for failover with some port forwarding and LAN rules putting important stuff on the faster WAN.

    I will try adjusting the MTU when not in production, but what bothers me is why all other servers are able to send emails just fine, and only this one fails. There are all behind the same pfSense box. By the way this server is much newer hardware compared to the other, if anything with the newer network card matters.



  • Tried with different MTUs- no luck, besides, our ISP adviced to stick with the default 1500.

    Any other ideas?



  • It seems to be the MTU after all.

    When the box was first installed I tried MTU 1472, then deleted the value, left the box blank and saved, expecting that this would revert to the default MTU.
    It appears, after you leave the MTU box blank, it does not revert back to the default value, but somehow remembers the last value used, if there was any.

    After entering 1500 and save, then make that box blank and save again, everything works properly.

    How I noticed that- opened  Diagnostics: Routing tables and noticed some routes with MTU 1472.