Broken port forwarding



  • Another story of Port forwarding not working. Curiously, this was working a while ago, but stopped working, possibly when I got my Site-to-Site OpenVPN connections working.

    Configuration is a physical pfSense box with two WAN connections:

    • 4G_WAN which ends up on a double-NAT out to the internet
    • ADSL_WAN which is bridged to the internet with a fixed IP address

    The default gateway is a gateway group with the 4G as the Tier 1, ADSL as Tier 2. Yeah, a little strange, but I get better 4G performance than ADSL here and it's relatively cheap.

    Since this stopped working, I've torn down everything and am starting fresh. So at the moment, I have one Port Forward rule for port 25:

    0_1545210297088_portforward1.png

    With the corresponding firewall rule:

    0_1545210361335_portforward2.png

    When I watch the logs on the mail server, there's no activity coming from the outside at all. On the firewall, pfSense is passing in the connections:

    0_1545210524374_firewallpasses25.png

    And on the server, netstat shows the following:

    tcp        0      0 192.168.10.209:smtp     mailhost11.lists.:56660 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     mxslcpool33.ebay.:40381 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     st43p00im-zteg100:40254 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     mail-ve1eur03lp205:6464 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     mail-ve1eur02lp20:18330 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     mail-eopbgr150105:27579 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     mxslcpool33.ebay.:38653 SYN_RECV   
    tcp        0      0 192.168.10.209:smtp     mail-eopbgr50102.:19091 SYN_RECV   
    

    And the corresponding states viewed by pfSense:
    0_1545210694558_states-timewait.png

    Now in order to get the OpenVPN routing working properly, on the LAN interface, I have an override for all traffic to the private subnets on the other OpenVPN sites set to the default gateway, while general internet traffic goes through the gateway group with the 4G Tier 1.

    0_1545210890758_LANrules.png

    A packet capture on port 25 on the ADSLWAN interface shows incoming connections but I don't see any corresponding responses:

    10:22:20.584803 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 50, id 15133, offset 0, flags [DF], proto TCP (6), length 60)
        198.2.136.56.36511 > 78.226.49.193.25: Flags [S], cksum 0x3711 (correct), seq 3429183177, win 14600, options [mss 1460,sackOK,TS val 783896680 ecr 0,nop,wscale 7], length 0
    10:22:31.271663 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41628, offset 0, flags [DF], proto TCP (6), length 60)
        83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9df8 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688106083 ecr 0,nop,wscale 7], length 0
    10:22:32.271203 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41629, offset 0, flags [DF], proto TCP (6), length 60)
        83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9a10 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688107083 ecr 0,nop,wscale 7], length 0
    10:22:32.721800 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 48, id 32553, offset 0, flags [DF], proto TCP (6), length 60)
        136.147.176.90.44592 > 78.226.49.193.25: Flags [S], cksum 0x1ee9 (correct), seq 2286167703, win 14600, options [mss 1460,sackOK,TS val 109957835 ecr 0,nop,wscale 7], length 0
    10:22:34.271054 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41630, offset 0, flags [DF], proto TCP (6), length 60)
        83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9240 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688109083 ecr 0,nop,wscale 7], length 0
    10:22:36.584715 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 50, id 15134, offset 0, flags [DF], proto TCP (6), length 60)
        198.2.136.56.36511 > 78.226.49.193.25: Flags [S], cksum 0xf890 (correct), seq 3429183177, win 14600, options [mss 1460,sackOK,TS val 783912680 ecr 0,nop,wscale 7], length 0
    10:22:38.271192 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41631, offset 0, flags [DF], proto TCP (6), length 60)
        83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x82a0 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688113083 ecr 0,nop,wscale 7], length 0
    

    On the LAN interface, I see the incoming requests and the outbound responses:

    09:40:48.371146 a0:ce:c8:04:16:e8 > 00:50:56:93:37:64, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 49, id 30184, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->25aa)!)
        204.92.22.84.18882 > 192.168.10.209.25: Flags [S], cksum 0xd473 (correct), seq 1971230723, win 29200, options [mss 1460,sackOK,TS val 3545508000 ecr 0,nop,wscale 7], length 0
    09:40:48.371574 00:50:56:93:37:64 > a0:ce:c8:04:16:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
        192.168.10.209.25 > 204.92.22.84.18882: Flags [S.], cksum 0x113d (correct), seq 2509475563, ack 1971230724, win 28960, options [mss 1460,sackOK,TS val 13751701 ecr 3545493968,nop,wscale 7], length 0
    

    However, checking the 4G interface there is no traffic on port 25, so I don't know where those responses are going to.

    Side notes:

    • I have full functioning access to the server from the internal network (SMTP, IMAP, etc), so it's not the machine that's not responding.
    • At the moment I have turned off (disabled) the OpenVPN connections to see if that has any impact on the behaviour and those networks are no longer noted in the routing table.
    • Outbound NAT is set to the default of Automatic rule generation
    • System Advanced > Firewall > Disable Reply-to is not activated


  • Add a rule on your lan and send the mail server ip straight to adsl, same way as your port forward is coming into and retest.
    Seems like the syn replies try to return via 4g...



  • @netblues That's what I think is happening as well, and I've added the following rule but I have the same behaviour:

    0_1545219293910_Screenshot 2018-12-19 at 12.33.59.png



  • @eableson Have you cleared states? Can you connect from mail server to the internet?
    Try changing the failover priority and use adsl first and see if it works.

    (No firewalls on mail server too I guess..)



  • @netblues Yup - The mail server has no problems going out and getting stuff and I can send email to it from the internal network which get properly forwarded along to the defined smarthost. So basically anything initiated by the mail server works fine, but the SYN ACK responses get to the pfSense box and it's trying to go out through a route that doesn't work for it.

    0_1545219812787_IMG_4791.jpeg

    Here I can see after each initiated connection that the gateway tried to return the ACK, but can't find a route to the originator.



  • @eableson Need to check nat outbound.
    Switch to manual and create rules for your two connections.



  • @netblues Like this?

    0_1545220543720_Screenshot 2018-12-19 at 12.55.21.png

    Still no go...

    Even with the two rules:

    0_1545220974316_Screenshot 2018-12-19 at 13.02.29.png



  • I don't get it. Why 104.47.9.51 is unreachable from pf?
    From the command line of pf, can you ping the world from adsl wan interface?
    Can you ping your adsl wan side from remote?



  • @netblues Ah, now things are getting interesting. Current state :

    • Disconnected the 4G connection (but still in the configuration)
    • Turned the OpenVPN connections back on to be sure I'm fixing the problem in a state that won't just get broken again once I turn it back on

    So at the moment from the pfSense box:

    • ping 8.8.8.8 from WANADSL IP: OK
    • ping 8.8.8.8 from LAN IP: No route to host

    But from a computer on the LAN, ping to 8.8.8.8 works fine... because I had set the priority routing up so that it would take the default route only when trying to go to other internal networks. When I look at the trace route, the first hop is the ADSL WAN Gateway, not the internal LAN IP address.

    So the current routing table looks like this:

    Destination        Gateway            Flags     Netif Expire
    78.226.49.0/24     link#2             U           re1
    78.226.49.193      link#2             UHS         lo0
    127.0.0.1          link#4             UH          lo0
    172.16.11.1        link#8             UH       ovpnc2
    172.16.11.2        link#8             UHS         lo0
    172.16.12.1        link#9             UH       ovpnc3
    172.16.12.2        link#9             UHS         lo0
    192.168.2.0/24     172.16.12.1        UGS      ovpnc3
    192.168.5.0/24     172.16.11.1        UGS      ovpnc2
    192.168.6.0/24     172.16.11.1        UGS      ovpnc2
    192.168.10.0/24    link#7             U           ue0
    192.168.10.1       link#7             UHS         lo0
    

    And attached are the current rules NAT & FW:

    0_1545226719685_pfctl-rules.txt

    So the issues seems to be that in a multi-wan setup that has VPN connections, the default routing table will only include known internal networks and if you need to go to the internet, it has to be via a gateway group. So the question I'm asking myself is if I should be declaring the OpenVPN connections as physical interfaces so that their routing rules get handled like regular interfaces.



  • @eableson @netblues Found it! After staring at the combination of the routing table and the "no route to host" message, I realised that there was no default route listed in the routing table.

    Digging around, this appears to be a side of effect of using gateway groups and setting the Default gateway IPv4 to a gateway group instead of setting it to a physical interface. Setting the Default gateway to the ADSL WAN connection add it as the default route and things start working again...



  • @eableson So, it has nothing to do with natting and port forward, as expected :)



  • @netblues As usual, the source of the problem was elsewhere, but brought to light by adding NAT and Port Forwarding to the mix. This one could stand to have a tech note - but before I ask I'm going to go back and reread the documentation about setting up multi-wan



  • @eableson Well, the norm is sending your default gw to the internet and not the vpn routes. (if being a server)



  • @netblues Well - it's not terribly clear about the correct use of the default gateway option when in multi wan mode. The trick is that in this configuration, most traffic should go out the 4G (high performance) connection, failing back to the ADSL when it drops (usually once every day or two). So in that case, the default configuration would be the default gateway as this gateway group, not a physical interface.

    Then there are services that have to run on the ADSL line since it's a direct connection with a fixed IP. But they are very much the exception. But in all cases, they point to the pfSense as the default gateway from their perspective and it's responsible for knowing the best route to take.



  • If we all got $1 for every "Gah! pfSense is hacked/broken/whatever!" and it turned out to be a configuration issue, we would all be able to retire.