Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Broken port forwarding

    NAT
    nat port forward
    3
    15
    1.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      eableson
      last edited by eableson

      Another story of Port forwarding not working. Curiously, this was working a while ago, but stopped working, possibly when I got my Site-to-Site OpenVPN connections working.

      Configuration is a physical pfSense box with two WAN connections:

      • 4G_WAN which ends up on a double-NAT out to the internet
      • ADSL_WAN which is bridged to the internet with a fixed IP address

      The default gateway is a gateway group with the 4G as the Tier 1, ADSL as Tier 2. Yeah, a little strange, but I get better 4G performance than ADSL here and it's relatively cheap.

      Since this stopped working, I've torn down everything and am starting fresh. So at the moment, I have one Port Forward rule for port 25:

      0_1545210297088_portforward1.png

      With the corresponding firewall rule:

      0_1545210361335_portforward2.png

      When I watch the logs on the mail server, there's no activity coming from the outside at all. On the firewall, pfSense is passing in the connections:

      0_1545210524374_firewallpasses25.png

      And on the server, netstat shows the following:

      tcp        0      0 192.168.10.209:smtp     mailhost11.lists.:56660 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mxslcpool33.ebay.:40381 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     st43p00im-zteg100:40254 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-ve1eur03lp205:6464 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-ve1eur02lp20:18330 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-eopbgr150105:27579 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mxslcpool33.ebay.:38653 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-eopbgr50102.:19091 SYN_RECV

      And the corresponding states viewed by pfSense:
      0_1545210694558_states-timewait.png

      Now in order to get the OpenVPN routing working properly, on the LAN interface, I have an override for all traffic to the private subnets on the other OpenVPN sites set to the default gateway, while general internet traffic goes through the gateway group with the 4G Tier 1.

      0_1545210890758_LANrules.png

      A packet capture on port 25 on the ADSLWAN interface shows incoming connections but I don't see any corresponding responses:

      10:22:20.584803 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 50, id 15133, offset 0, flags [DF], proto TCP (6), length 60)
    198.2.136.56.36511 > 78.226.49.193.25: Flags [S], cksum 0x3711 (correct), seq 3429183177, win 14600, options [mss 1460,sackOK,TS val 783896680 ecr 0,nop,wscale 7], length 0
10:22:31.271663 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41628, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9df8 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688106083 ecr 0,nop,wscale 7], length 0
10:22:32.271203 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41629, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9a10 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688107083 ecr 0,nop,wscale 7], length 0
10:22:32.721800 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 48, id 32553, offset 0, flags [DF], proto TCP (6), length 60)
    136.147.176.90.44592 > 78.226.49.193.25: Flags [S], cksum 0x1ee9 (correct), seq 2286167703, win 14600, options [mss 1460,sackOK,TS val 109957835 ecr 0,nop,wscale 7], length 0
10:22:34.271054 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41630, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9240 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688109083 ecr 0,nop,wscale 7], length 0
10:22:36.584715 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 50, id 15134, offset 0, flags [DF], proto TCP (6), length 60)
    198.2.136.56.36511 > 78.226.49.193.25: Flags [S], cksum 0xf890 (correct), seq 3429183177, win 14600, options [mss 1460,sackOK,TS val 783912680 ecr 0,nop,wscale 7], length 0
10:22:38.271192 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41631, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x82a0 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688113083 ecr 0,nop,wscale 7], length 0

      On the LAN interface, I see the incoming requests and the outbound responses:

      09:40:48.371146 a0:ce:c8:04:16:e8 > 00:50:56:93:37:64, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 49, id 30184, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->25aa)!)
    204.92.22.84.18882 > 192.168.10.209.25: Flags [S], cksum 0xd473 (correct), seq 1971230723, win 29200, options [mss 1460,sackOK,TS val 3545508000 ecr 0,nop,wscale 7], length 0
09:40:48.371574 00:50:56:93:37:64 > a0:ce:c8:04:16:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.209.25 > 204.92.22.84.18882: Flags [S.], cksum 0x113d (correct), seq 2509475563, ack 1971230724, win 28960, options [mss 1460,sackOK,TS val 13751701 ecr 3545493968,nop,wscale 7], length 0

      However, checking the 4G interface there is no traffic on port 25, so I don't know where those responses are going to.

      Side notes:

      • I have full functioning access to the server from the internal network (SMTP, IMAP, etc), so it's not the machine that's not responding.
      • At the moment I have turned off (disabled) the OpenVPN connections to see if that has any impact on the behaviour and those networks are no longer noted in the routing table.
      • Outbound NAT is set to the default of Automatic rule generation
      • System Advanced > Firewall > Disable Reply-to is not activated
      1 Reply Last reply Reply Quote 0
      • N
        netblues
        last edited by

        Add a rule on your lan and send the mail server ip straight to adsl, same way as your port forward is coming into and retest.
        Seems like the syn replies try to return via 4g...

        E 1 Reply Last reply Reply Quote 0
        • E
          eableson @netblues
          last edited by

          @netblues That's what I think is happening as well, and I've added the following rule but I have the same behaviour:

          0_1545219293910_Screenshot 2018-12-19 at 12.33.59.png

          N 1 Reply Last reply Reply Quote 0
          • N
            netblues @eableson
            last edited by

            @eableson Have you cleared states? Can you connect from mail server to the internet?
            Try changing the failover priority and use adsl first and see if it works.

            (No firewalls on mail server too I guess..)

            E 1 Reply Last reply Reply Quote 0
            • E
              eableson @netblues
              last edited by

              @netblues Yup - The mail server has no problems going out and getting stuff and I can send email to it from the internal network which get properly forwarded along to the defined smarthost. So basically anything initiated by the mail server works fine, but the SYN ACK responses get to the pfSense box and it's trying to go out through a route that doesn't work for it.

              0_1545219812787_IMG_4791.jpeg

              Here I can see after each initiated connection that the gateway tried to return the ACK, but can't find a route to the originator.

              N 1 Reply Last reply Reply Quote 0
              • N
                netblues @eableson
                last edited by

                @eableson Need to check nat outbound.
                Switch to manual and create rules for your two connections.

                E 1 Reply Last reply Reply Quote 0
                • E
                  eableson @netblues
                  last edited by eableson

                  @netblues Like this?

                  0_1545220543720_Screenshot 2018-12-19 at 12.55.21.png

                  Still no go...

                  Even with the two rules:

                  0_1545220974316_Screenshot 2018-12-19 at 13.02.29.png

                  1 Reply Last reply Reply Quote 0
                  • N
                    netblues
                    last edited by

                    I don't get it. Why 104.47.9.51 is unreachable from pf?
                    From the command line of pf, can you ping the world from adsl wan interface?
                    Can you ping your adsl wan side from remote?

                    E 1 Reply Last reply Reply Quote 0
                    • E
                      eableson @netblues
                      last edited by

                      @netblues Ah, now things are getting interesting. Current state :

                      • Disconnected the 4G connection (but still in the configuration)
                      • Turned the OpenVPN connections back on to be sure I'm fixing the problem in a state that won't just get broken again once I turn it back on

                      So at the moment from the pfSense box:

                      • ping 8.8.8.8 from WANADSL IP: OK
                      • ping 8.8.8.8 from LAN IP: No route to host

                      But from a computer on the LAN, ping to 8.8.8.8 works fine... because I had set the priority routing up so that it would take the default route only when trying to go to other internal networks. When I look at the trace route, the first hop is the ADSL WAN Gateway, not the internal LAN IP address.

                      So the current routing table looks like this:

                      Destination        Gateway            Flags     Netif Expire
78.226.49.0/24     link#2             U           re1
78.226.49.193      link#2             UHS         lo0
127.0.0.1          link#4             UH          lo0
172.16.11.1        link#8             UH       ovpnc2
172.16.11.2        link#8             UHS         lo0
172.16.12.1        link#9             UH       ovpnc3
172.16.12.2        link#9             UHS         lo0
192.168.2.0/24     172.16.12.1        UGS      ovpnc3
192.168.5.0/24     172.16.11.1        UGS      ovpnc2
192.168.6.0/24     172.16.11.1        UGS      ovpnc2
192.168.10.0/24    link#7             U           ue0
192.168.10.1       link#7             UHS         lo0

                      And attached are the current rules NAT & FW:

                      0_1545226719685_pfctl-rules.txt

                      So the issues seems to be that in a multi-wan setup that has VPN connections, the default routing table will only include known internal networks and if you need to go to the internet, it has to be via a gateway group. So the question I'm asking myself is if I should be declaring the OpenVPN connections as physical interfaces so that their routing rules get handled like regular interfaces.

                      E 1 Reply Last reply Reply Quote 0
                      • E
                        eableson @eableson
                        last edited by

                        @eableson @netblues Found it! After staring at the combination of the routing table and the "no route to host" message, I realised that there was no default route listed in the routing table.

                        Digging around, this appears to be a side of effect of using gateway groups and setting the Default gateway IPv4 to a gateway group instead of setting it to a physical interface. Setting the Default gateway to the ADSL WAN connection add it as the default route and things start working again...

                        N 1 Reply Last reply Reply Quote 0
                        • N
                          netblues @eableson
                          last edited by

                          @eableson So, it has nothing to do with natting and port forward, as expected :)

                          E 1 Reply Last reply Reply Quote 0
                          • E
                            eableson @netblues
                            last edited by

                            @netblues As usual, the source of the problem was elsewhere, but brought to light by adding NAT and Port Forwarding to the mix. This one could stand to have a tech note - but before I ask I'm going to go back and reread the documentation about setting up multi-wan

                            N 1 Reply Last reply Reply Quote 0
                            • N
                              netblues @eableson
                              last edited by

                              @eableson Well, the norm is sending your default gw to the internet and not the vpn routes. (if being a server)

                              E 1 Reply Last reply Reply Quote 0
                              • E
                                eableson @netblues
                                last edited by

                                @netblues Well - it's not terribly clear about the correct use of the default gateway option when in multi wan mode. The trick is that in this configuration, most traffic should go out the 4G (high performance) connection, failing back to the ADSL when it drops (usually once every day or two). So in that case, the default configuration would be the default gateway as this gateway group, not a physical interface.

                                Then there are services that have to run on the ADSL line since it's a direct connection with a fixed IP. But they are very much the exception. But in all cases, they point to the pfSense as the default gateway from their perspective and it's responsible for knowing the best route to take.

                                1 Reply Last reply Reply Quote 0
                                • KOMK
                                  KOM
                                  last edited by

                                  If we all got $1 for every "Gah! pfSense is hacked/broken/whatever!" and it turned out to be a configuration issue, we would all be able to retire.

                                  1 Reply Last reply Reply Quote 1
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.