Broken port forwarding

eableson · Dec 19, 2018, 9:39 AM

Another story of Port forwarding not working. Curiously, this was working a while ago, but stopped working, possibly when I got my Site-to-Site OpenVPN connections working.

Configuration is a physical pfSense box with two WAN connections:

4G_WAN which ends up on a double-NAT out to the internet
ADSL_WAN which is bridged to the internet with a fixed IP address

The default gateway is a gateway group with the 4G as the Tier 1, ADSL as Tier 2. Yeah, a little strange, but I get better 4G performance than ADSL here and it's relatively cheap.

Since this stopped working, I've torn down everything and am starting fresh. So at the moment, I have one Port Forward rule for port 25:

🔒 Log in to view

With the corresponding firewall rule:

🔒 Log in to view

When I watch the logs on the mail server, there's no activity coming from the outside at all. On the firewall, pfSense is passing in the connections:

🔒 Log in to view

And on the server, netstat shows the following:

tcp        0      0 192.168.10.209:smtp     mailhost11.lists.:56660 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mxslcpool33.ebay.:40381 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     st43p00im-zteg100:40254 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-ve1eur03lp205:6464 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-ve1eur02lp20:18330 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-eopbgr150105:27579 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mxslcpool33.ebay.:38653 SYN_RECV   
tcp        0      0 192.168.10.209:smtp     mail-eopbgr50102.:19091 SYN_RECV

And the corresponding states viewed by pfSense:
🔒 Log in to view

Now in order to get the OpenVPN routing working properly, on the LAN interface, I have an override for all traffic to the private subnets on the other OpenVPN sites set to the default gateway, while general internet traffic goes through the gateway group with the 4G Tier 1.

🔒 Log in to view

A packet capture on port 25 on the ADSLWAN interface shows incoming connections but I don't see any corresponding responses:

10:22:20.584803 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 50, id 15133, offset 0, flags [DF], proto TCP (6), length 60)
    198.2.136.56.36511 > 78.226.49.193.25: Flags [S], cksum 0x3711 (correct), seq 3429183177, win 14600, options [mss 1460,sackOK,TS val 783896680 ecr 0,nop,wscale 7], length 0
10:22:31.271663 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41628, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9df8 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688106083 ecr 0,nop,wscale 7], length 0
10:22:32.271203 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41629, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9a10 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688107083 ecr 0,nop,wscale 7], length 0
10:22:32.721800 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 48, id 32553, offset 0, flags [DF], proto TCP (6), length 60)
    136.147.176.90.44592 > 78.226.49.193.25: Flags [S], cksum 0x1ee9 (correct), seq 2286167703, win 14600, options [mss 1460,sackOK,TS val 109957835 ecr 0,nop,wscale 7], length 0
10:22:34.271054 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41630, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x9240 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688109083 ecr 0,nop,wscale 7], length 0
10:22:36.584715 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 50, id 15134, offset 0, flags [DF], proto TCP (6), length 60)
    198.2.136.56.36511 > 78.226.49.193.25: Flags [S], cksum 0xf890 (correct), seq 3429183177, win 14600, options [mss 1460,sackOK,TS val 783912680 ecr 0,nop,wscale 7], length 0
10:22:38.271192 f4:ca:e5:41:e2:e2 > 00:e0:4c:68:11:0e, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 41631, offset 0, flags [DF], proto TCP (6), length 60)
    83.68.134.65.10216 > 78.226.49.193.25: Flags [S], cksum 0x82a0 (correct), seq 309332223, win 14600, options [mss 1460,sackOK,TS val 688113083 ecr 0,nop,wscale 7], length 0

On the LAN interface, I see the incoming requests and the outbound responses:

09:40:48.371146 a0:ce:c8:04:16:e8 > 00:50:56:93:37:64, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 49, id 30184, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->25aa)!)
    204.92.22.84.18882 > 192.168.10.209.25: Flags [S], cksum 0xd473 (correct), seq 1971230723, win 29200, options [mss 1460,sackOK,TS val 3545508000 ecr 0,nop,wscale 7], length 0
09:40:48.371574 00:50:56:93:37:64 > a0:ce:c8:04:16:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.209.25 > 204.92.22.84.18882: Flags [S.], cksum 0x113d (correct), seq 2509475563, ack 1971230724, win 28960, options [mss 1460,sackOK,TS val 13751701 ecr 3545493968,nop,wscale 7], length 0

However, checking the 4G interface there is no traffic on port 25, so I don't know where those responses are going to.

Side notes:

I have full functioning access to the server from the internal network (SMTP, IMAP, etc), so it's not the machine that's not responding.
At the moment I have turned off (disabled) the OpenVPN connections to see if that has any impact on the behaviour and those networks are no longer noted in the routing table.
Outbound NAT is set to the default of Automatic rule generation
System Advanced > Firewall > Disable Reply-to is not activated

netblues · Dec 19, 2018, 11:28 AM

Add a rule on your lan and send the mail server ip straight to adsl, same way as your port forward is coming into and retest.
Seems like the syn replies try to return via 4g...

eableson · Dec 19, 2018, 11:34 AM

@netblues That's what I think is happening as well, and I've added the following rule but I have the same behaviour:

🔒 Log in to view

netblues · Dec 19, 2018, 11:38 AM

@eableson Have you cleared states? Can you connect from mail server to the internet?
Try changing the failover priority and use adsl first and see if it works.

(No firewalls on mail server too I guess..)

eableson · Dec 19, 2018, 11:44 AM

@netblues Yup - The mail server has no problems going out and getting stuff and I can send email to it from the internal network which get properly forwarded along to the defined smarthost. So basically anything initiated by the mail server works fine, but the SYN ACK responses get to the pfSense box and it's trying to go out through a route that doesn't work for it.

🔒 Log in to view

Here I can see after each initiated connection that the gateway tried to return the ACK, but can't find a route to the originator.

netblues · Dec 19, 2018, 11:52 AM

@eableson Need to check nat outbound.
Switch to manual and create rules for your two connections.

eableson · Dec 19, 2018, 12:03 PM

@netblues Like this?

🔒 Log in to view

Still no go...

Even with the two rules:

🔒 Log in to view

netblues · Dec 19, 2018, 1:12 PM

I don't get it. Why 104.47.9.51 is unreachable from pf?
From the command line of pf, can you ping the world from adsl wan interface?
Can you ping your adsl wan side from remote?

eableson · Dec 19, 2018, 1:42 PM

@netblues Ah, now things are getting interesting. Current state :

Disconnected the 4G connection (but still in the configuration)
Turned the OpenVPN connections back on to be sure I'm fixing the problem in a state that won't just get broken again once I turn it back on

So at the moment from the pfSense box:

ping 8.8.8.8 from WANADSL IP: OK
ping 8.8.8.8 from LAN IP: No route to host

But from a computer on the LAN, ping to 8.8.8.8 works fine... because I had set the priority routing up so that it would take the default route only when trying to go to other internal networks. When I look at the trace route, the first hop is the ADSL WAN Gateway, not the internal LAN IP address.

So the current routing table looks like this:

Destination        Gateway            Flags     Netif Expire
78.226.49.0/24     link#2             U           re1
78.226.49.193      link#2             UHS         lo0
127.0.0.1          link#4             UH          lo0
172.16.11.1        link#8             UH       ovpnc2
172.16.11.2        link#8             UHS         lo0
172.16.12.1        link#9             UH       ovpnc3
172.16.12.2        link#9             UHS         lo0
192.168.2.0/24     172.16.12.1        UGS      ovpnc3
192.168.5.0/24     172.16.11.1        UGS      ovpnc2
192.168.6.0/24     172.16.11.1        UGS      ovpnc2
192.168.10.0/24    link#7             U           ue0
192.168.10.1       link#7             UHS         lo0

And attached are the current rules NAT & FW:

0_1545226719685_pfctl-rules.txt

So the issues seems to be that in a multi-wan setup that has VPN connections, the default routing table will only include known internal networks and if you need to go to the internet, it has to be via a gateway group. So the question I'm asking myself is if I should be declaring the OpenVPN connections as physical interfaces so that their routing rules get handled like regular interfaces.

eableson · Dec 19, 2018, 2:04 PM

@eableson @netblues Found it! After staring at the combination of the routing table and the "no route to host" message, I realised that there was no default route listed in the routing table.

Digging around, this appears to be a side of effect of using gateway groups and setting the Default gateway IPv4 to a gateway group instead of setting it to a physical interface. Setting the Default gateway to the ADSL WAN connection add it as the default route and things start working again...

netblues · Dec 19, 2018, 2:08 PM

@eableson So, it has nothing to do with natting and port forward, as expected :)

eableson · Dec 19, 2018, 2:15 PM

@netblues As usual, the source of the problem was elsewhere, but brought to light by adding NAT and Port Forwarding to the mix. This one could stand to have a tech note - but before I ask I'm going to go back and reread the documentation about setting up multi-wan

netblues · Dec 19, 2018, 3:19 PM

@eableson Well, the norm is sending your default gw to the internet and not the vpn routes. (if being a server)

eableson · Dec 19, 2018, 3:55 PM

@netblues Well - it's not terribly clear about the correct use of the default gateway option when in multi wan mode. The trick is that in this configuration, most traffic should go out the 4G (high performance) connection, failing back to the ADSL when it drops (usually once every day or two). So in that case, the default configuration would be the default gateway as this gateway group, not a physical interface.

Then there are services that have to run on the ADSL line since it's a direct connection with a fixed IP. But they are very much the exception. But in all cases, they point to the pfSense as the default gateway from their perspective and it's responsible for knowing the best route to take.

KOM · Dec 19, 2018, 4:24 PM

If we all got $1 for every "Gah! pfSense is hacked/broken/whatever!" and it turned out to be a configuration issue, we would all be able to retire.