pfSense passing ICMP, not TCP

[bmcgover]

Bit of a newbie here, and I likely am hitting something obvious to the well initiated.

pfSense 2.6.0, community addition. I've set up an IPSEC tunnel with AWS using the instructions they create with with their VPN gateways, basically a fairly straight forward P1 and P2 setup. For reference, the link looks like:

10.0.2.16 <--Unknown --> VPN Gateway <--- 10.0.0.0/16 <--- Tunnel ---> 172.31.0.0/16 --> 172.31.6.41 <---> 172.31.6.42

10.0.2.16 and 172.31.6.42 are VMs. 172.31.6.41 is the pfSense LAN IP. I didn't add the WAN interface as it doesn't seem relevant. The CIDRs on ether side of the tunnel are what I plugged on to the P2 configurations, both as "Network" types.

When trying to run run traffic, ICMP echos from 10.0.2.16 to/from 172.31.6.42 is working fine. Both ends can initiate and be what is being pinged, no problems.

When trying to send TCP from 10.0.2.16 to 172.31.6.42, I can see the request on enc0, but it never comes out the LAN port (and obviously, the target host never sees it). Unfortunately, I can't test in the other direction.

It smells like there is a firewall in place coming off the tunnel, but looking at the rules in the UI, it looks like everything is allowed by default.

Any thoughts on what may be causing the "kinda working" tunnel?

[viragomann]

@bmcgover said in pfSense passing ICMP, not TCP:

Bit of a newbie here

To networking in general?

172.31.6.41 is the pfSense LAN IP. I didn't add the WAN interface as it doesn't seem relevant.

I assume pfSense isn't the default gateway in 172.31.0.0/16. Is it connection beside of the gateway?
If so I'm wondering how you do the routing.

[bmcgover]

@viragomann
172.31.6.42 is a multi-homed host. The default router here points upstream, but it has a route for 10.0.0.0/16 pointing at 172.31.6.41 (the pfSense instance).

From the pfSense appliance, the default gateway points at 172.31.6.42. It has two specific /32 routes for the WAN IPs also pointing upstream via the WAN interface. The two "upstream" paths eventually converge.

[viragomann]

@bmcgover said in pfSense passing ICMP, not TCP:

172.31.6.42 is a multi-homed host. The default router here points upstream, but it has a route for 10.0.0.0/16 pointing at 172.31.6.41 (the pfSense instance).

This is exactly what works for ICMP, but doesn't for TCP.
Other then ICMP is TCP a stateful protocol. But your set up runs into asymmetric routing.
That means request packets from your LAN device to the other site are sent to the router, then are forwarded to pfSense and passed over the VPN. Response packets are coming back and should go directly from pfSense to the LAN device, because of its destination IP.

If possible put pfSense out of the LAN, into a separate transit network. Then you can route properly.
If you cannot do this you can masquerade (S-NAT) the packets on your router. So the packets get the routers LAN IP as source and hence the return path is the same as the request.

[stephenw10]

Yup, this 'feels' like an asymmetric routing issue.

You should be seeing blocked traffic though if it is. It will have TCP flags set in the firewall log.

Steve

[bmcgover]

Ok, dug in a big more on the CLI to see whats up, and made a few configuration changes to get rid of any possible asymmetrical routing (at least locally; can't promise the Internet is clean).

So now
Internet <--> WAN -- pfSense -- Lan <---> 172.31.6.x local, 172.31.0.0/16 in a larger RFC1918 space.

Beyond setting up the tunnel, I added a "LANGW" gateway, as the pfSense device is 172.31.6.41, with a router on 172.31.6.42 leading to the rest of 172.31.0.0/16.

I added a route for 172.31.0.0/16

The tunnel P2 is set up with
Local network: Network 172.31.0.0/16
Remote Network 10.0.0.0/16

In /var/log/filter.log, I see repeats of (the other side of the link is trying to hit a local web server - this is hand copied, so apologies for any typos):

Jul 27 11:13:00 customer-vpn-server filterlog [11698]: 4,,,1000000103, enc0, match, block, in, 4,0x0,,254,12409,0,DF,6,tc,60,10.0.2.16,172.31.24.132,45550,9004,0,S,3229465956,,29200,,ms;sakOK;TS;nop;wscale

Thinking this might be from the "Block private networks and loopback addresses" or "Block bogon networks" options, I unchecked the LAN settings and double checked the WAN settings, and saved the changes. No changes.

Funking around, I got in to firewall rules->IPSEC, and not seeing any rules, added an any/any rule, the log messages stopped, and I now see packets heading out the LAN port.

Based on the earlier, similar configuration, I believe the change that mattered was adding the any/any rule to IPSEC. Re-adding the private network and bogon block doesn't seem to have an effect.

This would suggest that enc0 defaulting to blocking is a [recent?] change in behavior? In earlier versions (2.5.1 comes to mind, but it might be earlier), I didn't need this step in the past.

Thoughts?

[stephenw10]

Yeah, 1000000103 is the default block rule. And that is expected behaviour. There are no rules added to the IPSec interface by default, and never have been.

The reason ICMP appeared to work in this case is that the outgoing pings opened a state the incoming traffic was able to use. It's possible to get successful pings in that way even without any pass rules present on either end.
That can only happen if both ends are testing with Windows clients though because it uses icmp ID 1 for all pings allowing the match. Linux and FreeBSD do not.

Steve