TLS handshake failed error only on a specific network

Teddy 0

Hi guys, I cannot connect to my OpenVPN server over UDP on a specific network. (Also I'm pretty new to networking in general, so please bear with me).

Here's the scenario: I have an Actiontech T3200M from my ISP (Telus) that has Port one bridged to my pfsense VM. Ports 2-5, Wifi on the ISP modem are still used by other devices and are not behind pfsense. (I'm essentially trying to have 2 separate networks: one from the ISP modem, one from pfsense).

When I try to connect to my OpenVPN server over UDP using a device on the ISP modem's network, I get a TLS handshake failed error. However, I'm able to connect via TCP without issue.

OpenVPN logs show the port number changing midstream. I'm almost positive this isn't a firewall issue because I'm able to connect on any other external network (I've also completely disabled the firewall, but ran into the same issue), but I honestly don't have any idea what's going on apart from maybe some NAT issue.

So just to be clear, I am only getting the TLS handshake failed error when trying to connect to the OpenVPN server via UDP on the ISP modem's network. I'm able to connect to my OpenVPN server via UDP normally on any other outside network (cellular, coffee shop, etc).

Screenshots:

OpenVPN logs, with port number changing midstream.
- (In the screenshot, I've blanked out the public IP of the ISP modem with 2 different colors: Orange = TCP, Blue = UDP)
Here's the State table entry when I try to connect to my OpenVPN server
- (Blue = pfsense public IP, Green = ISP modem public IP)
My WAN Firewall rules
I don't have any NAT Port Forwarding, 1:1, Outbound, or NPt rules apart from those created automatically by pfsense/ OpenVPN

Any help is very much appreciated thanks!

johnpoz

@teddy-0 you have nat reflection issue.. When you hit your public IP, which is on your "isp device" its reflecting it back into pfsense, pfsense just answers directly.

If you want to connect to pfsense while on pfsense wan network, just use its IP whatever rfc1918 address that is, vs trying to reflect off your isp device which has your public IP on its wan.

Teddy 0

@johnpoz Thanks for the reply!

I think I understand what you're saying with the nat reflection, but why is this the case if both pfsense, and the ISP modem have different public IPs?

Also just to clarify:

if you want to connect to pfsense while on pfsense wan network

Sorry if this might be trivial, but just to clarify, do you mean if I'm trying to connect to pfsense from the devices connected directly to the ISP modem (devices on ports 2-5, and wifi)?

just use its IP whatever rfc1918 address that is

Aren't RFC1918 addresses just private addresses (10.x.x.x, 172.x.x.x, ...)? If the WAN interface has a public IP, how would you find the rfc1918 address? (Again sorry if this is trivial)