OpenVPN with pfSense on AWS VPC

[pgb]

@viragomann said in OpenVPN with pfSense on AWS VPC:

@pgb said in OpenVPN with pfSense on AWS VPC:

so the LAN interface in pfSense has a gateway configured at 192.168.16.1

What's the reason for that?
pfSense only need one default gateway. Since your WAN is connected to an upstream gateway, I guess, you should set a gateway on the WAN interface and set the gateway option to none on LAN.

No reason honestly. I just removed it, but the result is the same.

@pgb said in OpenVPN with pfSense on AWS VPC:

I ensured I have an outbound NAT rule on the LAN interface with 192.168.140.0/23 as source.

Did you also set the outbound NAT to hybrid mode?

Yes. I'm attaching a screenshot of the outbound rules.

@pgb said in OpenVPN with pfSense on AWS VPC:

I can use pfSense as a DNS

Do you really need this? Since pfSense is only used for OpenVPN, you may have another name server in your network.

I will need it later. I want certain domains to be forwarded to a DNS that is at a remote location, behind an IPSec site-to-site VPN. Using AWS solution for this is too expensive for our requirements, so I'd like to use pfSense instead.

In the context of this issue, though, I just mentioned it as a data point, meaning I can access pfSense itself, but not other servers on the same subnet.

[viragomann]

@pgb
What is the client?
Is it an access server or a site-to-site configuration?

Are the routes correct on the client?

[pgb]

@viragomann said in OpenVPN with pfSense on AWS VPC:

@pgb
What is the client?
Is it an access server or a site-to-site configuration?

Are the routes correct on the client?

It's an access server. I'm using TunnelBlick on a Mac, and the routes on seem correct to me:

192.168.16         192.168.140.1      UGSc         utun6
192.168.17         192.168.140.1      UGSc         utun6
192.168.140/23     192.168.140.2      UGSc         utun6
192.168.140.2      192.168.140.2      UH           utun6

Also, if I do a packet capture on the LAN interface while I try to SSH the remote host, I can see this:

18:23:31.904822 IP 192.168.17.10.16812 > 192.168.17.30.22: tcp 0

which as far as I understand, means pfSense is doing NAT as I expected and sending the SSH to my server (note that no other entries are captured in the log, that's where I'm getting lost).

[viragomann]

@pgb
An you don't see any response packet from the server in the capture?

It should be the same as ssh from the remote pfSense.

[pgb]

@viragomann said in OpenVPN with pfSense on AWS VPC:

@pgb
An you don't see any response packet from the server in the capture?

It should be the same as ssh from the remote pfSense.

No response.

Packet capture of remote SSH session (failing):

18:57:12.126970 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:13.131702 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:14.135982 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:15.145506 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:16.159995 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:17.152084 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:19.158465 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0

Packet capture of local (from pfSense) SSH session (working):

18:59:06.034748 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.034969 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.035013 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.035538 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 38
18:59:06.035705 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.060000 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 41
18:59:06.060098 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.060752 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 1432
18:59:06.060905 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.062725 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 1056
18:59:06.063141 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.065800 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 48
18:59:06.065956 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.070607 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 508
18:59:06.070772 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.074492 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 16
18:59:06.074616 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.074640 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 44
18:59:06.074802 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.074818 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 44
18:59:06.074868 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.074920 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 68
18:59:06.081411 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 44
18:59:06.081477 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.081628 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.082931 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.082959 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0

[viragomann]

@pgb
So there are no responses at all.
I just tested ssh over OpenVPN with NAT on the remote site as yours, works fine.

I'd suspect there would be something with AWS, but since the ssh packets have exactly the same source and destination addresses as when you connect from pfSense, that couldn't be the case.

Maybe something wrong with the OpenVPN settings? Have you special settings in the advanced options?

Something in the OpenVPN logs? Maybe increase the log level.

[pgb]

@viragomann nothing advanced. I just noticed I'd left IPv6 on even though I don't use it, so I disabled it.

Same thing... can connect, can SSH pfSense, but can't get to a server behind it.

OpenVPN logs in debug do not report anything strange to me.

[pgb]

Update with solution: Turns out, I had "hardware checksum offload" enabled, and it needs to be disabled when you use pfSense on AWS (it really should be a default setting on the official image).

Thank you @viragomann for the help.

[viragomann]

@pgb
Glad to hear that it works at last.

However, running pfSense on AWS is a special use case. On bare metal installations and as well some other virtualizations hardware checksum offloading gives a performance benefit.

[pgb]

@viragomann I agree, and we run pfSense with that turned on our on-premises hardware. However, when installing the official AMI in EC2 (and paying for it), I'd expect the defaults to be compatible with AWS' virtualized hardware edge-cases.