Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    OpenVPN with pfSense on AWS VPC

    Scheduled Pinned Locked Moved OpenVPN
    12 Posts 2 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pgb
      last edited by

      We are in the process of migrating our office servers to AWS to finally decommission. Since we are familiar with pfSense, we are using a mix of AWS services but want to use pfSense as a the OpenVPC server.

      Our VPC has two subnets:
      Public: 192.168.16.0/24
      Private: 192.168.17.0/24

      I created the an EC2 instance running in pfSense with two interfaces:
      WAN -> 192.168.16.10, LAN -> 192.168.17.10.

      Worth noting: we are not using pfSense as the NAT gateway for the private network, so the LAN interface in pfSense has a gateway configured at 192.168.16.1 (why so? because we only want to use pfSense as the VPN server, not as the NAT gateway at this point).

      Then, we configured OpenVPN to listen on the WAN interface and with a tunnel network of 192.168.140.0/23 and IPv4 local networks of 192.168.17.0/24.

      I ensured I have an outbound NAT rule on the LAN interface with 192.168.140.0/23 as source.

      On the EC2 instance, I stopped source destination check.

      What works:

      • I can connect to OpenVPN
      • I can use pfSense as a DNS
      • From a console in pfSense, I can ssh another server in my private network (192.168.17.30).
      • From said server (192.168.17.30), I can ssh to pfSense.

      What doesn't work:

      • I can't ssh to a server on the private network (192.168.17.30).
        • If I enable logs on my firewall rules, I can confirm that pfSense is letting the packets in.

      What I tried already:

      • Wide open firewall rules.
      • Wide open AWS security groups.
      • Adding 192.168.140.0/23 to the route tables in my VPC (although since it's NATing, I understand I should not need it).

      Any help would be appreciated. I can post any additional configuration / details needed.

      V 1 Reply Last reply Reply Quote 0
      • V
        viragomann @pgb
        last edited by

        @pgb said in OpenVPN with pfSense on AWS VPC:

        so the LAN interface in pfSense has a gateway configured at 192.168.16.1

        What's the reason for that?
        pfSense only need one default gateway. Since your WAN is connected to an upstream gateway, I guess, you should set a gateway on the WAN interface and set the gateway option to none on LAN.

        @pgb said in OpenVPN with pfSense on AWS VPC:

        I ensured I have an outbound NAT rule on the LAN interface with 192.168.140.0/23 as source.

        Did you also set the outbound NAT to hybrid mode?

        @pgb said in OpenVPN with pfSense on AWS VPC:

        I can use pfSense as a DNS

        Do you really need this? Since pfSense is only used for OpenVPN, you may have another name server in your network.

        P 1 Reply Last reply Reply Quote 0
        • P
          pgb @viragomann
          last edited by

          @viragomann said in OpenVPN with pfSense on AWS VPC:

          @pgb said in OpenVPN with pfSense on AWS VPC:

          so the LAN interface in pfSense has a gateway configured at 192.168.16.1

          What's the reason for that?
          pfSense only need one default gateway. Since your WAN is connected to an upstream gateway, I guess, you should set a gateway on the WAN interface and set the gateway option to none on LAN.

          No reason honestly. I just removed it, but the result is the same.

          @pgb said in OpenVPN with pfSense on AWS VPC:

          I ensured I have an outbound NAT rule on the LAN interface with 192.168.140.0/23 as source.

          Did you also set the outbound NAT to hybrid mode?

          Yes. I'm attaching a screenshot of the outbound rules.Screen Shot 2021-02-08 at 18.09.55.png

          @pgb said in OpenVPN with pfSense on AWS VPC:

          I can use pfSense as a DNS

          Do you really need this? Since pfSense is only used for OpenVPN, you may have another name server in your network.

          I will need it later. I want certain domains to be forwarded to a DNS that is at a remote location, behind an IPSec site-to-site VPN. Using AWS solution for this is too expensive for our requirements, so I'd like to use pfSense instead.

          In the context of this issue, though, I just mentioned it as a data point, meaning I can access pfSense itself, but not other servers on the same subnet.

          V 1 Reply Last reply Reply Quote 0
          • V
            viragomann @pgb
            last edited by

            @pgb
            What is the client?
            Is it an access server or a site-to-site configuration?

            Are the routes correct on the client?

            P 1 Reply Last reply Reply Quote 0
            • P
              pgb @viragomann
              last edited by

              @viragomann said in OpenVPN with pfSense on AWS VPC:

              @pgb
              What is the client?
              Is it an access server or a site-to-site configuration?

              Are the routes correct on the client?

              It's an access server. I'm using TunnelBlick on a Mac, and the routes on seem correct to me:

              192.168.16         192.168.140.1      UGSc         utun6
192.168.17         192.168.140.1      UGSc         utun6
192.168.140/23     192.168.140.2      UGSc         utun6
192.168.140.2      192.168.140.2      UH           utun6

              Also, if I do a packet capture on the LAN interface while I try to SSH the remote host, I can see this:

              18:23:31.904822 IP 192.168.17.10.16812 > 192.168.17.30.22: tcp 0

              which as far as I understand, means pfSense is doing NAT as I expected and sending the SSH to my server (note that no other entries are captured in the log, that's where I'm getting lost).

              V 1 Reply Last reply Reply Quote 0
              • V
                viragomann @pgb
                last edited by

                @pgb
                An you don't see any response packet from the server in the capture?

                It should be the same as ssh from the remote pfSense.

                P 1 Reply Last reply Reply Quote 0
                • P
                  pgb @viragomann
                  last edited by

                  @viragomann said in OpenVPN with pfSense on AWS VPC:

                  @pgb
                  An you don't see any response packet from the server in the capture?

                  It should be the same as ssh from the remote pfSense.

                  No response.

                  Packet capture of remote SSH session (failing):

                  18:57:12.126970 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:13.131702 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:14.135982 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:15.145506 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:16.159995 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:17.152084 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0
18:57:19.158465 IP 192.168.17.10.7955 > 192.168.17.30.22: tcp 0

                  Packet capture of local (from pfSense) SSH session (working):

                  18:59:06.034748 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.034969 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.035013 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.035538 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 38
18:59:06.035705 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.060000 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 41
18:59:06.060098 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.060752 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 1432
18:59:06.060905 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.062725 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 1056
18:59:06.063141 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.065800 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 48
18:59:06.065956 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.070607 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 508
18:59:06.070772 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.074492 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 16
18:59:06.074616 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.074640 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 44
18:59:06.074802 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.074818 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 44
18:59:06.074868 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.074920 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 68
18:59:06.081411 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 44
18:59:06.081477 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.081628 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
18:59:06.082931 IP 192.168.17.30.22 > 192.168.17.10.59085: tcp 0
18:59:06.082959 IP 192.168.17.10.59085 > 192.168.17.30.22: tcp 0
                  V 1 Reply Last reply Reply Quote 0
                  • V
                    viragomann @pgb
                    last edited by

                    @pgb
                    So there are no responses at all.
                    I just tested ssh over OpenVPN with NAT on the remote site as yours, works fine.

                    I'd suspect there would be something with AWS, but since the ssh packets have exactly the same source and destination addresses as when you connect from pfSense, that couldn't be the case.

                    Maybe something wrong with the OpenVPN settings? Have you special settings in the advanced options?

                    Something in the OpenVPN logs? Maybe increase the log level.

                    P 1 Reply Last reply Reply Quote 0
                    • P
                      pgb @viragomann
                      last edited by pgb

                      @viragomann nothing advanced. I just noticed I'd left IPv6 on even though I don't use it, so I disabled it.

                      Same thing... can connect, can SSH pfSense, but can't get to a server behind it.

                      OpenVPN logs in debug do not report anything strange to me.

                      1 Reply Last reply Reply Quote 0
                      • P
                        pgb
                        last edited by

                        Update with solution: Turns out, I had "hardware checksum offload" enabled, and it needs to be disabled when you use pfSense on AWS (it really should be a default setting on the official image).

                        Thank you @viragomann for the help.

                        V 1 Reply Last reply Reply Quote 0
                        • V
                          viragomann @pgb
                          last edited by

                          @pgb
                          Glad to hear that it works at last.

                          However, running pfSense on AWS is a special use case. On bare metal installations and as well some other virtualizations hardware checksum offloading gives a performance benefit.

                          P 1 Reply Last reply Reply Quote 0
                          • P
                            pgb @viragomann
                            last edited by

                            @viragomann I agree, and we run pfSense with that turned on our on-premises hardware. However, when installing the official AMI in EC2 (and paying for it), I'd expect the defaults to be compatible with AWS' virtualized hardware edge-cases.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.