Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    offloading OpenVPN using external gateway

    Routing and Multi WAN
    2
    70
    131
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      chrispazz @bingo600 last edited by

      @bingo600 Currently I am using a "per IP" policy so only specific clients and specific traffic is going thru VPN.
      I am using only firewall rules by indicating specific outgoing gateway (WAN or VPN).

      bingo600 1 Reply Last reply Reply Quote 0
      • bingo600
        bingo600 @chrispazz last edited by bingo600

        @chrispazz said in offloading OpenVPN using external gateway:

        @bingo600 Currently I am using a "per IP" policy so only specific clients and specific traffic is going thru VPN.
        I am using only firewall rules by indicating specific outgoing gateway (WAN or VPN).

        Then i suppose you have to make policy routing.

        Make an IP host alias Ie. OVPN_FORWARDS , and put your IP's in that one.

        Then make a policy route rule , AF IPv4 , Proto any , matching Source IP = OVPN_FORWARDS , dest any.

        Click advanced options
        6a23b88e-97f1-4fba-9c76-0d4315c68a47-image.png

        Set the Raspi Gateway as Gateway
        13adb891-1911-44ab-baac-403118ba3d55-image.png

        Pray that you have enabled forwarding in Raspbian Linux ๐Ÿ˜Š

        Remember that pfSense matches on inbound traffic , meaning the rule has to be applied on all interfaces where matching ip's could "enter" (Ingress traffic).

        /Bingo

        C 1 Reply Last reply Reply Quote 0
        • C
          chrispazz @bingo600 last edited by chrispazz

          @bingo600 here we are! this is exactly what I did before writing here! ahahahah
          Yes I enabled forwarding in Raspbian but I cannot understand why it is going outside without using VPN ๐Ÿ˜•

          I am sure it is using the new rule because of some tests I did....

          I am starting to suppose that the problem could be on the RPI side....strange since it is working from SSH....

          bingo600 1 Reply Last reply Reply Quote 0
          • bingo600
            bingo600 @chrispazz last edited by

            @chrispazz said in offloading OpenVPN using external gateway:

            Yes I enabled forwarding in Raspbian but I cannot understand why it is going outside without using VPN ๐Ÿ˜•

            Does the package reach the RasPI (using the gateway you set) ?
            Or does the pfSense never send it to the Raspi ?

            C 1 Reply Last reply Reply Quote 0
            • C
              chrispazz @bingo600 last edited by chrispazz

              @bingo600 I do not know how to check this on the RPI side.... ๐Ÿ˜

              I followed this guide:

              https://www.instructables.com/Raspberry-Pi-VPN-Gateway/

              bingo600 1 Reply Last reply Reply Quote 0
              • bingo600
                bingo600 @chrispazz last edited by

                @chrispazz said in offloading OpenVPN using external gateway:

                @bingo600 I do not know how to check this on the RPI side.... ๐Ÿ˜

                Quick hack
                Enable temporary logging on the Policy Rule , look for log entries to confirm your rule matches "interesting traffic".

                Real Packet capture (on pfSense).
                Diagnostic --> Packet Capture

                I suppose your pfSense Wan Interface is the one connecting to the "ISP Router LAN" ?

                Packet Capture on pfSense
                If you have a screen + kbd on the RasPi , you could install wireshark on that one , to do the packet capture.

                If only SSH , install tcpdump on the RasPI

                How about NAT ?
                Are you natting on your pfSense WAN IF ?

                /Bingo

                1 Reply Last reply Reply Quote 0
                • bingo600
                  bingo600 last edited by bingo600

                  You say your pfSense is 192.168.99.1 , but the drawing shows 99.250

                  Is the 99.1 interface a pfSense "inside Lan" ip ?

                  How does the 192.168.99.x interface relate on the drawing ?

                  I expect your pfSense to have a WAN ip on the ISP router LAN , meaning the 192.168.5.x range.

                  Since you say you can ping and prob SSH to the Raspi , i assume that is correct or ??

                  C 1 Reply Last reply Reply Quote 0
                  • C
                    chrispazz @bingo600 last edited by

                    @bingo600 NAS is 99.250.
                    Inside NAS (Synology) I run a VM with pfsense (99.1).

                    I tried enabling logging on the rule and I can confirm it is using it.
                    Yes, my pfsense WAN interface is the one connecting to the ISP router.
                    And yes, I am using NAT on the pfsense WAN if....

                    bingo600 2 Replies Last reply Reply Quote 0
                    • bingo600
                      bingo600 @chrispazz last edited by

                      @chrispazz

                      Just for completeness
                      What is your pfSense WAN IP ?

                      C 1 Reply Last reply Reply Quote 0
                      • C
                        chrispazz @bingo600 last edited by

                        @bingo600 it is 192.168.5.254

                        bingo600 1 Reply Last reply Reply Quote 0
                        • bingo600
                          bingo600 @chrispazz last edited by

                          @chrispazz

                          ??

                          Then what is your ISP router inside ip ?

                          C 1 Reply Last reply Reply Quote 0
                          • bingo600
                            bingo600 @chrispazz last edited by

                            @chrispazz said in offloading OpenVPN using external gateway:

                            I tried enabling logging on the rule and I can confirm it is using it.

                            Ok

                            Now try to do a packet trace on the pfSense WAN

                            Set host address to the Raspi IP

                            63beebf6-8231-43a9-965c-b3349295c4c7-image.png

                            And generate some traffic destined for VPN

                            C 1 Reply Last reply Reply Quote 0
                            • C
                              chrispazz @bingo600 last edited by

                              @bingo600 oops....sorry. Inside IP of router is 192.168.5.254.

                              Pf sense has 192.168.5.1 on the WAN side...

                              1 Reply Last reply Reply Quote 0
                              • C
                                chrispazz @bingo600 last edited by

                                @bingo600

                                Activated packet capture on host 192.168.5.9 and opened a webpage with www.google.it:

                                16:58:57.410983 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8815, length 9
                                16:58:57.920985 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8816, length 9

                                bingo600 1 Reply Last reply Reply Quote 0
                                • bingo600
                                  bingo600 @chrispazz last edited by

                                  @chrispazz said in offloading OpenVPN using external gateway:

                                  @bingo600

                                  Activated packet capture on host 192.168.5.9 and opened a webpage with www.google.it:

                                  16:58:57.410983 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8815, length 9
                                  16:58:57.920985 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8816, length 9

                                  Where did 192.168.5.9 come from ??
                                  did you mean 5.99

                                  C 1 Reply Last reply Reply Quote 0
                                  • C
                                    chrispazz @bingo600 last edited by chrispazz

                                    @bingo600 yes is 5.99 :) don't hate me eheheh

                                    bingo600 1 Reply Last reply Reply Quote 0
                                    • bingo600
                                      bingo600 @chrispazz last edited by

                                      @chrispazz

                                      So we have confirmed that the ICMP (ping) packages are actually being "policy routed" from 5.1 (pfSense) to 5.99 (RasPI)

                                      We don't see any Echo reply ... The issue is probably on the RasPI

                                      C 1 Reply Last reply Reply Quote 1
                                      • bingo600
                                        bingo600 last edited by

                                        On the Raspi (as root)

                                        What does a : sudo iptables -L show ?

                                        And this one ?
                                        92d9b6a4-89e6-402d-aeb5-c359a7092195-image.png

                                        C 1 Reply Last reply Reply Quote 0
                                        • C
                                          chrispazz @bingo600 last edited by

                                          This post is deleted!
                                          1 Reply Last reply Reply Quote 0
                                          • bingo600
                                            bingo600 last edited by

                                            Did you reboot the RasPI after enabling forwarding ?

                                            1 Reply Last reply Reply Quote 0
                                            • C
                                              chrispazz @bingo600 last edited by chrispazz

                                              @bingo600 yes, forward is activated.

                                              Chain INPUT (policy DROP)
                                              target prot opt source destination
                                              ACCEPT all -- anywhere anywhere
                                              ACCEPT icmp -- anywhere anywhere
                                              ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
                                              ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED

                                              Chain FORWARD (policy DROP)
                                              target prot opt source destination
                                              ACCEPT all -- anywhere anywhere
                                              ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED

                                              Chain OUTPUT (policy ACCEPT)
                                              target prot opt source destination

                                              and yes, I rebooted it a lot of times :)

                                              bingo600 1 Reply Last reply Reply Quote 0
                                              • bingo600
                                                bingo600 @chrispazz last edited by

                                                @chrispazz

                                                Does this one give the expected result , should be your (OpenVPN public ip) ?

                                                wget http://ipinfo.io/ip -qO -

                                                Can you ping the google.it from the RasPI ?

                                                /Bingo

                                                C 1 Reply Last reply Reply Quote 0
                                                • C
                                                  chrispazz @bingo600 last edited by

                                                  @bingo600 said in offloading OpenVPN using external gateway:

                                                  wget http://ipinfo.io/ip -qO -

                                                  Both yes.
                                                  And if I test a traceroute from raspi I can confirm it is going outside using VPN connection....

                                                  bingo600 1 Reply Last reply Reply Quote 0
                                                  • bingo600
                                                    bingo600 @chrispazz last edited by bingo600

                                                    @chrispazz

                                                    Now : I would connect a PC directly to the RasPI , or directly to the ISP inside LAN , and set def-gw to the RasPI , and maybe DNS to 8.8.8.8

                                                    Then start debugging directly between the PC & The Raspi.

                                                    This can come in handy ... Disables all iptables stuff on the Raspi.
                                                    But only for this "boot sesion"

                                                    iptables -F
                                                    iptables -X
                                                    iptables -t nat -F
                                                    iptables -t nat -X
                                                    iptables -t mangle -F
                                                    iptables -t mangle -X
                                                    iptables -P INPUT ACCEPT
                                                    iptables -P FORWARD ACCEPT
                                                    iptables -P OUTPUT ACCEPT
                                                    
                                                    

                                                    I use these rules on my Deb10 machines , for OVPN to a VPN provider.

                                                    Remember to replace ens192 with the name of your ethernet interface on the Raspi.

                                                    # Default policy to drop all incoming packets
                                                    iptables -P INPUT DROP
                                                    iptables -P FORWARD DROP
                                                    
                                                    # Accept incoming packets from localhost and the LAN interface
                                                    iptables -A INPUT -i lo -j ACCEPT
                                                    iptables -A INPUT -i ens192 -j ACCEPT
                                                    
                                                    # Accept incoming packets from the WAN if the router initiated
                                                    # the connection
                                                    iptables -A INPUT -i tun0 -m conntrack \
                                                        --ctstate ESTABLISHED,RELATED -j ACCEPT
                                                    
                                                    # Forward LAN packets to the WAN
                                                    iptables -A FORWARD -i ens192 -o tun0 -j ACCEPT
                                                    
                                                    # Forward WAN packets to the LAN if the LAN initiated the
                                                    # connection
                                                    iptables -A FORWARD -i tun0 -o ens192 -m conntrack \
                                                        --ctstate ESTABLISHED,RELATED -j ACCEPT
                                                    
                                                    # NAT traffic going out the WAN interface
                                                    iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
                                                    #
                                                    
                                                    
                                                    

                                                    Remember to save new rules if you change them

                                                    sudo systemctl enable netfilter-persistent
                                                    
                                                    C 1 Reply Last reply Reply Quote 0
                                                    • C
                                                      chrispazz @bingo600 last edited by chrispazz

                                                      @bingo600 said in offloading OpenVPN using external gateway:

                                                      sudo systemctl enable netfilter-persistent

                                                      I made some other tests.

                                                      Raspberry is working good if I set a client to use it as network interface gateway and connection will go thru VPN (using my own iptables rules or yours).

                                                      So it seems that when I go thru pfsense, WAN main gateway (ISP router) is getting used instead of firewall rule gateway...

                                                      bingo600 1 Reply Last reply Reply Quote 0
                                                      • bingo600
                                                        bingo600 @chrispazz last edited by

                                                        @chrispazz said in offloading OpenVPN using external gateway:

                                                        @bingo600 said in offloading OpenVPN using external gateway:

                                                        sudo systemctl enable netfilter-persistent

                                                        I made some other tests.

                                                        Raspberry is working good if I set a client to use it as network interface gateway and connection will go thru VPN (using my own iptables rules or yours).

                                                        So it seems that when I go thru pfsense, WAN main gateway (ISP router) is getting used instead of firewall rule gateway...

                                                        That doesn't make sense
                                                        The packet trace you made doesn't lie.

                                                        1 Reply Last reply Reply Quote 0
                                                        • bingo600
                                                          bingo600 last edited by

                                                          Now is the time to use tcpdump on the RasPI

                                                          https://kalitut.com/record-network-traffic-with-raspberry/

                                                          To see what it actually receives & forwards

                                                          C 1 Reply Last reply Reply Quote 0
                                                          • C
                                                            chrispazz @bingo600 last edited by

                                                            @bingo600
                                                            This is traceroute from the client with the firewall rule applies:

                                                            traceroute to www.google.it (216.58.215.227), 64 hops max, 52 byte packets
                                                            1 192.168.5.254 (192.168.5.254) 2.370 ms 1.583 ms 1.516 ms

                                                            The first hop go to ISP gateway....I can't see Raspi here...

                                                            bingo600 1 Reply Last reply Reply Quote 0
                                                            • bingo600
                                                              bingo600 @chrispazz last edited by bingo600

                                                              @chrispazz
                                                              But that is not consistent with the packet trace you made on the pfSense

                                                              Can you do another trace , and then the traceroute ?

                                                              C 1 Reply Last reply Reply Quote 0
                                                              • C
                                                                chrispazz @bingo600 last edited by

                                                                @bingo600 if I try a trace to www.google.it IP address as host I get this:

                                                                18:25:31.846958 IP 192.168.5.1.4704 > 172.217.168.35.443: UDP, length 1350
                                                                18:25:31.883433 IP 172.217.168.35.443 > 192.168.5.1.4704: UDP, length 1350

                                                                so it is using 192.168.5.1 as gateway....

                                                                C bingo600 2 Replies Last reply Reply Quote 0
                                                                • C
                                                                  chrispazz @chrispazz last edited by chrispazz

                                                                  @chrispazz Perhaps I understand the problem.

                                                                  My WAN interface is getting IP from ISP router (DHCP server on 192.168.5.0).
                                                                  The ISP gateway is getting on top of the firewall rule gateway....

                                                                  This is WAN if on pfsense:

                                                                  em1.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
                                                                  options=3<RXCSUM,TXCSUM>
                                                                  ether 02:11:32:25:4e:f9
                                                                  inet6 fe80::11:32ff:fe25:4ef9%em1.90 prefixlen 64 scopeid 0x8
                                                                  inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
                                                                  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                                                  media: Ethernet autoselect (1000baseT <full-duplex>)
                                                                  status: active
                                                                  vlan: 90 vlanpcp: 0 parent interface: em1
                                                                  groups: vlan

                                                                  C 1 Reply Last reply Reply Quote 0
                                                                  • C
                                                                    chrispazz @chrispazz last edited by

                                                                    No wrong.

                                                                    192.168.5.1 is pfsense.... :(

                                                                    1 Reply Last reply Reply Quote 0
                                                                    • bingo600
                                                                      bingo600 @chrispazz last edited by

                                                                      @chrispazz said in offloading OpenVPN using external gateway:

                                                                      @bingo600 if I try a trace to www.google.it IP address as host I get this:

                                                                      18:25:31.846958 IP 192.168.5.1.4704 > 172.217.168.35.443: UDP, length 1350
                                                                      18:25:31.883433 IP 172.217.168.35.443 > 192.168.5.1.4704: UDP, length 1350

                                                                      so it is using 192.168.5.1 as gateway....

                                                                      Nope
                                                                      It's using 5.1 as source IP (Consistent with your WAN NAT)

                                                                      If you enable logging on the Policy rule , do you see any log lines ?

                                                                      Now you are not doing the traceroute on the pfSense box ... Are you ?

                                                                      What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

                                                                      C 1 Reply Last reply Reply Quote 0
                                                                      • C
                                                                        chrispazz @bingo600 last edited by chrispazz

                                                                        @bingo600 said in offloading OpenVPN using external gateway:

                                                                        If you enable logging on the Policy rule , do you see any log lines ?

                                                                        Yes, it is the only rules that has logging

                                                                        Now you are not doing the traceroute on the pfSense box ... Are you ?

                                                                        No, I am doing it from iMac client (192.168.20.110)

                                                                        What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

                                                                        I am using a firewall rule like this on the same interface of the client (192.168.20):

                                                                        2.jpg

                                                                        and NVPNGTW is the gateway I added to pfsense in the gateway section.

                                                                        bingo600 1 Reply Last reply Reply Quote 0
                                                                        • bingo600
                                                                          bingo600 @chrispazz last edited by

                                                                          @chrispazz said in offloading OpenVPN using external gateway:

                                                                          @bingo600 said in offloading OpenVPN using external gateway:

                                                                          If you enable logging on the Policy rule , do you see any log lines ?

                                                                          Yes, it is the only rules that has logging

                                                                          Now you are not doing the traceroute on the pfSense box ... Are you ?

                                                                          No, I am doing it from iMac client (192.168.20.110)

                                                                          What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

                                                                          I am using a firewall rule like this on the same interface of the client (192.168.20):

                                                                          2.jpg

                                                                          and NVPNGTW is the gateway I added to pfsense in the gateway section.

                                                                          That's a (for me strange rule) why does it have WAN all the way to the right ?
                                                                          Are you using floating rules ?

                                                                          C 1 Reply Last reply Reply Quote 0
                                                                          • C
                                                                            chrispazz @bingo600 last edited by

                                                                            @bingo600 no no, sorry, it is only the description. Ignore it.

                                                                            bingo600 1 Reply Last reply Reply Quote 0
                                                                            • bingo600
                                                                              bingo600 @chrispazz last edited by

                                                                              @chrispazz

                                                                              Is that rule all the way at the top ?

                                                                              C 1 Reply Last reply Reply Quote 0
                                                                              • C
                                                                                chrispazz @bingo600 last edited by

                                                                                @bingo600 yes, sure

                                                                                bingo600 2 Replies Last reply Reply Quote 0
                                                                                • bingo600
                                                                                  bingo600 @chrispazz last edited by

                                                                                  @chrispazz
                                                                                  I have to pull in the tuff guyzz now.

                                                                                  @stephenw10
                                                                                  Any ideas here ??

                                                                                  1 Reply Last reply Reply Quote 1
                                                                                  • bingo600
                                                                                    bingo600 @chrispazz last edited by bingo600

                                                                                    @chrispazz

                                                                                    But just to summarize :

                                                                                    You say if you put a PC on the 192.168.5.x lan , and set 192.168.5.99 (RasPI) as def-gw , the PC can use the RasPI & OpenVPN tunnel ok ?

                                                                                    You have put all desired OPENVPN PC's in a IP HOST Alias
                                                                                    You made direct host entries in the rule, and made Policy based rules with the RasPI as Gateway , on all the interfaces where a "OPENVPN" PC could be connected ?

                                                                                    But the pfSense won't match the source ip's, and therefore won't policy route t the RasPI. This is also confirmed by , enabling logging in the Policy rule. And no loglines appear.

                                                                                    Q:
                                                                                    We did a trace (the first one above) , where we succesfully matched a ping to google.it , and you got both loglines , and we saw the policy routing in effect in the pfSense packet trace.

                                                                                    What has changed in the pfSense config since that ?

                                                                                    Was that test also made with the iMAC ?

                                                                                    Q:
                                                                                    You do use either static IP's or DHCP MAC Lock (Reservation) , in order to ensure that the PC's used for test & OPN always get the same ip ?.

                                                                                    C 1 Reply Last reply Reply Quote 0
                                                                                    • First post
                                                                                      Last post

                                                                                    Products

                                                                                    • Platform Overview
                                                                                    • TNSR
                                                                                    • pfSense Plus
                                                                                    • Appliances

                                                                                    Services

                                                                                    • Training
                                                                                    • Professional Services

                                                                                    Support

                                                                                    • Subscription Plans
                                                                                    • Contact Support
                                                                                    • Product Lifecycle
                                                                                    • Documentation

                                                                                    News

                                                                                    • Media Coverage
                                                                                    • Press
                                                                                    • Events

                                                                                    Resources

                                                                                    • Blog
                                                                                    • FAQ
                                                                                    • Find a Partner
                                                                                    • Resource Library
                                                                                    • Security Information

                                                                                    Company

                                                                                    • About Us
                                                                                    • Careers
                                                                                    • Partners
                                                                                    • Contact Us
                                                                                    • Legal
                                                                                    Our Mission

                                                                                    We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                                                                                    Subscribe to our Newsletter

                                                                                    Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                                                                                    © 2021 Rubicon Communications, LLC | Privacy Policy