Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    offloading OpenVPN using external gateway

    Routing and Multi WAN
    2
    70
    151
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bingo600
      bingo600 @chrispazz last edited by

      @chrispazz said in offloading OpenVPN using external gateway:

      @bingo600 said in offloading OpenVPN using external gateway:

      sudo systemctl enable netfilter-persistent

      I made some other tests.

      Raspberry is working good if I set a client to use it as network interface gateway and connection will go thru VPN (using my own iptables rules or yours).

      So it seems that when I go thru pfsense, WAN main gateway (ISP router) is getting used instead of firewall rule gateway...

      That doesn't make sense
      The packet trace you made doesn't lie.

      1 Reply Last reply Reply Quote 0
      • bingo600
        bingo600 last edited by

        Now is the time to use tcpdump on the RasPI

        https://kalitut.com/record-network-traffic-with-raspberry/

        To see what it actually receives & forwards

        C 1 Reply Last reply Reply Quote 0
        • C
          chrispazz @bingo600 last edited by

          @bingo600
          This is traceroute from the client with the firewall rule applies:

          traceroute to www.google.it (216.58.215.227), 64 hops max, 52 byte packets
          1 192.168.5.254 (192.168.5.254) 2.370 ms 1.583 ms 1.516 ms

          The first hop go to ISP gateway....I can't see Raspi here...

          bingo600 1 Reply Last reply Reply Quote 0
          • bingo600
            bingo600 @chrispazz last edited by bingo600

            @chrispazz
            But that is not consistent with the packet trace you made on the pfSense

            Can you do another trace , and then the traceroute ?

            C 1 Reply Last reply Reply Quote 0
            • C
              chrispazz @bingo600 last edited by

              @bingo600 if I try a trace to www.google.it IP address as host I get this:

              18:25:31.846958 IP 192.168.5.1.4704 > 172.217.168.35.443: UDP, length 1350
              18:25:31.883433 IP 172.217.168.35.443 > 192.168.5.1.4704: UDP, length 1350

              so it is using 192.168.5.1 as gateway....

              C bingo600 2 Replies Last reply Reply Quote 0
              • C
                chrispazz @chrispazz last edited by chrispazz

                @chrispazz Perhaps I understand the problem.

                My WAN interface is getting IP from ISP router (DHCP server on 192.168.5.0).
                The ISP gateway is getting on top of the firewall rule gateway....

                This is WAN if on pfsense:

                em1.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
                options=3<RXCSUM,TXCSUM>
                ether 02:11:32:25:4e:f9
                inet6 fe80::11:32ff:fe25:4ef9%em1.90 prefixlen 64 scopeid 0x8
                inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
                nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                media: Ethernet autoselect (1000baseT <full-duplex>)
                status: active
                vlan: 90 vlanpcp: 0 parent interface: em1
                groups: vlan

                C 1 Reply Last reply Reply Quote 0
                • C
                  chrispazz @chrispazz last edited by

                  No wrong.

                  192.168.5.1 is pfsense.... :(

                  1 Reply Last reply Reply Quote 0
                  • bingo600
                    bingo600 @chrispazz last edited by

                    @chrispazz said in offloading OpenVPN using external gateway:

                    @bingo600 if I try a trace to www.google.it IP address as host I get this:

                    18:25:31.846958 IP 192.168.5.1.4704 > 172.217.168.35.443: UDP, length 1350
                    18:25:31.883433 IP 172.217.168.35.443 > 192.168.5.1.4704: UDP, length 1350

                    so it is using 192.168.5.1 as gateway....

                    Nope
                    It's using 5.1 as source IP (Consistent with your WAN NAT)

                    If you enable logging on the Policy rule , do you see any log lines ?

                    Now you are not doing the traceroute on the pfSense box ... Are you ?

                    What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

                    C 1 Reply Last reply Reply Quote 0
                    • C
                      chrispazz @bingo600 last edited by chrispazz

                      @bingo600 said in offloading OpenVPN using external gateway:

                      If you enable logging on the Policy rule , do you see any log lines ?

                      Yes, it is the only rules that has logging

                      Now you are not doing the traceroute on the pfSense box ... Are you ?

                      No, I am doing it from iMac client (192.168.20.110)

                      What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

                      I am using a firewall rule like this on the same interface of the client (192.168.20):

                      2.jpg

                      and NVPNGTW is the gateway I added to pfsense in the gateway section.

                      bingo600 1 Reply Last reply Reply Quote 0
                      • bingo600
                        bingo600 @chrispazz last edited by

                        @chrispazz said in offloading OpenVPN using external gateway:

                        @bingo600 said in offloading OpenVPN using external gateway:

                        If you enable logging on the Policy rule , do you see any log lines ?

                        Yes, it is the only rules that has logging

                        Now you are not doing the traceroute on the pfSense box ... Are you ?

                        No, I am doing it from iMac client (192.168.20.110)

                        What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

                        I am using a firewall rule like this on the same interface of the client (192.168.20):

                        2.jpg

                        and NVPNGTW is the gateway I added to pfsense in the gateway section.

                        That's a (for me strange rule) why does it have WAN all the way to the right ?
                        Are you using floating rules ?

                        C 1 Reply Last reply Reply Quote 0
                        • C
                          chrispazz @bingo600 last edited by

                          @bingo600 no no, sorry, it is only the description. Ignore it.

                          bingo600 1 Reply Last reply Reply Quote 0
                          • bingo600
                            bingo600 @chrispazz last edited by

                            @chrispazz

                            Is that rule all the way at the top ?

                            C 1 Reply Last reply Reply Quote 0
                            • C
                              chrispazz @bingo600 last edited by

                              @bingo600 yes, sure

                              bingo600 2 Replies Last reply Reply Quote 0
                              • bingo600
                                bingo600 @chrispazz last edited by

                                @chrispazz
                                I have to pull in the tuff guyzz now.

                                @stephenw10
                                Any ideas here ??

                                1 Reply Last reply Reply Quote 1
                                • bingo600
                                  bingo600 @chrispazz last edited by bingo600

                                  @chrispazz

                                  But just to summarize :

                                  You say if you put a PC on the 192.168.5.x lan , and set 192.168.5.99 (RasPI) as def-gw , the PC can use the RasPI & OpenVPN tunnel ok ?

                                  You have put all desired OPENVPN PC's in a IP HOST Alias
                                  You made direct host entries in the rule, and made Policy based rules with the RasPI as Gateway , on all the interfaces where a "OPENVPN" PC could be connected ?

                                  But the pfSense won't match the source ip's, and therefore won't policy route t the RasPI. This is also confirmed by , enabling logging in the Policy rule. And no loglines appear.

                                  Q:
                                  We did a trace (the first one above) , where we succesfully matched a ping to google.it , and you got both loglines , and we saw the policy routing in effect in the pfSense packet trace.

                                  What has changed in the pfSense config since that ?

                                  Was that test also made with the iMAC ?

                                  Q:
                                  You do use either static IP's or DHCP MAC Lock (Reservation) , in order to ensure that the PC's used for test & OPN always get the same ip ?.

                                  C 1 Reply Last reply Reply Quote 0
                                  • C
                                    chrispazz @bingo600 last edited by chrispazz

                                    @bingo600 said in offloading OpenVPN using external gateway:

                                    @chrispazz

                                    But just to summarize :

                                    You say if you put a PC on the 192.168.5.x lan , and set 192.168.5.99 (RasPI) as def-gw , the PC can use the RasPI & OpenVPN tunnel ok ?

                                    Yes, correct.

                                    You have put all desired OPENVPN PC's in a IP HOST Alias
                                    You made direct host entries in the rule, and made Policy based rules with the RasPI as Gateway , on all the interfaces where a "OPENVPN" PC could be connected ?

                                    Yes.

                                    But the pfSense won't match the source ip's, and therefore won't policy route t the RasPI. This is also confirmed by , enabling logging in the Policy rule. And no loglines appear.

                                    No. The rule is correctly engaged and it is logged in firewall log but it seems to ignore the Raspi used as gateway.

                                    Q:
                                    We did a trace (the first one above) , where we succesfully matched a ping to google.it , and you got both loglines , and we saw the policy routing in effect in the pfSense packet trace.

                                    Currently, using 192.168.5.9 (Raspi gateway) as host, trace display the following when doing a ping to www.google.it:

                                    19:32:03.251588 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 26591, length 9
                                    19:32:03.252028 IP 192.168.5.9 > 192.168.5.1: ICMP echo reply, id 43267, seq 26591, length 9

                                    Doing packet trace using google IP as host gives the following:

                                    19:33:38.627397 IP 192.168.5.1 > 172.217.168.3: ICMP echo request, id 10411, seq 0, length 64
                                    19:33:38.642607 IP 172.217.168.3 > 192.168.5.1: ICMP echo reply, id 10411, seq 0, length 64

                                    Was that test also made with the iMAC ?

                                    The packet tracing is done pinging from Imac to www.google.it

                                    Q:
                                    You do use either static IP's or DHCP MAC Lock (Reservation) , in order to ensure that the PC's used for test & OPN always get the same ip ?.

                                    I have IP reservation for iMac and static IP for Pfsense.

                                    Just to give you more confirmation about the rule catched, consider that without that new rule, the connection has to go via the pfsense VPN client.
                                    If I deactivate the rule, the connection go with pfsense VPN client. if I enable the rule, the connection exit with ISP gateway (no raspi VPN).

                                    bingo600 1 Reply Last reply Reply Quote 0
                                    • bingo600
                                      bingo600 @chrispazz last edited by

                                      @chrispazz said in offloading OpenVPN using external gateway:
                                      packet trace.

                                      Currently, using 192.168.5.9 (Raspi gateway) as host, trace display the following when doing a ping to www.google.it:

                                      What does that mean ?
                                      Currently, using 192.168.5.9 (Raspi gateway) as host

                                      Wasnt the RasPI 192.168.5.99 ?

                                      What does as host mean (you mean you ping the raspi) ?

                                      The below 2 traces both show a req & a reply

                                      19:32:03.251588 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 26591, length 9
                                      19:32:03.252028 IP 192.168.5.9 > 192.168.5.1: ICMP echo reply, id 43267, seq 26591, length 9

                                      Doing packet trace using google IP as host gives the following:

                                      19:33:38.627397 IP 192.168.5.1 > 172.217.168.3: ICMP echo request, id 10411, seq 0, length 64
                                      19:33:38.642607 IP 172.217.168.3 > 192.168.5.1: ICMP echo reply, id 10411, seq 0, length 64

                                      In the Packet capture try to set detail to high or full

                                      2cedcbc3-da8f-40b9-92a1-e384cc29ec8f-image.png

                                      We need to see the mac addresses in those packets.

                                      And compare to the RasPI mac address

                                      C 1 Reply Last reply Reply Quote 0
                                      • C
                                        chrispazz @bingo600 last edited by

                                        @bingo600 said in offloading OpenVPN using external gateway:

                                        What does as host mean (you mean you ping the raspi) ?

                                        HOST is the captured Host address

                                        We need to see the mac addresses in those packets.

                                        b8:27:eb:8e:00:43 is Raspi MAC Address
                                        02:11:32:25:4e:f9 is pfsense WAN interface

                                        Full detailed trace is the following:

                                        192.168.5.1 > 192.168.5.99: ICMP echo request, id 43267, seq 28884, length 9
                                        19:51:42.213831 b8:27:eb:8e:00:43 > 02:11:32:25:4e:f9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 59852, offset 0, flags [none], proto ICMP (1), length 29)
                                        192.168.5.99 > 192.168.5.1: ICMP echo reply, id 43267, seq 28884, length 9
                                        19:51:42.723672 02:11:32:25:4e:f9 > b8:27:eb:8e:00:43, ethertype IPv4 (0x0800), length 43: (tos 0x0, ttl 64, id 65165, offset 0, flags [none], proto ICMP (1), length 29)

                                        bingo600 2 Replies Last reply Reply Quote 0
                                        • bingo600
                                          bingo600 @chrispazz last edited by bingo600

                                          @chrispazz

                                          Now do the same trace with the google ip as the destination

                                          Your iMAC isn't multihomed ?
                                          I mean both cable & WiFi

                                          C 1 Reply Last reply Reply Quote 0
                                          • C
                                            chrispazz @bingo600 last edited by chrispazz

                                            @bingo600 here it is:

                                            192.168.5.1 > 216.58.215.227: ICMP echo request, id 61288, seq 0, length 64
                                            20:09:20.262578 dc:39:6f:f2:4d:e4 > 02:11:32:25:4e:f9, ethertype IPv4 (0x0800), length 98: (tos 0x60, ttl 113, id 0, offset 0, flags [none], proto ICMP (1), length 84)
                                            216.58.215.227 > 192.168.5.1: ICMP echo reply, id 61288, seq 0, length 64
                                            20:09:21.238644 02:11:32:25:4e:f9 > dc:39:6f:f2:4d:e4, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 58651, offset 0, flags [none], proto ICMP (1), length 84)

                                            Trying to understand who is dc:39:6f:f2:4d:e4....

                                            Currently iMac is connected only with wifi so no multiple ip

                                            bingo600 1 Reply Last reply Reply Quote 0
                                            • bingo600
                                              bingo600 @chrispazz last edited by

                                              @chrispazz said in offloading OpenVPN using external gateway:

                                              Trying to understand who is dc:39:6f:f2:4d:e4....

                                              My money is on the ISP router 5.254

                                              pfSense Diagnostic -> arp table will reveal

                                              C 1 Reply Last reply Reply Quote 0
                                              • C
                                                chrispazz @bingo600 last edited by

                                                @bingo600 you are right!

                                                bingo600 1 Reply Last reply Reply Quote 0
                                                • bingo600
                                                  bingo600 @chrispazz last edited by

                                                  @chrispazz

                                                  In the capture , strange it matched ....
                                                  didn't you set 192.168.5.99 as ip match ?

                                                  C 1 Reply Last reply Reply Quote 0
                                                  • C
                                                    chrispazz @bingo600 last edited by

                                                    @bingo600 latest try was with google IP as you asked....am I wrong?

                                                    1 Reply Last reply Reply Quote 0
                                                    • bingo600
                                                      bingo600 @chrispazz last edited by

                                                      @chrispazz said in offloading OpenVPN using external gateway:

                                                      @bingo600 said in offloading OpenVPN using external gateway:

                                                      What does as host mean (you mean you ping the raspi) ?

                                                      HOST is the captured Host address

                                                      We need to see the mac addresses in those packets.

                                                      b8:27:eb:8e:00:43 is Raspi MAC Address
                                                      02:11:32:25:4e:f9 is pfsense WAN interface

                                                      Full detailed trace is the following:

                                                      192.168.5.1 > 192.168.5.99: ICMP echo request, id 43267, seq 28884, length 9
                                                      19:51:42.213831 b8:27:eb:8e:00:43 > 02:11:32:25:4e:f9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 59852, offset 0, flags [none], proto ICMP (1), length 29)
                                                      192.168.5.99 > 192.168.5.1: ICMP echo reply, id 43267, seq 28884, length 9
                                                      19:51:42.723672 02:11:32:25:4e:f9 > b8:27:eb:8e:00:43, ethertype IPv4 (0x0800), length 43: (tos 0x0, ttl 64, id 65165, offset 0, flags [none], proto ICMP (1), length 29)

                                                      What command did you use on the iMAC to make this trace , where it goes to the Raspi (but raspi as destination)

                                                      C 1 Reply Last reply Reply Quote 0
                                                      • C
                                                        chrispazz @bingo600 last edited by

                                                        @bingo600 only "ping www.google.it".

                                                        But in first trace I used the raspi as Host capture.

                                                        C bingo600 2 Replies Last reply Reply Quote 0
                                                        • C
                                                          chrispazz @chrispazz last edited by chrispazz

                                                          Ok this is interesting:

                                                          I added a floating rule with no filters but "out" direction and in advanced settings with the Raspi Gateway.

                                                          With this, the connection go correctly thru the Raspi VPN.

                                                          So, the external gateway thur pfsense is working correctly.
                                                          I have to understand why when using a firewall rule in a specific interface is not using Raspi gateway

                                                          1 Reply Last reply Reply Quote 0
                                                          • bingo600
                                                            bingo600 @chrispazz last edited by bingo600

                                                            @chrispazz
                                                            I don't think the first capture is showing the : ping www.google.it
                                                            I think something is pinging the Raspi ... And it's prob the gw-monitor.
                                                            I was fooled there

                                                            It was my mistake concluding the policy route was working, in the first trace we saw.

                                                            A working policy route package would be like the last one you made (google ip)
                                                            but with the RasPI MAC instead of the ISP-Router MAC

                                                            So at the moment pfSense policy routing is NOT working as intended

                                                            C 1 Reply Last reply Reply Quote 0
                                                            • C
                                                              chrispazz @bingo600 last edited by chrispazz

                                                              @bingo600 Yes but the rules is matched because it is logged in the pfsense firewall log.
                                                              It only is ignoring the gateway in advanced settings....
                                                              The same rule in "floating" mode is working...

                                                              C 1 Reply Last reply Reply Quote 0
                                                              • C
                                                                chrispazz @chrispazz last edited by

                                                                This is really strange, now again the floating rule is not working.
                                                                I will try to reboot pfsense....

                                                                bingo600 1 Reply Last reply Reply Quote 0
                                                                • bingo600
                                                                  bingo600 @chrispazz last edited by

                                                                  @chrispazz said in offloading OpenVPN using external gateway:

                                                                  This is really strange, now again the floating rule is not working.
                                                                  I will try to reboot pfsense....

                                                                  Noooooo
                                                                  Don't say you're using floating rules , i asked about that

                                                                  C 1 Reply Last reply Reply Quote 0
                                                                  • C
                                                                    chrispazz @bingo600 last edited by

                                                                    @bingo600
                                                                    I was NOT using floating rules at all.

                                                                    I tried now to see the same rule in floating mode if it works with raspi gateway or not and with floating IT IS working.....

                                                                    Why in floating mode use the specified gateway and in normal not floating mode ignores it?

                                                                    C bingo600 2 Replies Last reply Reply Quote 0
                                                                    • C
                                                                      chrispazz @chrispazz last edited by chrispazz

                                                                      This post is deleted!
                                                                      1 Reply Last reply Reply Quote 0
                                                                      • bingo600
                                                                        bingo600 @chrispazz last edited by

                                                                        @chrispazz

                                                                        It shouldn't
                                                                        But AFAIK floating rules are always parsed first.

                                                                        C 1 Reply Last reply Reply Quote 0
                                                                        • C
                                                                          chrispazz @bingo600 last edited by

                                                                          @bingo600 Should I open a bug in pfsense? 😧

                                                                          bingo600 1 Reply Last reply Reply Quote 0
                                                                          • bingo600
                                                                            bingo600 @chrispazz last edited by

                                                                            @chrispazz
                                                                            Not yet

                                                                            I think one of the mods / gurus should review this thread first.

                                                                            @stephenw10 @johnpoz
                                                                            Any words here

                                                                            1 Reply Last reply Reply Quote 0
                                                                            • bingo600
                                                                              bingo600 last edited by

                                                                              I don't hope we have been bitten by NAT 😰
                                                                              The policy rule rule was on a LAN ingress interface , looking at the HOST Source IP of the iMAC. And the Logging showed a match.

                                                                              Technically the source IP is later being set to the pfSense WAN IP, but the new gateway ought to have been set before that.

                                                                              I'm officialy confused 🤕

                                                                              I do hope your policy rule was the top most rule , and that you didn't have any rules before that on that interface , else we'we been chasing ghosts all night.

                                                                              A permit ICMP any any before the policy rule would have made that "ghost"

                                                                              /Bingo

                                                                              C 1 Reply Last reply Reply Quote 0
                                                                              • C
                                                                                chrispazz @bingo600 last edited by

                                                                                @bingo600 eheheh

                                                                                Yes, the policy rule is the first one on that interface.

                                                                                C 1 Reply Last reply Reply Quote 0
                                                                                • C
                                                                                  chrispazz @chrispazz last edited by

                                                                                  I tried some other tests but no luck. I am officially unable to apply that gateway

                                                                                  😧

                                                                                  1 Reply Last reply Reply Quote 0
                                                                                  • First post
                                                                                    Last post

                                                                                  Products

                                                                                  • Platform Overview
                                                                                  • TNSR
                                                                                  • pfSense
                                                                                  • Appliances

                                                                                  Services

                                                                                  • Training
                                                                                  • Professional Services

                                                                                  Support

                                                                                  • Subscription Plans
                                                                                  • Contact Support
                                                                                  • Product Lifecycle
                                                                                  • Documentation

                                                                                  News

                                                                                  • Media Coverage
                                                                                  • Press
                                                                                  • Events

                                                                                  Resources

                                                                                  • Blog
                                                                                  • FAQ
                                                                                  • Find a Partner
                                                                                  • Resource Library
                                                                                  • Security Information

                                                                                  Company

                                                                                  • About Us
                                                                                  • Careers
                                                                                  • Partners
                                                                                  • Contact Us
                                                                                  • Legal
                                                                                  Our Mission

                                                                                  We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                                                                                  Subscribe to our Newsletter

                                                                                  Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                                                                                  © 2021 Rubicon Communications, LLC | Privacy Policy