offloading OpenVPN using external gateway

bingo600

You say your pfSense is 192.168.99.1 , but the drawing shows 99.250

Is the 99.1 interface a pfSense "inside Lan" ip ?

How does the 192.168.99.x interface relate on the drawing ?

I expect your pfSense to have a WAN ip on the ISP router LAN , meaning the 192.168.5.x range.

Since you say you can ping and prob SSH to the Raspi , i assume that is correct or ??

chrispazz

@bingo600 NAS is 99.250.
Inside NAS (Synology) I run a VM with pfsense (99.1).

I tried enabling logging on the rule and I can confirm it is using it.
Yes, my pfsense WAN interface is the one connecting to the ISP router.
And yes, I am using NAT on the pfsense WAN if....

bingo600

@chrispazz

Just for completeness
What is your pfSense WAN IP ?

chrispazz

@bingo600 it is 192.168.5.254

bingo600

@chrispazz

??

Then what is your ISP router inside ip ?

bingo600

@chrispazz said in offloading OpenVPN using external gateway:

I tried enabling logging on the rule and I can confirm it is using it.

Ok

Now try to do a packet trace on the pfSense WAN

Set host address to the Raspi IP

And generate some traffic destined for VPN

chrispazz

@bingo600 oops....sorry. Inside IP of router is 192.168.5.254.

Pf sense has 192.168.5.1 on the WAN side...

chrispazz

@bingo600

Activated packet capture on host 192.168.5.9 and opened a webpage with www.google.it:

16:58:57.410983 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8815, length 9
16:58:57.920985 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8816, length 9

bingo600

@chrispazz said in offloading OpenVPN using external gateway:

@bingo600

Activated packet capture on host 192.168.5.9 and opened a webpage with www.google.it:

16:58:57.410983 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8815, length 9
16:58:57.920985 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 8816, length 9

Where did 192.168.5.9 come from ??
did you mean 5.99

chrispazz

@bingo600 yes is 5.99 :) don't hate me eheheh

bingo600

@chrispazz

So we have confirmed that the ICMP (ping) packages are actually being "policy routed" from 5.1 (pfSense) to 5.99 (RasPI)

We don't see any Echo reply ... The issue is probably on the RasPI

bingo600

On the Raspi (as root)

What does a : sudo iptables -L show ?

And this one ?

chrispazz

This post is deleted!

bingo600

Did you reboot the RasPI after enabling forwarding ?

chrispazz

@bingo600 yes, forward is activated.

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

and yes, I rebooted it a lot of times :)

bingo600

@chrispazz

Does this one give the expected result , should be your (OpenVPN public ip) ?

wget http://ipinfo.io/ip -qO -

Can you ping the google.it from the RasPI ?

/Bingo

chrispazz

@bingo600 said in offloading OpenVPN using external gateway:

wget http://ipinfo.io/ip -qO -

Both yes.
And if I test a traceroute from raspi I can confirm it is going outside using VPN connection....

bingo600

@chrispazz

Now : I would connect a PC directly to the RasPI , or directly to the ISP inside LAN , and set def-gw to the RasPI , and maybe DNS to 8.8.8.8

Then start debugging directly between the PC & The Raspi.

This can come in handy ... Disables all iptables stuff on the Raspi.
But only for this "boot sesion"

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

I use these rules on my Deb10 machines , for OVPN to a VPN provider.

Remember to replace ens192 with the name of your ethernet interface on the Raspi.

# Default policy to drop all incoming packets
iptables -P INPUT DROP
iptables -P FORWARD DROP

# Accept incoming packets from localhost and the LAN interface
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i ens192 -j ACCEPT

# Accept incoming packets from the WAN if the router initiated
# the connection
iptables -A INPUT -i tun0 -m conntrack \
    --ctstate ESTABLISHED,RELATED -j ACCEPT

# Forward LAN packets to the WAN
iptables -A FORWARD -i ens192 -o tun0 -j ACCEPT

# Forward WAN packets to the LAN if the LAN initiated the
# connection
iptables -A FORWARD -i tun0 -o ens192 -m conntrack \
    --ctstate ESTABLISHED,RELATED -j ACCEPT

# NAT traffic going out the WAN interface
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
#

Remember to save new rules if you change them

sudo systemctl enable netfilter-persistent

chrispazz

@bingo600 said in offloading OpenVPN using external gateway:

sudo systemctl enable netfilter-persistent

I made some other tests.

Raspberry is working good if I set a client to use it as network interface gateway and connection will go thru VPN (using my own iptables rules or yours).

So it seems that when I go thru pfsense, WAN main gateway (ISP router) is getting used instead of firewall rule gateway...

bingo600

@chrispazz said in offloading OpenVPN using external gateway:

@bingo600 said in offloading OpenVPN using external gateway:

sudo systemctl enable netfilter-persistent

I made some other tests.

Raspberry is working good if I set a client to use it as network interface gateway and connection will go thru VPN (using my own iptables rules or yours).

So it seems that when I go thru pfsense, WAN main gateway (ISP router) is getting used instead of firewall rule gateway...

That doesn't make sense
The packet trace you made doesn't lie.

bingo600

Now is the time to use tcpdump on the RasPI

https://kalitut.com/record-network-traffic-with-raspberry/

To see what it actually receives & forwards

chrispazz

@bingo600
This is traceroute from the client with the firewall rule applies:

traceroute to www.google.it (216.58.215.227), 64 hops max, 52 byte packets
1 192.168.5.254 (192.168.5.254) 2.370 ms 1.583 ms 1.516 ms

The first hop go to ISP gateway....I can't see Raspi here...

bingo600

@chrispazz
But that is not consistent with the packet trace you made on the pfSense

Can you do another trace , and then the traceroute ?

chrispazz

@bingo600 if I try a trace to www.google.it IP address as host I get this:

18:25:31.846958 IP 192.168.5.1.4704 > 172.217.168.35.443: UDP, length 1350
18:25:31.883433 IP 172.217.168.35.443 > 192.168.5.1.4704: UDP, length 1350

so it is using 192.168.5.1 as gateway....

chrispazz

@chrispazz Perhaps I understand the problem.

My WAN interface is getting IP from ISP router (DHCP server on 192.168.5.0).
The ISP gateway is getting on top of the firewall rule gateway....

This is WAN if on pfsense:

em1.90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
options=3<RXCSUM,TXCSUM>
ether 02:11:32:25:4e:f9
inet6 fe80::11:32ff:fe25:4ef9%em1.90 prefixlen 64 scopeid 0x8
inet 192.168.5.1 netmask 0xffffff00 broadcast 192.168.5.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 90 vlanpcp: 0 parent interface: em1
groups: vlan

chrispazz

No wrong.

192.168.5.1 is pfsense.... :(

bingo600

@chrispazz said in offloading OpenVPN using external gateway:

@bingo600 if I try a trace to www.google.it IP address as host I get this:

18:25:31.846958 IP 192.168.5.1.4704 > 172.217.168.35.443: UDP, length 1350
18:25:31.883433 IP 172.217.168.35.443 > 192.168.5.1.4704: UDP, length 1350

so it is using 192.168.5.1 as gateway....

Nope
It's using 5.1 as source IP (Consistent with your WAN NAT)

If you enable logging on the Policy rule , do you see any log lines ?

Now you are not doing the traceroute on the pfSense box ... Are you ?

What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

chrispazz

@bingo600 said in offloading OpenVPN using external gateway:

If you enable logging on the Policy rule , do you see any log lines ?

Yes, it is the only rules that has logging

Now you are not doing the traceroute on the pfSense box ... Are you ?

No, I am doing it from iMac client (192.168.20.110)

What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

I am using a firewall rule like this on the same interface of the client (192.168.20):

and NVPNGTW is the gateway I added to pfsense in the gateway section.

bingo600

@chrispazz said in offloading OpenVPN using external gateway:

@bingo600 said in offloading OpenVPN using external gateway:

If you enable logging on the Policy rule , do you see any log lines ?

Yes, it is the only rules that has logging

Now you are not doing the traceroute on the pfSense box ... Are you ?

No, I am doing it from iMac client (192.168.20.110)

What PC ip address are you doing the traceroute from , and are you 100% sure you have that ip in the OPENVPN Alias , and a policy route rule on the pfSense interface that that PC is connected to ?

I am using a firewall rule like this on the same interface of the client (192.168.20):

and NVPNGTW is the gateway I added to pfsense in the gateway section.

That's a (for me strange rule) why does it have WAN all the way to the right ?
Are you using floating rules ?

chrispazz

@bingo600 no no, sorry, it is only the description. Ignore it.

bingo600

@chrispazz

Is that rule all the way at the top ?

chrispazz

@bingo600 yes, sure

bingo600

@chrispazz
I have to pull in the tuff guyzz now.

@stephenw10
Any ideas here ??

bingo600

@chrispazz

But just to summarize :

You say if you put a PC on the 192.168.5.x lan , and set 192.168.5.99 (RasPI) as def-gw , the PC can use the RasPI & OpenVPN tunnel ok ?

~~You have put all desired OPENVPN PC's in a IP HOST Alias~~
You made direct host entries in the rule, and made Policy based rules with the RasPI as Gateway , on all the interfaces where a "OPENVPN" PC could be connected ?

But the pfSense won't match the source ip's, and therefore won't policy route t the RasPI. This is also confirmed by , enabling logging in the Policy rule. And no loglines appear.

Q:
We did a trace (the first one above) , where we succesfully matched a ping to google.it , and you got both loglines , and we saw the policy routing in effect in the pfSense packet trace.

What has changed in the pfSense config since that ?

Was that test also made with the iMAC ?

Q:
You do use either static IP's or DHCP MAC Lock (Reservation) , in order to ensure that the PC's used for test & OPN always get the same ip ?.

chrispazz

@bingo600 said in offloading OpenVPN using external gateway:

@chrispazz

But just to summarize :

You say if you put a PC on the 192.168.5.x lan , and set 192.168.5.99 (RasPI) as def-gw , the PC can use the RasPI & OpenVPN tunnel ok ?

Yes, correct.

~~You have put all desired OPENVPN PC's in a IP HOST Alias~~
You made direct host entries in the rule, and made Policy based rules with the RasPI as Gateway , on all the interfaces where a "OPENVPN" PC could be connected ?

Yes.

But the pfSense won't match the source ip's, and therefore won't policy route t the RasPI. This is also confirmed by , enabling logging in the Policy rule. And no loglines appear.

No. The rule is correctly engaged and it is logged in firewall log but it seems to ignore the Raspi used as gateway.

Q:
We did a trace (the first one above) , where we succesfully matched a ping to google.it , and you got both loglines , and we saw the policy routing in effect in the pfSense packet trace.

Currently, using 192.168.5.9 (Raspi gateway) as host, trace display the following when doing a ping to www.google.it:

19:32:03.251588 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 26591, length 9
19:32:03.252028 IP 192.168.5.9 > 192.168.5.1: ICMP echo reply, id 43267, seq 26591, length 9

Doing packet trace using google IP as host gives the following:

19:33:38.627397 IP 192.168.5.1 > 172.217.168.3: ICMP echo request, id 10411, seq 0, length 64
19:33:38.642607 IP 172.217.168.3 > 192.168.5.1: ICMP echo reply, id 10411, seq 0, length 64

Was that test also made with the iMAC ?

The packet tracing is done pinging from Imac to www.google.it

Q:
You do use either static IP's or DHCP MAC Lock (Reservation) , in order to ensure that the PC's used for test & OPN always get the same ip ?.

I have IP reservation for iMac and static IP for Pfsense.

Just to give you more confirmation about the rule catched, consider that without that new rule, the connection has to go via the pfsense VPN client.
If I deactivate the rule, the connection go with pfsense VPN client. if I enable the rule, the connection exit with ISP gateway (no raspi VPN).

bingo600

@chrispazz said in offloading OpenVPN using external gateway:
packet trace.

Currently, using 192.168.5.9 (Raspi gateway) as host, trace display the following when doing a ping to www.google.it:

What does that mean ?
Currently, using 192.168.5.9 (Raspi gateway) as host

Wasnt the RasPI 192.168.5.99 ?

What does as host mean (you mean you ping the raspi) ?

The below 2 traces both show a req & a reply

19:32:03.251588 IP 192.168.5.1 > 192.168.5.9: ICMP echo request, id 43267, seq 26591, length 9
19:32:03.252028 IP 192.168.5.9 > 192.168.5.1: ICMP echo reply, id 43267, seq 26591, length 9

Doing packet trace using google IP as host gives the following:

19:33:38.627397 IP 192.168.5.1 > 172.217.168.3: ICMP echo request, id 10411, seq 0, length 64
19:33:38.642607 IP 172.217.168.3 > 192.168.5.1: ICMP echo reply, id 10411, seq 0, length 64

In the Packet capture try to set detail to high or full

We need to see the mac addresses in those packets.

And compare to the RasPI mac address

chrispazz

@bingo600 said in offloading OpenVPN using external gateway:

What does as host mean (you mean you ping the raspi) ?

HOST is the captured Host address

We need to see the mac addresses in those packets.

b8:27:eb:8e:00:43 is Raspi MAC Address
02:11:32:25:4e:f9 is pfsense WAN interface

Full detailed trace is the following:

192.168.5.1 > 192.168.5.99: ICMP echo request, id 43267, seq 28884, length 9
19:51:42.213831 b8:27:eb:8e:00:43 > 02:11:32:25:4e:f9, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 59852, offset 0, flags [none], proto ICMP (1), length 29)
192.168.5.99 > 192.168.5.1: ICMP echo reply, id 43267, seq 28884, length 9
19:51:42.723672 02:11:32:25:4e:f9 > b8:27:eb:8e:00:43, ethertype IPv4 (0x0800), length 43: (tos 0x0, ttl 64, id 65165, offset 0, flags [none], proto ICMP (1), length 29)

bingo600

@chrispazz

Now do the same trace with the google ip as the destination

Your iMAC isn't multihomed ?
I mean both cable & WiFi

chrispazz

@bingo600 here it is:

192.168.5.1 > 216.58.215.227: ICMP echo request, id 61288, seq 0, length 64
20:09:20.262578 dc:39:6f:f2:4d:e4 > 02:11:32:25:4e:f9, ethertype IPv4 (0x0800), length 98: (tos 0x60, ttl 113, id 0, offset 0, flags [none], proto ICMP (1), length 84)
216.58.215.227 > 192.168.5.1: ICMP echo reply, id 61288, seq 0, length 64
20:09:21.238644 02:11:32:25:4e:f9 > dc:39:6f:f2:4d:e4, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 58651, offset 0, flags [none], proto ICMP (1), length 84)

Trying to understand who is dc:39:6f:f2:4d:e4....

Currently iMac is connected only with wifi so no multiple ip

bingo600

@chrispazz said in offloading OpenVPN using external gateway:

Trying to understand who is dc:39:6f:f2:4d:e4....

My money is on the ISP router 5.254

pfSense Diagnostic -> arp table will reveal

offloading OpenVPN using external gateway

Products

Services

Support

News

Resources

Company

Our Mission

Subscribe to our Newsletter