pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help

viragomann

@kpucko
It's well described on the OpenVPN community document which @Pippin linked above.

You are missing the iroute line:

iroute 192.168.0.0 255.255.255.0

KpuCko

Ok, added.

root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301

ifconfig-push 172.16.21.10 255.255.255.0

push "route 10.0.1.10 255.255.255.0"

iroute 192.168.0.0 255.255.255.0

root@frax1021dckr1110:~#

In the log of the OpenVPN server I see this LERAN line:

Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI_sva: pool returned IPv4=172.16.21.2, IPv6=(Not enabled)
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 OPTIONS IMPORT: reading client specific options from: ccd/sofn1010fw301
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: Learn: 172.16.21.10 -> sofn1010fw301:31634
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: primary virtual IP for sofn1010fw301:31634: 172.16.21.10
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: internal route 192.168.0.0/24 -> sofn1010fw301:31634
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: Learn: 192.168.0.0/24 -> sofn1010fw301:31634
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 Data Channel: using negotiated cipher 'AES-256-GCM'
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 SENT CONTROL [sofn1010fw301]: 'PUSH_REPLY,route-gateway 172.16.21.1,topology subnet,ping 10,ping-restart 60,route 10.0.1.10 255.255.255.0,ifconfig 172.16.21.10 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)

And now? What to test?

root@frax1021dckr1110:~# ping 172.16.21.1
PING 172.16.21.1 (172.16.21.1) 56(84) bytes of data.
64 bytes from 172.16.21.1: icmp_seq=1 ttl=64 time=0.018 ms
64 bytes from 172.16.21.1: icmp_seq=2 ttl=64 time=0.035 ms
^C
--- 172.16.21.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.018/0.026/0.035/0.008 ms
root@frax1021dckr1110:~# ping 172.16.21.10
PING 172.16.21.10 (172.16.21.10) 56(84) bytes of data.
64 bytes from 172.16.21.10: icmp_seq=1 ttl=64 time=27.5 ms
64 bytes from 172.16.21.10: icmp_seq=2 ttl=64 time=27.3 ms
^C
--- 172.16.21.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 27.332/27.400/27.468/0.068 ms
root@frax1021dckr1110:~# ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
^C
--- 192.168.0.1 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6124ms

root@frax1021dckr1110:~#

No difference in my opinion.

[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: ping -S 192.168.0.1 172.16.21.1
PING 172.16.21.1 (172.16.21.1) from 192.168.0.1: 56 data bytes
^C
--- 172.16.21.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root:

viragomann

@kpucko
Do you have this line in the server config?

route 192.168.0.0 255.255.255.0

If you still have the static route, remove it before your restart the server.

stephenw10

Yeah, you need a route and an iroute. Also appropriate firewall rules to pass that.

KpuCko

Ok, thanks to all of you guys.

I will find some time today to show the config of the OpenVPN server with all the routes and CCDs. Just bear with me, there are a lot of things happening in parallel.

KpuCko

Here it is:

root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301

ifconfig-push 172.16.21.10 255.255.255.0

push "route 10.0.1.10 255.255.255.0"

iroute 192.168.0.0 255.255.255.0
root@frax1021dckr1110:~#

And the OpenVPN server config:

# Service user and group
user nobody
group nogroup

# Set output verbosity
verb 3

# Tunnel config
proto udp
dev tun
port 1194
compress stub-v2
chroot /etc/openvpn/chroot/frax1021dckr1110/
tmp-dir tmp/
client-config-dir ccd
ccd-exclusive

persist-tun
persist-key

# Server config
topology subnet
server 172.16.21.0 255.255.255.0
tls-server
tls-version-min 1.3
ncp-ciphers AES-256-GCM
dh /etc/easyrsa/pki/dh.pem
ca /etc/easyrsa/pki/ca.crt
cert /etc/easyrsa/pki/issued/frax1021dckr1110.crt
key  /etc/easyrsa/pki/private/frax1021dckr1110.key
crl-verify crl.pem
verify-client-cert require
tls-auth /etc/openvpn/private/frax1021dckr1110-ta.key 0
auth SHA256
keepalive 10 60
float

#
# Extra params
#

root@frax1021dckr1110:~#

There is no static route for 192.168.0.0/24 on the server side:

root@frax1021dckr1110:~# route -n | grep 192.168
192.168.192.0   172.25.0.1      255.255.255.0   UG    20     0        0 tun0
192.168.193.0   172.25.0.1      255.255.255.0   UG    20     0        0 tun0
root@frax1021dckr1110:~#

So now you want me to put this on the server config?

route 192.168.0.0 255.255.255.0

And we are done?

Meantime let me share the rules for that interface:

Maybe this also make sense:

This is the config part of the OpenVPN client from the pfSense perspective

viragomann

@kpucko said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:

o now you want me to put this on the server config?
route 192.168.0.0 255.255.255.0

And we are done?

At least this is necessary to get it work.

Meantime let me share the rules for that interface:

Consider, the "VPNFRAX1 net" is only the tunnel (transit) network. This only applies to the server itself.
But as I got your initial post, you want to set up a site-to-site VPN. So I assume you need access from behind the Ubuntu server, don't you?
If yes, you have also ensure that there is a route for the clients Lan on the server side pointing the the server (in case the Ubuntu is not the default gateway).

Maybe this also make sense:

If you want to communicate with the network(s) behind the Ubuntu you have to add them to the "Remote networks" here.

stephenw10

@viragomann said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:

If you want to communicate with the network(s) behind the Ubuntu you have to add them to the "Remote networks" here.

It's an SSL/TLS server so it should push those networks to the client(s).
And it looks like it is configured to push 10.0.1.10/24. That should be 10.0.1.0/24 bu will probably still work.

Yes, those firewall rules will block everything except ping and BGP in the server to client direction. That might be what you want though.

Steve

viragomann

@stephenw10 said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:

It's an SSL/TLS server so it should push those networks to the client(s).
And it looks like it is configured to push 10.0.1.10/24. That should be 10.0.1.0/24 bu will probably still work.

Ahh, you're right.
I noticed this above.

KpuCko

@viragomann said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:

route 192.168.0.0 255.255.255.0

Yeah :-) all good now
Thanks guys!!! Everything is up and running.

Will leave the config here in case if anyone wonder howto do it:

root@frax1021dckr1110:~# grep route /etc/openvpn/server/frax1021dckr1110.conf
route 192.168.0.0 255.255.255.0
root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301

ifconfig-push 172.16.21.10 255.255.255.0

push "route 10.0.1.10 255.255.255.0"

iroute 192.168.0.0 255.255.255.0
root@frax1021dckr1110:~#

The tests:

C:\Users\KpuCko>ipconfig |FIND "192.168.0.15"
   IPv4 Address. . . . . . . . . . . : 192.168.0.15

C:\Users\KpuCko>ping 10.0.1.10

Pinging 10.0.1.10 with 32 bytes of data:
Reply from 10.0.1.10: bytes=32 time=30ms TTL=63
Reply from 10.0.1.10: bytes=32 time=30ms TTL=63
Reply from 10.0.1.10: bytes=32 time=30ms TTL=63
Reply from 10.0.1.10: bytes=32 time=30ms TTL=63

The confusing part for me was this route and iroute settings. I read in the tutorial linked above that they both are needed, check here:

Why the redundant route and iroute statements, you might ask? The reason is that route controls the routing from the kernel to the OpenVPN server (via the TUN interface) while iroute controls the routing from the OpenVPN server to the remote clients. Both are necessary.

The only thing left unclear for me is the line with the pool, check this:

Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: Could not determine IPv4/IPv6 protocol. Using AF_INET
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: UDPv4 link local (bound): [AF_INET][undef]:1194
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: UDPv4 link remote: [AF_UNSPEC]
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: chroot to '/etc/openvpn/chroot/frax1021dckr1110/' and cd to '/' succeeded
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: GID set to nogroup
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: UID set to nobody
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: MULTI: multi_init called, r=256 v=256
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: IFCONFIG POOL IPv4: base=172.16.21.2 size=253
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: Initialization Sequence Completed

So the pool starts from 172.16.21.2 because 172.16.21.1 is reserved for the local IP, but the client is instructed via CCD to get 172.16.21.10 which is shown below:

Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI_sva: pool returned IPv4=172.16.21.2, IPv6=(Not enabled)
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301/:63268 OPTIONS IMPORT: reading client specific options from: ccd/sofn1010fw301
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: Learn: 172.16.21.10 -> sofn1010fw301:63268
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: primary virtual IP for sofn1010fw301:63268: 172.16.21.10
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: internal route 192.168.0.0/24 -> sofn1010fw301:63268
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: Learn: 192.168.0.0/24 -> sofn1010fw301:63268
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 Data Channel: using negotiated cipher 'AES-256-GCM'
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 SENT CONTROL [sofn1010fw301]: 'PUSH_REPLY,route-gateway 172.16.21.1,topology subnet,ping 10,ping-restart 60,route 10.0.1.10 255.255.255.0,ifconfig 172.16.21.10 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)

Great 172.16.21.10 is assigned to the client, but see the local kernel routing table:

root@frax1021dckr1110:~# route -n |grep 192.168
192.168.0.0     172.16.21.2     255.255.255.0   UG    0      0        0 tun1
root@frax1021dckr1110:~#

Why 172.16.21.2, I don't have 172.16.21.2 anywhere..
At least it works, so I'm asking just for clarity :-)

stephenw10

OpenVPN always uses the first IP in the tunnel subnet to route traffic to like that. But usually at the server end. I expect to see .1 there at the client side if the server is configured as that. It seems to be passing that as a gateway.

It doesn't really matter since it's actually the OpenVPN daemon that decides where to route it to. The kernel route only has to get it to the daemon.

You should change the local subnet you are pushing to 10.0.1.0/24 instead of .10 to be 'correct'. Some clients might reject that as an invalid route.

Steve

KpuCko

@stephenw10

You should change the local subnet you are pushing to 10.0.1.0/24 instead of .10 to be 'correct'. Some clients might reject that as an invalid route.

Steve

Oo I see what you are speaking about:

push "route 10.0.1.10 255.255.255.0"

It should be 10.0.0.0 255.255.255.0, I even haven't noticed that I wrote it in this way..

stephenw10

It should be: 10.0.1.0 255.255.255.0

KpuCko

@stephenw10 said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:

It should be: 10.0.1.0 255.255.255.0

Of course :-)))

Sorry about that, I have really fast fingers =]]