pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help
-
Hi there, I need some help because I really don't have a clue what is happening..
The task is really simple, from on side I have Ubuntu server which is running OpenVPN server, and on the other side I have my pfSense box. So I try to establish site-to-site VPN and route 1 network via this tunnel. Unfortunately the things are not happening the way I want/know.
Let me describe what I have so far.
Ubuntu box:
- ens3 - 10.0.1.10
- tun1 - 172.16.21.1 (local OpenVPN interface)
pfSense box:
- VLAN30 (opt1) -> lagg0.30 -> v4: 192.168.0.1/24
- VPN2FRAX1 (opt11) -> ovpnc4 -> v4: 172.16.21.10/24
So if I try to ping the other side of the tunnel from the pfSense box I've got:
[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: ping 172.16.21.1 PING 172.16.21.1 (172.16.21.1): 56 data bytes 64 bytes from 172.16.21.1: icmp_seq=0 ttl=64 time=30.202 ms 64 bytes from 172.16.21.1: icmp_seq=1 ttl=64 time=29.565 ms ^C --- 172.16.21.1 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 29.565/29.883/30.202/0.318 ms [2.6.0-RELEASE][root@sofn-fw001.home.lan]/root:Same applies for the Ubuntu server:
root@frax1021dckr1110:~# ping 172.16.21.10 PING 172.16.21.10 (172.16.21.10) 56(84) bytes of data. 64 bytes from 172.16.21.10: icmp_seq=1 ttl=64 time=29.7 ms 64 bytes from 172.16.21.10: icmp_seq=2 ttl=64 time=29.6 ms 64 bytes from 172.16.21.10: icmp_seq=3 ttl=64 time=29.9 ms ^C --- 172.16.21.10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 29.627/29.719/29.856/0.098 ms root@frax1021dckr1110:~#Nothing unusual so far, right? Now let's involve additional network in the play.
I will try to ping 172.16.21.10 from the pfSense box, but this time I will use 192.168.0.1.
You immediately will say that it will fail, because you are right - the Ubuntu box don't know anything about 192.168.0.0/24. Let's check:[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: ping -S 192.168.0.1 172.16.21.1 PING 172.16.21.1 (172.16.21.1) from 192.168.0.1: 56 data bytes ^C --- 172.16.21.1 ping statistics --- 7 packets transmitted, 0 packets received, 100.0% packet loss [2.6.0-RELEASE][root@sofn-fw001.home.lan]/root:Ok, now if I understand the networking in general I have to add a static route for this network on the Ubuntu box. That is what I'm going to do:
root@frax1021dckr1110:~# route add -net 192.168.0.0/24 gw 172.16.21.10 root@frax1021dckr1110:~# route -n | grep tun1 172.16.21.0 0.0.0.0 255.255.255.0 U 0 0 0 tun1 192.168.0.0 172.16.21.10 255.255.255.0 UG 0 0 0 tun1 root@frax1021dckr1110:~#So I'm directing the traffic to destination 192.168.0.0/24 to go to the other site of the tunnel. That's why I have 172.16.21.10 this IP address. So now it should work, right?
[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: ping -S 192.168.0.1 172.16.21.1 PING 172.16.21.1 (172.16.21.1) from 192.168.0.1: 56 data bytes ^C --- 172.16.21.1 ping statistics --- 7 packets transmitted, 0 packets received, 100.0% packet loss [2.6.0-RELEASE][root@sofn-fw001.home.lan]/root:Unfortunately it doesn't work.. I even can't see any traffic coming from the pfSense box:
root@frax1021dckr1110:~# tcpdump icmp and src 192.168.0.1 -nvvv -i tun1 tcpdump: listening on tun1, link-type RAW (Raw IP), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@frax1021dckr1110:~#What about the routing table of the pfSense? Let me show you that piece of info:
[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: netstat -rn | grep 172.16.21 172.16.21.0/24 172.16.21.1 UGS ovpnc4 172.16.21.1 link#23 UH ovpnc4 172.16.21.10 link#23 UHS lo0 [2.6.0-RELEASE][root@sofn-fw001.home.lan]/root:So in my opinion all should be fine, but it isn't...
Any advice are more then welcomed.
Thanks. -
@kpucko said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
Ok, now if I understand the networking in general I have to add a static route for this network on the Ubuntu box.
You have to add a client specific override for the client. Otherwise the OpenVPN doesn't know, where to route 192.168.0.1 to.
Alternatively if there is only one single client you can use a /30 tunnel network.
-
Configure the Ubuntu OpenVPN server.
It is explained here:
https://community.openvpn.net/openvpn/wiki/HOWTO#ExpandingthescopeoftheVPNtoincludeadditionalmachinesoneithertheclientorserversubnetTo elaborate on @viragomann reply,
OpenVPN ccd (client config directory) is what pfSense calls CSO (Client Specific Overrides). -
Hmmm, this completely confuse me. As far as I know "client overwrites" are applicable only for OpenVPN servers.
I want to clarify, the pfSense box is acting as OpenVPN client, so there is no way to do "client overwrites" in that case.
Probably I can't get what you say. Sorry for my stupidity.
-
@kpucko said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
As far as I know "client overwrites" are applicable only for OpenVPN servers.
Yeah, exactly. I was talking about the server side, of course.
-
Ok, so as ccd so far I have this:
root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301 ifconfig-push 172.16.21.10 255.255.255.0 push "route 10.0.1.10 255.255.255.0" root@frax1021dckr1110:~#But I really don't know how this matter.
You want me here to put 192.168.0.0/24? But this network is not in the Ubuntu side. -
-
Ok, added.
root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301 ifconfig-push 172.16.21.10 255.255.255.0 push "route 10.0.1.10 255.255.255.0" iroute 192.168.0.0 255.255.255.0 root@frax1021dckr1110:~#In the log of the OpenVPN server I see this LERAN line:
Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI_sva: pool returned IPv4=172.16.21.2, IPv6=(Not enabled) Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 OPTIONS IMPORT: reading client specific options from: ccd/sofn1010fw301 Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: Learn: 172.16.21.10 -> sofn1010fw301:31634 Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: primary virtual IP for sofn1010fw301:31634: 172.16.21.10 Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: internal route 192.168.0.0/24 -> sofn1010fw301:31634 Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 MULTI: Learn: 192.168.0.0/24 -> sofn1010fw301:31634 Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 Data Channel: using negotiated cipher 'AES-256-GCM' Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key Oct 06 13:34:23 frax1021dckr1110 openvpn[51609]: sofn1010fw301:31634 SENT CONTROL [sofn1010fw301]: 'PUSH_REPLY,route-gateway 172.16.21.1,topology subnet,ping 10,ping-restart 60,route 10.0.1.10 255.255.255.0,ifconfig 172.16.21.10 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)And now? What to test?
root@frax1021dckr1110:~# ping 172.16.21.1 PING 172.16.21.1 (172.16.21.1) 56(84) bytes of data. 64 bytes from 172.16.21.1: icmp_seq=1 ttl=64 time=0.018 ms 64 bytes from 172.16.21.1: icmp_seq=2 ttl=64 time=0.035 ms ^C --- 172.16.21.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1009ms rtt min/avg/max/mdev = 0.018/0.026/0.035/0.008 ms root@frax1021dckr1110:~# ping 172.16.21.10 PING 172.16.21.10 (172.16.21.10) 56(84) bytes of data. 64 bytes from 172.16.21.10: icmp_seq=1 ttl=64 time=27.5 ms 64 bytes from 172.16.21.10: icmp_seq=2 ttl=64 time=27.3 ms ^C --- 172.16.21.10 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 27.332/27.400/27.468/0.068 ms root@frax1021dckr1110:~# ping 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. ^C --- 192.168.0.1 ping statistics --- 7 packets transmitted, 0 received, 100% packet loss, time 6124ms root@frax1021dckr1110:~#No difference in my opinion.
[2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: ping -S 192.168.0.1 172.16.21.1 PING 172.16.21.1 (172.16.21.1) from 192.168.0.1: 56 data bytes ^C --- 172.16.21.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss [2.6.0-RELEASE][root@sofn-fw001.home.lan]/root: -
@kpucko
Do you have this line in the server config?route 192.168.0.0 255.255.255.0If you still have the static route, remove it before your restart the server.
-
Yeah, you need a route and an iroute. Also appropriate firewall rules to pass that.
-
Ok, thanks to all of you guys.
I will find some time today to show the config of the OpenVPN server with all the routes and CCDs. Just bear with me, there are a lot of things happening in parallel.
-
Here it is:
root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301 ifconfig-push 172.16.21.10 255.255.255.0 push "route 10.0.1.10 255.255.255.0" iroute 192.168.0.0 255.255.255.0 root@frax1021dckr1110:~#And the OpenVPN server config:
# Service user and group user nobody group nogroup # Set output verbosity verb 3 # Tunnel config proto udp dev tun port 1194 compress stub-v2 chroot /etc/openvpn/chroot/frax1021dckr1110/ tmp-dir tmp/ client-config-dir ccd ccd-exclusive persist-tun persist-key # Server config topology subnet server 172.16.21.0 255.255.255.0 tls-server tls-version-min 1.3 ncp-ciphers AES-256-GCM dh /etc/easyrsa/pki/dh.pem ca /etc/easyrsa/pki/ca.crt cert /etc/easyrsa/pki/issued/frax1021dckr1110.crt key /etc/easyrsa/pki/private/frax1021dckr1110.key crl-verify crl.pem verify-client-cert require tls-auth /etc/openvpn/private/frax1021dckr1110-ta.key 0 auth SHA256 keepalive 10 60 float # # Extra params # root@frax1021dckr1110:~#There is no static route for 192.168.0.0/24 on the server side:
root@frax1021dckr1110:~# route -n | grep 192.168 192.168.192.0 172.25.0.1 255.255.255.0 UG 20 0 0 tun0 192.168.193.0 172.25.0.1 255.255.255.0 UG 20 0 0 tun0 root@frax1021dckr1110:~#So now you want me to put this on the server config?
route 192.168.0.0 255.255.255.0And we are done?
Meantime let me share the rules for that interface:
Maybe this also make sense:
This is the config part of the OpenVPN client from the pfSense perspective
-
@kpucko said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
o now you want me to put this on the server config?
route 192.168.0.0 255.255.255.0And we are done?
At least this is necessary to get it work.
Meantime let me share the rules for that interface:
Consider, the "VPNFRAX1 net" is only the tunnel (transit) network. This only applies to the server itself.
But as I got your initial post, you want to set up a site-to-site VPN. So I assume you need access from behind the Ubuntu server, don't you?
If yes, you have also ensure that there is a route for the clients Lan on the server side pointing the the server (in case the Ubuntu is not the default gateway).Maybe this also make sense:
If you want to communicate with the network(s) behind the Ubuntu you have to add them to the "Remote networks" here.
-
@viragomann said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
If you want to communicate with the network(s) behind the Ubuntu you have to add them to the "Remote networks" here.
It's an SSL/TLS server so it should push those networks to the client(s).
And it looks like it is configured to push 10.0.1.10/24. That should be 10.0.1.0/24 bu will probably still work.Yes, those firewall rules will block everything except ping and BGP in the server to client direction. That might be what you want though.
Steve
-
@stephenw10 said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
It's an SSL/TLS server so it should push those networks to the client(s).
And it looks like it is configured to push 10.0.1.10/24. That should be 10.0.1.0/24 bu will probably still work.Ahh, you're right.
I noticed this above. -
@viragomann said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
route 192.168.0.0 255.255.255.0
Yeah :-) all good now
Thanks guys!!! Everything is up and running.Will leave the config here in case if anyone wonder howto do it:
root@frax1021dckr1110:~# grep route /etc/openvpn/server/frax1021dckr1110.conf route 192.168.0.0 255.255.255.0 root@frax1021dckr1110:~# cat /etc/openvpn/chroot/frax1021dckr1110/ccd/sofn1010fw301 ifconfig-push 172.16.21.10 255.255.255.0 push "route 10.0.1.10 255.255.255.0" iroute 192.168.0.0 255.255.255.0 root@frax1021dckr1110:~#The tests:
C:\Users\KpuCko>ipconfig |FIND "192.168.0.15" IPv4 Address. . . . . . . . . . . : 192.168.0.15 C:\Users\KpuCko>ping 10.0.1.10 Pinging 10.0.1.10 with 32 bytes of data: Reply from 10.0.1.10: bytes=32 time=30ms TTL=63 Reply from 10.0.1.10: bytes=32 time=30ms TTL=63 Reply from 10.0.1.10: bytes=32 time=30ms TTL=63 Reply from 10.0.1.10: bytes=32 time=30ms TTL=63The confusing part for me was this route and iroute settings. I read in the tutorial linked above that they both are needed, check here:
Why the redundant route and iroute statements, you might ask? The reason is that route controls the routing from the kernel to the OpenVPN server (via the TUN interface) while iroute controls the routing from the OpenVPN server to the remote clients. Both are necessary.
The only thing left unclear for me is the line with the pool, check this:
Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: Could not determine IPv4/IPv6 protocol. Using AF_INET Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: Socket Buffers: R=[212992->212992] S=[212992->212992] Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: UDPv4 link local (bound): [AF_INET][undef]:1194 Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: UDPv4 link remote: [AF_UNSPEC] Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: chroot to '/etc/openvpn/chroot/frax1021dckr1110/' and cd to '/' succeeded Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: GID set to nogroup Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: UID set to nobody Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: MULTI: multi_init called, r=256 v=256 Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: IFCONFIG POOL IPv4: base=172.16.21.2 size=253 Oct 10 05:54:14 frax1021dckr1110 openvpn[61734]: Initialization Sequence CompletedSo the pool starts from 172.16.21.2 because 172.16.21.1 is reserved for the local IP, but the client is instructed via CCD to get 172.16.21.10 which is shown below:
Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI_sva: pool returned IPv4=172.16.21.2, IPv6=(Not enabled) Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301/:63268 OPTIONS IMPORT: reading client specific options from: ccd/sofn1010fw301 Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: Learn: 172.16.21.10 -> sofn1010fw301:63268 Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: primary virtual IP for sofn1010fw301:63268: 172.16.21.10 Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: internal route 192.168.0.0/24 -> sofn1010fw301:63268 Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 MULTI: Learn: 192.168.0.0/24 -> sofn1010fw301:63268 Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 Data Channel: using negotiated cipher 'AES-256-GCM' Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key Oct 10 05:54:51 frax1021dckr1110 openvpn[61734]: sofn1010fw301:63268 SENT CONTROL [sofn1010fw301]: 'PUSH_REPLY,route-gateway 172.16.21.1,topology subnet,ping 10,ping-restart 60,route 10.0.1.10 255.255.255.0,ifconfig 172.16.21.10 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)Great 172.16.21.10 is assigned to the client, but see the local kernel routing table:
root@frax1021dckr1110:~# route -n |grep 192.168 192.168.0.0 172.16.21.2 255.255.255.0 UG 0 0 0 tun1 root@frax1021dckr1110:~#Why 172.16.21.2, I don't have 172.16.21.2 anywhere..
At least it works, so I'm asking just for clarity :-) -
OpenVPN always uses the first IP in the tunnel subnet to route traffic to like that. But usually at the server end. I expect to see .1 there at the client side if the server is configured as that. It seems to be passing that as a gateway.
It doesn't really matter since it's actually the OpenVPN daemon that decides where to route it to. The kernel route only has to get it to the daemon.
You should change the local subnet you are pushing to 10.0.1.0/24 instead of .10 to be 'correct'. Some clients might reject that as an invalid route.
Steve
-
You should change the local subnet you are pushing to 10.0.1.0/24 instead of .10 to be 'correct'. Some clients might reject that as an invalid route.
Steve
Oo I see what you are speaking about:
push "route 10.0.1.10 255.255.255.0"It should be 10.0.0.0 255.255.255.0, I even haven't noticed that I wrote it in this way..
-
It should be: 10.0.1.0 255.255.255.0
-
@stephenw10 said in pfSense -> OpenVPN Ubuntu site-to-site throubleshooting help:
It should be: 10.0.1.0 255.255.255.0
Of course :-)))
Sorry about that, I have really fast fingers =]]
