VPN Stops Working after 12 hours

g18c

Dear All, I have a few sites but cant even get one VPN link working yet. I have used OpenVPN in server mode in the past with custom configs for different clients and it worked very well. I am trying to repeat the same with a number of pfsense boxes but having big problems.

IP addresses for the sites are as follows:

Site 1: 192.168.2.1/24 (Server HQ site)
Site 2: 192.168.10.1/24 (spoke site)
Site 3: 192.168.20.1/24 (spoke site)
… and so on

In the logs the client from site 1 is connecting:

Mar 1 11:17:32 openvpn[11702]: TUN/TAP device /dev/tun0 opened 
Mar 1 11:17:32 openvpn[11702]: /sbin/ifconfig tun0 172.16.1.1 172.16.1.2 mtu 1500 netmask 255.255.255.255 up 
Mar 1 11:17:32 openvpn[11702]: /etc/rc.filter_configure tun0 1500 1542 172.16.1.1 172.16.1.2 init 
Mar 1 11:17:34 openvpn[11729]: UDPv4 link local (bound): [undef]:1194 
Mar 1 11:17:34 openvpn[11729]: UDPv4 link remote: [undef] 
Mar 1 11:17:34 openvpn[11729]: Initialization Sequence Completed 
Mar 1 11:18:28 openvpn[11729]: 83.110.27.14:1194 Re-using SSL/TLS context 
Mar 1 11:18:28 openvpn[11729]: 83.110.27.14:1194 LZO compression initialized 
Mar 1 11:18:32 openvpn[11729]: 83.110.27.14:1194 [router.domain.com] Peer Connection Initiated with 83.110.27.14:1194

The routing log does not show any route to 192.168.10.0/24

The only routes listed are:
172.16.1/24 172.16.1.2 UGS 0 184 1500 tun0
172.16.1.2 172.16.1.1 UH 1 0 1500 tun0

Please note i have set a client-specific-connfiguration for the common name the router.domain.com.

The custom option i have is: iroute 192.168.10.0 255.255.255.0

On the OpenVPN server i have the following set:

Assume Dynamic IPs
Address pool: 172.16.1.0/24
Local network: 192.168.2.0/24
Remote network: this is blank as i will have multiple "remote networks"
Client to Client VPN: not set
Auth method: PKI with all certs generated for server, ca and clients

Is iroute supported by pfsense? I have used this in the past and this automatically added a route to 192.168.10.0/24 to go via the tunnel whenever a client connected.

I cant ping the remote client nor the gateway ip, neither 172.16.1.1 or 172.16.1.2… not sure whats going on? I havent got any rules in my firewall to block or allow otherwise, i dont think the firewall can block traffic over the tun adapter can it? All traffic is allowed out by default.

I really appreciate any pointers anyone can give me since i am quite stuck!

Many thanks,

Chris

g18c

Eeeek my bad!

I added the custom lines in the server:

push "route 192.168.2.0 255.255.255.0"; route 192.168.10.0 255.255.255.0

And i am now cooking with gas, works proper good now ;D

Thnx

g18c

Hello again, unfortunatly it seems the link is ok then after a while dies. Rebooting either router either end does not solve this problem.

I have a number of sites, 192.168.2.0/24 is the main HQ, 192.168.10.0/24 is site 1, 192.168.20.0/24 is site 2. I need to push the routes for all VPN subnets in the spoke configuration so i issue the following custom commands on the server config:

push "route 192.168.2.0 255.255.255.0";
push "route 192.168.10.0 255.255.255.0";
route 192.168.10.0 255.255.255.0;
push "route 192.168.20.0 255.255.255.0";
route 192.168.20.0 255.255.255.0;

Now the weird thing is this is ok for a good while, say 12 to 24 hours… then all of a sudden the pings will stop between subnets. I reboot and i can see from the logs that the site 1 is connecting ok, however pings are dead and there is no site connectivity.

I then change the config to the following:

push "route 192.168.2.0 255.255.255.0";
route 192.168.10.0 255.255.255.0;

I hit save then after a while the link comes up good and i can ping the remote site pfsense router on 192.168.10.1.

Again i can guarentee as i have seen before after 12 hours it will stop working. Like today if i change my config back to :

push "route 192.168.2.0 255.255.255.0";
push "route 192.168.10.0 255.255.255.0";
route 192.168.10.0 255.255.255.0;
push "route 192.168.20.0 255.255.255.0";
route 192.168.20.0 255.255.255.0;

It works, i did this just now and its ok!! Has anyone seen this before? This is a real shame because i need pfsense functionality however i cant operate like this its driving me nuts. Im pretty sure its a bug because why else would it work for a while then die after 12 hours?

I have been running 1.2 RC4 Embedded and recently upgraded to 1.2 RELEASE Embedded, same problem with both. All other functions are working fine.

Thanks in advance,

Chris

GruensFroeschli

Are your clients on a dynamic IP?
What does the log show, when you loose the link?

g18c

Yep, clients are on dynamic ip on adsl connections. HQ is on a static ip.

No messages in the clients or openvpn server logs, it seems as if the OpenVPN connection is still up but pings just stop working, apart from below which is from the site1 box (192.168.10.0/24):

Mar 6 07:04:44 openvpn[369]: OpenVPN 2.0.6 i386-portbld-freebsd6.2 [SSL] [LZO] built on Sep 13 2007
Mar 6 07:04:44 openvpn[369]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Mar 6 07:04:44 openvpn[369]: WARNING: file '/var/etc/openvpn_client0.key' is group or others accessible
Mar 6 07:04:44 openvpn[369]: LZO compression initialized
Mar 6 07:04:44 openvpn[370]: UDPv4 link local (bound): [undef]:1194
Mar 6 07:04:44 openvpn[370]: UDPv4 link remote: x.x.x.x:1194
Mar 6 07:04:55 openvpn[370]: [router-hq.rotaryhumm.com] Peer Connection Initiated with x.x.x.x:1194
Mar 6 07:04:56 openvpn[370]: gw 169.254.100.2
Mar 6 07:04:56 openvpn[370]: TUN/TAP device /dev/tun0 opened
Mar 6 07:04:56 openvpn[370]: /sbin/ifconfig tun0 172.16.1.6 172.16.1.5 mtu 1500 netmask 255.255.255.255 up
Mar 6 07:04:56 openvpn[370]: /etc/rc.filter_configure tun0 1500 1542 172.16.1.6 172.16.1.5 init
Mar 6 07:04:57 openvpn[370]: ERROR: FreeBSD route add command failed: shell command exited with error status: 1
Mar 6 07:04:57 openvpn[370]: Initialization Sequence Completed

I presume the route add command failed is due to it the site 1 client (192.68.10.0/24) getting pushed the route 192.168.10.0/24 however this needs to be sent so all clients know they can connect to this network via the openvpn tunnel as per the hub/spoke setup.

The routing table on the HQ pfsense box is:

172.16.1/24 172.16.1.2 UGS 0 56 1500 tun0
172.16.1.2 172.16.1.1 UH 3 0 1500 tun0
192.168.10 172.16.1.2 UGS 0 14191 1500 tun0
192.168.20 172.16.1.2 UGS 0 4036 1500 tun0

The above seems correct.

On the client pfsense box (site 1) the routing table is:

default 169.254.10.1 UGS 0 162 1500 rl3
127.0.0.1 127.0.0.1 UH 0 0 16384 lo0
169.254.10/30 link#4 UC 0 0 1500 rl3
172.16.1/24 172.16.1.5 UGS 0 0 1500 tun0
172.16.1.5 172.16.1.6 UH 3 0 1500 tun0
192.168.2 172.16.1.5 UGS 0 16 1500 tun0
192.168.10 link#1 UC 0 0 1500 rl0
192.168.20 172.16.1.5 UGS 0 0 1500 tun0

Again, to me the above seems correct also! The OpenVPN link is definetly coming up as i can see that in the client and server logs.

The strangest thing is the config works, stops working, gets changed & works, returned to original config and works again! I tried again repeating this process and now its not working. Im pulling my hair out, have run OpenVPN before and not had as many problems as this.

Any ideas much appreciated,

Thanks

GruensFroeschli

Could you post the server and the client configs?
It could be that when a client changes his IP/port/whatever the link stops working.

Btw: are you using a PKI? It seems to me like this because of: Mar 6 07:04:56 openvpn[370]: /etc/rc.filter_configure tun0 1500 1542 172.16.1.6 172.16.1.5 init

If you are using site-to-site better use a shared key and not a PKI.

With a PKI you would need client specific iroute command and just normal /push's and /route's

g18c

Hi, dont have the setup xml files to hand as im at home now but in summary:

Server has local network set to 192.168.2.0/24.
Dynamic DHCP clients are enabled.
Addresses are assigned to clients 172.16.1.0/24
No remote address has been entered (this is since is a multi-site to site config)
The following routes are used:

push "route 192.168.2.0 255.255.255.0";
push "route 192.168.10.0 255.255.255.0";
route 192.168.10.0 255.255.255.0;
push "route 192.168.20.0 255.255.255.0";
route 192.168.20.0 255.255.255.0;

Under the custom client config for the server box i have entered the CN name of the site 1 pfsense boxes certificate and under that i have entered:

iroute 192.168.10.0 255.255.255.0

i understand this to mean that this tells openvpn internally that 192.168.10.0 sits behind this client. The route in the main config of 192.168.10.0 is to direct traffic to the tun0 interface and then the internal routing of openvpn can apply.

I believe i do need to run openvpn in PKI as i would not be able to assign specific subnets to specific clients like the above scenario of using iroute. From my perhaps blinkered understanding i dont think this is possible with shared keys? Please see this link: http://openvpn.net/archive/openvpn-users/2005-02/msg00261.html

Thanks so much for the time so far.

GruensFroeschli

You can find the openVPN configs in the path: /var/etc/
(just use the command line to less the files)

Regarding PKI vs. SKI
What you are doing right now should work. But in my opinion the roadwarriors should be separated from the site-to-site links.
–> running multiple instances of OpenVPN with different tasks.
PKI for RoadWarriors
SKI for site-to-site

g18c

Hi, setup a Debian box at home and got in first time no problem as a client. Very strange.

Config file is as follows:

writepid /var/run/openvpn_server0.pid
#user nobody
#group nobody
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
dev tun
proto udp
cipher BF-CBC
up /etc/rc.filter_configure
down /etc/rc.filter_configure
client-to-client
server 172.16.1.0 255.255.255.0
client-config-dir /var/etc/openvpn_csc
push "route 192.168.2.0 255.255.255.0"
lport 1194
push "dhcp-option DNS 192.168.2.10"
push "dhcp-option WINS 192.168.2.10"
ca /var/etc/openvpn_server0.ca
cert /var/etc/openvpn_server0.cert
key /var/etc/openvpn_server0.key
dh /var/etc/openvpn_server0.dh
comp-lzo
persist-remote-ip
float
push "route 192.168.2.0 255.255.255.0"
route 192.168.10.0 255.255.255.0
push "route 192.168.20.0 255.255.255.0"
route 192.168.20.0 255.255.255.0

I checked the routing table of my client and it seems ok.

For the site to site link i believe we have to use PKI and have no other choice, this is borne from the fact OpenVPN needs a way of identifying what subnet lies behind what client… having the CN name of the clients certificate and identifying a client against this gives us this ability. So i dont think the multi-site to site link would work with static keys? I do agree i will separate the road warriors from site to site links, i will do this by running a new instance on a separate port should the need arise for road warriors.

Anyways, i will setup the same config with a debian box at both ends and see if i get any joy. Debian Sarge -> pfSense seems to work.

Will put on ping test for a while and see if i get any packet loss.

Regards,

Chris

GruensFroeschli

For the site to site link i believe we have to use PKI and have no other choice, this is borne from the fact OpenVPN needs a way of identifying what subnet lies behind what client… having the CN name of the clients certificate and identifying a client against this gives us this ability. So i dont think the multi-site to site link would work with static keys? I do agree i will separate the road warriors from site to site links, i will do this by running a new instance on a separate port should the need arise for road warriors.

I disagree. You can setup a SKI for every site-to-site connection.
Given the administration effort is a bit bigger but troubleshooting gets easier.
In your case where you have 2 offsites that would mean 2 SKI's for the site-to-site and 1 PKI for the Roadwarriors
–> 3 Separate Server instances.

OpenVPN can distinguish the different site-to-sites because like this, the different connections are on different virtual interfaces.

Anyway: i think i know why your config is not working:
persist-remote-ip

–persist-remote-ip
Preserve most recently authenticated remote IP address and port number across SIGUSR1 or --ping-restart restarts.

SIGUSR1
Like SIGHUP, except don't re-read configuration file, and possibly don't close and reopen TUN/TAP device, re-read key files, preserve local IP address/port, or preserve most recently authenticated remote IP address/port based on –persist-tun, --persist-key, --persist-local-ip, and --persist-remote-ip options respectively (see above).

This signal may also be internally generated by a timeout condition, governed by the --ping-restart option.

This signal, when combined with --persist-remote-ip, may be sent when the underlying parameters of the host's network interface change such as when the host is a DHCP client and is assigned a new IP address. See --ipchange above for more information.

I remember there was once a hread about something similar.
It can well be that this is an error in the auto-generation of the config file.

Can the developer who did the OpenVPN-stuff take a look at that?

g18c

Thnx. I didnt touch the config file all settings were done through the web interface. ???

Thanks again for the help, lets hope this is a bug becuase the debian box has been conntect rock solid all day to the pfsense HQ box.

Cheers

sullrich

If this was a bug I am sure others would have reported it by now….

GruensFroeschli

I'll try to recreate this (as soon as i find the time….)

g18c

Well i can confirm Debian box as OpenVPN client to pfsense server has been up solid for over 24 hours now no problem. Link is still solid. This is probably something to do with the client, i will post the client configs tomorrow.

I appreciate what you say about bugs and reporting, i am sure it would have been reported also and maybe this is something i have done wrong but one things for sure i have seen weird stuff like this before like with OpenWRT and netfilter working ok with NAT redirects for 24 hours and then randomly remapping to a different port for no reason!

Thanks for the time really appreciate it.

Regards,

Chris