OpenVPN on 2.3.2 "Exiting due to fatal error"

tsmalmbe · Sep 27, 2016, 10:16 PM

If I reboot pfsense, the OpenVPN server starts. If I make a change and save the server config, I get an error. The same goes for the client. I also never get a OpenVPN connection to work, but this "Exiting due to fatal error" behavior makes it impossible to assess whether this error is the problem, part of the problem or something completely different. The firewall passes the traffic to UDP:1194 succesfully. My connection problem might be a UDP-double-nat issue perhaps, but again, impossible to debug when the OpenVPN server and interface goes down all the time. Why is it exiting with a "fatal error"?

I have copied my configuration exactly (well, not the exact IP-addresses of course) from a working pfsense 2.3.2 where this setup works just fine, no errors. Both the client and the server configurations. Just to let you know, the working setup I have copied (or rather mimicked) the configuration from runs on VULTR, the other nonworking one is on OpenStack.

This is my OpenVPN log:
Sep 28 01:04:13 openvpn 45407 Exiting due to fatal error
Sep 28 01:04:13 openvpn 45407 FreeBSD ifconfig failed: external program exited with error status: 1
Sep 28 01:04:13 openvpn 45407 /sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up
Sep 28 01:04:13 openvpn 45407 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Sep 28 01:04:13 openvpn 45407 TUN/TAP device /dev/tun1 opened
Sep 28 01:04:13 openvpn 45407 TUN/TAP device ovpns1 exists previously, keep at program end
Sep 28 01:04:13 openvpn 45407 Socket Buffers: R=[42080->42080] S=[57344->57344]
Sep 28 01:04:13 openvpn 45407 TLS-Auth MTU parms [ L:1570 D:1172 EF:78 EB:0 ET:0 EL:3 ]
Sep 28 01:04:13 openvpn 45407 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
Sep 28 01:04:13 openvpn 45407 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
Sep 28 01:04:13 openvpn 45407 Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file
Sep 28 01:04:13 openvpn 45407 Diffie-Hellman initialized with 2048 bit key
Sep 28 01:04:13 openvpn 45407 Initializing OpenSSL support for engine 'cryptodev'

This is from the general log:
Sep 28 01:04:14 xinetd 17389 Swapping defaults
Sep 28 01:04:14 xinetd 17389 Starting reconfiguration
Sep 28 01:04:13 kernel ovpns1: link state changed to DOWN
Sep 28 01:04:13 check_reload_status Reloading filter
Sep 28 01:04:13 kernel ifa_add_loopback_route: insertion failed: 17
Sep 28 01:04:13 kernel ovpns1: link state changed to UP
Sep 28 01:04:13 check_reload_status Reloading filter
Sep 28 01:04:13 check_reload_status Syncing firewall
Sep 28 01:04:13 kernel ovpns1: link state changed to DOWN

tsmalmbe · Sep 27, 2016, 10:27 PM

Also

/sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up
ifconfig: ioctl (SIOCAIFADDR): Address already in use

tsmalmbe · Sep 28, 2016, 6:35 AM

The only addons are OpenVPN client export and HAProxy (HAProxy is disabled currently).

tsmalmbe · Sep 28, 2016, 6:16 PM

Anyone? Can I do something to get more logs? I'm not doing much out of theordinary - if anything - so what could be the issue?

KOM · Sep 28, 2016, 6:58 PM

Can I do something to get more logs?

Increase your Verbosity level in VPN - OpenVPN - Servers - Advanced Configuration.

tsmalmbe · Sep 28, 2016, 8:40 PM

Yes, I pulled that to eleven and basically got nothing more.

tsmalmbe · Sep 28, 2016, 8:42 PM

Sep 28 23:41:21 openvpn 78304 Exiting due to fatal error
Sep 28 23:41:21 openvpn 78304 FreeBSD ifconfig failed: external program exited with error status: 1
Sep 28 23:41:21 openvpn 78304 /sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up
Sep 28 23:41:21 openvpn 78304 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Sep 28 23:41:21 openvpn 78304 TUN/TAP device /dev/tun1 opened
Sep 28 23:41:21 openvpn 78304 TUN/TAP device ovpns1 exists previously, keep at program end
Sep 28 23:41:21 openvpn 78304 Socket Buffers: R=[42080->42080] S=[57344->57344]
Sep 28 23:41:21 openvpn 78304 MTU DYNAMIC mtu=1450, flags=2, 1570 -> 1450
Sep 28 23:41:21 openvpn 78304 TLS-Auth MTU parms [ L:1570 D:1172 EF:78 EB:0 ET:0 EL:3 ]
Sep 28 23:41:21 openvpn 78304 crypto_adjust_frame_parameters: Adjusting frame parameters for crypto by 40 bytes
Sep 28 23:41:21 openvpn 78304 crypto_adjust_frame_parameters: Adjusting frame parameters for crypto by 68 bytes
Sep 28 23:41:21 openvpn 78304 Incoming Control Channel Authentication: HMAC size=32 block_size=32
Sep 28 23:41:21 openvpn 78304 Incoming Control Channel Authentication: HMAC KEY: cb932dba 4b6549a4 d1b4c692 3ebc625f e32956b0 c8b840fe 96f67504 fe8341ad
Sep 28 23:41:21 openvpn 78304 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
Sep 28 23:41:21 openvpn 78304 Outgoing Control Channel Authentication: HMAC size=32 block_size=32
Sep 28 23:41:21 openvpn 78304 Outgoing Control Channel Authentication: HMAC KEY: 0f4bdecd adc73bd7 1c54931a da4f3522 bcad3008 6af1fd93 3c35bc23 a141d073
Sep 28 23:41:21 openvpn 78304 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
Sep 28 23:41:21 openvpn 78304 Control Channel Authentication: using '/var/etc/openvpn/server1.tls-auth' as a OpenVPN static key file
Sep 28 23:41:21 openvpn 78304 PRNG init md=SHA1 size=36
Sep 28 23:41:21 openvpn 78304 Diffie-Hellman initialized with 2048 bit key
Sep 28 23:41:21 openvpn 78304 Initializing OpenSSL support for engine 'cryptodev'
Sep 28 23:41:21 openvpn 78304 NOTE: the current –script-security setting may allow this configuration to call user-defined scripts
Sep 28 23:41:21 openvpn 78304 MANAGEMENT: unix domain socket listening on /var/etc/openvpn/server1.sock
Sep 28 23:41:21 openvpn 78304 PO_INIT maxevents=1 flags=0x00000002
Sep 28 23:41:21 openvpn 78021 library versions: OpenSSL 1.0.1s-freebsd 1 Mar 2016, LZO 2.09
Sep 28 23:41:21 openvpn 78021 OpenVPN 2.3.11 amd64-portbld-freebsd10.3 [SSL (OpenSSL)] [LZO] [MH] [IPv6] built on Jul 19 2016
Sep 28 23:41:21 openvpn 78021 auth_user_pass_file = '[UNDEF]'
Sep 28 23:41:21 openvpn 78021 pull = DISABLED
Sep 28 23:41:21 openvpn 78021 client = DISABLED
Sep 28 23:41:21 openvpn 78021 port_share_port = 0
Sep 28 23:41:21 openvpn 78021 port_share_host = '[UNDEF]'
Sep 28 23:41:21 openvpn 78021 auth_user_pass_verify_script_via_file = DISABLED
Sep 28 23:41:21 openvpn 78021 auth_user_pass_verify_script = '[UNDEF]'
Sep 28 23:41:21 openvpn 78021 max_routes_per_client = 256
Sep 28 23:41:21 openvpn 78021 max_clients = 1024
Sep 28 23:41:21 openvpn 78021 cf_per = 0
Sep 28 23:41:21 openvpn 78021 cf_max = 0
Sep 28 23:41:21 openvpn 78021 duplicate_cn = DISABLED
Sep 28 23:41:21 openvpn 78021 enable_c2c = DISABLED
Sep 28 23:41:21 openvpn 78021 push_ifconfig_ipv6_remote = ::
Sep 28 23:41:21 openvpn 78021 push_ifconfig_ipv6_local = ::/0
Sep 28 23:41:21 openvpn 78021 push_ifconfig_ipv6_defined = DISABLED
Sep 28 23:41:21 openvpn 78021 push_ifconfig_remote_netmask = 0.0.0.0
Sep 28 23:41:21 openvpn 78021 push_ifconfig_local = 0.0.0.0
Sep 28 23:41:21 openvpn 78021 push_ifconfig_defined = DISABLED
Sep 28 23:41:21 openvpn 78021 tmp_dir = '/tmp'
Sep 28 23:41:21 openvpn 78021 ccd_exclusive = DISABLED
Sep 28 23:41:21 openvpn 78021 client_config_dir = '/var/etc/openvpn-csc/server1'
Sep 28 23:41:21 openvpn 78021 client_disconnect_script = '[UNDEF]'
Sep 28 23:41:21 openvpn 78021 learn_address_script = '[UNDEF]'

KOM · Sep 29, 2016, 1:06 PM

I'm no expert by far, but I'm wondering if the following command is what's causing the error:

Sep 28 23:41:21 openvpn 78304 FreeBSD ifconfig failed: external program exited with error status: 1
Sep 28 23:41:21 openvpn 78304 /sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up

I wonder if your problem is caused by a misconfiguration which can't be processed successfully by ifconfig. Can you run that command by itself without error?

heper · Sep 29, 2016, 4:32 PM

you get that error for all kinds of problems such as:

that route already exists (check routing table when openvpn service is disabled)
ip already in use ( don't set that ip on an interface, not even a vpn interface)

try to run the /sbin/ifconfig command manually, perhaps you'll get more debugging info that way

tsmalmbe · Sep 29, 2016, 7:57 PM

This I already tested previously

/sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up
ifconfig: ioctl (SIOCAIFADDR): Address already in use

Yes, the address is in use - somewhere - I have NOT used that address anywhere. The first time the address was ever in touch with the firewall was when I added a OpenVPN server.

tsmalmbe · Sep 29, 2016, 8:05 PM

This all done postboot immediately.

Disable the OpenVPN server, keep the client:

route -rn

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 172.17.255.254 UGS vtnet0
10.11.12.0/24 link#3 U vtnet2
10.11.12.5 link#3 UHS lo0
10.172.0.1 link#9 UH ovpnc2
127.0.0.1 link#7 UH lo0
172.17.0.0/16 link#1 U vtnet0
172.17.0.1 fa:16:3e:b6:f4:b8 UHS vtnet0
172.17.0.16 link#1 UHS lo0
192.168.168.0/24 link#2 U vtnet1
192.168.168.5 link#2 UHS lo0

Enable the OpenVPN server

route -rn

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 172.17.255.254 UGS vtnet0
10.11.12.0/24 link#3 U vtnet2
10.11.12.5 link#3 UHS lo0
10.172.0.1 link#9 UH ovpnc2
10.172.0.2 link#8 UH ovpns1
127.0.0.1 link#7 UH lo0
172.17.0.0/16 link#1 U vtnet0
172.17.0.1 fa:16:3e:b6:f4:b8 UHS vtnet0
172.17.0.16 link#1 UHS lo0
192.168.168.0/24 link#2 U vtnet1
192.168.168.5 link#2 UHS lo0

Logs:

Sep 29 23:01:00 openvpn 22695 Exiting due to fatal error
Sep 29 23:01:00 openvpn 22695 FreeBSD ifconfig failed: external program exited with error status: 1
Sep 29 23:01:00 openvpn 22695 /sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up
Sep 29 23:01:00 openvpn 22695 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Sep 29 23:01:00 openvpn 22695 TUN/TAP device /dev/tun1 opened
Sep 29 23:01:00 openvpn 22695 TUN/TAP device ovpns1 exists previously, keep at program end

Disable the OpenVPN server AND delete the client

route -rn

Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 172.17.255.254 UGS vtnet0
10.11.12.0/24 link#3 U vtnet2
10.11.12.5 link#3 UHS lo0
10.172.0.1 link#9 UH tun2
10.172.0.2 link#8 UH ovpns1
127.0.0.1 link#7 UH lo0
172.17.0.0/16 link#1 U vtnet0
172.17.0.1 fa:16:3e:b6:f4:b8 UHS vtnet0
172.17.0.16 link#1 UHS lo0
192.168.168.0/24 link#2 U vtnet1
192.168.168.5 link#2 UHS lo0

and then ifconfig

ovpns1: flags=8010 <pointopoint,multicast>metric 0 mtu 1500
options=80000 <linkstate>nd6 options=21 <performnud,auto_linklocal>tun2: flags=8010 <pointopoint,multicast>metric 0 mtu 1500
options=80000 <linkstate>nd6 options=21 <performnud,auto_linklocal></performnud,auto_linklocal></linkstate></pointopoint,multicast></performnud,auto_linklocal></linkstate></pointopoint,multicast>

tsmalmbe · Sep 29, 2016, 8:06 PM

@KOM:

I'm no expert by far, but I'm wondering if the following command is what's causing the error:

Sep 28 23:41:21 openvpn 78304 FreeBSD ifconfig failed: external program exited with error status: 1
Sep 28 23:41:21 openvpn 78304 /sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up

I wonder if your problem is caused by a misconfiguration which can't be processed successfully by ifconfig. Can you run that command by itself without error?

Already tested previously, do not understand the result

/sbin/ifconfig ovpns1 10.172.0.1 10.172.0.2 mtu 1500 netmask 255.255.255.0 up
ifconfig: ioctl (SIOCAIFADDR): Address already in use

tsmalmbe · Sep 29, 2016, 8:13 PM

Any idea what this post implies?

http://peter-mao.blogspot.fi/2015/09/freebsd-openvpn-bug.html

"Mon Sep 7 11:28:19 2015 /sbin/ifconfig tun0 172.23.238.249 172.23.238.250 mtu 1500 netmask 172.23.238.250 up
ifconfig: ioctl (SIOCAIFADDR): Address already in use

Mon Sep 7 11:25:45 2015 /sbin/ifconfig tun0 172.23.238.249 172.23.238.250 mtu 1500 netmask 172.23.238.250 up
ifconfig: ioctl (SIOCAIFADDR): File exists

solution net.link.ether.inet.useloopback=0 "

heper · Sep 29, 2016, 9:39 PM

it appears you are running an openvpn server & an openvpn client on the same subnet …. that is more then likely what the ifconfig error is about.

pick a different subnet for either of them

tsmalmbe · Sep 29, 2016, 9:41 PM

OK, you have got me there. Because this is how I have configured all my OpenVPNs always everywhere I thought this is the correct way to do it - to make sure the client connecting to the server is using the same subnet to be able to connect? This is not so?

tsmalmbe · Sep 29, 2016, 9:44 PM

My server has a "tunnel network" defined, my client has that field empty. So the server assigns the tunnel network automatically to the client connected? Is this not intended behavior?

heper · Sep 30, 2016, 6:40 AM

clients receive their tunnel network when connecting to a server.
servers have a tunnel network defined in the config.

in no circumstance should any interface have the same subnet and/or ip.

Because this is how I have configured all my OpenVPNs always everywhere I thought this is the correct way to do it - to make sure the client connecting to the server is using the same subnet to be able to connect? This is not so?

hmm? i'm unable to understand what you want todo.
clients connect to a server instance. there is no point in running both a client & server instance and connecting them to themselfs ????

a tunnel network is a transit network. It honestly doesn't matter what subnet you use, as long as it doesn't conflict with others.

tsmalmbe · Sep 30, 2016, 10:39 AM

Summa summarum. The server component has a transit/tunnel network defined, the client configuration does not. However that translates into interface configurations, that I do not know. In no place have I defined the same network twice in any configuration.

heper · Sep 30, 2016, 12:23 PM

Where does the client connect to?

tsmalmbe · Sep 30, 2016, 12:35 PM

@heper:

Where does the client connect to?

I have defined one OpenVPN server and one Client. The client connects to the server.