OpenConnect + split routing
-
I am attempting to achieve the following setup:
- All traffic, by default, goes through WAN
- OpenConnect VPN is always up
- Traffic only routes to VPN interface on specified nets, the most important being 10.0.0.0/8
- There are two sets of DNS servers: for WAN and VPN
Reading a handful of guides, I've done the following:
- Install Shellcmd and run an earlyshellcmd,```
/sbin/ifconfig tun create; /sbin/ifconfig tun0 name vpn0
- Modify vpnc-script to rename tun0 to vpn0 to get past the pfSense GUI hiding tun* interfaces - Set the WAN-specific DNS servers to use the WAN interface, but leave the interface field blank for VPN because it's "directly connected" - VPN has an assigned interface, and is set to static IPv4 with custom gateway, static IPv6 with no gateway - The VPN GW is not set as default. The WAN GW is set as default. - There are several dozen static routes set to be routed through the VPN GW. I haven't written a cron job to start openconnect yet; for now I'm starting it manually. An example run looks like:
echo "$password" |
openconnect
--pid-file=/var/run/openconnect.pid
--non-inter
--user="$username"
--passwd-on-stdin
--cafile=/usr/share/openconnect/myserver.pem
--interface="$tunif"
--script=/usr/share/openconnect/vpnc-script-latest
--timestamp
--no-proxy
--pfs
--reconnect-timeout 60
mycorpvpn.com[2017-08-03 15:42:40] POST https://mycorpvpn.com/
...- env
- sort
BLOCKSIZE=K
CISCO_CSTP_OPTIONS=X-CSTP-Version=1
CISCO_DEF_DOMAIN=mycorpvpn.com
CISCO_PROXY_PAC=http://wpad/wpad.dat
CLICOLOR=true
FTP_PASSIVE_MODE=YES
GROUP=wheel
HOME=/root
HOST=puffball.house.lan
HOSTTYPE=FreeBSD
INTERNAL_IP4_ADDRESS=10.65.164.129
INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
INTERNAL_IP4_MTU=1300
INTERNAL_IP4_NETADDR=10.65.160.0
INTERNAL_IP4_NETMASK=255.255.224.0
INTERNAL_IP4_NETMASKLEN=19
INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
LOGNAME=admin
LSCOLORS=exfxcxdxbxegedabagacad
MACHTYPE=x86_64
MAIL=/var/mail/admin
OSTYPE=FreeBSD
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
PWD=/usr/share/openconnect
REMOTEHOST=192.168.1.100
SHELL=/etc/rc.initial
SHLVL=1
SSH_CLIENT=192.168.1.100 55906 22
SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
SSH_TTY=/dev/pts/0
TERM=xterm-256color
USER=admin
VENDOR=amd
VPNGATEWAY=129.xxx.xxx.247
X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
X-CSTP-Address=10.65.164.129
X-CSTP-Client-Bypass-Protocol=false
X-CSTP-DNS=10.209.76.197
X-CSTP-DNS=10.209.76.198
X-CSTP-DPD=30
X-CSTP-Default-Domain=mycorp.com
X-CSTP-Disable-Always-On-VPN=false
X-CSTP-Disconnected-Timeout=21600
X-CSTP-Hostname=vpn.mycorp.com
X-CSTP-Idle-Timeout=21600
X-CSTP-Keep=true
X-CSTP-Keepalive=20
X-CSTP-Lease-Duration=86400
X-CSTP-MSIE-Proxy-Lockdown=false
X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
X-CSTP-MTU=1300
X-CSTP-Netmask=255.255.224.0
X-CSTP-Post-Auth-XML=
X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
X-CSTP-Quarantine=false
X-CSTP-Routing-Filtering-Ignore=false
X-CSTP-Session-Timeout=86400
X-CSTP-Smartcard-Removal-Disconnect=true
X-CSTP-TCP-Keepalive=true
X-CSTP-Tunnel-All-DNS=false
reason=pre-init - set -x
- PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
- uname -s
- OS=FreeBSD
- HOOKS_DIR=/etc/vpnc
- DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
- RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
- basename /usr/share/openconnect/vpnc-script-latest
- SCRIPTNAME=vpnc-script-latest
- [ ! -d /var/run/vpnc ]
- mkdir -p /var/run/vpnc
- [ -x /sbin/restorecon ]
- which ip
- grep ^/
- IPROUTE=''
- ifconfig --help
- grep BusyBox
- ifconfig_syntax_inet=inet
- [ FreeBSD = Linux ]
- ifconfig_syntax_ptp=''
- route_syntax_gw=''
- route_syntax_del=delete
- route_syntax_netmask=-netmask
- [ FreeBSD = SunOS ]
- route_syntax_interface=''
- ifconfig_syntax_ptpv6=''
- [ -r /etc/openwrt_release ]
- [ -x /usr/bin/busctl ]
- [ -x /sbin/resolvconf ]
- MODIFYRESOLVCONF=modify_resolvconf_manager
- RESTORERESOLVCONF=restore_resolvconf_manager
- [ -z '' ]
- TUNDEV=tun0
- OLDTUNDEV=tun0
- TUNDEV=vpn0
- [ -n '' ]
- AF_INET=2
- [ -z pre-init ]
- run_hooks pre-init
- HOOK=pre-init
- [ -d /etc/vpnc/pre-init.d ]
- do_pre_init
- [ FreeBSD = Linux ]
- [ FreeBSD = FreeBSD ]
- kldstat -q -m if_tun
- ifconfig tun0
ifconfig: interface tun0 does not exist - ifconfig tun0 create
ifconfig: SIOCIFCREATE2: File exists - ifconfig tun0 name vpn0
ifconfig: interface tun0 does not exist - exit 0
[2017-08-03 15:42:55] SIOCSIFMTU: Device not configured - env
- sort
BLOCKSIZE=K
CISCO_CSTP_OPTIONS=X-CSTP-Version=1
CISCO_DEF_DOMAIN=mycorp.com
CISCO_PROXY_PAC=http://wpad/wpad.dat
CLICOLOR=true
FTP_PASSIVE_MODE=YES
GROUP=wheel
HOME=/root
HOST=puffball.house.lan
HOSTTYPE=FreeBSD
INTERNAL_IP4_ADDRESS=10.65.164.129
INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
INTERNAL_IP4_MTU=1300
INTERNAL_IP4_NETADDR=10.65.160.0
INTERNAL_IP4_NETMASK=255.255.224.0
INTERNAL_IP4_NETMASKLEN=19
INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
LOGNAME=admin
LSCOLORS=exfxcxdxbxegedabagacad
MACHTYPE=x86_64
MAIL=/var/mail/admin
OSTYPE=FreeBSD
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
PWD=/usr/share/openconnect
REMOTEHOST=192.168.1.100
SHELL=/etc/rc.initial
SHLVL=1
SSH_CLIENT=192.168.1.100 55906 22
SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
SSH_TTY=/dev/pts/0
TERM=xterm-256color
TUNDEV=tun0
USER=admin
VENDOR=amd
VPNGATEWAY=129.xx.xx.247
X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
X-CSTP-Address=10.65.164.129
X-CSTP-Client-Bypass-Protocol=false
X-CSTP-DNS=10.209.76.197
X-CSTP-DNS=10.209.76.198
X-CSTP-DPD=30
X-CSTP-Default-Domain=mycorp.com
X-CSTP-Disable-Always-On-VPN=false
X-CSTP-Disconnected-Timeout=21600
X-CSTP-Hostname=vpn.mycorp.com
X-CSTP-Idle-Timeout=21600
X-CSTP-Keep=true
X-CSTP-Keepalive=20
X-CSTP-Lease-Duration=86400
X-CSTP-MSIE-Proxy-Lockdown=false
X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
X-CSTP-MTU=1300
X-CSTP-Netmask=255.255.224.0
X-CSTP-Post-Auth-XML=
X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
X-CSTP-Quarantine=false
X-CSTP-Routing-Filtering-Ignore=false
X-CSTP-Session-Timeout=86400
X-CSTP-Smartcard-Removal-Disconnect=true
X-CSTP-TCP-Keepalive=true
X-CSTP-Tunnel-All-DNS=false
reason=connect - set -x
- PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
- uname -s
- OS=FreeBSD
- HOOKS_DIR=/etc/vpnc
- DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
- RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
- basename /usr/share/openconnect/vpnc-script-latest
- SCRIPTNAME=vpnc-script-latest
- [ ! -d /var/run/vpnc ]
- which ip
- grep ^/
- IPROUTE=''
- ifconfig --help
- grep BusyBox
- ifconfig_syntax_inet=inet
- [ FreeBSD = Linux ]
- ifconfig_syntax_ptp=''
- route_syntax_gw=''
- route_syntax_del=delete
- route_syntax_netmask=-netmask
- [ FreeBSD = SunOS ]
- route_syntax_interface=''
- ifconfig_syntax_ptpv6=''
- [ -r /etc/openwrt_release ]
- [ -x /usr/bin/busctl ]
- [ -x /sbin/resolvconf ]
- MODIFYRESOLVCONF=modify_resolvconf_manager
- RESTORERESOLVCONF=restore_resolvconf_manager
- [ -z tun0 ]
- OLDTUNDEV=tun0
- TUNDEV=vpn0
- [ -n '' ]
- AF_INET=2
- [ -z connect ]
- run_hooks connect
- HOOK=connect
- [ -d /etc/vpnc/connect.d ]
- do_connect
- [ -n '' ]
- set_vpngateway_route
- get_default_gw
- netstat -r -n
- awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
- route add -host 129.xx.xx.247 192.168.0.1
add host 129.xx.xx.247: gateway 192.168.0.1 - do_ifconfig
- [ -n 1300 ]
- MTU=1300
- [ -z 1300 ]
- [ -n '' ]
- ifconfig vpn0 inet 10.65.164.129 10.65.164.129 netmask 255.255.255.255 mtu 1300 up
- [ -n 255.255.224.0 ]
- set_network_route 10.65.160.0 255.255.224.0 19
- NETWORK=10.65.160.0
- NETMASK=255.255.224.0
- NETMASKLEN=19
- del_network_route 10.65.160.0 255.255.224.0 19
- NETWORK=10.65.160.0
- NETMASK=255.255.224.0
- NETMASKLEN=19
- route delete -net 10.65.160.0 -netmask 255.255.224.0 10.65.164.129
route: writing to routing socket: No such process
delete net 10.65.160.0: gateway 10.65.164.129 fib 0: not in table - route add -net 10.65.160.0 -netmask 255.255.224.0 10.65.164.129
add net 10.65.160.0: gateway 10.65.164.129 - [ -n 2606:b400:2050:24:8000::8d ]
- [ -z 2606:b400:2050:24:8000::8d/64 ]
- [ -n 2606:b400:2050:24:8000::8d/64 ]
- [ -n '' ]
- ifconfig vpn0 inet6 2606:b400:2050:24:8000::8d/64 mtu 1300 up
- [ -n '' ]
- [ -n 10.65.164.129 ]
- set_default_route
- get_default_gw
- netstat -r -n
- awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
- DEFAULTGW=192.168.0.1
- echo 192.168.0.1
- route delete default 192.168.0.1
delete net default: gateway 192.168.0.1 - route add default 10.65.164.129
add net default: gateway 10.65.164.129 - [ -n '' ]
- [ -n 2606:b400:2050:24:8000::8d/64 -o -n 2606:b400:2050:24:8000::8d ]
- set_ipv6_default_route
- route add -inet6 default 2606:b400:2050:24:8000::8d
route: writing to routing socket: File exists
add net default: gateway 2606:b400:2050:24:8000::8d fib 0: route already in table - [ -n '10.209.76.198 10.209.76.197' ]
- modify_resolvconf_manager
- NEW_RESOLVCONF=''
- NEW_RESOLVCONF='
nameserver 10.209.76.198' - NEW_RESOLVCONF='
nameserver 10.209.76.198
nameserver 10.209.76.197' - [ -n mycorp.com ]
- NEW_RESOLVCONF='
nameserver 10.209.76.198
nameserver 10.209.76.197
domain mycorp.com' - echo '
nameserver 10.209.76.198
nameserver 10.209.76.197
domain mycorp.com' - /sbin/resolvconf -a vpn0
cp: /dev/null.bak: Operation not supported - run_hooks post-connect
- HOOK=post-connect
- [ -d /etc/vpnc/post-connect.d ]
- exit 0
[2017-08-03 15:42:55] Connected tun0 as 10.65.164.129 + 2606:b400:2050:24:8000::8d/64, using SSL
[2017-08-03 15:42:56] Established DTLS connection (using OpenSSL). Ciphersuite AES256-SHA.
^C[2017-08-03 15:46:27] Send BYE packet: Aborted by caller - env
- sort
BLOCKSIZE=K
CISCO_CSTP_OPTIONS=X-CSTP-Version=1
CISCO_DEF_DOMAIN=mycorp.com
CISCO_PROXY_PAC=http://wpad/wpad.dat
CLICOLOR=true
FTP_PASSIVE_MODE=YES
GROUP=wheel
HOME=/root
HOST=puffball.house.lan
HOSTTYPE=FreeBSD
INTERNAL_IP4_ADDRESS=10.65.164.129
INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
INTERNAL_IP4_MTU=1300
INTERNAL_IP4_NETADDR=10.65.160.0
INTERNAL_IP4_NETMASK=255.255.224.0
INTERNAL_IP4_NETMASKLEN=19
INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
LOGNAME=admin
LSCOLORS=exfxcxdxbxegedabagacad
MACHTYPE=x86_64
MAIL=/var/mail/admin
OSTYPE=FreeBSD
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
PWD=/usr/share/openconnect
REMOTEHOST=192.168.1.100
SHELL=/etc/rc.initial
SHLVL=1
SSH_CLIENT=192.168.1.100 55906 22
SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
SSH_TTY=/dev/pts/0
TERM=xterm-256color
TUNDEV=tun0
USER=admin
VENDOR=amd
VPNGATEWAY=129.xx.xx.247
X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
X-CSTP-Address=10.65.164.129
X-CSTP-Client-Bypass-Protocol=false
X-CSTP-DNS=10.209.76.197
X-CSTP-DNS=10.209.76.198
X-CSTP-DPD=30
X-CSTP-Default-Domain=mycorp.com
X-CSTP-Disable-Always-On-VPN=false
X-CSTP-Disconnected-Timeout=21600
X-CSTP-Hostname=vpn.mycorp.com
X-CSTP-Idle-Timeout=21600
X-CSTP-Keep=true
X-CSTP-Keepalive=20
X-CSTP-Lease-Duration=86400
X-CSTP-MSIE-Proxy-Lockdown=false
X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
X-CSTP-MTU=1300
X-CSTP-Netmask=255.255.224.0
X-CSTP-Post-Auth-XML=
X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
X-CSTP-Quarantine=false
X-CSTP-Routing-Filtering-Ignore=false
X-CSTP-Session-Timeout=86400
X-CSTP-Smartcard-Removal-Disconnect=true
X-CSTP-TCP-Keepalive=true
X-CSTP-Tunnel-All-DNS=false
reason=disconnect - set -x
- PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
- uname -s
- OS=FreeBSD
- HOOKS_DIR=/etc/vpnc
- DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
- RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
- basename /usr/share/openconnect/vpnc-script-latest
- SCRIPTNAME=vpnc-script-latest
- [ ! -d /var/run/vpnc ]
- which ip
- grep ^/
- IPROUTE=''
- ifconfig --help
- grep BusyBox
- ifconfig_syntax_inet=inet
- [ FreeBSD = Linux ]
- ifconfig_syntax_ptp=''
- route_syntax_gw=''
- route_syntax_del=delete
- route_syntax_netmask=-netmask
- [ FreeBSD = SunOS ]
- route_syntax_interface=''
- ifconfig_syntax_ptpv6=''
- [ -r /etc/openwrt_release ]
- [ -x /usr/bin/busctl ]
- [ -x /sbin/resolvconf ]
- MODIFYRESOLVCONF=modify_resolvconf_manager
- RESTORERESOLVCONF=restore_resolvconf_manager
- [ -z tun0 ]
- OLDTUNDEV=tun0
- TUNDEV=vpn0
- [ -n '' ]
- AF_INET=2
- [ -z disconnect ]
- run_hooks disconnect
- HOOK=disconnect
- [ -d /etc/vpnc/disconnect.d ]
- do_disconnect
- [ -n '' ]
- reset_default_route
- [ -s /var/run/vpnc/defaultroute ]
- get_default_gw
- netstat -r -n
- awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
- route delete default 10.65.164.129
delete net default: gateway 10.65.164.129 - cat /var/run/vpnc/defaultroute
- route add default 192.168.0.1
add net default: gateway 192.168.0.1 - rm -f -- /var/run/vpnc/defaultroute
- [ -n '' ]
- [ -n 2606:b400:2050:24:8000::8d/64 -o -n 2606:b400:2050:24:8000::8d ]
- reset_ipv6_default_route
- route delete -inet6 default 2606:b400:2050:24:8000::8d
delete net default: gateway 2606:b400:2050:24:8000::8d - :
- del_vpngateway_route
- get_default_gw
- netstat -r -n
- awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
- route delete -host 129.xx.xx.247 192.168.0.1
delete host 129.xx.xx.247: gateway 192.168.0.1 - [ -n '10.209.76.198 10.209.76.197' ]
- restore_resolvconf_manager
- /sbin/resolvconf -d vpn0
- [ -n '' ]
- [ -n 10.65.164.129 ]
- ifconfig vpn0 0.0.0.0
ifconfig: ioctl (SIOCAIFADDR): Destination address required - [ -n 2606:b400:2050:24:8000::8d ]
- [ -z 2606:b400:2050:24:8000::8d/64 ]
- [ -n 2606:b400:2050:24:8000::8d/64 ]
- ifconfig vpn0 inet6 del 2606:b400:2050:24:8000::8d/64
ifconfig: del: bad value - destroy_tun_device
- run_hooks post-disconnect
- HOOK=post-disconnect
- ifconfig vpn0 destroy
- [ -d /etc/vpnc/post-disconnect.d ]
- exit 0
[2017-08-03 15:46:28] User cancelled (SIGINT); exiting.
Even though the VPN looks like it's brought up correctly according to openconnect, it's still fairly broken. I cannot resolve any of the hostnames on the far end using ping. If I attempt to use dig with the server set explicitly:
dig @10.209.76.197 somehost.lan
It successfully resolves and I can then ping the IP of somehost.lan. So what should I be doing to fix DNS? Additionally, netstat -nr is missing all of the static routes that I had set to be routed through the VPN. So I wonder why the static route config is getting ignored.
-
first a clarifying question or two….
It sounds like you want clients to connect to a OpenVPN server. Is that accurate?
If so, are the clients things like phones, computers, etc? Or is the client also a PF box?It also sounds like you want the clients to use their own wan connection unless trying to reach a host on the other end of the OpenVPN. Is that accurate?
If so - that's basically how OpenVPN works without modification.
So it looks like a route issue. I don't see (but only skimmed) any route statements in the logs. Are you declaring remote networks?
-
How is the client supposed to know which DNS server to use?
Before it knows the answer to the query it has no idea if the destination is out on the internet or over the VPN.
You are probably best off sending the queries to the DNS server over the VPN and letting it return the appropriate answer.