OpenConnect + split routing

[reinderien]

I am attempting to achieve the following setup:

• All traffic, by default, goes through WAN
• OpenConnect VPN is always up
• Traffic only routes to VPN interface on specified nets, the most important being 10.0.0.0/8
• There are two sets of DNS servers: for WAN and VPN

Reading a handful of guides, I've done the following:

• Install Shellcmd and run an earlyshellcmd,```
  /sbin/ifconfig tun create; /sbin/ifconfig tun0 name vpn0

- Modify vpnc-script to rename tun0 to vpn0 to get past the pfSense GUI hiding tun* interfaces
- Set the WAN-specific DNS servers to use the WAN interface, but leave the interface field blank for VPN because it's "directly connected"
- VPN has an assigned interface, and is set to static IPv4 with custom gateway, static IPv6 with no gateway
- The VPN GW is not set as default. The WAN GW is set as default.
- There are several dozen static routes set to be routed through the VPN GW.

I haven't written a cron job to start openconnect yet; for now I'm starting it manually.

An example run looks like:

echo "$password" |
openconnect
  --pid-file=/var/run/openconnect.pid
  --non-inter
  --user="$username"
  --passwd-on-stdin
  --cafile=/usr/share/openconnect/myserver.pem
  --interface="$tunif"
  --script=/usr/share/openconnect/vpnc-script-latest
  --timestamp
  --no-proxy
  --pfs
  --reconnect-timeout 60
  mycorpvpn.com

[2017-08-03 15:42:40] POST https://mycorpvpn.com/
...

• env
• sort
  BLOCKSIZE=K
  CISCO_CSTP_OPTIONS=X-CSTP-Version=1
  CISCO_DEF_DOMAIN=mycorpvpn.com
  CISCO_PROXY_PAC=http://wpad/wpad.dat
  CLICOLOR=true
  FTP_PASSIVE_MODE=YES
  GROUP=wheel
  HOME=/root
  HOST=puffball.house.lan
  HOSTTYPE=FreeBSD
  INTERNAL_IP4_ADDRESS=10.65.164.129
  INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
  INTERNAL_IP4_MTU=1300
  INTERNAL_IP4_NETADDR=10.65.160.0
  INTERNAL_IP4_NETMASK=255.255.224.0
  INTERNAL_IP4_NETMASKLEN=19
  INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
  INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
  LOGNAME=admin
  LSCOLORS=exfxcxdxbxegedabagacad
  MACHTYPE=x86_64
  MAIL=/var/mail/admin
  OSTYPE=FreeBSD
  PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
  PWD=/usr/share/openconnect
  REMOTEHOST=192.168.1.100
  SHELL=/etc/rc.initial
  SHLVL=1
  SSH_CLIENT=192.168.1.100 55906 22
  SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
  SSH_TTY=/dev/pts/0
  TERM=xterm-256color
  USER=admin
  VENDOR=amd
  VPNGATEWAY=129.xxx.xxx.247
  X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
  X-CSTP-Address=10.65.164.129
  X-CSTP-Client-Bypass-Protocol=false
  X-CSTP-DNS=10.209.76.197
  X-CSTP-DNS=10.209.76.198
  X-CSTP-DPD=30
  X-CSTP-Default-Domain=mycorp.com
  X-CSTP-Disable-Always-On-VPN=false
  X-CSTP-Disconnected-Timeout=21600
  X-CSTP-Hostname=vpn.mycorp.com
  X-CSTP-Idle-Timeout=21600
  X-CSTP-Keep=true
  X-CSTP-Keepalive=20
  X-CSTP-Lease-Duration=86400
  X-CSTP-MSIE-Proxy-Lockdown=false
  X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
  X-CSTP-MTU=1300
  X-CSTP-Netmask=255.255.224.0
  X-CSTP-Post-Auth-XML=
  X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
  X-CSTP-Quarantine=false
  X-CSTP-Routing-Filtering-Ignore=false
  X-CSTP-Session-Timeout=86400
  X-CSTP-Smartcard-Removal-Disconnect=true
  X-CSTP-TCP-Keepalive=true
  X-CSTP-Tunnel-All-DNS=false
  reason=pre-init
• set -x
• PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
• uname -s
• OS=FreeBSD
• HOOKS_DIR=/etc/vpnc
• DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
• RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
• basename /usr/share/openconnect/vpnc-script-latest
• SCRIPTNAME=vpnc-script-latest
• [ ! -d /var/run/vpnc ]
• mkdir -p /var/run/vpnc
• [ -x /sbin/restorecon ]
• which ip
• grep ^/
• IPROUTE=''
• ifconfig --help
• grep BusyBox
• ifconfig_syntax_inet=inet
• [ FreeBSD = Linux ]
• ifconfig_syntax_ptp=''
• route_syntax_gw=''
• route_syntax_del=delete
• route_syntax_netmask=-netmask
• [ FreeBSD = SunOS ]
• route_syntax_interface=''
• ifconfig_syntax_ptpv6=''
• [ -r /etc/openwrt_release ]
• [ -x /usr/bin/busctl ]
• [ -x /sbin/resolvconf ]
• MODIFYRESOLVCONF=modify_resolvconf_manager
• RESTORERESOLVCONF=restore_resolvconf_manager
• [ -z '' ]
• TUNDEV=tun0
• OLDTUNDEV=tun0
• TUNDEV=vpn0
• [ -n '' ]
• AF_INET=2
• [ -z pre-init ]
• run_hooks pre-init
• HOOK=pre-init
• [ -d /etc/vpnc/pre-init.d ]
• do_pre_init
• [ FreeBSD = Linux ]
• [ FreeBSD = FreeBSD ]
• kldstat -q -m if_tun
• ifconfig tun0
  ifconfig: interface tun0 does not exist
• ifconfig tun0 create
  ifconfig: SIOCIFCREATE2: File exists
• ifconfig tun0 name vpn0
  ifconfig: interface tun0 does not exist
• exit 0
  [2017-08-03 15:42:55] SIOCSIFMTU: Device not configured
• env
• sort
  BLOCKSIZE=K
  CISCO_CSTP_OPTIONS=X-CSTP-Version=1
  CISCO_DEF_DOMAIN=mycorp.com
  CISCO_PROXY_PAC=http://wpad/wpad.dat
  CLICOLOR=true
  FTP_PASSIVE_MODE=YES
  GROUP=wheel
  HOME=/root
  HOST=puffball.house.lan
  HOSTTYPE=FreeBSD
  INTERNAL_IP4_ADDRESS=10.65.164.129
  INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
  INTERNAL_IP4_MTU=1300
  INTERNAL_IP4_NETADDR=10.65.160.0
  INTERNAL_IP4_NETMASK=255.255.224.0
  INTERNAL_IP4_NETMASKLEN=19
  INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
  INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
  LOGNAME=admin
  LSCOLORS=exfxcxdxbxegedabagacad
  MACHTYPE=x86_64
  MAIL=/var/mail/admin
  OSTYPE=FreeBSD
  PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
  PWD=/usr/share/openconnect
  REMOTEHOST=192.168.1.100
  SHELL=/etc/rc.initial
  SHLVL=1
  SSH_CLIENT=192.168.1.100 55906 22
  SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
  SSH_TTY=/dev/pts/0
  TERM=xterm-256color
  TUNDEV=tun0
  USER=admin
  VENDOR=amd
  VPNGATEWAY=129.xx.xx.247
  X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
  X-CSTP-Address=10.65.164.129
  X-CSTP-Client-Bypass-Protocol=false
  X-CSTP-DNS=10.209.76.197
  X-CSTP-DNS=10.209.76.198
  X-CSTP-DPD=30
  X-CSTP-Default-Domain=mycorp.com
  X-CSTP-Disable-Always-On-VPN=false
  X-CSTP-Disconnected-Timeout=21600
  X-CSTP-Hostname=vpn.mycorp.com
  X-CSTP-Idle-Timeout=21600
  X-CSTP-Keep=true
  X-CSTP-Keepalive=20
  X-CSTP-Lease-Duration=86400
  X-CSTP-MSIE-Proxy-Lockdown=false
  X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
  X-CSTP-MTU=1300
  X-CSTP-Netmask=255.255.224.0
  X-CSTP-Post-Auth-XML=
  X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
  X-CSTP-Quarantine=false
  X-CSTP-Routing-Filtering-Ignore=false
  X-CSTP-Session-Timeout=86400
  X-CSTP-Smartcard-Removal-Disconnect=true
  X-CSTP-TCP-Keepalive=true
  X-CSTP-Tunnel-All-DNS=false
  reason=connect
• set -x
• PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
• uname -s
• OS=FreeBSD
• HOOKS_DIR=/etc/vpnc
• DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
• RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
• basename /usr/share/openconnect/vpnc-script-latest
• SCRIPTNAME=vpnc-script-latest
• [ ! -d /var/run/vpnc ]
• which ip
• grep ^/
• IPROUTE=''
• ifconfig --help
• grep BusyBox
• ifconfig_syntax_inet=inet
• [ FreeBSD = Linux ]
• ifconfig_syntax_ptp=''
• route_syntax_gw=''
• route_syntax_del=delete
• route_syntax_netmask=-netmask
• [ FreeBSD = SunOS ]
• route_syntax_interface=''
• ifconfig_syntax_ptpv6=''
• [ -r /etc/openwrt_release ]
• [ -x /usr/bin/busctl ]
• [ -x /sbin/resolvconf ]
• MODIFYRESOLVCONF=modify_resolvconf_manager
• RESTORERESOLVCONF=restore_resolvconf_manager
• [ -z tun0 ]
• OLDTUNDEV=tun0
• TUNDEV=vpn0
• [ -n '' ]
• AF_INET=2
• [ -z connect ]
• run_hooks connect
• HOOK=connect
• [ -d /etc/vpnc/connect.d ]
• do_connect
• [ -n '' ]
• set_vpngateway_route
• get_default_gw
• netstat -r -n
• awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
• route add -host 129.xx.xx.247 192.168.0.1
  add host 129.xx.xx.247: gateway 192.168.0.1
• do_ifconfig
• [ -n 1300 ]
• MTU=1300
• [ -z 1300 ]
• [ -n '' ]
• ifconfig vpn0 inet 10.65.164.129 10.65.164.129 netmask 255.255.255.255 mtu 1300 up
• [ -n 255.255.224.0 ]
• set_network_route 10.65.160.0 255.255.224.0 19
• NETWORK=10.65.160.0
• NETMASK=255.255.224.0
• NETMASKLEN=19
• del_network_route 10.65.160.0 255.255.224.0 19
• NETWORK=10.65.160.0
• NETMASK=255.255.224.0
• NETMASKLEN=19
• route delete -net 10.65.160.0 -netmask 255.255.224.0 10.65.164.129
  route: writing to routing socket: No such process
  delete net 10.65.160.0: gateway 10.65.164.129 fib 0: not in table
• route add -net 10.65.160.0 -netmask 255.255.224.0 10.65.164.129
  add net 10.65.160.0: gateway 10.65.164.129
• [ -n 2606:b400:2050:24:8000::8d ]
• [ -z 2606:b400:2050:24:8000::8d/64 ]
• [ -n 2606:b400:2050:24:8000::8d/64 ]
• [ -n '' ]
• ifconfig vpn0 inet6 2606:b400:2050:24:8000::8d/64 mtu 1300 up
• [ -n '' ]
• [ -n 10.65.164.129 ]
• set_default_route
• get_default_gw
• netstat -r -n
• awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
• DEFAULTGW=192.168.0.1
• echo 192.168.0.1
• route delete default 192.168.0.1
  delete net default: gateway 192.168.0.1
• route add default 10.65.164.129
  add net default: gateway 10.65.164.129
• [ -n '' ]
• [ -n 2606:b400:2050:24:8000::8d/64 -o -n 2606:b400:2050:24:8000::8d ]
• set_ipv6_default_route
• route add -inet6 default 2606:b400:2050:24:8000::8d
  route: writing to routing socket: File exists
  add net default: gateway 2606:b400:2050:24:8000::8d fib 0: route already in table
• [ -n '10.209.76.198 10.209.76.197' ]
• modify_resolvconf_manager
• NEW_RESOLVCONF=''
• NEW_RESOLVCONF='
  nameserver 10.209.76.198'
• NEW_RESOLVCONF='
  nameserver 10.209.76.198
  nameserver 10.209.76.197'
• [ -n mycorp.com ]
• NEW_RESOLVCONF='
  nameserver 10.209.76.198
  nameserver 10.209.76.197
  domain mycorp.com'
• echo '
  nameserver 10.209.76.198
  nameserver 10.209.76.197
  domain mycorp.com'
• /sbin/resolvconf -a vpn0
  cp: /dev/null.bak: Operation not supported
• run_hooks post-connect
• HOOK=post-connect
• [ -d /etc/vpnc/post-connect.d ]
• exit 0
  [2017-08-03 15:42:55] Connected tun0 as 10.65.164.129 + 2606:b400:2050:24:8000::8d/64, using SSL
  [2017-08-03 15:42:56] Established DTLS connection (using OpenSSL). Ciphersuite AES256-SHA.
  ^C[2017-08-03 15:46:27] Send BYE packet: Aborted by caller
• env
• sort
  BLOCKSIZE=K
  CISCO_CSTP_OPTIONS=X-CSTP-Version=1
  CISCO_DEF_DOMAIN=mycorp.com
  CISCO_PROXY_PAC=http://wpad/wpad.dat
  CLICOLOR=true
  FTP_PASSIVE_MODE=YES
  GROUP=wheel
  HOME=/root
  HOST=puffball.house.lan
  HOSTTYPE=FreeBSD
  INTERNAL_IP4_ADDRESS=10.65.164.129
  INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
  INTERNAL_IP4_MTU=1300
  INTERNAL_IP4_NETADDR=10.65.160.0
  INTERNAL_IP4_NETMASK=255.255.224.0
  INTERNAL_IP4_NETMASKLEN=19
  INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
  INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
  LOGNAME=admin
  LSCOLORS=exfxcxdxbxegedabagacad
  MACHTYPE=x86_64
  MAIL=/var/mail/admin
  OSTYPE=FreeBSD
  PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
  PWD=/usr/share/openconnect
  REMOTEHOST=192.168.1.100
  SHELL=/etc/rc.initial
  SHLVL=1
  SSH_CLIENT=192.168.1.100 55906 22
  SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
  SSH_TTY=/dev/pts/0
  TERM=xterm-256color
  TUNDEV=tun0
  USER=admin
  VENDOR=amd
  VPNGATEWAY=129.xx.xx.247
  X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
  X-CSTP-Address=10.65.164.129
  X-CSTP-Client-Bypass-Protocol=false
  X-CSTP-DNS=10.209.76.197
  X-CSTP-DNS=10.209.76.198
  X-CSTP-DPD=30
  X-CSTP-Default-Domain=mycorp.com
  X-CSTP-Disable-Always-On-VPN=false
  X-CSTP-Disconnected-Timeout=21600
  X-CSTP-Hostname=vpn.mycorp.com
  X-CSTP-Idle-Timeout=21600
  X-CSTP-Keep=true
  X-CSTP-Keepalive=20
  X-CSTP-Lease-Duration=86400
  X-CSTP-MSIE-Proxy-Lockdown=false
  X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
  X-CSTP-MTU=1300
  X-CSTP-Netmask=255.255.224.0
  X-CSTP-Post-Auth-XML=
  X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
  X-CSTP-Quarantine=false
  X-CSTP-Routing-Filtering-Ignore=false
  X-CSTP-Session-Timeout=86400
  X-CSTP-Smartcard-Removal-Disconnect=true
  X-CSTP-TCP-Keepalive=true
  X-CSTP-Tunnel-All-DNS=false
  reason=disconnect
• set -x
• PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
• uname -s
• OS=FreeBSD
• HOOKS_DIR=/etc/vpnc
• DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
• RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
• basename /usr/share/openconnect/vpnc-script-latest
• SCRIPTNAME=vpnc-script-latest
• [ ! -d /var/run/vpnc ]
• which ip
• grep ^/
• IPROUTE=''
• ifconfig --help
• grep BusyBox
• ifconfig_syntax_inet=inet
• [ FreeBSD = Linux ]
• ifconfig_syntax_ptp=''
• route_syntax_gw=''
• route_syntax_del=delete
• route_syntax_netmask=-netmask
• [ FreeBSD = SunOS ]
• route_syntax_interface=''
• ifconfig_syntax_ptpv6=''
• [ -r /etc/openwrt_release ]
• [ -x /usr/bin/busctl ]
• [ -x /sbin/resolvconf ]
• MODIFYRESOLVCONF=modify_resolvconf_manager
• RESTORERESOLVCONF=restore_resolvconf_manager
• [ -z tun0 ]
• OLDTUNDEV=tun0
• TUNDEV=vpn0
• [ -n '' ]
• AF_INET=2
• [ -z disconnect ]
• run_hooks disconnect
• HOOK=disconnect
• [ -d /etc/vpnc/disconnect.d ]
• do_disconnect
• [ -n '' ]
• reset_default_route
• [ -s /var/run/vpnc/defaultroute ]
• get_default_gw
• netstat -r -n
• awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
• route delete default 10.65.164.129
  delete net default: gateway 10.65.164.129
• cat /var/run/vpnc/defaultroute
• route add default 192.168.0.1
  add net default: gateway 192.168.0.1
• rm -f -- /var/run/vpnc/defaultroute
• [ -n '' ]
• [ -n 2606:b400:2050:24:8000::8d/64 -o -n 2606:b400:2050:24:8000::8d ]
• reset_ipv6_default_route
• route delete -inet6 default 2606:b400:2050:24:8000::8d
  delete net default: gateway 2606:b400:2050:24:8000::8d
• :
• del_vpngateway_route
• get_default_gw
• netstat -r -n
• awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
• route delete -host 129.xx.xx.247 192.168.0.1
  delete host 129.xx.xx.247: gateway 192.168.0.1
• [ -n '10.209.76.198 10.209.76.197' ]
• restore_resolvconf_manager
• /sbin/resolvconf -d vpn0
• [ -n '' ]
• [ -n 10.65.164.129 ]
• ifconfig vpn0 0.0.0.0
  ifconfig: ioctl (SIOCAIFADDR): Destination address required
• [ -n 2606:b400:2050:24:8000::8d ]
• [ -z 2606:b400:2050:24:8000::8d/64 ]
• [ -n 2606:b400:2050:24:8000::8d/64 ]
• ifconfig vpn0 inet6 del 2606:b400:2050:24:8000::8d/64
  ifconfig: del: bad value
• destroy_tun_device
• run_hooks post-disconnect
• HOOK=post-disconnect
• ifconfig vpn0 destroy
• [ -d /etc/vpnc/post-disconnect.d ]
• exit 0
  [2017-08-03 15:46:28] User cancelled (SIGINT); exiting.

Even though the VPN looks like it's brought up correctly according to openconnect, it's still fairly broken.

I cannot resolve any of the hostnames on the far end using ping. If I attempt to use dig with the server set explicitly:

dig @10.209.76.197 somehost.lan

It successfully resolves and I can then ping the IP of somehost.lan. So what should I be doing to fix DNS?

Additionally, netstat -nr is missing all of the static routes that I had set to be routed through the VPN. So I wonder why the static route config is getting ignored.

[SpaceBass]

first a clarifying question or two….

It sounds like you want clients to connect to a OpenVPN server. Is that accurate?
If so, are the clients things like phones, computers, etc? Or is the client also a PF box?

It also sounds like you want the clients to use their own wan connection unless trying to reach a host on the other end of the OpenVPN. Is that accurate?

If so - that's basically how OpenVPN works without modification.

So it looks like a route issue. I don't see (but only skimmed) any route statements in the logs. Are you declaring remote networks?

[Derelict]

How is the client supposed to know which DNS server to use?

Before it knows the answer to the query it has no idea if the destination is out on the internet or over the VPN.

You are probably best off sending the queries to the DNS server over the VPN and letting it return the appropriate answer.