OpenConnect + split routing

OpenVPN
Log in to reply
  • R

    I am attempting to achieve the following setup:

    • All traffic, by default, goes through WAN
    • OpenConnect VPN is always up
    • Traffic only routes to VPN interface on specified nets, the most important being 10.0.0.0/8
    • There are two sets of DNS servers: for WAN and VPN

    Reading a handful of guides, I've done the following:

    • Install Shellcmd and run an earlyshellcmd,```
      /sbin/ifconfig tun create; /sbin/ifconfig tun0 name vpn0
    - Modify vpnc-script to rename tun0 to vpn0 to get past the pfSense GUI hiding tun* interfaces
- Set the WAN-specific DNS servers to use the WAN interface, but leave the interface field blank for VPN because it's "directly connected"
- VPN has an assigned interface, and is set to static IPv4 with custom gateway, static IPv6 with no gateway
- The VPN GW is not set as default. The WAN GW is set as default.
- There are several dozen static routes set to be routed through the VPN GW.

I haven't written a cron job to start openconnect yet; for now I'm starting it manually.

An example run looks like:

    echo "$password" |
    openconnect
      --pid-file=/var/run/openconnect.pid
      --non-inter
      --user="$username"
      --passwd-on-stdin
      --cafile=/usr/share/openconnect/myserver.pem
      --interface="$tunif"
      --script=/usr/share/openconnect/vpnc-script-latest
      --timestamp
      --no-proxy
      --pfs
      --reconnect-timeout 60
      mycorpvpn.com

    [2017-08-03 15:42:40] POST https://mycorpvpn.com/
    ...

    • env
    • sort
      BLOCKSIZE=K
      CISCO_CSTP_OPTIONS=X-CSTP-Version=1
      CISCO_DEF_DOMAIN=mycorpvpn.com
      CISCO_PROXY_PAC=http://wpad/wpad.dat
      CLICOLOR=true
      FTP_PASSIVE_MODE=YES
      GROUP=wheel
      HOME=/root
      HOST=puffball.house.lan
      HOSTTYPE=FreeBSD
      INTERNAL_IP4_ADDRESS=10.65.164.129
      INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
      INTERNAL_IP4_MTU=1300
      INTERNAL_IP4_NETADDR=10.65.160.0
      INTERNAL_IP4_NETMASK=255.255.224.0
      INTERNAL_IP4_NETMASKLEN=19
      INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
      INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
      LOGNAME=admin
      LSCOLORS=exfxcxdxbxegedabagacad
      MACHTYPE=x86_64
      MAIL=/var/mail/admin
      OSTYPE=FreeBSD
      PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
      PWD=/usr/share/openconnect
      REMOTEHOST=192.168.1.100
      SHELL=/etc/rc.initial
      SHLVL=1
      SSH_CLIENT=192.168.1.100 55906 22
      SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
      SSH_TTY=/dev/pts/0
      TERM=xterm-256color
      USER=admin
      VENDOR=amd
      VPNGATEWAY=129.xxx.xxx.247
      X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
      X-CSTP-Address=10.65.164.129
      X-CSTP-Client-Bypass-Protocol=false
      X-CSTP-DNS=10.209.76.197
      X-CSTP-DNS=10.209.76.198
      X-CSTP-DPD=30
      X-CSTP-Default-Domain=mycorp.com
      X-CSTP-Disable-Always-On-VPN=false
      X-CSTP-Disconnected-Timeout=21600
      X-CSTP-Hostname=vpn.mycorp.com
      X-CSTP-Idle-Timeout=21600
      X-CSTP-Keep=true
      X-CSTP-Keepalive=20
      X-CSTP-Lease-Duration=86400
      X-CSTP-MSIE-Proxy-Lockdown=false
      X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
      X-CSTP-MTU=1300
      X-CSTP-Netmask=255.255.224.0
      X-CSTP-Post-Auth-XML=
      X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
      X-CSTP-Quarantine=false
      X-CSTP-Routing-Filtering-Ignore=false
      X-CSTP-Session-Timeout=86400
      X-CSTP-Smartcard-Removal-Disconnect=true
      X-CSTP-TCP-Keepalive=true
      X-CSTP-Tunnel-All-DNS=false
      reason=pre-init
    • set -x
    • PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
    • uname -s
    • OS=FreeBSD
    • HOOKS_DIR=/etc/vpnc
    • DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
    • RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
    • basename /usr/share/openconnect/vpnc-script-latest
    • SCRIPTNAME=vpnc-script-latest
    • [ ! -d /var/run/vpnc ]
    • mkdir -p /var/run/vpnc
    • [ -x /sbin/restorecon ]
    • which ip
    • grep ^/
    • IPROUTE=''
    • ifconfig --help
    • grep BusyBox
    • ifconfig_syntax_inet=inet
    • [ FreeBSD = Linux ]
    • ifconfig_syntax_ptp=''
    • route_syntax_gw=''
    • route_syntax_del=delete
    • route_syntax_netmask=-netmask
    • [ FreeBSD = SunOS ]
    • route_syntax_interface=''
    • ifconfig_syntax_ptpv6=''
    • [ -r /etc/openwrt_release ]
    • [ -x /usr/bin/busctl ]
    • [ -x /sbin/resolvconf ]
    • MODIFYRESOLVCONF=modify_resolvconf_manager
    • RESTORERESOLVCONF=restore_resolvconf_manager
    • [ -z '' ]
    • TUNDEV=tun0
    • OLDTUNDEV=tun0
    • TUNDEV=vpn0
    • [ -n '' ]
    • AF_INET=2
    • [ -z pre-init ]
    • run_hooks pre-init
    • HOOK=pre-init
    • [ -d /etc/vpnc/pre-init.d ]
    • do_pre_init
    • [ FreeBSD = Linux ]
    • [ FreeBSD = FreeBSD ]
    • kldstat -q -m if_tun
    • ifconfig tun0
      ifconfig: interface tun0 does not exist
    • ifconfig tun0 create
      ifconfig: SIOCIFCREATE2: File exists
    • ifconfig tun0 name vpn0
      ifconfig: interface tun0 does not exist
    • exit 0
      [2017-08-03 15:42:55] SIOCSIFMTU: Device not configured
    • env
    • sort
      BLOCKSIZE=K
      CISCO_CSTP_OPTIONS=X-CSTP-Version=1
      CISCO_DEF_DOMAIN=mycorp.com
      CISCO_PROXY_PAC=http://wpad/wpad.dat
      CLICOLOR=true
      FTP_PASSIVE_MODE=YES
      GROUP=wheel
      HOME=/root
      HOST=puffball.house.lan
      HOSTTYPE=FreeBSD
      INTERNAL_IP4_ADDRESS=10.65.164.129
      INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
      INTERNAL_IP4_MTU=1300
      INTERNAL_IP4_NETADDR=10.65.160.0
      INTERNAL_IP4_NETMASK=255.255.224.0
      INTERNAL_IP4_NETMASKLEN=19
      INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
      INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
      LOGNAME=admin
      LSCOLORS=exfxcxdxbxegedabagacad
      MACHTYPE=x86_64
      MAIL=/var/mail/admin
      OSTYPE=FreeBSD
      PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
      PWD=/usr/share/openconnect
      REMOTEHOST=192.168.1.100
      SHELL=/etc/rc.initial
      SHLVL=1
      SSH_CLIENT=192.168.1.100 55906 22
      SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
      SSH_TTY=/dev/pts/0
      TERM=xterm-256color
      TUNDEV=tun0
      USER=admin
      VENDOR=amd
      VPNGATEWAY=129.xx.xx.247
      X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
      X-CSTP-Address=10.65.164.129
      X-CSTP-Client-Bypass-Protocol=false
      X-CSTP-DNS=10.209.76.197
      X-CSTP-DNS=10.209.76.198
      X-CSTP-DPD=30
      X-CSTP-Default-Domain=mycorp.com
      X-CSTP-Disable-Always-On-VPN=false
      X-CSTP-Disconnected-Timeout=21600
      X-CSTP-Hostname=vpn.mycorp.com
      X-CSTP-Idle-Timeout=21600
      X-CSTP-Keep=true
      X-CSTP-Keepalive=20
      X-CSTP-Lease-Duration=86400
      X-CSTP-MSIE-Proxy-Lockdown=false
      X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
      X-CSTP-MTU=1300
      X-CSTP-Netmask=255.255.224.0
      X-CSTP-Post-Auth-XML=
      X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
      X-CSTP-Quarantine=false
      X-CSTP-Routing-Filtering-Ignore=false
      X-CSTP-Session-Timeout=86400
      X-CSTP-Smartcard-Removal-Disconnect=true
      X-CSTP-TCP-Keepalive=true
      X-CSTP-Tunnel-All-DNS=false
      reason=connect
    • set -x
    • PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
    • uname -s
    • OS=FreeBSD
    • HOOKS_DIR=/etc/vpnc
    • DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
    • RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
    • basename /usr/share/openconnect/vpnc-script-latest
    • SCRIPTNAME=vpnc-script-latest
    • [ ! -d /var/run/vpnc ]
    • which ip
    • grep ^/
    • IPROUTE=''
    • ifconfig --help
    • grep BusyBox
    • ifconfig_syntax_inet=inet
    • [ FreeBSD = Linux ]
    • ifconfig_syntax_ptp=''
    • route_syntax_gw=''
    • route_syntax_del=delete
    • route_syntax_netmask=-netmask
    • [ FreeBSD = SunOS ]
    • route_syntax_interface=''
    • ifconfig_syntax_ptpv6=''
    • [ -r /etc/openwrt_release ]
    • [ -x /usr/bin/busctl ]
    • [ -x /sbin/resolvconf ]
    • MODIFYRESOLVCONF=modify_resolvconf_manager
    • RESTORERESOLVCONF=restore_resolvconf_manager
    • [ -z tun0 ]
    • OLDTUNDEV=tun0
    • TUNDEV=vpn0
    • [ -n '' ]
    • AF_INET=2
    • [ -z connect ]
    • run_hooks connect
    • HOOK=connect
    • [ -d /etc/vpnc/connect.d ]
    • do_connect
    • [ -n '' ]
    • set_vpngateway_route
    • get_default_gw
    • netstat -r -n
    • awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
    • route add -host 129.xx.xx.247 192.168.0.1
      add host 129.xx.xx.247: gateway 192.168.0.1
    • do_ifconfig
    • [ -n 1300 ]
    • MTU=1300
    • [ -z 1300 ]
    • [ -n '' ]
    • ifconfig vpn0 inet 10.65.164.129 10.65.164.129 netmask 255.255.255.255 mtu 1300 up
    • [ -n 255.255.224.0 ]
    • set_network_route 10.65.160.0 255.255.224.0 19
    • NETWORK=10.65.160.0
    • NETMASK=255.255.224.0
    • NETMASKLEN=19
    • del_network_route 10.65.160.0 255.255.224.0 19
    • NETWORK=10.65.160.0
    • NETMASK=255.255.224.0
    • NETMASKLEN=19
    • route delete -net 10.65.160.0 -netmask 255.255.224.0 10.65.164.129
      route: writing to routing socket: No such process
      delete net 10.65.160.0: gateway 10.65.164.129 fib 0: not in table
    • route add -net 10.65.160.0 -netmask 255.255.224.0 10.65.164.129
      add net 10.65.160.0: gateway 10.65.164.129
    • [ -n 2606:b400:2050:24:8000::8d ]
    • [ -z 2606:b400:2050:24:8000::8d/64 ]
    • [ -n 2606:b400:2050:24:8000::8d/64 ]
    • [ -n '' ]
    • ifconfig vpn0 inet6 2606:b400:2050:24:8000::8d/64 mtu 1300 up
    • [ -n '' ]
    • [ -n 10.65.164.129 ]
    • set_default_route
    • get_default_gw
    • netstat -r -n
    • awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
    • DEFAULTGW=192.168.0.1
    • echo 192.168.0.1
    • route delete default 192.168.0.1
      delete net default: gateway 192.168.0.1
    • route add default 10.65.164.129
      add net default: gateway 10.65.164.129
    • [ -n '' ]
    • [ -n 2606:b400:2050:24:8000::8d/64 -o -n 2606:b400:2050:24:8000::8d ]
    • set_ipv6_default_route
    • route add -inet6 default 2606:b400:2050:24:8000::8d
      route: writing to routing socket: File exists
      add net default: gateway 2606:b400:2050:24:8000::8d fib 0: route already in table
    • [ -n '10.209.76.198 10.209.76.197' ]
    • modify_resolvconf_manager
    • NEW_RESOLVCONF=''
    • NEW_RESOLVCONF='
      nameserver 10.209.76.198'
    • NEW_RESOLVCONF='
      nameserver 10.209.76.198
      nameserver 10.209.76.197'
    • [ -n mycorp.com ]
    • NEW_RESOLVCONF='
      nameserver 10.209.76.198
      nameserver 10.209.76.197
      domain mycorp.com'
    • echo '
      nameserver 10.209.76.198
      nameserver 10.209.76.197
      domain mycorp.com'
    • /sbin/resolvconf -a vpn0
      cp: /dev/null.bak: Operation not supported
    • run_hooks post-connect
    • HOOK=post-connect
    • [ -d /etc/vpnc/post-connect.d ]
    • exit 0
      [2017-08-03 15:42:55] Connected tun0 as 10.65.164.129 + 2606:b400:2050:24:8000::8d/64, using SSL
      [2017-08-03 15:42:56] Established DTLS connection (using OpenSSL). Ciphersuite AES256-SHA.
      ^C[2017-08-03 15:46:27] Send BYE packet: Aborted by caller
    • env
    • sort
      BLOCKSIZE=K
      CISCO_CSTP_OPTIONS=X-CSTP-Version=1
      CISCO_DEF_DOMAIN=mycorp.com
      CISCO_PROXY_PAC=http://wpad/wpad.dat
      CLICOLOR=true
      FTP_PASSIVE_MODE=YES
      GROUP=wheel
      HOME=/root
      HOST=puffball.house.lan
      HOSTTYPE=FreeBSD
      INTERNAL_IP4_ADDRESS=10.65.164.129
      INTERNAL_IP4_DNS=10.209.76.198 10.209.76.197
      INTERNAL_IP4_MTU=1300
      INTERNAL_IP4_NETADDR=10.65.160.0
      INTERNAL_IP4_NETMASK=255.255.224.0
      INTERNAL_IP4_NETMASKLEN=19
      INTERNAL_IP6_ADDRESS=2606:b400:2050:24:8000::8d
      INTERNAL_IP6_NETMASK=2606:b400:2050:24:8000::8d/64
      LOGNAME=admin
      LSCOLORS=exfxcxdxbxegedabagacad
      MACHTYPE=x86_64
      MAIL=/var/mail/admin
      OSTYPE=FreeBSD
      PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
      PWD=/usr/share/openconnect
      REMOTEHOST=192.168.1.100
      SHELL=/etc/rc.initial
      SHLVL=1
      SSH_CLIENT=192.168.1.100 55906 22
      SSH_CONNECTION=192.168.1.100 55906 192.168.1.1 22
      SSH_TTY=/dev/pts/0
      TERM=xterm-256color
      TUNDEV=tun0
      USER=admin
      VENDOR=amd
      VPNGATEWAY=129.xx.xx.247
      X-CSTP-Address-IP6=2606:b400:2050:24:8000::8d/64
      X-CSTP-Address=10.65.164.129
      X-CSTP-Client-Bypass-Protocol=false
      X-CSTP-DNS=10.209.76.197
      X-CSTP-DNS=10.209.76.198
      X-CSTP-DPD=30
      X-CSTP-Default-Domain=mycorp.com
      X-CSTP-Disable-Always-On-VPN=false
      X-CSTP-Disconnected-Timeout=21600
      X-CSTP-Hostname=vpn.mycorp.com
      X-CSTP-Idle-Timeout=21600
      X-CSTP-Keep=true
      X-CSTP-Keepalive=20
      X-CSTP-Lease-Duration=86400
      X-CSTP-MSIE-Proxy-Lockdown=false
      X-CSTP-MSIE-Proxy-PAC-URL=http://wpad/wpad.dat
      X-CSTP-MTU=1300
      X-CSTP-Netmask=255.255.224.0
      X-CSTP-Post-Auth-XML=
      X-CSTP-Protocol=Copyright (c) 2004 Cisco Systems, Inc.
      X-CSTP-Quarantine=false
      X-CSTP-Routing-Filtering-Ignore=false
      X-CSTP-Session-Timeout=86400
      X-CSTP-Smartcard-Removal-Disconnect=true
      X-CSTP-TCP-Keepalive=true
      X-CSTP-Tunnel-All-DNS=false
      reason=disconnect
    • set -x
    • PATH=/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
    • uname -s
    • OS=FreeBSD
    • HOOKS_DIR=/etc/vpnc
    • DEFAULT_ROUTE_FILE=/var/run/vpnc/defaultroute
    • RESOLV_CONF_BACKUP=/var/run/vpnc/resolv.conf-backup
    • basename /usr/share/openconnect/vpnc-script-latest
    • SCRIPTNAME=vpnc-script-latest
    • [ ! -d /var/run/vpnc ]
    • which ip
    • grep ^/
    • IPROUTE=''
    • ifconfig --help
    • grep BusyBox
    • ifconfig_syntax_inet=inet
    • [ FreeBSD = Linux ]
    • ifconfig_syntax_ptp=''
    • route_syntax_gw=''
    • route_syntax_del=delete
    • route_syntax_netmask=-netmask
    • [ FreeBSD = SunOS ]
    • route_syntax_interface=''
    • ifconfig_syntax_ptpv6=''
    • [ -r /etc/openwrt_release ]
    • [ -x /usr/bin/busctl ]
    • [ -x /sbin/resolvconf ]
    • MODIFYRESOLVCONF=modify_resolvconf_manager
    • RESTORERESOLVCONF=restore_resolvconf_manager
    • [ -z tun0 ]
    • OLDTUNDEV=tun0
    • TUNDEV=vpn0
    • [ -n '' ]
    • AF_INET=2
    • [ -z disconnect ]
    • run_hooks disconnect
    • HOOK=disconnect
    • [ -d /etc/vpnc/disconnect.d ]
    • do_disconnect
    • [ -n '' ]
    • reset_default_route
    • [ -s /var/run/vpnc/defaultroute ]
    • get_default_gw
    • netstat -r -n
    • awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
    • route delete default 10.65.164.129
      delete net default: gateway 10.65.164.129
    • cat /var/run/vpnc/defaultroute
    • route add default 192.168.0.1
      add net default: gateway 192.168.0.1
    • rm -f -- /var/run/vpnc/defaultroute
    • [ -n '' ]
    • [ -n 2606:b400:2050:24:8000::8d/64 -o -n 2606:b400:2050:24:8000::8d ]
    • reset_ipv6_default_route
    • route delete -inet6 default 2606:b400:2050:24:8000::8d
      delete net default: gateway 2606:b400:2050:24:8000::8d
    • :
    • del_vpngateway_route
    • get_default_gw
    • netstat -r -n
    • awk '/:/ { next; } /^(default|0.0.0.0)/ { print $2; }'
    • route delete -host 129.xx.xx.247 192.168.0.1
      delete host 129.xx.xx.247: gateway 192.168.0.1
    • [ -n '10.209.76.198 10.209.76.197' ]
    • restore_resolvconf_manager
    • /sbin/resolvconf -d vpn0
    • [ -n '' ]
    • [ -n 10.65.164.129 ]
    • ifconfig vpn0 0.0.0.0
      ifconfig: ioctl (SIOCAIFADDR): Destination address required
    • [ -n 2606:b400:2050:24:8000::8d ]
    • [ -z 2606:b400:2050:24:8000::8d/64 ]
    • [ -n 2606:b400:2050:24:8000::8d/64 ]
    • ifconfig vpn0 inet6 del 2606:b400:2050:24:8000::8d/64
      ifconfig: del: bad value
    • destroy_tun_device
    • run_hooks post-disconnect
    • HOOK=post-disconnect
    • ifconfig vpn0 destroy
    • [ -d /etc/vpnc/post-disconnect.d ]
    • exit 0
      [2017-08-03 15:46:28] User cancelled (SIGINT); exiting.
    
Even though the VPN looks like it's brought up correctly according to openconnect, it's still fairly broken.

I cannot resolve any of the hostnames on the far end using ping. If I attempt to use dig with the server set explicitly:

    dig @10.209.76.197 somehost.lan

    
It successfully resolves and I can then ping the IP of somehost.lan. So what should I be doing to fix DNS?

Additionally, netstat -nr is missing all of the static routes that I had set to be routed through the VPN. So I wonder why the static route config is getting ignored.
    0
  • S

    first a clarifying question or two….

    It sounds like you want clients to connect to a OpenVPN server. Is that accurate?
    If so, are the clients things like phones, computers, etc? Or is the client also a PF box?

    It also sounds like you want the clients to use their own wan connection unless trying to reach a host on the other end of the OpenVPN. Is that accurate?

    If so - that's basically how OpenVPN works without modification.

    So it looks like a route issue. I don't see (but only skimmed) any route statements in the logs. Are you declaring remote networks?

    0
  • Derelict
    LAYER 8 Netgate

    How is the client supposed to know which DNS server to use?

    Before it knows the answer to the query it has no idea if the destination is out on the internet or over the VPN.

    You are probably best off sending the queries to the DNS server over the VPN and letting it return the appropriate answer.

    0

Products

Services

Support

News

Resources

Company

Our Mission

We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

© 2021 Rubicon Communications, LLC | Privacy Policy