[SOLVED] Unable to reach pfsense or any computer on its subnet from VPN server



  • UPDATE:  I solved this by using tap instead of tun.
    see below



  • see below


  • LAYER 8 Global Moderator

    How about some actual details of your setup??  For starters WTF would you be doing wan rules for a vpn client to ping stuff for???

    And your wan is rfc1918.. From your thead over at openvpn this seems to be 1 side is in the google compute engine, and the other is where exactly - where is pfsense running?

    If all you want is a site to site vpn, then look at the freaking docs..
    https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site



  • see below



  • Per request:

    My setup:

    
    GCE CentOS OVPN Server                                                                                        pfSense Router
    eth0 10.250.0.2                                            Google Gateway                                   rl0 (LAN) 10.0.0.1 
    10.250.255.255                                                                                                      10.0.0.255
    tun0  10.254.254.1,2           <=VPN=>               ip addr:  10.250.0.1         <=WAN/VPN=>              ovpnc1 (tun) 10.254.254.5,6
    Static Public IP (no if)                                                                                              sis0  Dynamic Public IP
    No NAT                                                                                                                   NAT
    
    

    Firewall Rules on GCE Network allow all ports (1-65535) for tcp, udp, and icmp on the 10.250.0.0/16 subnet for all instances (there is only the server).

    SELinux is disabled on server.

    firewalld is disabled on server.

    IP Forwarding on Server:

    
    vpn-server-1 etc]$ cat /proc/sys/net/ipv4/ip_forward
    1
    
    

    Server Routing Table:

    
    $ route
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    default         gateway         0.0.0.0         UG    100    0        0 eth0
    10.0.0.0        10.254.254.2    255.255.255.0   UG    0      0        0 tun0
    gateway         0.0.0.0         255.255.255.255 UH    100    0        0 eth0
    ads-vpn-server- 0.0.0.0         255.255.255.255 UH    100    0        0 eth0
    10.254.254.0    10.254.254.2    255.255.255.0   UG    0      0        0 tun0
    10.254.254.2    0.0.0.0         255.255.255.255 UH    0      0        0 tun0
    metadata.google gateway         255.255.255.255 UGH   100    0        0 eth0
    

    Server's Gateway's Routes

    
    Name	                                 Destination IP ranges  	Priority	Instance tags	Next hop	                        Network
    
    ads-??-vpn-route	                 10.0.0.0/24	                 500	         None	        10.250.0.2	                        Default
    
    ads-vpn-server-1-tun-route	         10.254.254.0/24	         500	         None	        10.250.0.2	                        Default
    
    default-route-0dbf2173481c8cf2	 10.250.0.0/16	               1000	         None	         Virtual network	                Default
    
    default-route-6befe203e9e08025	   0.0.0.0/0	               1000	         None	         Default internet gateway	Default
    
    

    pfSense VPN Client's Route's

    
    # netstat -r
    Routing tables
    
    Internet:
    Destination                            Gateway                         Flags      Netif Expire
    default                                  ool-45936001.dyn.o         UGS        sis0
    10.0.0.0                                link#2                             U            rl0
    adsllc--pfse                            link#2                            UHS         lo0
    10.250.0.0                             10.254.254.5                 UGS      ovpnc1
    10.254.254.0                         10.254.254.5                 UGS      ovpnc1
    10.254.254.5                         link#7                            UH       ovpnc1
    10.254.254.6                         link#7                           UHS         lo0
    69.115.144.0/20                      link#1                            U          sis0
    ool-45936d3c.dyn.o                link#1                           UHS         lo0
    localhost                                link#5                           UH          lo0
    vdnssec1.srv.prnyn                 00:0f:b5:8a:b4:76         UHS        sis0
    vdnssec2.srv.prnyn                 00:0f:b5:8a:b4:76         UHS        sis0
    
    

    /etc/openvpn/server.conf

    
    proto udp
    dev tun
    ca /etc/openvpn/keys/ca.crt
    cert /etc/openvpn/keys/server.crt
    key /etc/openvpn/keys/server.key  # This file should be kept secret
    dh /etc/openvpn/keys/dh2048.pem
    server 10.254.254.0 255.255.255.0
    push "route 10.250.0.0 255.255.0.0"
    client-config-dir ccd
    route 10.0.0.0 255.255.255.0
    client-to-client
    keepalive 10 120
    comp-lzo
    persist-key
    persist-tun
    status openvpn-status.log
    verb 5
    
    

    /etc/openvpn/ccd/client.conf

    
    iroute 10.0.0.0 255.255.255.0
    
    

    /var/etc/openvpn/client1.conf (client config, autogenerated by GUI)

    
    dev ovpnc1
    verb 4
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_client1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp
    cipher BF-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 69.115.144.60
    tls-client
    client
    lport 0
    management /var/etc/openvpn/client1.sock unix
    remote 104.196.144.148 1194
    ifconfig 10.254.254.2 10.254.254.1
    ca /var/etc/openvpn/client1.ca
    cert /var/etc/openvpn/client1.cert
    key /var/etc/openvpn/client1.key
    comp-lzo adaptive
    resolv-retry infinite
    
    

    From server console:

    
     sudo openvpn server.conf 
    Mon Mar 21 02:05:07 2016 us=690291 Current Parameter Settings:
    Mon Mar 21 02:05:07 2016 us=690330   config = 'server.conf'
    Mon Mar 21 02:05:07 2016 us=690337   mode = 1
    Mon Mar 21 02:05:07 2016 us=690342   persist_config = DISABLED
    Mon Mar 21 02:05:07 2016 us=690346   persist_mode = 1
    Mon Mar 21 02:05:07 2016 us=690351   show_ciphers = DISABLED
    Mon Mar 21 02:05:07 2016 us=690355   show_digests = DISABLED
    Mon Mar 21 02:05:07 2016 us=690359   show_engines = DISABLED
    Mon Mar 21 02:05:07 2016 us=690363   genkey = DISABLED
    Mon Mar 21 02:05:07 2016 us=690367   key_pass_file = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690374   show_tls_ciphers = DISABLED
    Mon Mar 21 02:05:07 2016 us=690378 Connection profiles [default]:
    Mon Mar 21 02:05:07 2016 us=690383   proto = udp
    Mon Mar 21 02:05:07 2016 us=690387   local = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690391   local_port = 1194
    Mon Mar 21 02:05:07 2016 us=690395   remote = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690400   remote_port = 1194
    Mon Mar 21 02:05:07 2016 us=690404   remote_float = DISABLED
    Mon Mar 21 02:05:07 2016 us=690408   bind_defined = DISABLED
    Mon Mar 21 02:05:07 2016 us=690412   bind_local = ENABLED
    Mon Mar 21 02:05:07 2016 us=690416   connect_retry_seconds = 5
    Mon Mar 21 02:05:07 2016 us=690421   connect_timeout = 10
    Mon Mar 21 02:05:07 2016 us=690425   connect_retry_max = 0
    Mon Mar 21 02:05:07 2016 us=690429   socks_proxy_server = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690434   socks_proxy_port = 0
    Mon Mar 21 02:05:07 2016 us=690438   socks_proxy_retry = DISABLED
    Mon Mar 21 02:05:07 2016 us=690442   tun_mtu = 1500
    Mon Mar 21 02:05:07 2016 us=690446   tun_mtu_defined = ENABLED
    Mon Mar 21 02:05:07 2016 us=690451   link_mtu = 1500
    Mon Mar 21 02:05:07 2016 us=690455   link_mtu_defined = DISABLED
    Mon Mar 21 02:05:07 2016 us=690459   tun_mtu_extra = 0
    Mon Mar 21 02:05:07 2016 us=690463   tun_mtu_extra_defined = DISABLED
    Mon Mar 21 02:05:07 2016 us=690468   mtu_discover_type = -1
    Mon Mar 21 02:05:07 2016 us=690472   fragment = 0
    Mon Mar 21 02:05:07 2016 us=690476   mssfix = 1450
    Mon Mar 21 02:05:07 2016 us=690480   explicit_exit_notification = 0
    Mon Mar 21 02:05:07 2016 us=690485 Connection profiles END
    Mon Mar 21 02:05:07 2016 us=690490   remote_random = DISABLED
    Mon Mar 21 02:05:07 2016 us=690494   ipchange = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690498   dev = 'tun'
    Mon Mar 21 02:05:07 2016 us=690502   dev_type = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690506   dev_node = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690510   lladdr = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690514   topology = 1
    Mon Mar 21 02:05:07 2016 us=690518   tun_ipv6 = DISABLED
    Mon Mar 21 02:05:07 2016 us=690522   ifconfig_local = '10.254.254.1'
    Mon Mar 21 02:05:07 2016 us=690526   ifconfig_remote_netmask = '10.254.254.2'
    Mon Mar 21 02:05:07 2016 us=690530   ifconfig_noexec = DISABLED
    Mon Mar 21 02:05:07 2016 us=690534   ifconfig_nowarn = DISABLED
    Mon Mar 21 02:05:07 2016 us=690538   ifconfig_ipv6_local = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690543   ifconfig_ipv6_netbits = 0
    Mon Mar 21 02:05:07 2016 us=690547   ifconfig_ipv6_remote = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690551   shaper = 0
    Mon Mar 21 02:05:07 2016 us=690555   mtu_test = 0
    Mon Mar 21 02:05:07 2016 us=690559   mlock = DISABLED
    Mon Mar 21 02:05:07 2016 us=690563   keepalive_ping = 10
    Mon Mar 21 02:05:07 2016 us=690567   keepalive_timeout = 120
    Mon Mar 21 02:05:07 2016 us=690571   inactivity_timeout = 0
    Mon Mar 21 02:05:07 2016 us=690575   ping_send_timeout = 10
    Mon Mar 21 02:05:07 2016 us=690579   ping_rec_timeout = 240
    Mon Mar 21 02:05:07 2016 us=690583   ping_rec_timeout_action = 2
    Mon Mar 21 02:05:07 2016 us=690587   ping_timer_remote = DISABLED
    Mon Mar 21 02:05:07 2016 us=690591   remap_sigusr1 = 0
    Mon Mar 21 02:05:07 2016 us=690595   persist_tun = ENABLED
    Mon Mar 21 02:05:07 2016 us=690599   persist_local_ip = DISABLED
    Mon Mar 21 02:05:07 2016 us=690603   persist_remote_ip = DISABLED
    Mon Mar 21 02:05:07 2016 us=690607   persist_key = ENABLED
    Mon Mar 21 02:05:07 2016 us=690612   passtos = DISABLED
    Mon Mar 21 02:05:07 2016 us=690616   resolve_retry_seconds = 1000000000
    Mon Mar 21 02:05:07 2016 us=690620   username = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690624   groupname = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690628   chroot_dir = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690632   cd_dir = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690636   writepid = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690639   up_script = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690643   down_script = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=690647   down_pre = DISABLED
    Mon Mar 21 02:05:07 2016 us=693362   duplicate_cn = DISABLED
    Mon Mar 21 02:05:07 2016 us=693366   cf_max = 0
    Mon Mar 21 02:05:07 2016 us=693370   cf_per = 0
    Mon Mar 21 02:05:07 2016 us=693374   max_clients = 1024
    Mon Mar 21 02:05:07 2016 us=693378   max_routes_per_client = 256
    Mon Mar 21 02:05:07 2016 us=693430   auth_user_pass_verify_script = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=693436   auth_user_pass_verify_script_via_file = DISABLED
    Mon Mar 21 02:05:07 2016 us=693440   port_share_host = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=693445   port_share_port = 0
    Mon Mar 21 02:05:07 2016 us=693449   client = DISABLED
    Mon Mar 21 02:05:07 2016 us=693453   pull = DISABLED
    Mon Mar 21 02:05:07 2016 us=693457   auth_user_pass_file = '[UNDEF]'
    Mon Mar 21 02:05:07 2016 us=693476 OpenVPN 2.3.10 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Jan  4 2016
    Mon Mar 21 02:05:07 2016 us=693485 library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
    Mon Mar 21 02:05:07 2016 us=699218 Diffie-Hellman initialized with 2048 bit key
    Mon Mar 21 02:05:07 2016 us=699565 TLS-Auth MTU parms [ L:1542 D:1212 EF:38 EB:0 ET:0 EL:3 ]
    Mon Mar 21 02:05:07 2016 us=699582 Socket Buffers: R=[212992->212992] S=[212992->212992]
    Mon Mar 21 02:05:07 2016 us=699650 ROUTE_GATEWAY 10.250.0.1
    Mon Mar 21 02:05:07 2016 us=699852 TUN/TAP device tun0 opened
    Mon Mar 21 02:05:07 2016 us=699864 TUN/TAP TX queue length set to 100
    Mon Mar 21 02:05:07 2016 us=699873 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
    Mon Mar 21 02:05:07 2016 us=699890 /usr/sbin/ip link set dev tun0 up mtu 1500
    Mon Mar 21 02:05:07 2016 us=703282 /usr/sbin/ip addr add dev tun0 local 10.254.254.1 peer 10.254.254.2
    Mon Mar 21 02:05:07 2016 us=704819 /usr/sbin/ip route add 10.0.0.0/24 via 10.254.254.2
    Mon Mar 21 02:05:07 2016 us=710749 /usr/sbin/ip route add 10.254.254.0/24 via 10.254.254.2
    Mon Mar 21 02:05:07 2016 us=712357 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:143 ET:0 EL:3 AF:3/1 ]
    Mon Mar 21 02:05:07 2016 us=712377 UDPv4 link local (bound): [undef]
    Mon Mar 21 02:05:07 2016 us=712383 UDPv4 link remote: [undef]
    Mon Mar 21 02:05:07 2016 us=712391 MULTI: multi_init called, r=256 v=256
    Mon Mar 21 02:05:07 2016 us=712435 IFCONFIG POOL: base=10.254.254.4 size=62, ipv6=0
    Mon Mar 21 02:05:07 2016 us=712451 Initialization Sequence Completed
    Mon Mar 21 02:05:10 2016 us=123321 MULTI: multi_create_instance called
    Mon Mar 21 02:05:10 2016 us=123366 69.115.144.60:65005 Re-using SSL/TLS context
    Mon Mar 21 02:05:10 2016 us=123394 69.115.144.60:65005 LZO compression initialized
    Mon Mar 21 02:05:10 2016 us=123491 69.115.144.60:65005 Control Channel MTU parms [ L:1542 D:1212 EF:38 EB:0 ET:0 EL:3 ]
    Mon Mar 21 02:05:10 2016 us=123500 69.115.144.60:65005 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:143 ET:0 EL:3 AF:3/1 ]
    Mon Mar 21 02:05:10 2016 us=123522 69.115.144.60:65005 Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
    Mon Mar 21 02:05:10 2016 us=123533 69.115.144.60:65005 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
    Mon Mar 21 02:05:10 2016 us=123551 69.115.144.60:65005 Local Options hash (VER=V4): '530fdded'
    Mon Mar 21 02:05:10 2016 us=123559 69.115.144.60:65005 Expected Remote Options hash (VER=V4): '41690919'
    RMon Mar 21 02:05:10 2016 us=123586 69.115.144.60:65005 TLS: Initial packet from [AF_INET]69.115.144.60:65005, sid=dab3460f a9ab573f
    WRRWRWRWRWWWWRRRRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRMon Mar 21 02:05:10 2016 us=842168 69.115.144.60:65005 VERIFY OK: depth=1, C=US, ST=CA, L=Newark, O=ADS, OU=MyOrganizationalUni
    t, CN=ads-vpn-server-1, name=EasyRSA, emailAddress=me@myemail.com
    Mon Mar 21 02:05:10 2016 us=842359 69.115.144.60:65005 VERIFY OK: depth=0, C=US, ST=CA, L=Newark, O=ADS, OU=MyOrganizationalUnit, CN=ads--pfsense, name=EasyRSA, emailAddress=me@myemail.com
    WRWRWRWRWRWRWRWRWRWRMon Mar 21 02:05:10 2016 us=916578 69.115.144.60:65005 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
    Mon Mar 21 02:05:10 2016 us=916609 69.115.144.60:65005 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    Mon Mar 21 02:05:10 2016 us=916655 69.115.144.60:65005 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
    Mon Mar 21 02:05:10 2016 us=916662 69.115.144.60:65005 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    WRMon Mar 21 02:05:10 2016 us=949581 69.115.144.60:65005 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
    Mon Mar 21 02:05:10 2016 us=949618 69.115.144.60:65005 [ads--pfsense] Peer Connection Initiated with [AF_INET]69.115.144.60:65005
    Mon Mar 21 02:05:10 2016 us=949655 ads-??-pfsense/69.115.144.60:65005 MULTI_sva: pool returned IPv4=10.254.254.6, IPv6=(Not enabled)
    Mon Mar 21 02:05:10 2016 us=949685 ads-??-pfsense/69.115.144.60:65005 MULTI: Learn: 10.254.254.6 -> ads--pfsense/69.115.144.60:65005
    Mon Mar 21 02:05:10 2016 us=949692 ads-??-pfsense/69.115.144.60:65005 MULTI: primary virtual IP for ads--pfsense/69.115.144.60:65005: 10.254.254.6
    RMon Mar 21 02:05:13 2016 us=117978 ads-??-pfsense/69.115.144.60:65005 PUSH: Received control message: 'PUSH_REQUEST'
    Mon Mar 21 02:05:13 2016 us=118012 ads-??-pfsense/69.115.144.60:65005 send_push_reply(): safe_cap=940
    Mon Mar 21 02:05:13 2016 us=118030 ads-??-pfsense/69.115.144.60:65005 SENT CONTROL [ads--pfsense]: 'PUSH_REPLY,route 10.250.0.0 255.255.0.0,route 10.254.254.0 255.255.255.0,topology net30,ping 10,ping-restart 120,ifconfig 10.254
    .254.6 10.254.254.5' (status=1)
    WWRRWRWWRWRWRWR
    

    Client Log

    
    Mar 21 02:05:02	openvpn[17113]: [server] Inactivity timeout (--ping-restart), restarting
    Mar 21 02:05:02	openvpn[17113]: TCP/UDP: Closing socket
    Mar 21 02:05:02	openvpn[17113]: SIGUSR1[soft,ping-restart] received, process restarting
    Mar 21 02:05:02	openvpn[17113]: Restart pause, 2 second(s)
    Mar 21 02:05:04	openvpn[17113]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
    Mar 21 02:05:04	openvpn[17113]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
    Mar 21 02:05:04	openvpn[17113]: Re-using SSL/TLS context
    Mar 21 02:05:04	openvpn[17113]: LZO compression initialized
    Mar 21 02:05:04	openvpn[17113]: Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:3 ]
    Mar 21 02:05:04	openvpn[17113]: Socket Buffers: R=[42080->65536] S=[57344->65536]
    Mar 21 02:05:04	openvpn[17113]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:143 ET:0 EL:3 AF:3/1 ]
    Mar 21 02:05:04	openvpn[17113]: Local Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
    Mar 21 02:05:04	openvpn[17113]: Expected Remote Options String: 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
    Mar 21 02:05:04	openvpn[17113]: Local Options hash (VER=V4): '41690919'
    Mar 21 02:05:04	openvpn[17113]: Expected Remote Options hash (VER=V4): '530fdded'
    Mar 21 02:05:04	openvpn[17113]: UDPv4 link local (bound): [AF_INET]69.115.144.60
    Mar 21 02:05:04	openvpn[17113]: UDPv4 link remote: [AF_INET]104.196.144.148:1194
    Mar 21 02:05:10	openvpn[17113]: TLS: Initial packet from [AF_INET]104.196.144.148:1194, sid=37518aa9 5fd4ad99
    Mar 21 02:05:10	openvpn[17113]: VERIFY OK: depth=1, C=US, ST=New Jersey, L=Newark, O=Atlantic Digital Solutions, LLC, OU=MyOrganizationalUnit, CN=ads-vpn-server-1, name=EasyRSA, emailAddress=brian@atlanticdigitalsolutions.com
    Mar 21 02:05:10	openvpn[17113]: VERIFY OK: depth=0, C=US, ST=New Jersey, L=Newark, O=Atlantic Digital Solutions, LLC, OU=MyOrganizationalUnit, CN=server, name=EasyRSA, emailAddress=brian@atlanticdigitalsolutions.com
    Mar 21 02:05:10	openvpn[17113]: Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
    Mar 21 02:05:10	openvpn[17113]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    Mar 21 02:05:10	openvpn[17113]: Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
    Mar 21 02:05:10	openvpn[17113]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    Mar 21 02:05:10	openvpn[17113]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
    Mar 21 02:05:10	openvpn[17113]: [server] Peer Connection Initiated with [AF_INET]104.196.144.148:1194
    Mar 21 02:05:13	openvpn[17113]: SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
    Mar 21 02:05:13	openvpn[17113]: PUSH: Received control message: 'PUSH_REPLY,route 10.250.0.0 255.255.0.0,route 10.254.254.0 255.255.255.0,topology net30,ping 10,ping-restart 120,ifconfig 10.254.254.6 10.254.254.5'
    Mar 21 02:05:13	openvpn[17113]: OPTIONS IMPORT: timers and/or timeouts modified
    Mar 21 02:05:13	openvpn[17113]: OPTIONS IMPORT: --ifconfig/up options modified
    Mar 21 02:05:13	openvpn[17113]: OPTIONS IMPORT: route options modified
    Mar 21 02:05:13	openvpn[17113]: Preserving previous TUN/TAP instance: ovpnc1
    Mar 21 02:05:13	openvpn[17113]: Initialization Sequence Completed
    

    From client-side windows machine:

    
    >ping 10.250.0.2
    
    Pinging 10.250.0.2 with 32 bytes of data:
    Reply from 10.250.0.2: bytes=32 time=33ms TTL=63
    Reply from 10.250.0.2: bytes=32 time=30ms TTL=63
    Reply from 10.250.0.2: bytes=32 time=37ms TTL=63
    Reply from 10.250.0.2: bytes=32 time=65ms TTL=63
    
    Ping statistics for 10.250.0.2:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 30ms, Maximum = 65ms, Average = 41ms
    

    Works!

    From server:

    
    $ ping 10.0.0.1   <---  pfsense address
    PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
    

    Doesn't work.

    While pinging:

    
    vpn-server-1 etc]$ sudo tcpdump -vv -n -i tun0|grep 10.0
    tcpdump: listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 52, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 53, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 54, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 55, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 56, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 57, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 58, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 59, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 60, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 61, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 62, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 63, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 64, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 65, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 66, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 67, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 68, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 69, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 70, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 71, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 72, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 73, length 64
        10.254.254.1 > 10.0.0.1: ICMP echo request, id 3931, seq 74, length 64
    

    Looks good, I think:

    But on the pfSense (client) side (still pinging):

    
    # tcpdump -vv -n -i sis0|grep 10.0.0
    tcpdump: listening on sis0, link-type EN10MB (Ethernet), capture size 65535 bytes
    capability mode sandbox enabled
    ^C843 packets captured
    851 packets received by filter
    0 packets dropped by kernel
    
    (nothing)
    
    # tcpdump -vv -n -i ovpnc1|grep 10.0.0
    tcpdump: listening on ovpnc1, link-type NULL (BSD loopback), capture size 65535 bytes
    capability mode sandbox enabled
    ^C0 packets captured
    0 packets received by filter
    0 packets dropped by kernel
    
    (nothing)
    
    # ping 10.250.0.2
    PING 10.250.0.2 (10.250.0.2): 56 data bytes
    64 bytes from 10.250.0.2: icmp_seq=0 ttl=64 time=31.611 ms
    64 bytes from 10.250.0.2: icmp_seq=1 ttl=64 time=29.781 ms
    ^C
    --- 10.250.0.2 ping statistics ---
    2 packets transmitted, 2 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 29.781/30.696/31.611/0.915 ms
    
    (other direction still works)
    
    

    pfSense firewall rules

    
    pfctl -sr
    scrub on sis0 all fragment reassemble
    scrub on rl0 all fragment reassemble
    scrub on ovpnc1 all fragment reassemble
    anchor "relayd/*" all
    anchor "openvpn/*" all
    anchor "ipsec/*" all
    block drop in log quick inet from 169.254.0.0/16 to any label "Block IPv4 link-local"
    block drop in log quick inet from any to 169.254.0.0/16 label "Block IPv4 link-local"
    block drop in log inet all label "Default deny rule IPv4"
    block drop out log inet all label "Default deny rule IPv4"
    block drop in log inet6 all label "Default deny rule IPv6"
    block drop out log inet6 all label "Default deny rule IPv6"
    pass quick inet6 proto ipv6-icmp all icmp6-type unreach keep state
    pass quick inet6 proto ipv6-icmp all icmp6-type toobig keep state
    pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state
    pass quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echorep keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echorep keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echoreq keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type echoreq keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routersol keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routeradv keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbrsol keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbradv keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echoreq keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state
    block drop log quick inet proto tcp from any port = 0 to any label "Block traffic from port 0"
    block drop log quick inet proto udp from any port = 0 to any label "Block traffic from port 0"
    block drop log quick inet proto tcp from any to any port = 0 label "Block traffic to port 0"
    block drop log quick inet proto udp from any to any port = 0 label "Block traffic to port 0"
    block drop log quick inet6 proto tcp from any port = 0 to any label "Block traffic from port 0"
    block drop log quick inet6 proto udp from any port = 0 to any label "Block traffic from port 0"
    block drop log quick inet6 proto tcp from any to any port = 0 label "Block traffic to port 0"
    block drop log quick inet6 proto udp from any to any port = 0 label "Block traffic to port 0"
    block drop log quick from <snort2c>to any label "Block snort2c hosts"
    block drop log quick from any to <snort2c>label "Block snort2c hosts"
    block drop in log quick proto tcp from <sshlockout>to (self) port = ssh label "sshlockout"
    block drop in log quick proto tcp from <webconfiguratorlockout>to (self) port = https label "webConfiguratorlockout"
    block drop in log quick from <virusprot>to any label "virusprot overload table"
    pass in quick on sis0 inet6 proto udp from fe80::/10 port = dhcpv6-client to fe80::/10 port = dhcpv6-client keep state label "allow dhcpv6 client in WAN"
    pass in quick on sis0 proto udp from any port = dhcpv6-server to any port = dhcpv6-client keep state label "allow dhcpv6 client in WAN"
    pass out quick on sis0 proto udp from any port = dhcpv6-client to any port = dhcpv6-server keep state label "allow dhcpv6 client out WAN"
    block drop in log quick on sis0 from <bogons>to any label "block bogon IPv4 networks from WAN"
    block drop in log quick on sis0 from <bogonsv6>to any label "block bogon IPv6 networks from WAN"
    block drop in log on ! sis0 inet from 69.115.144.0/20 to any
    block drop in log inet from 69.115.144.60 to any
    block drop in log on sis0 inet6 from fe80::20f:b5ff:fe8a:b476 to any
    pass in on sis0 proto udp from any port = bootps to any port = bootpc keep state label "allow dhcp client out WAN"
    pass out on sis0 proto udp from any port = bootpc to any port = bootps keep state label "allow dhcp client out WAN"
    block drop in log quick on rl0 from <bogons>to any label "block bogon IPv4 networks from LAN"
    block drop in log quick on rl0 from <bogonsv6>to any label "block bogon IPv6 networks from LAN"
    block drop in log on ! rl0 inet from 10.0.0.0/24 to any
    block drop in log inet from 10.0.0.1 to any
    block drop in log on rl0 inet6 from fe80::220:18ff:fed5:fd75 to any
    pass in quick on rl0 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server"
    pass in quick on rl0 inet proto udp from any port = bootpc to 10.0.0.1 port = bootps keep state label "allow access to DHCP server"
    pass out quick on rl0 inet proto udp from 10.0.0.1 port = bootps to any port = bootpc keep state label "allow access to DHCP server"
    block drop in log on ! ovpnc1 inet from 10.254.254.6 to any
    block drop in log inet from 10.254.254.6 to any
    block drop in log on ovpnc1 inet6 from fe80::20f:b5ff:fe8a:b476 to any
    pass in on lo0 inet all flags S/SA keep state label "pass IPv4 loopback"
    pass out on lo0 inet all flags S/SA keep state label "pass IPv4 loopback"
    pass in on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback"
    pass out on lo0 inet6 all flags S/SA keep state label "pass IPv6 loopback"
    pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
    pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
    pass out route-to (sis0 69.115.144.1) inet from 69.115.144.60 to ! 69.115.144.0/20 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass out route-to (ovpnc1 10.254.254.5) inet from 10.254.254.6 to ! 10.254.254.6 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass in quick on rl0 proto tcp from any to (rl0) port = https flags S/SA keep state label "anti-lockout rule"
    pass in quick on rl0 proto tcp from any to (rl0) port = http flags S/SA keep state label "anti-lockout rule"
    pass in quick on rl0 proto tcp from any to (rl0) port = ssh flags S/SA keep state label "anti-lockout rule"
    anchor "userrules/*" all
    pass in quick on openvpn inet all flags S/SA keep state label "USER_RULE"
    pass in quick on sis0 reply-to (sis0 69.115.144.1) inet proto tcp from any to 10.0.0.10 port = ssh flags S/SA keep state label "USER_RULE: NAT SSH to Server"
    pass in quick on sis0 reply-to (sis0 69.115.144.1) inet proto tcp from any to 10.0.0.10 port = 3389 flags S/SA keep state label "USER_RULE: NAT RDP to Server"
    pass in quick on sis0 reply-to (sis0 69.115.144.1) inet from 10.250.0.0/16 to any flags S/SA keep state label "USER_RULE"
    pass in quick on sis0 reply-to (sis0 69.115144.1) inet from 10.254.254.0/24 to any flags S/SA keep state label "USER_RULE"
    pass in quick on sis0 reply-to (sis0 69.115.144.1) inet proto tcp from any to 10.0.0.10 port = 25565 flags S/SA keep state label "USER_RULE: NAT "
    pass in quick on rl0 inet from 10.0.0.0/24 to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule"
    pass in quick on rl0 inet from 10.250.0.0/16 to any flags S/SA keep state label "USER_RULE"
    pass in quick on rl0 inet from 10.254.254.0/24 to any flags S/SA keep state label "USER_RULE"
    pass in quick on ovpnc1 reply-to (ovpnc1 10.254.254.5) inet all flags S/SA keep state label "USER_RULE: Allow all"
    anchor "tftp-proxy/*" all</bogonsv6></bogons></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c> 
    

    I am at a total loss as to what the problem might be at this point.  Any help is much appreciated.


  • LAYER 8 Global Moderator

    Are you trying to do 2 vpns??

    Did you go over their doc?  They do not show 2 vpns like your showing with vpn from you device in the google cloud to the gateway, and then one from your pfsense to the gateway.. There would be only 1 vpn from gateway to pfsense

    https://cloud.google.com/compute/docs/vpn/

    Why are you showing a 10.250 address for the gateway.  Its going to have to have a public IP..

    Are you trying to do a openvpn tunnel to your vpn server through a ipsec tunnel through the gateway??

    I have a funny feeling you have not even breezed over their doc???



  • That's not what I'm trying to do.  I'm just running a an OpenVPN server, as per the docs, and the built in Google IPSec VPN won't work for what I need anyway.

    I showed VPN between my gateway and and the OVPN server because the gateway is not my OVPN server, but it is the next hop from it.


  • LAYER 8 Global Moderator

    And where do they state that is supported??

    Your going to have to put a public IP on your instance that is running, not some port forward..

    What is it that you need btw??  Can I fire up a google compute instance for low cost or free for testing?

    I see they have a $300 60 day free trial, signing up..  So what is it exactly your wanting to accomplish?



  • @johnpoz:

    And where do they state that is supported??

    Your going to have to put a public IP on your instance that is running, not some port forward..

    I don't want to assume it isn't.

    @johnpoz:

    What is it that you need btw??  Can I fire up a google compute instance for low cost or free for testing?

    No but I can make one for you.

    @johnpoz:

    I see they have a $300 60 day free trial, signing up..  So what is it exactly your wanting to accomplish?

    That's for support.  An instance might only cost you $5 per month if you get the teeny tiny one.

    As to second part:  I need to add more subnets as well as do site-to-client (which google's VPN server doesn't do).

    Currently I'm trying to get it working with a tap interface.



  • Sent you PM.


  • LAYER 8 Global Moderator

    Well I got in in like 5 minutes

    fired up an instance, wget the openvpn as package

    Boom connected




  • I fixed it on my end.

    Set up server for tap.  Set up interface accordingly (needed to reboot as ovpn client was failing to ifconfig).  Set up bridge interface with LAN and OPT1.  Was able to ping vitrual IP of pfSense client from GCE server, but not pfSense's LAN IP or anything behind it.

    did a # sudo ip route add 10.0.0.0/24 dev br0 on server and voila.

    Not sure why it is not working with tun, maybe a bug of some sort with GCE.  Not sure what you did different to get it working on your end.


  • LAYER 8 Global Moderator

    I didn't do anything special, installed openvpn as - connected.. using TUN.  I had to change the IP that was in the profile to the external IP..



  • @johnpoz:

    Well I got in in like 5 minutes

    fired up an instance, wget the openvpn as package

    Boom connected

    I had no problem connecting.  Can you ping pfSense or anything behind its nat, assuming there is NAT.

    (BTW, I erroneously said there was no NAT on my GCE slice earlier, but now I think it is 1:1 NAT.  I'm new to all this stuff.)


  • LAYER 8 Global Moderator

    I am routing my traffic over the connection..

    What exactly are you wanting to accomplish with the vpn connection??




  • @johnpoz:

    I am routing my traffic over the connection..

    What exactly are you wanting to accomplish with the vpn connection??

    I have a funny feeling you only breezed through my post :P

    For now I have accomplished what I wanted to accomplish, which is a site-to-site VPN.

    Subnets are going to be added from various physical locations with lans behind pfsense and dd-wrt (in most cases).  There will be some modestly intricate routing between them.  In this case, the default gateway is always the local one.

    On the GCE subnet side some services will service.

    There will also be client-to-server connections which will do what you are doing.

    I think I would rather try and run pfSense on GCE.  It appears to be possible and there is some documentation, but it involves making a KVM virtual disk and loading it into a new instance in GCE, and I don't have a spare PC with VT-d needed to build it.

    See here:  https://gist.github.com/mkhon/0d8867e07c6b325ae228

    Who can I bribe to make one for me?  Maybe I'll start a new thread later.



  • By the way:  anyone trying to do what I'm doing should know that windows firewall by default blocks pings from other subnets, android phones and linux servers do not (not sure about iOS).  That might have really screwed me up had I not read it in the tons of time I spent trying and failing to get tun to work.


  • LAYER 8 Global Moderator

    So your going to have multiple machines on gce?  An they are going to use this vpn machine as their gateway to your network?  Can you setup the GCE networking that way for their instances?


Log in to reply