[Solved] LAN to LAN not routing



  • First off, I apologize if this is the wrong forum for this.

    Here's my scenario.

    I've set up 2 openVPN connections between two sites in my environment. I do not specify any subnets as part of the openVPN configuration. I instead use Quagga OSPF to distribute routes between the two.

    I've confirmed Quagga is working and reaching each firewall. Below you can see that the routes are being populated.

    Sample from one site (192.168.7.0/24 network)

    Quagga OSPF Routes
    ============ OSPF network routing table ============
    N    10.0.7.0/24           [10] area: 0.0.0.0
                               directly attached to ovpns1
    N    10.0.17.0/24          [20] area: 0.0.0.0
                               directly attached to ovpns2
    N    192.168.7.0/24        [10] area: 0.0.0.0
                               directly attached to hn1
    N    192.168.8.0/24        [20] area: 0.0.0.0
                               via 10.0.7.2, ovpns1
    
    

    Routes Snippet

    
    Destination	Gateway	Flags	Use	Mtu	Netif	Expire
    192.168.8.0/24	10.0.7.2	UG1	0	1500	ovpns1	
    
    

    Each pfSense can ping and ICMP tracert to the remote network and clients in it, but once the source is changed to LAN, the connection dies at the other end of the tunnel.

    Source Any ICMP Tracert

     1  10.0.7.2  99.323 ms  96.635 ms  87.962 ms
     2  192.168.8.15  99.241 ms  98.373 ms  105.315 ms
    

    Source LAN ICMP Tracert

     1  10.0.7.2  88.038 ms  86.730 ms  100.667 ms
     2  * * *
     3  * * *
    

    10.0.7.2 is the remote site's VPN endpoint so I know it's crossing the VPN, but it seems traffic dies right after passing through it. The default OpenVPN interface rules (IPv4, allow any) are in place so nothing should be blocking traffic at this point. The LAN rules are allow all as well, with not specified Gateway. Naturally, clients on the LAN experience the same drop when trying to ping to the remote LAN network.

    Any ideas? A possible outbound NAT issue?


  • Netgate

    You shouldn't need Outbound NAT in most Site-to-Site cases.

    First thing to check is for any firewalls on the target hosts themselves. Pinging the far side pfSense interface is a good test for that. If that succeeds and pinging hosts on the far side LAN fail, it means all the necessary routing is in place but something about accessing the target hosts from the source network isn't working.

    The two most common causes are:

    Local firewall on the target host
    Target host's default gateway is pointed someplace else



  • @Derelict:

    You shouldn't need Outbound NAT in most Site-to-Site cases.

    First thing to check is for any firewalls on the target hosts themselves. Pinging the far side pfSense interface is a good test for that. If that succeeds and pinging hosts on the far side LAN fail, it means all the necessary routing is in place but something about accessing the target hosts from the source network isn't working.

    The two most common causes are:

    Local firewall on the target host
    Target host's default gateway is pointed someplace else

    At this point most troubleshooting I've been doing is on the pfSense firewalls themselves. But still, I did check the LAN test clients and their default gateway is pointed their local pfsense instance.

    Summary of IPs
    source pfSense: 192.168.7.1
    source pfSense OpenVPN IP: 10.0.7.1
    target pfSense: 192.168.8.1
    target pfSense Open VPN IP: 10.0.7.2

    Current state
    The source pfSense (192.168.7.1) can only ping the target pfSense (192.168.8.1) if I select the OpenVPN connection as it's interface.
    The source pfSense CAN ping the target openVPN interface (10.0.7.2) on the LAN interface.

    The LAN ICMP tracert above implies that the LAN knows to route the traffic over the VPN, but drops off after that point. The LAN client firewalls can accept pings from other remote sites.

    Attached are screenshots of my LAN & openVPN rules on both ends. It's very minimal so nothing should be blocking this traffic.

    I'm a bit at a loss at this point since it appears that routing is functioning correctly.

    ![192.168.7.1 LAN Rules.png](/public/imported_attachments/1/192.168.7.1 LAN Rules.png)
    ![192.168.7.1 LAN Rules.png_thumb](/public/imported_attachments/1/192.168.7.1 LAN Rules.png_thumb)
    ![192.168.8.1 LAN rules.png](/public/imported_attachments/1/192.168.8.1 LAN rules.png)
    ![192.168.8.1 LAN rules.png_thumb](/public/imported_attachments/1/192.168.8.1 LAN rules.png_thumb)
    ![Both - OpenVPN Rules.png](/public/imported_attachments/1/Both - OpenVPN Rules.png)
    ![Both - OpenVPN Rules.png_thumb](/public/imported_attachments/1/Both - OpenVPN Rules.png_thumb)


  • Netgate

    Shouldn't matter to this issue but what are you trying to do with those SMB rules? They don't make a lot of sense. WAN net is not the internet. WAN net is the subnet of the WAN interface. Any is the internet.

    You are going to have to post more data. Like the OpenVPN connection profiles on each side, the routing tables, etc.



  • @Derelict:

    Shouldn't matter to this issue but what are you trying to do with those SMB rules? They don't make a lot of sense. WAN net is not the internet. WAN net is the subnet of the WAN interface. Any is the internet.

    Looking at them again you're right that they're not accomplishing what I want (which is blocking SMB out to the internet.) Most of my firewall experience is with Watchguard and Cyberoam which does things a bit differently.

    For the OpenVPN connection profiles, would that be the config file under /var/etc/openvpn? I just want to ensure I'm getting you the information you want :).


  • Netgate

    Or screen shots. I would actually rather look at screen shots than the OpenVPN .conf files.



  • OK here you go!

    Server: https://imgur.com/a/k6PqT
    Client: https://imgur.com/a/sd1CX


  • Netgate

    Ya know, this forum allows uploading of screenshots as attachments.

    SSL/TLS with larger than a /30 tunnel network also requires client-specific overrides.

    Try changing the tunnel network to a /30.


  • Rebel Alliance Global Moderator

    "SSL/TLS with larger than a /30 tunnel network also requires client-specific overrides."

    Why would you set a tunnel in a peer to peer to anything other than a 30?  But curious to what client specific overrides you would put in place.

    I don't see any sort of mention of this in the note on the tunnel network or on the doc linked to with the ? on the openvpn page..

    Since the note gives "e.g. 10.0.8.0/24" in the tunnel note I could see where this could get a bit confusing if your saying a /30 is required without extra work?

    edit:  Even the book gives a /24 as example so this statement is throwing me Derelict could you expand on what you mean exactly by requires client-specific overrides?

    Confused as to why he has his peer to peer in tap mode as well?  But that explains why his topology setting goes away.



  • In remote access mode the /30 refers to the complicated way of arranging the tunnel network (net30 in OpenVPN options) that is required on MS Windows in some cases. In peer-to-peer mode you can use a subnet of your choice, a /24 is fine, even a /31 should work because a TUN interface requires only two addresses and it's point to point.

    Please don't use TAP for peer-to-peer, it makes no sense at all.


  • Netgate

    In an SSL/TLS server with a tunnel network as a /29 or larger, the server is in server mode, not peer-to-peer mode.

    The Remote Networks in the Server configuration establish kernel routes into the OpenVPN server instance.

    When the packet is in OpenVPN it needs an iroute to know which client to send the traffic to even if there is only one.

    These iroutes are most-easily-established using the Remote Networks in a Client-Specific Override.

    The same holds true for a Remote Access SSL/TLS server but rarely comes into play because the server is only exchanging traffic with the client's tunnel address, not a network routed behind the tunnel address. Routing to the tunnel address does not require an iroute because the server knows to which client each tunnel address belongs.

    In shared-key mode you can only have one client. Even if you give a /24 as a tunnel address it is treated as a /30.

    From the book in the SSL/TLS Server Section:

    The last piece of the puzzle is to add Client Specific Overrides for each client site. These are needed to tie a client subnet to a particular certificate for a site so that it may be properly routed.

    Navigate to VPN > OpenVPN, Client Specific Overrides tab
        Click + to add a new override
        Fill in the fields on this screen as follows:

    Common Name: Enter the CN of the first client site. In this example, that is clientB.
    IPv4 Remote Network:
    This field sets up the required iroute so enter the clientB LAN subnet, 10.5.0.0/24

    Click Save

    I didn't notice the TAP mode. Yeah. Use tun mode.



  • Thank you for all the replies. I know understand why to use a /30 after looking into iroutes a bit. The end goal is to eventually have multiple sites connect in to this one server, but for now I just want a basic working connection.

    The reason for using TAP mode was that it allowed Quagga OSPF packets through to my firewalls. That was originally what was populating my routing tables. I hadn't specified the networks in the server or client configuration.

    I've modified both ends to use a /30 tunnel network now in Tun mode. I've also populated the local and remote networks at both ends in the configuration now that OSPF traffic won't pass. Does the client settings topology setting matter with a /30? I tried both and it didn't seem to behave any differently.

    I see the appropriate routes in my routing tables on both ends. With these changes, I'm still seeing the same behavior. Pings through to the LAN work fine, until I change the source to LAN. A tracert from the LAN shows it hitting the remote tunnel endpoint then dying. Any other suggestions for places to look?


  • Netgate

    Packet capture for the pings that are not making it.

    First on the OpenVPN interface to be sure they are coming across the tunnel. Then on the local interface they are supposed to go out of.

    All pretty hard to help when you are not posting actual config shots, routing tables, etc. More like guessing from here.

    You can run OSPF across OpenVPN as long as it is in peer-to-peer and not server mode (because ospf cannot insert iroutes into OpenVPN).

    All of this is in the book and in the hangout videos that come with gold.



  • I took a step back and redid my configuration to simplify troubleshooting. I'm now using a shared key peer to peer configuration with a /30 tunnel to avoid iroutes, certificates, and client overrides entirely. I've disabled Quagga and have put the network information into the configuration itself.

    I'm still seeing the same behavior. The two pfsense instances can ping each other fine, and ping through to the LAN. Anything coming from the LAN itself does not cross the tunnel.

    I initiated a persistent ping on a workstation on the "server" side LAN and the only interface that detected it in a packet trace was the LAN interface. It was not detected on any WAN or OpenVPN tunnels. The traffic appears to be dying once it hits the pfSense.

    To Recap
    Summary of IPs
    server pfSense: 192.168.7.1
    server pfSense OpenVPN IP: 10.0.7.1
    client pfSense: 192.168.8.1
    client pfSense Open VPN IP: 10.0.7.2

    Server config file

    dev ovpns1
    verb 1
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_server1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local <redacted>ifconfig 10.0.7.1 10.0.7.2
    lport 1194
    management /var/etc/openvpn/server1.sock unix
    route 192.168.8.0 255.255.255.0
    secret /var/etc/openvpn/server1.secret</redacted> 
    

    Client config file

    dev ovpnc1
    verb 1
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_client1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local <redacted>lport 0
    management /var/etc/openvpn/client1.sock unix
    remote <redacted>1194
    ifconfig 10.0.7.2 10.0.7.1
    route 192.168.7.0 255.255.255.0
    secret /var/etc/openvpn/client1.secret 
    resolv-retry infinite</redacted></redacted> 
    

    Attached are screenshots of the GUI configs, routes, firewall rules and the LAN packet capture showing it saw the pings. It should also be noted that I have gateway groups on both firewalls for WAN failover. The configuration of the gateway group for the Server side is attached as well.

    Thoughts?

    ![Client Config 1.png](/public/imported_attachments/1/Client Config 1.png)
    ![Client Config 1.png_thumb](/public/imported_attachments/1/Client Config 1.png_thumb)
    ![Client Config 2.png](/public/imported_attachments/1/Client Config 2.png)
    ![Client Config 2.png_thumb](/public/imported_attachments/1/Client Config 2.png_thumb)
    ![Client Config 3.png](/public/imported_attachments/1/Client Config 3.png)
    ![Client Config 3.png_thumb](/public/imported_attachments/1/Client Config 3.png_thumb)
    ![Client Config 4.png](/public/imported_attachments/1/Client Config 4.png)
    ![Client Config 4.png_thumb](/public/imported_attachments/1/Client Config 4.png_thumb)
    ![Client OpenVPN Rules.png](/public/imported_attachments/1/Client OpenVPN Rules.png)
    ![Client OpenVPN Rules.png_thumb](/public/imported_attachments/1/Client OpenVPN Rules.png_thumb)
    ![Client Routes.png](/public/imported_attachments/1/Client Routes.png)
    ![Client Routes.png_thumb](/public/imported_attachments/1/Client Routes.png_thumb)
    ![Server Config 1.png](/public/imported_attachments/1/Server Config 1.png)
    ![Server Config 1.png_thumb](/public/imported_attachments/1/Server Config 1.png_thumb)
    ![Server Config 2.png](/public/imported_attachments/1/Server Config 2.png)
    ![Server Config 2.png_thumb](/public/imported_attachments/1/Server Config 2.png_thumb)
    ![Server Config 3.png](/public/imported_attachments/1/Server Config 3.png)
    ![Server Config 3.png_thumb](/public/imported_attachments/1/Server Config 3.png_thumb)
    ![Server Config 4.png](/public/imported_attachments/1/Server Config 4.png)
    ![Server Config 4.png_thumb](/public/imported_attachments/1/Server Config 4.png_thumb)
    ![Server LAN Rules.png](/public/imported_attachments/1/Server LAN Rules.png)
    ![Server LAN Rules.png_thumb](/public/imported_attachments/1/Server LAN Rules.png_thumb)
    ![Server OpenVPN Rules.png](/public/imported_attachments/1/Server OpenVPN Rules.png)
    ![Server OpenVPN Rules.png_thumb](/public/imported_attachments/1/Server OpenVPN Rules.png_thumb)
    ![Server Routes.png](/public/imported_attachments/1/Server Routes.png)
    ![Server Routes.png_thumb](/public/imported_attachments/1/Server Routes.png_thumb)
    ![Server WAN Gateway Groups.png](/public/imported_attachments/1/Server WAN Gateway Groups.png)
    ![Server WAN Gateway Groups.png_thumb](/public/imported_attachments/1/Server WAN Gateway Groups.png_thumb)
    ![Server LAN Packet Capture.png](/public/imported_attachments/1/Server LAN Packet Capture.png)
    ![Server LAN Packet Capture.png_thumb](/public/imported_attachments/1/Server LAN Packet Capture.png_thumb)



  • For anyone who stumbles across this thread, the solution was to add the OpenVPN connection as a Interface on the client side. After creating the interface, restart the OpenVPN service and add allow firewall  rules for the interface. For OSPF to work, you need to add the interface on both ends.

    It's also advised to remove/disable the default OpenVPN rules as they'll supersede the interface rules if matched first.