Site-to-site tunnel, only endpoints can ping other side. [SOLVED]
I run IT for a local construction company. I've set up a pfsense router (running on our old Dell server) as our core router at the office. We also have one remote worker, who connects his phone and computer via a SonicWall (not my choice), IPSECed into the main router. Various other employees connect via OpenVPN on an irregular basis (need a file out in the field, etc.)
What I'm trying to set up now is a "Mobile Office Kit", using an SG-1000, a PoE switch, a wifi AP and a phone (Avaya to a locally hosted PBX). The idea is that, wherever you are (in real life or in network space - as long as you can hit the internet), you should be able to plug in the WAN port on the sg-1000, and have a phone, some ethernet ports, and a wifi connection that are all "on our network". After realizing IPSEC was going to be far more of a hassle to set up than it should be (and maybe that's still the right option), I decided to try an OpenVPN setup instead.
Right now I have the SG-1000 set up on my phone's hotspot, like so:
Laptop (192.168.26.10) - SG-1000 (192.168.26.1 lan / 192.168.1.10 wan) - dd-wrt router (192.168.1.1 lan / 192.168.43.10 wifi) - phone (192.168.43.1 wifi / 66.x.x.x wan).
The basic setup is fine, everyone has internet access.
The openvpn link is set up as 192.168.27.0/24 (so the main router is 192.168.27.1 and the sg-1000 is 192.168.27.2). For testing I have removed the IPSEC tunnel and the associated SPDs on the main router (I know I'll have to do some NAT wizardry to make this work as expected with the IPSEC tunnel up). The openVPN connection is set to access a few of our VLANs (192.168.x.0/24 where x is 2, 3, 4, and 10. There is also VLAN 11 and 99 which I don't want connected).
Where I'm sitting at now, the two pfsense routers can ping each other by OpenVPN IP (192.168.27.x). The SG-1000 can ping hosts on the network as it should (192.168.3.2 for example). However, even though all the routes are set up correctly, the main router can't ping the sg-1000 or anything on its subnet of 192.168.26.0/24. The SG-1000, pinging as 192.168.26.1 (or anything else in 192.168.26.0/24) cannot reach anything across the link. Likewise, the main router can't hit 192.168.27.2 pinging as any of its other addresses.
Anything that I'm trying to ping has the proper gateway set up (checked with route show). There are no entries in the firewall blocking logs related to these pings (and there's enough allow all rules in there that there really shouldn't be a firewall problem). Packet capture reveals the real problem: for any of the pings that don't work, I can see them leaving the openvpn interface on one side, but the requests never make it to the other side's openvpn interface. The pings that do work transfer properly, of course.
I know there's plenty of config and logs that you want to see, let me know and I'll get them in here (I'm not sure what's important at this point). What would cause the link itself to drop traffic like this? And, in general, how do I get this thing to work as-is?
@mkernalcon Couple of observations/comments and a basic question or two:
In general your concept of a "field VPN kit" is good. You'll need to watch out for the fact that you'll likely be "dual NATting" (or even triple) your field router in some cases - your router will get it's internet behind someone else's router.
Try to avoid pre-assigning/using the "common" 192.168.0.0/24 & 192.168.1.0/24 ranges in your setup - likely you'll end up behind someone else's router that assigns you a WAN address in those ranges which will cause grief when it overlaps with a range you're trying to use elsewhere. I think your example setup is OK in this regard, but you could try changing out the dd-wrt subnet range to eliminate the possibility.
It would definitely be helpful if you could try and diagram your intended setup a little more completely including the Home Office end and the subnet's you intend to use, so we can all be on the same page. You may find this gives you a better picture of the whole issue and might identify things you didn't think of initially.
You don't mention what type of OpenVPN connection you've created. For Site-Site links I personally prefer PKI (dedicated certificates) and I would suggest the SG-1000 be configured as the client. You'll need a Client Specific Option to make sure the Main router can send the appropriate traffic to the SG-1000
From what you've described so far, the only firewall rules needed would be an "allow UDP" on the main router's S2S OpenVPN port and an "allow all" on the OpenVPN tab of both the Main router and SG-1000.
@divsys The dd-wrt router isn't a permanent part of the kit, for testing I've been in the office, using my phone's wireless hotspot feature, and the dd-wrt is just put in place as a client (so I can "plug in" to that internet). 192.168.0.0/24 and 192.168.1.0/24 are not actually used anywhere in the setup. I'm a bit worried about .2., but I can't eliminate every possible collision without some dark scripting magic..
The kit side is just 192.168.26.0/24, plus whatever subnet the WAN gets DHCP on (plus 192.168.27.2 for the OpenVPN tunnel)
The office side has 192.168.x.0/24 where x is: 2, 3, 4, 10, 11, and 99, plus 25 for the other OpenVPN clients (for computer connections), 50 for the IPSEC vpn, 27.1 for the tunnel, and a static route to a weird 172.16.0.0/24 network behind my NVR.
I would like the SG-1000 to route any traffic aimed at 2, 3, 4 or 10 through the tunnel and eventually onto its host on the office side. I'd like the office router to route traffic aimed at 26 through the tunnel to the SG-1000 (and onto the correct host), but this is less important.
Firewall rules are set up as you say. It's acting as though neither side is routing unless the source is on .27. (and there aren't dropped packets in the firewall logs).
Server mode is "Peer to Peer (SSL/TLS)", no TLS, and layer 3 tun mode.
Do you have a Client Specific option on the server side to route the subnets appropriately?
@divsys I didn't think that I needed a client-specific option at all, since this is the only client connecting to this OpenVPN Server instance. In the main config for the server instance (and on the sg-1000, but backwards of course), I have the local and remote networks set up properly, and on connect, all the proper routes are created on both sides. I have no custom options. Is there a need to create a client specific override for this setup?
You always need a CSO to route subnets in an SSL/TLS server mode with a tunnel network larger than /30.
The typical remote access OpenVPN server does not include routed subnets - only tunnel addresses. The CSO is not necessary for the tunnel addresses.
The Remote Networks in the server establish the kernel routes into OpenVPN. The Remote Networks in the CSOs establish iroutes in OpenVPN so it knows which client to route that traffic to.
@derelict Yup, this was it. The routing even seems to work with my IPSEC tunnel still in place. If this was mentioned in the book, I must have read right over it!