Interface on Site-to-site VPN client cannot reach remote network

Telex

I hope this is the appropriate forum for this issue. I’m not sure if this is a VPN issue, a firewall issue, or a routing issue. However, being that setting up a VPN properly involves all 3, I thought this was the best forum for my problem.

I have two sites connected together by two routers running the latest stable release of pfSense. The router at Site A operates as an OpenVPN server, while the router at Site B is an OpenVPN client.

Site A has servers on the subnet 10.0.10.0/24.

Site B has two subnets configured:
Wired - 10.10.0.0/24
Wireless - 10.10.20.0/24

The OpenVPN server uses 10.254.0.0/24 as the tunnel network.

Both networks at Site B must be able to access the server subnet at Site A via an OpenVPN site-to-site tunnel.

The wired network at Site B has absolutely no problems accessing the servers at Site A. However, the Wireless network is unable to access anything across the VPN.

The wired and wireless networks at Site B have no problems routing between each other.

Up until about two weeks ago, everything worked beautifully. For a whole year prior to that, Site B’s wireless network was always able to access Site A’s server subnet. I cannot remember making any changes that could have affected this, other than modifying a few firewall rules.

I have made every attempt at troubleshooting this strange problem. I have tried setting all of the relevant firewall rules to allow ALL TRAFFIC from any source to any destination.

I have checked the routes, and the route to 10.0.10.0/24 through the VPN connection appears there.

Here is a traceroute comparison of wired versus wireless. As you can see, the wired wireless connection times out after hitting the incoming interface for the wireless network.
Wired

Tracing route to 10.0.10.1 over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  10.10.0.1
  2    66 ms    66 ms    65 ms  10.254.0.1
  3    73 ms    71 ms    68 ms  10.0.10.1
Trace complete.

Wireless

Tracing route to 10.0.10.1 over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  10.10.20.1
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.

To me, it appears that pfSense at Site B is attempting to route the traffic over the VPN to 10.254.0.1. However, something at Site A must be blocking. But what could that be? I have the firewall rules under the OpenVPN tab set the same for allowing both the wired and wireless traffic. And how could the same rule be passing traffic from the wired network but rejecting it from the wireless network?

Here are some configuration screenshots from the router at “Site B” (VPN client).
Wired firewall rules: http://i.imgur.com/nVx2s.png
Wireless firewall rules: http://i.imgur.com/vl5DV.png
OpenVPN client firewall rules: http://i.imgur.com/625Kf.png
Wireless interface: http://i.imgur.com/1MkCG.png
Routes: http://i.imgur.com/edqku.png

And here are the “Site A” (VPN server) OpenVPN firewall rules: http://i.imgur.com/Ktw25.png

**Note: the order of rules shown (allow all at the top) is only temporary for the purpose of diagnosing.

Telex

Any guesses?

Honestly, I don't even know what is going on anymore. For the hell of it, I just tried disabling ALL of the "pass" rules I created under the OpenVPN tab, and yet I am still able to communicate from a client on 10.10.0.0/24 to 10.0.10.1. How is this even possible? It's as if there's a phantom rule allowing some traffic, and a phantom rule blocking other traffic.

Edit: Apparently with all of the rules disabled, I can access pfsense at 10.0.10.1 but not any of the servers on that network.

Edit 2: Sorry, nevermind about that. I had to rest for a small period between ping requests before the pings began timing out. I thought that it would have been effective immediately after clicking apply and refreshing the filter. The issue with wireless being unable to access the 10.0.10.0/24 network still exists, however.

cmb

Firewall rule changes, on every firewall, don't apply to already-established connections. You have to kill states if you want them to.

Sounds like you're missing a route on the remote end for the wireless subnet. Add advanced option "route 10.10.20.0 255.255.255.0" sans quotes to the remote side.

Telex

Thanks for the reply cmb.

I added that to the advanced options under the OpenVPN server settings and let the VPN connection re-establish itself. I observed in the pfSense IPv4 routes table that the following route was added:
Destination: 10.10.20.0/24
Gateway: 10.254.0.2
Flags: UGS
Netif: ovpns1

However, I'm still unable to communicate between networks in either direction. I am still testing with a firewall rules at the top configured to "Pass" any traffic.

The Site B Wireless > Site A remote server traceroute times out at the second hop (should be 10.254.0.1), and the Site A > Site B Wireless traceroute times out at the second hop. I cannot ping the VPN gateway (10.254.0.1) from Site B's wireless network, while a ping from the wired network succeeds.

I think that the router is attempting to send the traffic through the VPN interface, since if I attempt to traceroute a different subnet not in my network (for instance, 10.0.155.1), the router will just pass it out of its WAN interface.

On a side note, I was definitely missing the 10.10.20.0 route, though. There was never a need to access the wireless network from the server net in the past, but it's better (and makes more sense) to have it anyway…as long as we can get this to work. :p

cmb

Can you ping the remote end's LAN IP? If yes, your routing is good and your firewall rules are likely good, may be passing out and not getting a response from the destination (host firewall, broken host routing table, other possibilities).

Telex

cmb, here's a breakdown of every scenario I have tried…

Site A: 10.0.10.0/24
Site B Wired: 10.10.0.0/24
Site B Wireless: 10.10.20.0/24

Site A > Site B Wired - works! (always has)
Site A > Site B Wireless - doesn't work
Site B Wired > Site A - works! can ping router's interface and host on network (always has)
Site B Wireless > Site A - doesn't work! unable to ping Site A's LAN router interface or host on network, even with host firewall disabled

One thing worth noting is that the firewall logs at remote ends are not displaying any activity. That is, if I attempt to access a webpage on port 80 at Site A from Site B's Wireless network, the Site B firewall logs say that it is PASSING traffic from my wireless host to the server at Site A. However, there is absolutely no firewall entry at Site A that reflects traffic coming from the host on Site B's wireless network. The same can be said in reverse for traffic from Site A to Site B's wireless.

Also, just throwing out there that I have tried rebooting everything.

heper

probably a route missing for the 10.10.20.0/24 at site A. this considering you've noticed the the webpage logging.

you should check the openvpn config at site A, check if you have a route for the wireless subnet there

Telex

That was actually the first thing I tried, after cmb's original reply.
@Telex:

I added that to the advanced options under the OpenVPN server settings and let the VPN connection re-establish itself. I observed in the pfSense IPv4 routes table that the following route was added:
Destination: 10.10.20.0/24
Gateway: 10.254.0.2
Flags: UGS
Netif: ovpns1

It appears to me that the route was added and it should work, but for whatever reason, I still can't successfully communicate in either direction.

Also, I noticed that if I attempt to ping 10.10.20.1 from Site A, the integer for that route's row in the "Use" column increases.

But even if I was missing a route at Site A, shouldn't I still be able to communicate from Site B's Wireless > Site A, as long as the route to 10.0.10.0/24 exists at Site B (just not in reverse)? That route already exists, and since Site B's Wired network can communicate with Site A, I know the route is properly set up.

bardelot

What are your advanced OpenVPN configuration options?

e.g server (global):
route 10.10.0.0 255.255.255.0;
route 10.10.20.0 255.255.255.0;

and server (client specific):
iroute 10.10.0.0 255.255.255.0;
iroute 10.10.20.0 255.255.255.0;

Telex

Oh my god, I feel like an idiot.

You guys were correct about the routes. Unfortunately, I made an error while adding the route, to the VPN settings.

While I added the route under the Server settings, I forgot to add the 10.10.20.0/24 route under the Client Specific Overrides. As soon as the route information was added there, communication worked bi-directionally.

Thanks so much for your help! This was a great learning experience.