OpenVPN (Routing?) Issue (SOLVED)
-
Hello all,
I try to establish a VPN tunnel between my two offices. There both endpoints are pfSense. The setup so far is:
Site A:
(1) OpenVPN Server, port 1194 udp / TUN / Remote Access for roadworriers. This just works.
(2) OpenVPN Server, port 443 tcp / TUN / Peer to Peer for Site B. This works kind of (see below).Site B:
OpenVPN Client, port 443 tcp / TUN / Peer to Peer that connects to (2) on Site A.The pfSense box on Site B has two physical NICs - one connected to/as WAN and one that transports multiple VLANs. I want the OpenVPN tunnel to route one of these VLANs (VLAN_1) to Site A.
My box on Site B does connect to the OpenVPN server on Site A and I can ping from the pfSense box itself. But I cannot ping from a client on VLAN_1.
For testing I have an "allow all" rule on VLAN_1. I have also "allow all" rules on OpenVPN and OVPN (the name I gave the OPT interface) (btw: what exactly is the difference?).When I ping from pfSense (Site B) itself I get an answer and can observe packets beeing sent. The strange thing is if I ping from a client on VLAN_1 and watch pfInfo I can see packets flow in (In4/Pass) at the VLAN_1 adapter and flow out (Out4/Pass) at ovpnc1. BUT: I can not observe any packets at Site A coming in (also using pfInfo).
If I ping from the pfSense box at Site B itself I can observe traffic reaching Site A (and pfSense on Site A can answer back to Site B). I can also ping from Site A to Site B (but of course not beyond pfSense itself).
In my desperation I even tried to bridge VLAN_1 to ovpn (and yes, I did a reboot) but that a) did not work and b) is not what I want.
I think I am missing something obvious but honestly I don't know where to go from here. What else can I try?
PS: the whole pfSense setup on Site B runs inside a virtual machine (vmware esxi) but I think that is unrelated as the tunnel itself seems to work (I can ping across from the box itself) and the ping from the client computer on VLAN_1 reaches pfSense also.
EDIT: Here is the solution:
https://forum.pfsense.org/index.php?topic=108785.msg659344#msg659344 -
What kind of Site-Site did you setup for SiteA Server #2, PKI or Shared Key?
-
What kind of Site-Site did you setup for SiteA Server #2, PKI or Shared Key?
Shared Key. But how would that make a difference? The tunnel does (kind of) work and only my routing (or whatever it is) is borked.
-
For PKI, I'd tell you to look for a missing iroute entry or a specific CSO entry for the connecting site.
I don't think that's needed normally for Shared Key setups.The other obvious thing to watch out for is an "Allow All" rule for the OpenVPN Firewall tab on both the server and remote ends.
BTW the OpenVPN Firewall tab gets created when you start OpenVPN because a new interface is created for the VPN tunnel.
It has nothing to do with OPT1 and renaming that to OVPN is only going to cause you confusion IMHO, I'd rename it back so you're clear on what you're dealing with.Can you post your full OpenVPN Server screen shot?
-
If you diagram your network treating "OpenVPN" as a router between the two units you can look at it like this:
-
Every time a decision must be made where to send traffic there has to be a route. These are "routes" outside of OpenVPN and "iroutes" inside of OpenVPN.
-
Every time a new connection ENTERS pfSense, there has to be a firewall rule passing it.
It sounds like you don't have everything that needs to be in the Local and Remote networks in the right places, or CSOs like divsys said.
Maybe I'll diagram this tonight. Once it clicks OpenVPN is pretty simple.
For testing I have an "allow all" rule on VLAN_1. I have also "allow all" rules on OpenVPN and OVPN (the name I gave the OPT interface) (btw: what exactly is the difference?).
The OpenVPN tab is really just an interface group containing all OpenVPN clients and servers. It is processed BEFORE interfaces, including OpenVPN assigned interfaces. If is fine to use for endpoints that have few if any rules but for more complicated sites with multiple clients and servers I like assigned interfaces. You can't perform neat things like NAT and policy routing on the OpenVPN group (it's a group, not an interface).
Note that if traffic is matched by the OpenVPN group rules, they're first so your assigned interface rules won't ever match.
I generally delete all OpenVPN group rules (or move them to the assigned interface) when I start using interfaces. -
-
… on the OpenVPN group (it's a group, not an interface)
That was the missing bit :)
As for CSOs: I don't have any.
I have attached a screenshot of my configuration and (just in case) the firewall rules. They are the same for OVPN_WORK, VLAN_WORK (which I described before as VLAN_1 for brevity) and OpenVPN.
-
The "IPV4 Tunnel Network" and "IPv4 Remote network" are key pieces we need to see to verify if all the pieces are right.
They should both be internal addresses and are not accessible via the internet, so it's safe to post them. -
The "IPV4 Tunnel Network" and "IPv4 Remote network" are key pieces we need to see to verify if all the pieces are right.
They should both be internal addresses and are not accessible via the internet, so it's safe to post them.You are of course right. I have attached screenshots of both client and server configurations.
I just want to repeat that the tunnel does work from the pfSense box (Site B) itself. And I can observe traffic flowing out (Out4/Pass) at ovpnc1 (as seen in pfInfo). I fear there is a subtle bug/incompatibility with my vlan setup or something but maybe that is just a red herring.
Thanks for your efforts so far :)
-
On the client side, remove the 192.168.0.0/24 entry from the "IPv4 Remote networks" field.
That box should be normally filled on the server side only.Restart the client and see if that helps.
-
On the client side, remove the 192.168.0.0/24 entry from the "IPv4 Remote networks" field.
That box should be normally filled on the server side only.Restart the client and see if that helps.
Good suggestion but same behavior as befor: When pinging from a computer on VLAN_WORK I see packets coming "In4/Pass" at the VLAN_WORK nic and "Out4/Pass" at ovpnc1 at the pfSense client but nothing coming in at the server. When I ping from the pfSense client box itself I see "Out4/Pass" at ovpnc1 at the client and "In4/Pass" at ovpnc at the server.
Thanks for your effort. Any other ideas?
-
Any chance you're hitting the (stupid) Windoze firewall issue blocking "unknown" external subnets?
-
Any chance you're hitting the (stupid) Windoze firewall issue blocking "unknown" external subnets?
Nope. Traffic does enter the pfSense box so any routing up to this box is correct. Also no filtering otherwise I would not see traffic flowing in.
I have an entirely other suspicion right now: how exactly does pfSense handle VLAN tagging? My setup in ASCII art is like this:
–----------------- Site-B ---------------------- --------------------- Site-A --------------------------
VLAN_WORK ??? ??? LAN
(ClientComputer) -------> pfSenseB -------------> OpenVPN ----------> pfSenseA ---------> 192.168.0.0/24 networkSo I am trying to route vlan tagged packets through an OpenVPN tunnel to a normal and simple non-vlan network. For this to work properly one of the pfSense boxes has to untag the VLAN packets but I have not seen any option to route tagged or untagged packets. My suspicion is that either pfSenseB or pfSenseA does not what I expect it to do.
I'll see if I can install another NIC in my pfSense box to verify.
-
It's 802.1q. Traffic is either untagged (em0) or tagged (em0_vlan999).
You can't "route" tagged "packets (frames)". Tagged frames are layer 2, not layer 3.
If the layer 2 traffic hits the right interface and should be routed into OpenVPN, it will be.
-
If the layer 2 traffic hits the right interface and should be routed into OpenVPN, it will be.
This is how I understood IP layering to work. But this OpenVPN gives me headaches - it should be dead simple (and it was for the roadworrier setup) but this one gets over my head :(
I am a developer and not so much a network guy and I am feeling blind - how am I supposed to debug this issue?
Sorry for ranting :)
-
My box on Site B does connect to the OpenVPN server on Site A and I can ping from the pfSense box itself. But I cannot ping from a client on VLAN_1.
For testing I have an "allow all" rule on VLAN_1. I have also "allow all" rules on OpenVPN and OVPN (the name I gave the OPT interface)Every time a decision must be made where to send traffic there has to be a route. These are "routes" outside of OpenVPN and "iroutes" inside of OpenVPN.
Every time a new connection ENTERS pfSense, there has to be a firewall rule passing it.–----------------- Site-B ---------------------- --------------------- Site-A --------------------------
VLAN_WORK ??? ??? LAN
(ClientComputer) -------> pfSenseB -------------> OpenVPN ----------> pfSenseA ---------> 192.168.0.0/24 networkI am a developer and not so much a network guy and I am feeling blind - how am I supposed to debug this issue?
So on Client computer apply the tests. No firewall rules to check here. Is there a route to 192.168.0.0/24? Probably default gateway.
On to pfSense B. Is there a firewall rule on that interface allowing traffic from Client B to 192.168.0.0/24? Is there a route to 192.168.0.0/24? Should be one into OpenVPN.
On to OpenVPN. No firewall rules to check here. Is there a route (iroute) to 192.168.0.0/24?
On to pfSense A. Is there a firewall rule on OpenVPN allowing the traffic from 10.0.43.0/24 to 192.168.0.0/24? Is there a route to 192.168.0.0/24 (should be a connected subnet).
On to the host on 192.168.0.0/24. Is there a firewall rule passing traffic from 10.0.43.0/24 to the host interface? This trips up a lot of people because Windows firewall is easy to forget about until you try to connect to it from another subnet.
You have to do the same in reverse at every hop for routes to 10.0.43.0/24. You generally don't have to worry about firewall rules for the return traffic because pfSense is stateful.
Make sure your firewall rules pass all traffic, and not just TCP-only or something (which doesn't pass pings). That sometimes happens too.
Diagnostics > Routes
Status > OpenVPN
netstat -r command in windows.
netstat -rn -finet command on Mac and pfSense command line. -
Thank you very much for this detailed checklist. I appreciate your help. Unfortunately no progress so far:
So on Client computer apply the tests. No firewall rules to check here. Is there a route to 192.168.0.0/24? Probably default gateway.
I have a default route with pfSenseB as gateway. I have tried 192.168.0.0/24 with pfSenseB as gateway without success too.
On to pfSense B. Is there a firewall rule on that interface allowing traffic from Client B to 192.168.0.0/24? Is there a route to 192.168.0.0/24? Should be one into OpenVPN.
"Allow all" on OVPN_WORK as well as on the group OpenVPN.
On to OpenVPN. No firewall rules to check here. Is there a route (iroute) to 192.168.0.0/24?
Where do I check this? on "Status > OpenVPN" I have:
Name Status Virtual Addr Remote Host
Client TCP up 10.0.43.6 Site-A-public-IPOn to pfSense A. Is there a firewall rule on OpenVPN allowing the traffic from 10.0.43.0/24 to 192.168.0.0/24? Is there a route to 192.168.0.0/24 (should be a connected subnet).
"Allow all" on everything but WAN. WAN allows TCP on port 443 (as well as UDP just in case).
What do you mean with "should be a connected subnet"? I know what this is from a networking point of view but what do you mean with regard to pfSense/OpenVPN in this context?
On to the host on 192.168.0.0/24. Is there a firewall rule passing traffic from 10.0.43.0/24 to the host interface? This trips up a lot of people because Windows firewall is easy to forget about until you try to connect to it from another subnet.
I agree. Windows Firewall can trip you up so badly… but as I am able to ping from pfSense B I think I can rule this out. Nontheless I have tried to disable the firewall completely without success.
You have to do the same in reverse at every hop for routes to 10.0.43.0/24. You generally don't have to worry about firewall rules for the return traffic because pfSense is stateful.
I think these are OK also (see below).
Make sure your firewall rules pass all traffic, and not just TCP-only or something (which doesn't pass pings). That sometimes happens too.
Very good point but unfortunately did not apply to me.
Interestingly I can ping from a computer on 192.168.0.0 (Site A) to the far side of the tunnel (Windows machine):
Pinging 10.0.43.6 with 32 bytes of data:
Reply from 10.0.43.6: bytes=32 time=41ms TTL=63When I try the same from my local net on Site B I get (FreeBSD)
PING 10.0.43.1 (10.0.43.1): 56 data bytes
^C
–- 10.0.43.1. ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss -
Digging this thread from its grave to post my solution: I enabled "Client Specific Overrides" and literally copy-and-pasted my configuration from the "Servers" tab. I have no idea whatsoever why this would be needed but everything works now.
If someone could explain why I needed doing so this could maybe help another poor soul with the same problem.