Routing specific LAN segment via OpenVPN tunnel

kroem

If I can't explain this in a proper manner - please ask and I'll add more info :o :o

I'm trying to achieve routing a specific LAN-net (VLAN 99) out over a OpenVPN tunnel, and only over that tunnel.

In order to do so I have a firewall rule on VLAN99 accepting any / any with the OpenVPN tunnel gateway set. I also have a outbound NAT tied to the VLAN's subnet to translate it to the OpenVPN address.

This is a all fine and well, it works - I ping out towards an external host from a VM inside VLAN99 and all traffic in/out are flowing over the OpenVPN tunnel. (browsing also works)
However, if I try to access a service running on a VM within VLAN99 externally then the traffic from the VM back to Internet is flowing over the ordinary WAN port (probably following the default gw).

(the testing scenario that I have now is a iperf server running on a VM ) I see traffic coming inbound on the OpenVPN tunnel, reaching the server, and being sent back towards the external host over the WAN interface.

Any idea what I'm doing wrong here?

tsmalmbe

Show us a screenshot of your local/remote networks you have defined in OpenVPN. These will define what is routed.

kroem

@tsmalmbe:

Show us a screenshot of your local/remote networks you have defined in OpenVPN. These will define what is routed.

Hi! Could you clarify what you mean - defined where(I have a OpenVPN client, not server)? I have a FW-rule for VLAN99/10.99.1.0/24 to use the OpenVPN gateway.

marvosa

We would need to see your LAN firewall rules to offer more help, but I saw this posted as a solution to another post… and I wonder if may help you as well:

https://doc.pfsense.org/index.php/Bypassing_Policy_Routing

kroem

@marvosa:

We would need to see your LAN firewall rules to offer more help, but I saw this posted as a solution to another post… and I wonder if may help you as well:

https://doc.pfsense.org/index.php/Bypassing_Policy_Routing

Attached are LAN rules, the Gateway and Outbound NAT.

I'll look into the wiki post, thanks!

![Skärmklipp 2017-05-24 07.52.23.png](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.52.23.png)
![Skärmklipp 2017-05-24 07.52.23.png_thumb](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.52.23.png_thumb)
![Skärmklipp 2017-05-24 07.52.41.png](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.52.41.png)
![Skärmklipp 2017-05-24 07.52.41.png_thumb](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.52.41.png_thumb)
![Skärmklipp 2017-05-24 07.53.31.png](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.53.31.png)
![Skärmklipp 2017-05-24 07.53.31.png_thumb](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.53.31.png_thumb)
![Skärmklipp 2017-05-24 07.56.23.png](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.56.23.png)
![Skärmklipp 2017-05-24 07.56.23.png_thumb](/public/imported_attachments/1/Skärmklipp 2017-05-24 07.56.23.png_thumb)

kroem

Hmm. I dont really get the suggested setup inform wiki, if I do that it's going to match any to go out on default gateway.

(I tried, and there's no change to my situation) :|

marvosa

I don't see your LAN rules….. or did you rename one of your physical interfaces as "OPVN_99"?

johnpoz

"if I try to access a service running on a VM within VLAN99 externally then the traffic from the VM back to Internet is flowing over the ordinary WAN port (probably following the default gw). "

So sounds like to me your routing on your end is fine you rules look good, anything not local go out the vpn tunnel. Where your issue is the host your trying to talk to on the remote end is not taking the route back.

So you need to make sure this remote host routes back through your tunnel. So where does the other end of this vpn terminate? At that hosts default gateway? If so you need to make sure it routes your traffic back through the tunnel. Normally when doing a site to site you do not nat outbound traffic to be the vpn interfaces IP.

What does your routing look like at the other end?

kroem

@marvosa:

I don't see your LAN rules….. or did you rename one of your physical interfaces as "OPVN_99"?

It's a VLAN segment, OVPN_99 is the local VLAN 99 which will contain hosts/services I want to be routed out on the tunnel.

@johnpoz:

"if I try to access a service running on a VM within VLAN99 externally then the traffic from the VM back to Internet is flowing over the ordinary WAN port (probably following the default gw). "

So sounds like to me your routing on your end is fine you rules look good, anything not local go out the vpn tunnel. Where your issue is the host your trying to talk to on the remote end is not taking the route back.

So you need to make sure this remote host routes back through your tunnel. So where does the other end of this vpn terminate? At that hosts default gateway? If so you need to make sure it routes your traffic back through the tunnel. Normally when doing a site to site you do not nat outbound traffic to be the vpn interfaces IP.

What does your routing look like at the other end?

I dont really agree. The routing from the remote end is correct. I access the VM (on my LAN, sorry, maybe that was not clear…) through the tunnel but the return is routed back over the normal WAN-interface - IF the connection is established from remote. IF I establish connection from the local end - ping/trace/browser out - then the routing is out/in over the tunnel.

EDIT: Appreciate all the help/efforts :)

marvosa

Ok, OVPN_99 is a VLAN, gotcha.

Just curious, what is in "pfB ET comprblock Spamhaus drop" alias?

Something's not adding up… because according to what I'm looking at, there are no rules allowing outbound traffic to the internet at all, so I'm not sure how ANY traffic sourced from this VLAN (99) is making it anywhere. It should all be blocked by the firewall.

Also, if you only want this VLAN accessible and routed thru your VPN, then why are there rules configured with the default gateway on this tab?

Another mystery, rule #3 (at the bottom), reads... allow traffic sourced from VLAN99 destined to other LAN interfaces and route that traffic over the VPN? I'm not sure why that's there..... why would you route local traffic between two interfaces on the same machine over a VPN tunnel?

I know you've already stated this, but I'm going to ask this specific question anyway because I hate to assume anything... is there any chance the rules you've listed are actually configured on the interface assigned to the tunnel instead of the VLAN interface? Please post the rules from both rules from both the VLAN interface and the interface assigned to the tunnel.

I also just noticed that your "LOCAL NETS" alias has vlan99 listed at 10.99.0.0/24, but your NAT mappings show "OVPN_99" lists the source as 10.99.1.0/24. Is there a simple typo in your alias? Or is there another reason for this discrepancy?

CuteBoi

https://forum.pfsense.org/index.php?topic=131074.0

This smells like the same exact issue I'm having. Word for word, port forwarding, the packets are attempting to go out my WAN, while the 2 hosts on the LAN that have rules assigning the VPN gateway work as intended.

I'll follow this thread as well.

kroem

@marvosa:

Ok, OVPN_99 is a VLAN, gotcha.

Just curious, what is in "pfB ET comprblock Spamhaus drop" alias?

That's a pfBlocker list. I only recently added it, it was the same without it.

@marvosa:

Something's not adding up… because according to what I'm looking at, there are no rules allowing outbound traffic to the internet at all, so I'm not sure how ANY traffic sourced from this VLAN (99) is making it anywhere. It should all be blocked by the firewall.

Also, if you only want this VLAN accessible and routed thru your VPN, then why are there rules configured with the default gateway on this tab?

But there is, actually any traffic from the network 10.99.1.0/24 is allowed out to Internet via OVPN Gateway. I've tried a any/any (source any) rule going to the OVPN gateway and it's not working either.
The local nets rule is to access it from my other networks.

@marvosa:

Another mystery, rule #3 (at the bottom), reads… allow traffic sourced from VLAN99 destined to other LAN interfaces and route that traffic over the VPN? I'm not sure why that's there..... why would you route local traffic between two interfaces on the same machine over a VPN tunnel?

No, that's sourced from VLAN99 not to other local nets. I.e. LAN traffic.

@marvosa:

I know you've already stated this, but I'm going to ask this specific question anyway because I hate to assume anything… is there any chance the rules you've listed are actually configured on the interface assigned to the tunnel instead of the VLAN interface? Please post the rules from both rules from both the VLAN interface and the interface assigned to the tunnel.

No, it's the VLAN interface :(

I tried adding the OVPN gateway on the tunnel interface rule - no difference (as a test). The OVPN rules are attached.

@marvosa:

I also just noticed that your "LOCAL NETS" alias has vlan99 listed at 10.99.0.0/24, but your NAT mappings show "OVPN_99" lists the source as 10.99.1.0/24. Is there a simple typo in your alias? Or is there another reason for this discrepancy?

Typo from me! godfuckingdammit - that's like the 5th time :-[ :-[ Anyways, it's the same with the correct subnet.

As a reference - here's the tcpdump from OpenVPN interface, VLAN and WAN interface. First. I'm trying to establish a iperf session towards my VM1 host, then I ping out from VM1 to the server which tried to establish connection:

OVPN_99 interface, traffic flowing "correctly"
[code]

[2.3.3-RELEASE][root@fw-pf.kroem.eu]/root: tcpdump -i igb1_vlan99 -nn host 80.239.147.29
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1_vlan99, link-type EN10MB (Ethernet), capture size 65535 bytes
22:10:55.944376 IP 80.239.147.29.42200 > 10.99.1.201.59362: Flags ~~, seq 3526785419, win 29200, options [mss 1349,sackOK,TS val 34928838 ecr 0,nop,wscale 7], length 0
22:10:55.944659 IP 10.99.1.201.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33974369 ecr 34928838,nop,wscale 7], length 0
22:10:56.943438 IP 80.239.147.29.42200 > 10.99.1.201.59362: Flags ~~, seq 3526785419, win 29200, options [mss 1349,sackOK,TS val 34929088 ecr 0,nop,wscale 7], length 0
22:10:56.943652 IP 10.99.1.201.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33974619 ecr 34928838,nop,wscale 7], length 0
22:10:58.141497 IP 10.99.1.201.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33974919 ecr 34928838,nop,wscale 7], length 0
22:10:58.947546 IP 80.239.147.29.42200 > 10.99.1.201.59362: Flags ~~, seq 3526785419, win 29200, options [mss 1349,sackOK,TS val 34929589 ecr 0,nop,wscale 7], length 0
22:10:58.947812 IP 10.99.1.201.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33975120 ecr 34928838,nop,wscale 7], length 0
22:11:01.341273 IP 10.99.1.201.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33975719 ecr 34928838,nop,wscale 7], length 0
22:11:02.955574 IP 10.99.1.201 > 80.239.147.29: ICMP echo request, id 2099, seq 1, length 64
22:11:02.977335 IP 80.239.147.29 > 10.99.1.201: ICMP echo reply, id 2099, seq 1, length 64
22:11:03.956783 IP 10.99.1.201 > 80.239.147.29: ICMP echo request, id 2099, seq 2, length 64
22:11:03.978472 IP 80.239.147.29 > 10.99.1.201: ICMP echo reply, id 2099, seq 2, length 64
22:11:04.957965 IP 10.99.1.201 > 80.239.147.29: ICMP echo request, id 2099, seq 3, length 64
22:11:04.979579 IP 80.239.147.29 > 10.99.1.201: ICMP echo reply, id 2099, seq 3, length 64
22:11:05.959203 IP 10.99.1.201 > 80.239.147.29: ICMP echo request, id 2099, seq 4, length 64
22:11:05.980780 IP 80.239.147.29 > 10.99.1.201: ICMP echo reply, id 2099, seq 4, length 64

OpenVPN tunnel interface, you see the traffic incoming from the remote host trying to establish connection, but no replies. The ping request and replies are both here:


[2.3.3-RELEASE][root@fw-pf.kroem.eu]/root: tcpdump -i ovpnc3 -nn host 80.239.147.29
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovpnc3, link-type NULL (BSD loopback), capture size 65535 bytes
22:10:55.944344 IP 80.239.147.29.42200 > 10.128.164.2.59362: Flags [s], seq 3526785419, win 29200, options [mss 1349,sackOK,TS val 34928838 ecr 0,nop,wscale 7], length 0
22:10:56.943427 IP 80.239.147.29.42200 > 10.128.164.2.59362: Flags [s], seq 3526785419, win 29200, options [mss 1349,sackOK,TS val 34929088 ecr 0,nop,wscale 7], length 0
22:10:58.947523 IP 80.239.147.29.42200 > 10.128.164.2.59362: Flags [s], seq 3526785419, win 29200, options [mss 1349,sackOK,TS val 34929589 ecr 0,nop,wscale 7], length 0
22:11:02.955619 IP 10.128.164.2 > 80.239.147.29: ICMP echo request, id 31135, seq 1, length 64
22:11:02.977311 IP 80.239.147.29 > 10.128.164.2: ICMP echo reply, id 31135, seq 1, length 64
22:11:03.956792 IP 10.128.164.2 > 80.239.147.29: ICMP echo request, id 31135, seq 2, length 64
22:11:03.978450 IP 80.239.147.29 > 10.128.164.2: ICMP echo reply, id 31135, seq 2, length 64
22:11:04.957975 IP 10.128.164.2 > 80.239.147.29: ICMP echo request, id 31135, seq 3, length 64
22:11:04.979558 IP 80.239.147.29 > 10.128.164.2: ICMP echo reply, id 31135, seq 3, length 64
22:11:05.959207 IP 10.128.164.2 > 80.239.147.29: ICMP echo request, id 31135, seq 4, length 64
22:11:05.980760 IP 80.239.147.29 > 10.128.164.2: ICMP echo reply, id 31135, seq 4, length 64

WAN interface, only the replies to the iperf session is here...
[code]
[2.3.3-RELEASE][root@fw-pf.kroem.eu]/root: tcpdump -i igb0 -nn host 80.239.147.29 and port not ssh
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 65535 bytes
22:10:55.944666 IP 10.128.164.2.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33974369 ecr 34928838,nop,wscale 7], length 0
22:10:56.943658 IP 10.128.164.2.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33974619 ecr 34928838,nop,wscale 7], length 0
22:10:58.141509 IP 10.128.164.2.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33974919 ecr 34928838,nop,wscale 7], length 0
22:10:58.947823 IP 10.128.164.2.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33975120 ecr 34928838,nop,wscale 7], length 0
22:11:01.341281 IP 10.128.164.2.59362 > 80.239.147.29.42200: Flags [S.], seq 2629731937, ack 3526785420, win 28960, options [mss 1460,sackOK,TS val 33975719 ecr 34928838,nop,wscale 7], length 0
^C
[/code]

It does look strange...

![Skärmklipp 2017-05-25 22.02.41.png](/public/_imported_attachments_/1/Skärmklipp 2017-05-25 22.02.41.png)
![Skärmklipp 2017-05-25 22.02.41.png_thumb](/public/_imported_attachments_/1/Skärmklipp 2017-05-25 22.02.41.png_thumb)[/s][/s][/s]
```~~~~~~

kroem

@CuteBoi:

https://forum.pfsense.org/index.php?topic=131074.0

This smells like the same exact issue I'm having. Word for word, port forwarding, the packets are attempting to go out my WAN, while the 2 hosts on the LAN that have rules assigning the VPN gateway work as intended.

I'll follow this thread as well.

Seems fishy - however, I'm on 2.3.3, if that really makes a difference. Dod you setup work before you upgraded?

johnpoz

Why are you doing a pfblocker rule on your vpn connection… In what scenario would you have traffic you don't want coming from some spam source via your vpn connection??

I am confused on how your seeing traffic on your vpn tunnel with that private address 80.x.x.x

So your using using public IP space on one end of this tunnel? Your vpn sniff on that interface should only show your internal IPs
15:24:06.293864 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1086, length 40
15:24:06.294602 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1086, length 40
15:24:07.282752 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1087, length 40
15:24:07.283160 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1087, length 40
15:24:08.289218 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1088, length 40
15:24:08.289814 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1088, length 40
15:24:09.295496 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1089, length 40
15:24:09.295787 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1089, length 40

kroem

@johnpoz:

Why are you doing a pfblocker rule on your vpn connection… In what scenario would you have traffic you don't want coming from some spam source via your vpn connection??

I am confused on how your seeing traffic on your vpn tunnel with that private address 80.x.x.x

So your using using public IP space on one end of this tunnel? Your vpn sniff on that interface should only show your internal IPs
15:24:06.293864 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1086, length 40
15:24:06.294602 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1086, length 40
15:24:07.282752 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1087, length 40
15:24:07.283160 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1087, length 40
15:24:08.289218 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1088, length 40
15:24:08.289814 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1088, length 40
15:24:09.295496 IP 10.0.8.100 > 192.168.9.100: ICMP echo request, id 1, seq 1089, length 40
15:24:09.295787 IP 192.168.9.100 > 10.0.8.100: ICMP echo reply, id 1, seq 1089, length 40

Well it's not a site to site vpn, it's a vpn tunnel wan service, so basically a privacy service. The openvpn server is not hosted on the machine I'm establishing a connection from, it's hosted in a DC in the middle.

Regarding the pfblockerng rule, since it's a WAN connection, I want the same ruleset for this interface as any other. Anyways I've tried with and without it - no difference.

marvosa

Unless this is a bug (which it very well may be), my guess is the firewall rules are just over complicated and not being followed like you'd expect.

What VPN service are you using? I'd like to test the same scenario on my end.

kroem

@marvosa:

Unless this is a bug (which it very well may be), my guess is the firewall rules are just over complicated and not being followed like you'd expect.

What VPN service are you using? I'd like to test the same scenario on my end.

Yes, but I need to know how the packet forwarding differs and what rules are being looked at differently when traffic is established from the outside versus inside.

The service is OVPN (https://www.ovpn.com/en) you can get a test account for a few hours I believe…

johnpoz

"it's a vpn tunnel wan service"

So in that case you would be natting to what the exit point is on that vpn service. If your not seeing return traffic..

"if I try to access a service running on a VM within VLAN99 externally"

This would have to be done via a port forward on the vpn service.. Your asking for unsolicited traffic to be sent down their tunnel.. Which you would then have to port forward on pfsense to what IP you want it to go to.. With a vpn privacy service your going to be behind a double nat..

So your client

192.168.1.100 –-> pfsense privateTunnelIP ---- vpn ----> VPNservice --- VPNpublicIP ----> internet

So to the internet you look like VPNpublicIP, answers get sent back to it, vpnservice state tables says oh that is privateTunnelIP so gets sent back to pfsense, pfsense says oh that is 192.168.1.100

What your asking for is unsolicated traffic..

Im on the internet and I talk to VPNpublicIP on port X.. That service has to have a port forward that says hey send that to privatedTunnelIP on pfsense. Pfsense has to have a port forward that says oh traffic hitting me on my privateTunnelIP on port X gets forwarded to IP 192.168.1.?

kroem

@johnpoz:

"it's a vpn tunnel wan service"

So in that case you would be natting to what the exit point is on that vpn service. If your not seeing return traffic..

"if I try to access a service running on a VM within VLAN99 externally"

This would have to be done via a port forward on the vpn service.. Your asking for unsolicited traffic to be sent down their tunnel.. Which you would then have to port forward on pfsense to what IP you want it to go to.. With a vpn privacy service your going to be behind a double nat..

So your client

192.168.1.100 –-> pfsense privateTunnelIP ---- vpn ----> VPNservice --- VPNpublicIP ----> internet

So to the internet you look like VPNpublicIP, answers get sent back to it, vpnservice state tables says oh that is privateTunnelIP so gets sent back to pfsense, pfsense says oh that is 192.168.1.100

What your asking for is unsolicated traffic..

Im on the internet and I talk to VPNpublicIP on port X.. That service has to have a port forward that says hey send that to privatedTunnelIP on pfsense. Pfsense has to have a port forward that says oh traffic hitting me on my privateTunnelIP on port X gets forwarded to IP 192.168.1.?

Yes, the traffic is port forwarded on the VPN service provider ingress, it's routed via my tunnel and port forwarded to the local host.

All seems fine here, since I do see traffic incoming (as per the tcpdumps).

The same for an established session from local host.

CuteBoi

Per traffic in the tcpdump, the traffic IS arriving to the VPN ip, and arrives at the private IP in his vlan 99 group, and even responds, but the response is attempting to go out through the WAN, instead of the VPN gateway.

Yet he has the rules right there to use the gateway.