Working OpenVPN tun that can access the LAN reliably

[TD22057]

I thought I would post this since it took me several days to get working correctly and I've seen other people having similar problems.  Hopefully it can help someone else.

I want to access my LAN via OpenVPN from my Android smart phone. Android doesn't support TAP (except through a paid app) so I'd prefer to use TUN.  Here's what I did:

I'm running pfSense 2.3.1.  My local LAN is 192.168.1.x, the router is at 192.168.1.1, and the VPN network is 192.168.2.x.

Create a user account and password to use for vpn connections (system->user manager).  Use the OpenVPN wizard to create the connection.  I used local user access and created new certificates.  Inputs not in this list were left as the default (blank usually).

Interface: WAN
Protocol: UDP
Port: whatever you want
Tunnel network: 192.168.2.0/24
IPv4 Local Network: 192.168.1.0/24
Compression: adaptive
Dynamic IP: checked
Address pool: checked
DSN server: 192.168.1.1

Let the wizard add firewall rules for the server and clients.  Then edit the OpenVPN settings (vpn->openvpn) and scroll all the way to the bottom and add this to the custom options:

route add 192.168.1.0 mask 255.255.255.0 gw 192.168.1.1

and hit save.  Check the firewall rules (firewall->rules).  Under WAN, there should be a rule for the WAN->server port.  Then install the open vpn export plugin (system->package manager).  Go to vpn->openvpn->client export, pick the server name, change the host name to dynamic DNS (in my case - yours may vary).  Then export the archive which will download a zip file.  Unpack it which should give three files (.ovpn config and certs).

I moved the three files to my phone and imported them into the OpenVPN app.  I used the user/password to connect and it worked fine (don't try and connect with your local wifi - make sure your phone is using the cellular connection).  I could ping hosts on the LAN, they could ping my phone, and I could ssh to a linux server on my LAN.  I could also access the internet with no problem.

However, when I tried to access a local web server on my LAN, the browser would hang.  Trivial files worked fine, but any normal page just hung.  The main reason I enabled VPN was to access local security cameras that I didn't want to have any access to the WAN (unknown firmware security).  The camera app would never connect to the local cameras and just sat there.

After doing some research, I found out that the MTU size needs to be reduced for slow (cellular) connections.  Here is a good link which explains how to do this.  After connecting via vpn, I pinged my phone using the instructions on that link and found that a MTU of ~1300 worked fine.  So edit the VPN server (vpn->openvpn) and change the custom options to look something like this:

route add 192.168.1.0 mask 255.255.255.0 gw 192.168.1.1;fragment 1250;mssfix 1250;

Then edit the .ovpn file that was sent to the client.  Add two lines "fragment 1250" and "mssfix 1250" to the end of that file and save it.  Delete the connection on the phone and re-import it.

Once I did that, I can now access everything on my LAN.

[johnpoz]

So in other words you ran the wizard.. And changed the mtu for your crappy cell service.. ;)

[TD22057]

@johnpoz:

So in other words you ran the wizard.. And changed the mtu for your crappy cell service.. ;)

Yes, but…  when you can't get something to work and you're banging your against the wall for days and days trying different things, it's very frustrating.  So when I finally got something that works, I was very excited about it and since I've seen other people have the same problem, I decided to share.  Maybe more experienced people already know the the mtu trick, but for me it wasn't obvious at all.

I actually think there is something wrong w/ OpenVPN and/or pfSense in this case.  My "crappy cell service" gets 50Mbps at my house and 25Mbps over VPN so it's faster than many people's broadband connections.  The problem I have is that large web pages load just fine from my WAN connection.  The only problem is on the LAN side.  I have a reasonably fast Ubuntu machine running Apache serving a fairly small page over a gigabit wired connection on the LAN that can't be accessed at all.

So why can my phone, over VPN, download large web pages from the WAN side without any problem and then completely hang when downloading a much smaller page over the LAN?  The "crappy cell" is common in both cases - so that's not the problem.  Either OpenVPN is doing something or pfSense is doing something in moving from the LAN to the VPN subnets that's different than the WAN.

I'm sure there is a logical answer to this, but it certainly looks like a problem from my end...

[TD22057]

OK - somehow typing that last reply got me thinking about what's different about my home server.  I'm guessing the answer is jumbo frames.  I enabled that a long time ago on my server to get faster file transfer speeds to/from the server.  That could explain why I can't access web pages on that machine over the VPN.

That doesn't explain my security camera problem though.  If I turn on a port forward to the camera access port, I can connect to the camera and watch video fine over the VPN connection.  If I try a direct connection to the camera (same port, just on the LAN), it won't work unless I adjust the MTU down.

[Pippin]

Just a remark:
Some "crappy cell service" do not allow fragmented packets…  ;)

[TD22057]

@Pippin:

Just a remark:
Some "crappy cell service" do not allow fragmented packets…  ;)

Agreed.  But why can I VPN in from my phone, access an IP camera feed from a port forward through the WAN, but not access the same port through the LAN without lowering the MTU in the VPN client?  The only thing I can think of is that pfSense has a different MTU set on the WAN port than the LAN port?

Running ifconfig would indicate that isn't the case though… (em0=WAN, em1=LAN)

[2.3.1-RELEASE][admin@router.localdomain]/root: ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether x:x:x:x:x:x
        inet6 x::x:x:x:x%em0 prefixlen 64 scopeid 0x1
        inet x.x.x.xnetmask 0xffffe000 broadcast 255.255.255.255
        nd6 options=23 <performnud,accept_rtadv,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether x:x:x:x:x:x
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::1:1%em1 prefixlen 64 scopeid 0x2
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast> 

[Pippin]

route add 192.168.1.0 mask 255.255.255.0 gw 192.168.1.1

Never seen that before and as far as I know thats not how its done in config file of server or client.
I guess if you look at the server log, which you surely did while troubleshooting :), you will probably see error.

If you want to push that local network, you add

push "route 192.168.1.0 255.255.255.0"

to the costum options.

Or, if then still necessary:

push "route 192.168.1.0 255.255.255.0";fragment 1250;mssfix 1250

Not sure if gateway is needed in this case but if so then:
push "route 192.168.1.0 255.255.255.0 192.168.1.1"

Good luck.

[johnpoz]

I'm guessing the answer is jumbo frames.  I enabled that a long time ago on my server to get faster file transfer speeds to/from the server.

Clearly not, since

em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500</up,broadcast,running,simplex,multicast>

So where do you have jumbo frames enabled?  Between your machine and your server on the same network?  What is the point when anything off that segment is not going to be using jumbo.. So your increase in performance for file transfer was what exactly??

Not sure what your "crappy cell service" is but I can stream video on my phone via my cell connection via vpn into my network.. Without any need to modify any mtu, etc.

Attached is screenshot from my phone - notice only 1 bar, and the vpn icon..  This is streaming off my plex server on my home network..

[TD22057]

Thanks for the replies folks.  I have jumbo frames enabled on my windows machines and ubuntu server because it gave me better file transfer throughput when I was setting things up.  Changing those settings doesn't have any affect on the problems I'm having w/ VPN.  And the route statement in my settings was left over from a failed attempt at fixing my problem - it should be removed.

I was quoting "crappy" because I don't actually have crappy cell service - mine is quite good.  That was the point I was trying (unsuccessfully obviously) to make.  Over the cell (4G) network, I can:

• stream video from a netflix, amazon, and a plex server on my home network
• download files from my home network over sftp
• access any site on the internet w/o issue
• stream video from security cameras via port forward through pfSense to the camera

On my wifi network, the same is true plus I can see my local web server pages, and the cameras just fine.  However, if connect using the same 4G network using OpenVPN, things change.

• external (WAN) web sites, even large pages, work fine.
• internal (LAN) web pages connect, but hang after they start.  Very small pages work but anything normal sized hangs.  Web server logs show the connection but that's it.
• internal (LAN) access to the cameras hang
• external (WAN->port forward->camera) access to the cameras still works

If I adjust the MTU lower in the OpenVPN settings, then all of the problems go away.  That's all fine and good but I'd like to understand why.  With normal MTU (1500) and VPN, why can I access a camera feed using the WAN port forward but not using the LAN port?  It's the same port on the camera, so the only difference I can think of is that pfSense is treating the WAN/LAN interfaces differently.

[johnpoz]

"With normal MTU (1500) and VPN, why can I access a camera feed using the WAN port forward but not using the LAN port? "

Huh?  If your vpn'd into your network what would you be accessing on pfsense at all?  You would go direct to the IP address of whatever it is on your network - that is the whole point of the vpn.. You would not be hitting any IP on pfsense at all and having it forward anywhere be it your wan IP or LAN???

So your saying you vpn, and then through the vpn hit your pfsense WAN IP so it can forward you into whatever it is you want to access??  That makes no sense at all - none!

"ubuntu server because it gave me better file transfer throughput"

Yeah most likely placebo.

[TD22057]

@johnpoz:

Huh?  If your vpn'd into your network what would you be accessing on pfsense at all?  You would go direct to the IP address of whatever it is on your network - that is the whole point of the vpn.. You would not be hitting any IP on pfsense at all and having it forward anywhere be it your wan IP or LAN???

So your saying you vpn, and then through the vpn hit your pfsense WAN IP so it can forward you into whatever it is you want to access??  That makes no sense at all - none!

I agree completely - I only did that as a test.  If I VPN in and try to access the camera IP on the LAN, it does not work.  Accessing it through the WAN is stupid, but it works.  I was hoping that if I understand why it works that way will help me understand why it doesn't work directly.

[johnpoz]

So does not work on your lan even now or does with your mtu reduction? Or is that through your port forward?

So lets get a basic drawing so all on the same speed here..

So we have a pfsense that has public on its wan (internet) and using 192.168.1.0/24 on its lan.  We use a tunnel network of 10.0.1.0/24, so vpn client gets a 10.0.1.x ip

VPN client wants to talk to something on your network, 192.168.1.100 in this drawing.  Does not matter if that is camera, webserver, ssh, rdp, whatever.

Client goes to 192.168.1.100, his vpn connection says hey that network is on the other end of my tunnel and sends traffic to pfsense.  Pfsense says hey your wanting to go to 192.168.1.0/24 yup that is my locally attached network and send the traffic on the wire..  Client answers and say hey there 10.0.1.100 here is my answer, which he sends back to pfsense lets call it 192.168.1.1 his IP on that network.  Pfsense says oh you want to talk to 10.0.1.100 sure that is down this tunnel I have with him.

There is no NAT..  The firewall rules that will come into play are the vpn rules and your lan rules. So please post those up.  You can see mine. Now on my vpn tab the remote aliases is made up of my 10.0.8 and 10.0.200 networks, since i have 2 vpn servers running 1 on tcp and one on udp they use different tunnel networks.  So this rule says hey your coming from either of those networks come on in.. And there is nothing in my lan rules that send traffic down some other gateway or block talking to rfc1918 addresses.  So clients on my lan network can create conversations with vpn clients..  The lan rules only would come into play if lan is creating the connection to vpn client.  If only vpn client starts the conversations then only your vpn tab rules would be of concern.

This is your typical setup, so what your doing that requires you to hit your wan of pfsense makes no sense at all.. And to be honest points to you not even using the vpn but just normal port forward over the internet to access what your accessing.

Do a traceroute to your device on your network your trying to access from your vpn client..  Should be 2 hops, pfsense end of the tunnel and then the device.  What does your route table look like on your remote client?  So attached you see a traceroute from my phone connected to vpn via cell connection.

[TD22057]

I finally had some time to do some more exhaustive testing and you were right.  For some reason, the default flag in my Android VPN client was not routing all traffic over the VPN.  The route to the LAN was as expected (through pfsense) and the route to the WAN was over the cell network.  Once I set the flag to force everything over the VPN, the behavior (and routes) are the same.

So in the end, I really just needed to lower the MTU to get a reliable connection.  I'm just happy it's working  :)