Routing LAN traffic through OpenVPN

[schroedinger]

Hi everyone,

I am aware that this question hast been asked probably a million times but even with all this information out there I do not succeed in what I want to do:
I want to connect an APU router running pfSense 2.2.2 via openVPN to a Linux workstation (Fedora 19/3.12.9). In the routers LAN I have 5-10 local computers in the 192.168.1.x/24 subnetwork. These computers MUST be able to communicate with one another without going to the VPN server and back (that would be too slow) at the same time all internet traffic should be routed through the VPN.

What I have done so far:

• The VPN connection between APU router and Linux workstation is running and stable.
  In pfsense in Interface->Rules->Lan->Advanced Features->Gateway I have selected the VPN connection.
  iptables settings

The problem:
Computers in the subnetwork can't access the internet if the openVPN gateway is activated in pfsense. On the server side I get:
Jul 14 14:45:16 <#server#> openvpn[14334]: router/<#addr#> MULTI: bad source address from client [192.168.1.104], packet dropped
I have tried pretty much all settings that I have found on the internet so far :/

Here are some outputs on the server side, please let me know if you need some more information:

watch systemctl status openvpn@server.service
openvpn@server.service - OpenVPN Robust And Highly Flexible Tunneling Application On server
  Loaded: loaded (/usr/lib/systemd/system/openvpn@.service; enabled)
  Active: active (running) since Tue 2015-06-16 15:26:46 CEST; 3 weeks 6 days ago
  Process: 14328 ExecStart=/usr/sbin/openvpn –daemon --writepid /var/run/openvpn/%i.pid --cd /etc/openvpn/ --config %i.conf (code=exited, status=0/SUCCESS)
Main PID: 14334 (openvpn)
  CGroup: name=systemd:/system/openvpn@.service/openvpn@server.service
          └─14334 /usr/sbin/openvpn --daemon --writepid /var/run/openvpn/server.pid --cd /etc/openvpn/ --config server.conf

Jul 14 13:32:28 <##> openvpn[14334]: router/<##> Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Jul 14 13:32:28 <##> openvpn[14334]: router/<##> Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> TLS: soft reset sec=0 bytes=489927/0 pkts=3881/0
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> VERIFY OK: depth=1, C=<##>, ST=<##>, L=<##>, O=<##>, OU=<##>, CN=kinder, name=router, emailAddress=<##>
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> VERIFY OK: depth=0, C=<##>, ST=<##>, L=<##>, O=<##>, OU=<##>, CN=router, name=router, emailAddress=<##>
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
.
.
Jul 14 14:44:48 <##> openvpn[14334]: router/<##> MULTI: bad source address from client [192.168.1.104], packet dropped

more /etc/openvpn/server.conf
port 443
dev tun

proto tcp-server
mode server

tls-server
tls-auth /etc/openvpn/keys/ta.key 0
cipher AES-256-CBC
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/kinder.crt
key /etc/openvpn/keys/kinder.key
dh /etc/openvpn/keys/dh2048.pem

#virtual endpoints
ifconfig 10.8.0.1 10.8.0.2
ifconfig-pool 10.8.0.4 10.8.0.255

keepalive 10 60

user openvpn
group openvpn

persist-tun
persist-key

verb 4

#push traffic  route to the client
push "route 192.168.1.0 255.255.255.0"

#assign a specific address to our client (aka router)
#client-config-dir ccd
route 192.168.1.0 255.255.255.0

iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 82M packets, 14G bytes)
pkts bytes target    prot opt in    out    source              destination

Chain INPUT (policy ACCEPT 6887K packets, 787M bytes)
pkts bytes target    prot opt in    out    source              destination

Chain OUTPUT (policy ACCEPT 19883 packets, 4494K bytes)
pkts bytes target    prot opt in    out    source              destination

Chain POSTROUTING (policy ACCEPT 19883 packets, 4494K bytes)
pkts bytes target    prot opt in    out    source              destination       
    0    0 MASQUERADE  all  –  *      eth+    0.0.0.0/0            0.0.0.0/0         
    0    0 MASQUERADE  all  --  *      em1    10.8.0.0/24          0.0.0.0/0

These are not complete anymore - some settings previousely used:
iptables -I FORWARD -i tun0 -o em1          -s 10.8.0.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED          -j ACCEPT
iptables -t nat -I POSTROUTING -o em1          -s 10.8.0.0/24 -j MASQUERADE

Now my questions are:
How do I get rid of the 'packet dropped' on the server (probably more related to iptables than to what the router does OR do I need NAT on the router)?
How do I only pick internet requests to go to the VPN but not local communication in pfSense?

Thank you for your time!
And I will definitely make a tutorial on how to do it once you have helped me to figure it out.

[johnpoz]

huh?

"In the routers LAN I have 5-10 local computers in the 192.168.1.x/24 These computers MUST be able to communicate with one another without going to the VPN server and back"

What would pfsense have to do with local computers talking to each other??  You do understand that local computers only talk to the gateway (pfsense) then they want to talk to devices not on their local network.

[schroedinger]

Sorry let me rephrase it:
Packages from one 192.168.1.x/24 address to another 192.168.1.x/24 address should not go through the gateway (which I believe would mean that they are send to my VPN server and back) but should be routed directly to the destination address in the LAN.

On the other hand packages that have a destination address outside of 192.168.1.x/24 should be routed through the gateway and consequently through the VPN.

My whole reasoning behind this is that I really need GBit-Lan locally (VPN is only 100Mbit).

Thanks

[johnpoz]

Again why do you think pfsense would care if 192.168.1.100 is talking to 192.168.1.101??

Pfsense has no control over devices talking amongst themselves on the same network..  Only when say they want to go to 192.168.2.0/24 would they talk their gateway pfsense on 192.168.1.1 say for example..

So see my little drawing..  When cpu A is talking to B pfsense is not involved.  If A or B wanted to talk to C then yes, if they want to talk to internet or anything else off the 192.168.1.0/24 then yes pfsense is involved..  Talking to themselves pfsense doesn't even see the traffic..  Nothing pfsense can do to allow or block it.

[schroedinger]

Ok, I understand.
Now I grasped the concept of LAN..
So it is fair to say that any package that has an address different from the subnetwork 192.168.1.x/24 stays in the LAN while everything else has to go over the gateway. So far so good - that solved the first problem/my missunderstanding.

Now, how do I get the server to handle the packages it receives from my pfSense router correctly. Does pfSense has to do some sort of NAT on the packages?
In my understanding the packages need to have a sender from the 10.8.0.0/24 network otherwise I will get this message from the openVPN server as it does not know what to do with them:

Jul 14 14:44:48 <##> openvpn[14334]: router/<##> MULTI: bad source address from client [192.168.1.104], packet dropped

Thank you!

[johnpoz]

"So it is fair to say that any package that has an address different from the subnetwork 192.168.1.x/24 stays in the LAN"

This is typo or your still not grasping it..

traffic that is in the same network, say 192.168.1.0/24 stays in the lan..

How would you be getting traffic from a bad source IP?  Can you please draw up your network..

Where is this?  "The VPN connection between APU router and Linux workstation is running and stable."

[schroedinger]

Sorry, yes that was a typo.

Attached you will find an improvised drawing.
In the setup I have an APU router with pfSense that sits behind a firewall that only allows port 80/443. I also have a Linux workstation that is directly connected to the internet with no additional firewall in front of it. Now the idea was to make a VPN tunnel from the APU router to the Linux workstation and direct all internet traffic from any of the users in the 192.168.1.x/24 network through that tunnel to prevent the port limitation. I do have a working VPN connection between APU router (openVPN tunnel IP: 10.8.0.6) and Linux workstation (openVPN tunnel IP: 10.8.0.1).

I believe the problem is that if a computer from the 192.168.1.x/24 network sends an internet request it will be forwarded through the tunnel (10.8.0.x/24) to the Linux workstation without the source IP address being altered. The Linux workstation will then get a package from 192.168.1.x in it's 10.8.0.x/24 network and does not know what to do with it and drops it.

If that is true, how do I do NAT on outgoing packages in pfSense?

Thanks


[johnpoz]

Did you setup your remote networks in your openvpn wizard, this then tells the clients hey you want to get to 192.168.1.0/24 go down your tunnel.

So for example.. I use 10.0.8 as my vpn network.  My client currently connected, lets call it your linux box..

Connection-specific DNS Suffix  . : local.lan                                 
  Description . . . . . . . . . . . : TAP-Windows Adapter V9                   
  Physical Address. . . . . . . . . : 00-FF-5A-2F-7E-EA                         
  DHCP Enabled. . . . . . . . . . . : Yes                                       
  Autoconfiguration Enabled . . . . : Yes                                       
  IPv4 Address. . . . . . . . . . . : 10.0.8.6(Preferred)                       
  Subnet Mask . . . . . . . . . . . : 255.255.255.252                           
  Lease Obtained. . . . . . . . . . : Wednesday, July 15, 2015 9:24:38 AM       
  Lease Expires . . . . . . . . . . : Thursday, July 14, 2016 9:24:38 AM       
  Default Gateway . . . . . . . . . :                                           
  DHCP Server . . . . . . . . . . . : 10.0.8.5                                 
  DNS Servers . . . . . . . . . . . : 192.168.9.253                             
  NetBIOS over Tcpip. . . . . . . . : Enabled

One of my networks behind pfsense is 192.168.9/24  So you see the route in the route table on my vpn client.

Route Print
      192.168.2.0    255.255.255.0        10.0.8.5        10.0.8.6    20
      192.168.3.0    255.255.255.0        10.0.8.5        10.0.8.6    20
      192.168.9.0    255.255.255.0        10.0.8.5        10.0.8.6    20

So if I traceroute from here I see this
C:>tracert -d 192.168.9.100

Tracing route to 192.168.9.100 over a maximum of 30 hops

1  105 ms    98 ms    99 ms  10.0.8.1
  2  101 ms  108 ms    99 ms  192.168.9.100

Trace complete.

I snipped out the ipv6 info, I also have that routing through my vpn connection - but that is beyond the scope of your question.  There is no reason to nat these private addresses..  That would just complicate it for no reason.  If you have a firewall on the remote client you would need to allow that traffic on it..  Are you do unsolicited traffic from your lan to your vpn client?  Or just answering stuff that your vpn client starts the conversation.

So for example here is a box on my lan segment routing and pinging my remote vpn client on the address 10.0.8.6 you saw from above

user@ubuntu:~$ traceroute -I 10.0.8.6
traceroute to 10.0.8.6 (10.0.8.6), 64 hops max
  1  192.168.9.253  0.587ms  0.244ms  0.243ms
  2  10.0.8.6  261.488ms  161.872ms  177.313ms
user@ubuntu:~$ ping 10.0.8.6
PING 10.0.8.6 (10.0.8.6) 56(84) bytes of data.
64 bytes from 10.0.8.6: icmp_seq=1 ttl=127 time=251 ms
64 bytes from 10.0.8.6: icmp_seq=2 ttl=127 time=103 ms
64 bytes from 10.0.8.6: icmp_seq=3 ttl=127 time=98.6 ms
^C
–- 10.0.8.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 98.613/151.269/251.501/70.906 ms
user@ubuntu:~$

Latency is not reall good for my vpn, because of the way work is currently using a proxy in hou area that I have to bounce my vpn off of..  So I am in chicago but my vpn routes through houston area ;)  Just to come back to chicago area to my house.

[schroedinger]

Thanks for all the information. Unfortunately the problem persists.
After my vacation I will study networking from bottom up and hopefully solve the problem in an acceptable amount of time.

[johnpoz]

Well without what doesn't work and what you have done there is little more to help you.

[Guest]

(which I believe would mean that they are send to my VPN server and back) but should be routed
directly to the destination address in the LAN.
Would not be running, because the VPN must be having on both ends a different LAN address!

• 192.168.1.0/24 –--VPN ----192.168.1.0/24 - will not be running
• 192.168.1.0/24 ----VPN ----172.xxx.xxx.xxx/24 - will be running

On the other hand packages that have a destination address outside of 192.168.1.x/24 should be
routed through the gateway and consequently through the VPN.
If the destination is on the other VPN end yes, if not no.

My whole reasoning behind this is that I really need GBit-Lan locally
Then we should be are talking about other things and perhaps other hardware also.
If the pfSense is doing all, what is very popular for many users, it slows down even a little bit
more how much more the pfsense have to do, for sure this is also by other vendors and systems
Let us see a MikroTik Router it deliveres full speed at first time and after SPI, NAT and 20 firewall
rules, VLANs and QoS it is delivering something around 25% of its full power, for sure not at all
models but at the most ones. And fore sure it would be also running with all other systems on
mother earth! So if you install some Layer3 Switch in your network and stack them instead of
only uplink them you would be at these days doing the best as you are able to do.

The whole and entire LAN traffic will be routed only be the Layer3 Switches and the pfSense
is now free of this work. This is often very speeding up many network constructions.
And if you bind your servers over 10 GBit/s to the Switches you will be getting out
of creating a so called bottleneck. Or plain LAG (LACP) them perhaps would also bringing
more throughput near by.

(VPN is only 100Mbit).
There fore you will be able to do also some things to speed up the throughput a lot.

• The CPU has to do the most, so if you spend pfSense a really powerful CPU you
  get the most of, and then perhaps also some more ECC RAM it would be the best
  point to start speeding up the WAN throughput. Using Intel server network adapters
  would bring you also more stability and gaining once more again the throughput a bit.

Inserting then perhaps a compression card on both ends of the VPN (not only at one side)
would be increasing the entire throughput once more again. Comtech AHA362PCIe is able
to buy over eBay for something around likes ~$30 - $60.

AES-NI at the CPU would be the best option today and a 4 Core Intel Xeon E3-12xxv3 at a
minimum of 3,0GHz would do the job.

For sure there are also other options out to insert but I am in Germany and the most
companies of those equipment are in the USA. So if you are a citizen of the USA you
could try starting around your search for a Exar DX1700 crypto accelerator that will
be speeding VPNs up mostly really wicked. If this card is supported.