Routing LAN traffic through OpenVPN



  • Hi everyone,

    I am aware that this question hast been asked probably a million times but even with all this information out there I do not succeed in what I want to do:
    I want to connect an APU router running pfSense 2.2.2 via openVPN to a Linux workstation (Fedora 19/3.12.9). In the routers LAN I have 5-10 local computers in the 192.168.1.x/24 subnetwork. These computers MUST be able to communicate with one another without going to the VPN server and back (that would be too slow) at the same time all internet traffic should be routed through the VPN.

    What I have done so far:

    • The VPN connection between APU router and Linux workstation is running and stable.
      In pfsense in Interface->Rules->Lan->Advanced Features->Gateway I have selected the VPN connection.
      iptables settings

    The problem:
    Computers in the subnetwork can't access the internet if the openVPN gateway is activated in pfsense. On the server side I get:
    Jul 14 14:45:16 <#server#> openvpn[14334]: router/<#addr#> MULTI: bad source address from client [192.168.1.104], packet dropped
    I have tried pretty much all settings that I have found on the internet so far :/

    Here are some outputs on the server side, please let me know if you need some more information:

    watch systemctl status openvpn@server.service
    openvpn@server.service - OpenVPN Robust And Highly Flexible Tunneling Application On server
      Loaded: loaded (/usr/lib/systemd/system/openvpn@.service; enabled)
      Active: active (running) since Tue 2015-06-16 15:26:46 CEST; 3 weeks 6 days ago
      Process: 14328 ExecStart=/usr/sbin/openvpn –daemon --writepid /var/run/openvpn/%i.pid --cd /etc/openvpn/ --config %i.conf (code=exited, status=0/SUCCESS)
    Main PID: 14334 (openvpn)
      CGroup: name=systemd:/system/openvpn@.service/openvpn@server.service
              └─14334 /usr/sbin/openvpn --daemon --writepid /var/run/openvpn/server.pid --cd /etc/openvpn/ --config server.conf

    Jul 14 13:32:28 <##> openvpn[14334]: router/<##> Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    Jul 14 13:32:28 <##> openvpn[14334]: router/<##> Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> TLS: soft reset sec=0 bytes=489927/0 pkts=3881/0
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> VERIFY OK: depth=1, C=<##>, ST=<##>, L=<##>, O=<##>, OU=<##>, CN=kinder, name=router, emailAddress=<##>
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> VERIFY OK: depth=0, C=<##>, ST=<##>, L=<##>, O=<##>, OU=<##>, CN=router, name=router, emailAddress=<##>
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
    Jul 14 14:32:28 <##> openvpn[14334]: router/<##> Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
    .
    .
    Jul 14 14:44:48 <##> openvpn[14334]: router/<##> MULTI: bad source address from client [192.168.1.104], packet dropped

    more /etc/openvpn/server.conf
    port 443
    dev tun

    proto tcp-server
    mode server

    tls-server
    tls-auth /etc/openvpn/keys/ta.key 0
    cipher AES-256-CBC
    ca /etc/openvpn/keys/ca.crt
    cert /etc/openvpn/keys/kinder.crt
    key /etc/openvpn/keys/kinder.key
    dh /etc/openvpn/keys/dh2048.pem

    #virtual endpoints
    ifconfig 10.8.0.1 10.8.0.2
    ifconfig-pool 10.8.0.4 10.8.0.255

    keepalive 10 60

    user openvpn
    group openvpn

    persist-tun
    persist-key

    verb 4

    #push traffic  route to the client
    push "route 192.168.1.0 255.255.255.0"

    #assign a specific address to our client (aka router)
    #client-config-dir ccd
    route 192.168.1.0 255.255.255.0

    iptables -L -n -v -t nat
    Chain PREROUTING (policy ACCEPT 82M packets, 14G bytes)
    pkts bytes target    prot opt in    out    source              destination

    Chain INPUT (policy ACCEPT 6887K packets, 787M bytes)
    pkts bytes target    prot opt in    out    source              destination

    Chain OUTPUT (policy ACCEPT 19883 packets, 4494K bytes)
    pkts bytes target    prot opt in    out    source              destination

    Chain POSTROUTING (policy ACCEPT 19883 packets, 4494K bytes)
    pkts bytes target    prot opt in    out    source              destination       
        0    0 MASQUERADE  all  –  *      eth+    0.0.0.0/0            0.0.0.0/0         
        0    0 MASQUERADE  all  --  *      em1    10.8.0.0/24          0.0.0.0/0

    These are not complete anymore - some settings previousely used:
    iptables -I FORWARD -i tun0 -o em1          -s 10.8.0.0/24 -m conntrack --ctstate NEW -j ACCEPT
    iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED          -j ACCEPT
    iptables -t nat -I POSTROUTING -o em1          -s 10.8.0.0/24 -j MASQUERADE

    Now my questions are:
    How do I get rid of the 'packet dropped' on the server (probably more related to iptables than to what the router does OR do I need NAT on the router)?
    How do I only pick internet requests to go to the VPN but not local communication in pfSense?

    Thank you for your time!
    And I will definitely make a tutorial on how to do it once you have helped me to figure it out.


  • LAYER 8 Global Moderator

    huh?

    "In the routers LAN I have 5-10 local computers in the 192.168.1.x/24 These computers MUST be able to communicate with one another without going to the VPN server and back"

    What would pfsense have to do with local computers talking to each other??  You do understand that local computers only talk to the gateway (pfsense) then they want to talk to devices not on their local network.



  • Sorry let me rephrase it:
    Packages from one 192.168.1.x/24 address to another 192.168.1.x/24 address should not go through the gateway (which I believe would mean that they are send to my VPN server and back) but should be routed directly to the destination address in the LAN.

    On the other hand packages that have a destination address outside of 192.168.1.x/24 should be routed through the gateway and consequently through the VPN.

    My whole reasoning behind this is that I really need GBit-Lan locally (VPN is only 100Mbit).

    Thanks


  • LAYER 8 Global Moderator

    Again why do you think pfsense would care if 192.168.1.100 is talking to 192.168.1.101??

    Pfsense has no control over devices talking amongst themselves on the same network..  Only when say they want to go to 192.168.2.0/24 would they talk their gateway pfsense on 192.168.1.1 say for example..

    So see my little drawing..  When cpu A is talking to B pfsense is not involved.  If A or B wanted to talk to C then yes, if they want to talk to internet or anything else off the 192.168.1.0/24 then yes pfsense is involved..  Talking to themselves pfsense doesn't even see the traffic..  Nothing pfsense can do to allow or block it.




  • Ok, I understand.
    Now I grasped the concept of LAN..
    So it is fair to say that any package that has an address different from the subnetwork 192.168.1.x/24 stays in the LAN while everything else has to go over the gateway. So far so good - that solved the first problem/my missunderstanding.

    Now, how do I get the server to handle the packages it receives from my pfSense router correctly. Does pfSense has to do some sort of NAT on the packages?
    In my understanding the packages need to have a sender from the 10.8.0.0/24 network otherwise I will get this message from the openVPN server as it does not know what to do with them:

    Jul 14 14:44:48 <##> openvpn[14334]: router/<##> MULTI: bad source address from client [192.168.1.104], packet dropped

    Thank you!


  • LAYER 8 Global Moderator

    "So it is fair to say that any package that has an address different from the subnetwork 192.168.1.x/24 stays in the LAN"

    This is typo or your still not grasping it..

    traffic that is in the same network, say 192.168.1.0/24 stays in the lan..

    How would you be getting traffic from a bad source IP?  Can you please draw up your network..

    Where is this?  "The VPN connection between APU router and Linux workstation is running and stable."



  • Sorry, yes that was a typo.

    Attached you will find an improvised drawing.
    In the setup I have an APU router with pfSense that sits behind a firewall that only allows port 80/443. I also have a Linux workstation that is directly connected to the internet with no additional firewall in front of it. Now the idea was to make a VPN tunnel from the APU router to the Linux workstation and direct all internet traffic from any of the users in the 192.168.1.x/24 network through that tunnel to prevent the port limitation. I do have a working VPN connection between APU router (openVPN tunnel IP: 10.8.0.6) and Linux workstation (openVPN tunnel IP: 10.8.0.1).

    I believe the problem is that if a computer from the 192.168.1.x/24 network sends an internet request it will be forwarded through the tunnel (10.8.0.x/24) to the Linux workstation without the source IP address being altered. The Linux workstation will then get a package from 192.168.1.x in it's 10.8.0.x/24 network and does not know what to do with it and drops it.

    If that is true, how do I do NAT on outgoing packages in pfSense?

    Thanks

    ![Screen Shot 2015-07-15 at 16.39.00.png](/public/imported_attachments/1/Screen Shot 2015-07-15 at 16.39.00.png)
    ![Screen Shot 2015-07-15 at 16.39.00.png_thumb](/public/imported_attachments/1/Screen Shot 2015-07-15 at 16.39.00.png_thumb)


  • LAYER 8 Global Moderator

    Did you setup your remote networks in your openvpn wizard, this then tells the clients hey you want to get to 192.168.1.0/24 go down your tunnel.

    So for example.. I use 10.0.8 as my vpn network.  My client currently connected, lets call it your linux box..

    Connection-specific DNS Suffix  . : local.lan                                 
      Description . . . . . . . . . . . : TAP-Windows Adapter V9                   
      Physical Address. . . . . . . . . : 00-FF-5A-2F-7E-EA                         
      DHCP Enabled. . . . . . . . . . . : Yes                                       
      Autoconfiguration Enabled . . . . : Yes                                       
      IPv4 Address. . . . . . . . . . . : 10.0.8.6(Preferred)                       
      Subnet Mask . . . . . . . . . . . : 255.255.255.252                           
      Lease Obtained. . . . . . . . . . : Wednesday, July 15, 2015 9:24:38 AM       
      Lease Expires . . . . . . . . . . : Thursday, July 14, 2016 9:24:38 AM       
      Default Gateway . . . . . . . . . :                                           
      DHCP Server . . . . . . . . . . . : 10.0.8.5                                 
      DNS Servers . . . . . . . . . . . : 192.168.9.253                             
      NetBIOS over Tcpip. . . . . . . . : Enabled

    One of my networks behind pfsense is 192.168.9/24  So you see the route in the route table on my vpn client.

    Route Print
          192.168.2.0    255.255.255.0        10.0.8.5        10.0.8.6    20
          192.168.3.0    255.255.255.0        10.0.8.5        10.0.8.6    20
          192.168.9.0    255.255.255.0        10.0.8.5        10.0.8.6    20

    So if I traceroute from here I see this
    C:>tracert -d 192.168.9.100

    Tracing route to 192.168.9.100 over a maximum of 30 hops

    1  105 ms    98 ms    99 ms  10.0.8.1
      2  101 ms  108 ms    99 ms  192.168.9.100

    Trace complete.

    I snipped out the ipv6 info, I also have that routing through my vpn connection - but that is beyond the scope of your question.  There is no reason to nat these private addresses..  That would just complicate it for no reason.  If you have a firewall on the remote client you would need to allow that traffic on it..  Are you do unsolicited traffic from your lan to your vpn client?  Or just answering stuff that your vpn client starts the conversation.

    So for example here is a box on my lan segment routing and pinging my remote vpn client on the address 10.0.8.6 you saw from above

    user@ubuntu:~$ traceroute -I 10.0.8.6
    traceroute to 10.0.8.6 (10.0.8.6), 64 hops max
      1  192.168.9.253  0.587ms  0.244ms  0.243ms
      2  10.0.8.6  261.488ms  161.872ms  177.313ms
    user@ubuntu:~$ ping 10.0.8.6
    PING 10.0.8.6 (10.0.8.6) 56(84) bytes of data.
    64 bytes from 10.0.8.6: icmp_seq=1 ttl=127 time=251 ms
    64 bytes from 10.0.8.6: icmp_seq=2 ttl=127 time=103 ms
    64 bytes from 10.0.8.6: icmp_seq=3 ttl=127 time=98.6 ms
    ^C
    –- 10.0.8.6 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2002ms
    rtt min/avg/max/mdev = 98.613/151.269/251.501/70.906 ms
    user@ubuntu:~$

    Latency is not reall good for my vpn, because of the way work is currently using a proxy in hou area that I have to bounce my vpn off of..  So I am in chicago but my vpn routes through houston area ;)  Just to come back to chicago area to my house.




  • Thanks for all the information. Unfortunately the problem persists.
    After my vacation I will study networking from bottom up and hopefully solve the problem in an acceptable amount of time.


  • LAYER 8 Global Moderator

    Well without what doesn't work and what you have done there is little more to help you.



  • (which I believe would mean that they are send to my VPN server and back) but should be routed
    directly to the destination address in the LAN.
    Would not be running, because the VPN must be having on both ends a different LAN address!

    • 192.168.1.0/24 –--VPN ----192.168.1.0/24 - will not be running
    • 192.168.1.0/24 ----VPN ----172.xxx.xxx.xxx/24 - will be running

    On the other hand packages that have a destination address outside of 192.168.1.x/24 should be
    routed through the gateway and consequently through the VPN.
    If the destination is on the other VPN end yes, if not no.

    My whole reasoning behind this is that I really need GBit-Lan locally
    Then we should be are talking about other things and perhaps other hardware also.
    If the pfSense is doing all, what is very popular for many users, it slows down even a little bit
    more how much more the pfsense have to do, for sure this is also by other vendors and systems
    Let us see a MikroTik Router it deliveres full speed at first time and after SPI, NAT and 20 firewall
    rules, VLANs and QoS it is delivering something around 25% of its full power, for sure not at all
    models but at the most ones. And fore sure it would be also running with all other systems on
    mother earth! So if you install some Layer3 Switch in your network and stack them instead of
    only uplink them you would be at these days doing the best as you are able to do.

    The whole and entire LAN traffic will be routed only be the Layer3 Switches and the pfSense
    is now free of this work. This is often very speeding up many network constructions.
    And if you bind your servers over 10 GBit/s to the Switches you will be getting out
    of creating a so called bottleneck. Or plain LAG (LACP) them perhaps would also bringing
    more throughput near by.

    (VPN is only 100Mbit).
    There fore you will be able to do also some things to speed up the throughput a lot.

    • The CPU has to do the most, so if you spend pfSense a really powerful CPU you
      get the most of, and then perhaps also some more ECC RAM it would be the best
      point to start speeding up the WAN throughput. Using Intel server network adapters
      would bring you also more stability and gaining once more again the throughput a bit.

    Inserting then perhaps a compression card on both ends of the VPN (not only at one side)
    would be increasing the entire throughput once more again. Comtech AHA362PCIe is able
    to buy over eBay for something around likes ~$30 - $60.

    AES-NI at the CPU would be the best option today and a 4 Core Intel Xeon E3-12xxv3 at a
    minimum of 3,0GHz would do the job.

    For sure there are also other options out to insert but I am in Germany and the most
    companies of those equipment are in the USA. So if you are a citizen of the USA you
    could try starting around your search for a Exar DX1700 crypto accelerator that will
    be speeding VPNs up mostly really wicked. If this card is supported.


Log in to reply