[2.1] site2site vpn stops to work after Multi VPN server firmware upgrade



  • Hello,
    I got the following opnevpn Peer2Peer SSL/TSL conf:

    site1<–->MAINSITE<--->site2

    On MAINSITE pfSenseOpenVPNServer
    On site1  pfSenseOpenVPNClient1
    On site2  pfSenseOpenVPNClient2

    (site2 vpn was/is configured specifying "route 10.106.100.0/24" openVPN command on advanced configuration options of the Server)

    Everything worked fine on 2.0.3, 2.1-RC0 and RC1.
    Since I upgraded Server from 2.1-RC1 to 2.1 site2 vpn stops to work.

    I can see both vpns UP on menu Status->OpenVPN but only site1 "ping".

    Moreover if I

    • disable site1 & site2 vpn client;

    • enable site2 vpn (No connectivity gained);

    • (re)disable site2 vpn;

    • enable site1 vpn: it does not work anymore: I HAVE TO reboot pfSenseOpenVPNServer to regain site1 connection

    traceroute diagnosis find out (perhaps) it's a routing problem:
    If I try to tracerouting a site1 IP routing is ok: passing through pfSenseOpenVPNServer gw and pfSenseOpenVPNClient1 Gw;
    If I try to tracerouting a site2 IP pfsense server seems to route traffic through default gw instead of vpn tunnel.

    No changes were made during upgrade.
    I had to restore 2.1-RC1 to get the conf working.

    Thank you in advance.



  • This might not apply to your configuration, but if it's similar to mine:
    Do you have the correct GW chosen on the LAN in the Firewall Rules?  I've noticed 2.1 does some funky naming of the GW's created when you create the VPN Interface.  So you would have to go back into the Firewall Rule and make sure the correct GW is chosen under advanced options.



  • if you throw some screen shots up it might be easier to help. Check to make sure you have port 1195 open on your wan. Are you seeing any blocks in your logs?



  • @ckraimer:

    This might not apply to your configuration, but if it's similar to mine:
    Do you have the correct GW chosen on the LAN in the Firewall Rules?  I've noticed 2.1 does some funky naming of the GW's created when you create the VPN Interface.  So you would have to go back into the Firewall Rule and make sure the correct GW is chosen under advanced options.

    First of all… thanks!
    That is a good question!

    Because I got a multi wan failover configuration as you can see on picture below.

    Is this conf similar to yours?

    If yes what do you mean? Would I have to change GW on advanced option of the fw rule in the picture?
    Sorry to be annoying but I reverted to 2.1-RC1 and at the moment I can not check  "GW funky naming".
    Thanks again!




  • @mikeisfly:

    if you throw some screen shots up it might be easier to help. Check to make sure you have port 1195 open on your wan. Are you seeing any blocks in your logs?

    Thanks for you reply mikeisfly!

    I use tcp port 1194 and as I stated on my post I can see both vpns up and running on Status->OpenVPN, but I can ping just site1 one. Unfortunately I did not check firewall logs… and now I had reverted to 2.1-RC1... sorry.

    Meanwhile I miss to describe the following thing:
    If I create a new "Server" with site2 remote lan into the IPv4 Remote Network/s field and site1 in "advanced option" field (see OpenVPN_Server3.JPG) both vpns start but this time site1 do not ping.
    I tried to insert both sites lans on IPv4 Remote Network/s (OpenVPN server conf side) as described under the IPv4 Remote Network/s field without success.

    It sounds like the second vpn was not added to routing table correctly.

    Here are the VPN Server and Site1 Client conf screenshots. Site2 conf is identical (lan net apart obviously)
    Hope this help…
    Thank you in advance.














  • @vielfede:

    ….
    Everything worked fine on 2.0.3, 2.1-RC0 and RC1.
    ......
    traceroute diagnosis find out (perhaps) it's a routing problem:
    If I try to tracerouting a site1 IP routing is ok: passing through pfSenseOpenVPNServer gw and pfSenseOpenVPNClient1 Gw;
    If I try to tracerouting a site2 IP pfsense server seems to route traffic through default gw instead of vpn tunnel.
    ….

    Searching into the forum I found out the following:
    http://forum.pfsense.org/index.php/topic,66776.30.html
    And this could confirm my hypothesis of a 2.1-RELEASE routing bug.



  • Saw you post in the other thread and came here.

    Did you create an explicit LAN route from the LAN to the 10.106.100.0/24 net?  Try doing that and see if it fixes it.



  • @tim.mcmanus:

    Saw you post in the other thread and came here.

    Did you create an explicit LAN route from the LAN to the 10.106.100.0/24 net?  Try doing that and see if it fixes it.

    I did not! But it was one of the things to do it has came in my mind….

    but....

    I remember (on 2.1-RELEASE) I checked routes on Diagnostic->Routes and everything seemed ok...
    I mean no difference between vpn site1 and site2 routing...

    As I reverted to RC1 I can not try it at the moment... It is a production env and I'd to wait 'til monday afternoon to try..
    Thanks by now



  • I see above that folks mentioned checking your firewall logs.  I cannot emphasize enough to do exactly that.  Most of my multi-WAN routing issues were resolved using ping, traceroute, and checking the FW logs.  That motivated me to create explicit routes between LANs to resolve those routing issues.  I have a hunch that 2.1 requires explicit routes for nearly everything, which isn't a bad thing, but it is more maintenance and engineering.



  • I see what you are trying to do But what I think is happening is that you are sending traffic that is meant for one site to the other site. What I would do is a /30 to each site from the main site.

    main -> site1 - udp port 1194
    main -> site 2 - udp port 1195

    That way you can send 10.106.100.0/24 to site 2 and 10.116.100.0/24 to site1.
    What I do is just make a point to point from main site to remote sites. If you need the remote sites to talk to each other I guess you can setup static routes to accomplish this. One other thing I noticed too is make sure you don't have any IPSec tunnels configured because from my testing PfSense perfers IPsec tunnels over OpenVPN tunnels.

    On a side note is there a reason that you are using TCP over UDP? It would seem to me that you will get better performance with UDP.



  • @mikeisfly:

    I see what you are trying to do But what I think is happening is that you are sending traffic that is meant for one site to the other site.

    First of all Thanks for your answer mikeisfly, but I don't think so… sorry.
    As I stated on my first post: If I try to tracerouting a site2 IP pfsense server seems to route traffic through default gw instead of vpn tunnel (and hence nor the first vpn tunnel) .

    @mikeisfly:

    What I would do is a /30 to each site from the main site.

    main -> site1 - udp port 1194
    main -> site 2 - udp port 1195

    That way you can send 10.106.100.0/24 to site 2 and 10.116.100.0/24 to site1.
    What I do is just make a point to point from main site to remote sites. If you need the remote sites to talk to each other I guess you can setup static routes to accomplish this.

    That would be a solution, but by this way I'd to open a new port, reconfigure nat etc… I think not a good thing: remember the current config work flawlessly on RC1.
    Static route could solve the problem, but I checked them on Routes and all seemed ok.

    @mikeisfly:

    One other thing I noticed too is make sure you don't have any IPSec tunnels configured because from my testing PfSense perfers IPsec tunnels over OpenVPN tunnels.

    On a side note is there a reason that you are using TCP over UDP? It would seem to me that you will get better performance with UDP.

    I do not use IPSEC;
    I use tcp  because a friend suggestion.



  • Why don't you try to put both your remote networks in the remote network field just separate them with a comma instead of using the advanced section?

    I tried to insert both sites lans on IPv4 Remote Network/s (OpenVPN server conf side) as described under the IPv4 Remote Network/s  field without success.

    and then create a static route for each network using the 192.168.x.x you made as your OpenVPN network. I have never tried it this way but i'm thinking that it should work.

    If you were to do it as two separate servers the only thing that you would have to do is expand your port range on your Wan Rule to 1194-1195. This might actually give you some granularity as you can easily disable sites on a site per site basis. Also on the dash board you can quickly see which sites are up and down. I know this doesn't answer the question of why it worked before and not now.



  • @mikeisfly:

    Why don't you try to put both your remote networks in the remote network field just separate them with a comma instead of using the advanced section?

    I tried to insert both sites lans on IPv4 Remote Network/s (OpenVPN server conf side) as described under the IPv4 Remote Network/s  field without success.

    and then create a static route for each network using the 192.168.x.x you made as your OpenVPN network. I have never tried it this way but i'm thinking that it should work.

    If you were to do it as two separate servers the only thing that you would have to do is expand your port range on your Wan Rule to 1194-1195. This might actually give you some granularity as you can easily disable sites on a site per site basis. Also on the dash board you can quickly see which sites are up and down. I know this doesn't answer the question of why it worked before and not now.

    Thanks mikeisfly…
    You are right... no way to answer that question at this time.
    I'd add static route to 192.168.... if I did not remember the following: (on 2.1-RELASE psense) the 2 static routes to vpn net  were been already added and they seemed like that ones on the image attached of the current RC1 working conf.
    Indeed I'd better to upgrade back to 2.1-RELEASE and check.... but I have to schedule it. Maybe It's better to wait a lil'bit and see if something arise...
    I agree with you... 2 separate servers  would be a  working workaround... but considering this conf (1 server allowing 2+ remote site2site clients to connect) a "common" conf, I'd like to find out a definitive solution.
    Do you agree?




  • Isn't that your problem, shouldn't your static routes be pointing to your individual sites? 192.168.100.2 and the other site to maybe 192.168.100.3? Not sure what IP your other site is pulling from your main site. Did you downgrade? If you did then I would agree with you I would wait until a more convenient time.



  • @mikeisfly:

    Isn't that your problem, shouldn't your static routes be pointing to your individual sites? 192.168.100.2 and the other site to maybe 192.168.100.3? Not sure what IP your other site is pulling from your main site. Did you downgrade? If you did then I would agree with you I would wait until a more convenient time.

    192.168.100.2?
    Perhaps you mean 192.168.100.102!
    192.168.100.102 is IP of my 5th interface, routing some "private traffic"…
    yes, I downgraded.
    I'm following the following thread where  a similar routing problem is described.
    http://forum.pfsense.org/index.php/topic,66776.30.html

    I hope my thread can be a reference for OpnenVPN with 1 Server and 2+ clients site2site VPNs.



  • Sorry I meant 192.168.12.3 :).

    Here is a simplified view of your network, please let me know if I have made a mistake here:

    So I'm thinking when you make your static routes, when you want to get to the 10.106.100.0/24 network you need to send traffic to 192.168.12.2. Similarly when you want to send traffic to 10.116.100.0/24 you need to send traffic to 192.168.12.3. I'm thinking that you can just make a static route to accomplish this and no further configurations are need under OpenVPN. Like I stated before I have never done it this way before but seems like it should work. I'm very interested to see if this works, if it does seems like with a little tweaking you could make a fully mesh OpenVPN network as well. I prefer to do it where every site has  a connection to every other site, that way if the main site goes down then the remote sites still have connection to each other. But that is getting beyond what you are looking to accomplish here. So what do you think? The only problem here is to insure that the remote sites maintain the same IP on there Ovpn interfaces which I guess could be done with a static IP. Just another question if you are only going to have three sites connected why not use 192.168.12.0/29 or even a /28 if you think you might expand in the future? Just seems like a waist to use /24 for point to point links or a small network like you are using.



  • Great job mikeisfly!  :)
    But there is an error: on site 2. It got the same virtual VPN IP of site 1: 192.168.12.2 (as you can see on my attachment)

    @mikeisfly:

    So I'm thinking when you make your static routes, when you want to get to the 10.106.100.0/24 network you need to send traffic to 192.168.12.2. Similarly when you want to send traffic to 10.116.100.0/24 you need to send traffic to 192.168.12.3. I'm thinking that you can just make a static route to accomplish this and no further configurations are need under OpenVPN.

    I agree… but as I stated on my prev message, on RC1 routes are added automatically! (as you can see on figure routes_2.1-RC1.jpg), hence I do not understand why it shouldn't work On 2.1-RELEASE. Moreover as stated before, I remember the 2 routes on Diagnostic->routes panel on 2.1 RELEASE automatically

    I do not need site1 to site2 link (hub&spoke). I use hub&spoke on RoadWarrior connection(mainsite): my roadWarrior VPN is configurend to allow RW cliients get access to mainsite LAN, site1 LAN and site2 LAN.
    That by means of iroute/route OpenVPN commands. Those commands add routes on each site as needed, without any other custom static route or something like that.



  • But there is an error: on site 2. It got the same virtual VPN IP of site 1: 192.168.12.2 (as you can see on my attachment)

    That's interesting? Not sure how that is working when two sites have the same virtual IP. Doesn't seem like it should work. Maybe this is the issue with 2.1 Release. I will try to set something up in my lab and let you know the results. Like you said if we can figure out what's going on others can use this as resource for future issues.



  • @mikeisfly:

    But there is an error: on site 2. It got the same virtual VPN IP of site 1: 192.168.12.2 (as you can see on my attachment)

    That's interesting? Not sure how that is working when two sites have the same virtual IP. Doesn't seem like it should work. Maybe this is the issue with 2.1 Release. I will try to set something up in my lab and let you know the results. Like you said if we can figure out what's going on others can use this as resource for future issues.

    I don't know… It has worked flawlessly on 2.01, 2.02, 2.03, 2.1-RC0, 2.1-RC1.
    Thank you for your interest.
    I hope this can help other pfSense users..



  • [UPDATE]
    This afternoon I set a test 2.1-RELEASE pfSense on my mainsite and….

    As I stated before no difference in routing... (I got some images but they are useless beeing identical to the 2.1-RC1).
    Moreover:

    • I can ping site1 from inside mainsite pfsensefw

    • I can ping site2 from inside mainsite pfsensefw

    Indeed the problem seems to be on multiwan gw.
    Disabling failover/loadbal on lan net both vpn start to work.
    I found a similar problem here: http://forum.pfsense.org/index.php/topic,68494.0.html



  • What do your firewall logs show?  Where is this traffic getting blocked if at all?

    What do the traceroute logs show?



  • Here It's the traceroute…
    I did not look to the firewall logs... as I thought no block was on! Tomorrow I'll take a look...




  • This morning I have checked up the fw logs….
    No block at all...



  • I have no experience with doing a site to site over a MultiWAN setup but I would just make sure that you have 1194 opened up on both interfaces. I would also switch to UDP as TCP could be a source of your problems. Just thinking out loud could there be a problem traffic leaving one WAN interface and then coming back on Anohter? If you disable one of your WAN interfaces does this solve your issues. Is something that is even possible for you to do?



  • No way… at the moment I do not use the 2nd WAN in vpn conf (just internet conn). And the vpn is up and running (=> no fw problem) hence as I stated above disabling just the "multigw" allow vpn to "ping"…
    Indeed there is something wrong on routing in 2.1-RELEASE when  vpn is coupled with multigw.

    I don't know if there is something other we can do....

    I hope in some admin/developer help...
    Pleeeeeeeeeeese!  :)

    Otherwise I (we?) have just to wait 2.1.1...



  • Summarizing I thought It can be only a bug: how is possible routing differently 2 nets with the same gw?

    finally… I submit a bug on https://redmine.pfsense.org/issues/3309

    I hope this help...



  • [SOLVED]
    Fix will be available on 2.1.1