Site-to-site loop - How to stop the loop?



  • Hey guys I have been doing some looking around to try and find a way to stop this site-to-site loop. I am a bit new to the site-to-site thing and wanting to know more about it so I set one up to test it out and so that my home network has access to my colo network both using Pfsense & Openvpn. So my issue is that when I ping a box from OTHER (C) to HOME (A) I am getting TTL expired in transit, tracert is a loop of 10.0.100.2 - 10.0.100.1 at the site OTHER (C).

    Networks / Sites / Tunnels
    Home (A):192.168.3.x / Openvpn tunnel:10.1.200.0/24 - Clients
    Colo (B):192.168.2.x - Openvpn server
    Other (C):192.168.1.x / Openvpn tunnel:10.0.100.0/24 - Clients

    Layout (I know could be better ::) )
    (A-192.168.3.x)==10.1.200.0/24==(B-192.168.2.x)==10.0.100.0/24==(C-192.168.1.x)

    Firewall Ruels
    Home (A): ANY-ANY
    Colo (B): ANY-ANY (The tab openvpn on this one has no settings, I was having issue where I could not get a hold of anything and after removing all settings under this tab all worked and has been like this for some time)
    Other (C): ANY-ANY

    –---

    Packet Capture from Colo (B) w/ openvpn interface
    20:22:22.168581 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:22:22.182551 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48409, length 8
    20:22:22.197295 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48409, length 8
    20:22:22.244038 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0
    20:22:22.453026 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38033, length 8
    20:22:22.453039 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38033, length 8
    20:22:22.683856 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48410, length 8
    20:22:22.697768 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48410, length 8
    20:22:22.956090 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38034, length 8
    20:22:22.956100 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38034, length 8
    20:22:23.168486 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:22:23.184740 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48411, length 8
    20:22:23.199135 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48411, length 8
    20:22:23.214819 IP 192.168.3.12.59694 > 192.168.1.55.3389: UDP, length 12
    20:22:23.230688 IP 192.168.1.55.3389 > 192.168.3.12.59694: UDP, length 16
    20:22:23.242777 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0
    20:22:23.469242 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38035, length 8
    20:22:23.469252 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38035, length 8
    20:22:23.685558 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48412, length 8
    20:22:23.700208 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48412, length 8
    20:22:23.981996 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38036, length 8
    20:22:23.982006 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38036, length 8
    20:22:24.168330 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:22:24.186957 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48413, length 8
    20:22:24.200678 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48413, length 8
    20:22:24.242025 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0
    20:22:24.501941 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38037, length 8
    20:22:24.501951 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38037, length 8
    20:22:24.687777 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48414, length 8
    20:22:24.702551 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48414, length 8
    20:22:24.774977 IP 192.168.3.2.62646 > 192.168.1.1.53: UDP, length 42
    20:22:24.790620 IP 192.168.1.1.53 > 192.168.3.2.62646: UDP, length 12
    20:22:25.028577 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38038, length 8
    20:22:25.028586 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38038, length 8
    
    
    Packet Capture from Other (C) w/ openvpn interface
    20:20:17.966349 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:17.966358 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:17.966361 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:17.966365 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:17.966368 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 157
    20:20:18.031131 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0
    20:20:18.031158 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0
    20:20:18.059369 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 493
    20:20:18.059384 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 0
    20:20:18.059889 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:18.059894 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:18.059896 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:18.059898 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:18.059900 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350
    20:20:18.059902 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 637
    20:20:18.061946 IP 192.168.3.12.57357 > 192.168.1.1.2486: tcp 493
    20:20:18.061960 IP 192.168.1.1.2486 > 192.168.3.12.57357: tcp 0
    20:20:18.062120 IP 192.168.1.1.2486 > 192.168.3.12.57357: tcp 1128
    20:20:18.065188 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 464
    20:20:18.065206 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 0
    20:20:18.065661 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350
    20:20:18.065665 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350
    20:20:18.065667 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350
    20:20:18.065669 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350
    20:20:18.065671 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350
    20:20:18.065672 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 838
    20:20:18.113443 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0
    20:20:18.114551 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0
    20:20:18.115181 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0
    20:20:18.117586 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 0
    20:20:18.117608 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 0
    20:20:18.118340 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 0
    20:20:18.136722 IP 192.168.3.12.57357 > 192.168.1.1.2486: tcp 0
    20:20:18.185364 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37790, length 8
    20:20:18.201056 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37790, length 8
    20:20:18.370729 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48162, length 8
    20:20:18.370739 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48162, length 8
    20:20:18.532704 IP 192.168.1.55.3389 > 192.168.3.12.59695: UDP, length 16
    20:20:18.705085 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37791, length 8
    20:20:18.720357 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37791, length 8
    20:20:18.872663 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48163, length 8
    20:20:18.872673 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48163, length 8
    20:20:18.876489 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:20:18.965559 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0
    20:20:19.224799 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37792, length 8
    20:20:19.239245 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37792, length 8
    20:20:19.372774 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48164, length 8
    20:20:19.372784 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48164, length 8
    20:20:19.728696 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37793, length 8
    20:20:19.742884 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37793, length 8
    20:20:19.873835 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48165, length 8
    20:20:19.873845 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48165, length 8
    20:20:19.876529 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:20:19.967339 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0
    20:20:20.238013 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37794, length 8
    20:20:20.252919 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37794, length 8
    20:20:20.374606 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48166, length 8
    20:20:20.374615 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48166, length 8
    20:20:20.752359 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37795, length 8
    20:20:20.766346 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37795, length 8
    20:20:20.875899 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48167, length 8
    20:20:20.875912 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48167, length 8
    20:20:20.876545 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:20:20.969120 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0
    20:20:21.258031 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37796, length 8
    20:20:21.276826 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37796, length 8
    20:20:21.376922 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48168, length 8
    20:20:21.376932 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48168, length 8
    20:20:21.768005 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37797, length 8
    20:20:21.782205 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37797, length 8
    20:20:21.876570 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101
    20:20:21.877929 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48169, length 8
    20:20:21.877938 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48169, length 8
    
    
    Tracing route from Other (C) to a box on my Home (A) network.
    Tracing route to 192.168.3.12 over a maximum of 30 hops
    
      1    <1 ms    <1 ms    <1 ms  192.168.1.1
      2    14 ms    14 ms    15 ms  10.0.100.1
      3    15 ms    14 ms    15 ms  10.0.100.2
      4    29 ms    28 ms    28 ms  10.0.100.1
      5    29 ms    29 ms    29 ms  10.0.100.2
      6    43 ms    54 ms    43 ms  10.0.100.1
      7    44 ms    45 ms    44 ms  10.0.100.2
      8    75 ms    58 ms    58 ms  10.0.100.1
      9    57 ms    57 ms    66 ms  10.0.100.2
     10    73 ms    79 ms    75 ms  10.0.100.1
     11    84 ms    76 ms    80 ms  10.0.100.2
     12    94 ms    94 ms    87 ms  10.0.100.1
     13    88 ms    88 ms   104 ms  10.0.100.2
     14   106 ms   123 ms   104 ms  10.0.100.1
     15   106 ms   115 ms   104 ms  10.0.100.2
     16   128 ms   118 ms   135 ms  10.0.100.1
     17   118 ms   119 ms   122 ms  10.0.100.2
     18   151 ms   136 ms   132 ms  10.0.100.1
     19   157 ms   140 ms   131 ms  10.0.100.2
     20   150 ms   160 ms   147 ms  10.0.100.1
     21   157 ms   164 ms   170 ms  10.0.100.2
     22   162 ms   167 ms   163 ms  10.0.100.1
     23   158 ms   165 ms   162 ms  10.0.100.2
     24   176 ms   182 ms   187 ms  10.0.100.1
     25   177 ms   181 ms   177 ms  10.0.100.2
     26   198 ms   204 ms   208 ms  10.0.100.1
     27   198 ms   194 ms   195 ms  10.0.100.2
     28   202 ms   212 ms   216 ms  10.0.100.1
     29   203 ms   213 ms   218 ms  10.0.100.2
     30   233 ms   237 ms   229 ms  10.0.100.1
    
    

    Umm, I don't know what all to post so just ask!!

    (I know some of this can be fixed up my ips and so on could be better but this is more for real world testing/playing around an when I get more time I will fix it all up nice 8) )


  • Rebel Alliance Global Moderator

    So you setup site to site vpn between A and B and C and B?  Or A an C are just road warriors into B?

    So you created a route that Tells A hey to get to C go to B?  Or your just default routing it out?



  • @johnpoz:

    A an C are just road warriors into B?

    A an C are just road warriors into B

    Just default routing it out?

    Yep, I added the ip's of Home (A):192.168.3.x & Other (C):192.168.1.x into filed "IPv4 Remote network(s)" under the OpenVPN server tab on Colo (B) … Then did the same on A & C under the client tab but I left out the LAN or local network of that site and only added the remote networks. Should have I created a route??

    just to add some more info.

    2.3.4-RELEASE-p1 (i386)
    Colo (B) the firewall and server for openvpn on a Watchguard X550e

    Thanks for your reply johnpoz  :)



  • Rebel Alliance Global Moderator

    "Yep, I added the ip's of Home (A):192.168.3.x & Other (C):192.168.1.x into filed "IPv4 Remote network(s)" "

    Huh?  If your server is road warrior server, why would you setup any remote networks??  If your going to setup your server as s2s then the clients should be s2s as well, not remote access..

    You have pfsense at site A and B right?

    https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site



  • @johnpoz:

    "Yep, I added the ip's of Home (A):192.168.3.x & Other (C):192.168.1.x into filed "IPv4 Remote network(s)" "

    Huh?  If your server is road warrior server, why would you setup any remote networks??  If your going to setup your server as s2s then the clients should be s2s as well, not remote access..

    Sorry. I must have miss understood "road warrior" (Did have my coffee yep :o an just so I know im on the same page. road warrior=mobile? ), no the server and clients are not "road warrior"…
    So you setup site to site vpn between A and B and C and B?yes!? if I am reading this right this time… (look at pic in post up there^)

    You have pfsense at site A and B right?

    Yes

    https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site

    I did follow this tutorial to get everything up and running, as it said to add remote networks..

    Server settings: "IPv4 Remote networks : Enter the remote (Client Side) LAN here. To access more than one network, add them all here separated by a comma (e.g. 10.10.10.0/24, 192.168.10.0/24)."

    Client Settings: "IPv4 Remote networks : Enter the remote (Server Side) LAN here. To access more than one network, add them all here separated by a comma (e.g. 10.10.9.0/24, 192.168.9.0/24)."


    Screenshots of server settings and client settings.

    Screenshot_25 is the server side B
    Screenshot_26 is client side C





  • Rebel Alliance Global Moderator

    "I did follow this tutorial to get everything up and running, a"

    And did you see where the other site is also s2s and not road warrior..

    according to your drawing your C site is 192.168.1.. But you have on its client setting that 192.168.1 is remote!  That is not the case the networks that are remote from it or B and A, not itself.



  • @johnpoz:

    And did you see where the other site is also s2s and not road warrior..

    Quote what you are talking about from the tut.. By road warrior do you mean client?

    according to your drawing your C site is 192.168.1.. But you have on its client setting that 192.168.1 is remote!  That is not the case the networks that are remote from it or B and A, not itself.

    That is a typo (C is 192.168.2.x) see Attachment.



  • Rebel Alliance Global Moderator

    "Quote what you are talking about from the tut.. By road warrior do you mean client?"

    Yes the CLIENT!!  You stated it was road warrior or remote access..  That is not how you do it..

    @Fmslick:

    [A an C are just road warriors into B
    [/quote]

    "That is a typo (C is 192.168.2.x) see Attachment."

    Your new attachment shows 192.168.1 - dude can not help you without correct info…




  • Sorry, lets start over!

    I have listed all of my setting on all the box's , if I have some wrong settings or just effed up just say so lol.

    A: 192.168.3.x - Home network
    B: 192.168.2.x - Openvpn servers
    C: 192.168.1.x - Other network

    A: 192.168.3.x - Home network
    General Information
    Client A
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.1.200.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.1.0/24,192.168.2.0/24
    Disable IPv6: yes
    –--------------------------------------------------------------------------------------
    B: 192.168.2.x - Openvpn servers
    Server settings to (A)
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.1.200.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.3.0/24,192.168.1.0/24
    Disable IPv6: yes

    General Information
    Server to other (C)
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.0.100.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.1.0/24,192.168.3.0/24
    Disable IPv6: yes

    [[b]Why did I use two OPENVPN servers you may ask? Well I tried to use one but could not get it to work so I googled and found this video https://youtu.be/8f13lfnEKY8 and it worked. is using two OPENVPN servers wrong? ]
    –---------------------------------------------------------------------------------------
    C: 192.168.1.x - Other network
    General Information
    Client C
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.0.100.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.2.0/24,192.168.3.0/24
    Disable IPv6: yes

    Is there more info you need johnpoz?



  • Netgate

    Do this:

    B: 192.168.2.x - Openvpn servers
    Server settings to (A)
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.1.200.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.3.0/24~~,192.168.1.0/24~~
    Disable IPv6: yes

    Server setting to(C)
    General Information
    Server to other (C)
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.0.100.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.1.0/24~~,192.168.3.0/24~~
    Disable IPv6: yes

    You are attempting to create an OpenVPN route for both remote sites on both openvpn tunnels. You only need to define the remote networks that are actually remote on that connection.

    You are probably seeing all kinds of strangeness because the OpenVPN process that starts first gets both routes and the other one fails to add the routes because they already exist.



  • @Derelict:

    Do this:

    B: 192.168.2.x - Openvpn servers
    Server settings to (A)
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.1.200.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.3.0/24~~,192.168.1.0/24~~
    Disable IPv6: yes

    Server setting to(C)
    General Information
    Server to other (C)
    Server mode: p2p (Shared Key)
    Device mode: tun
    Interface: WAN
    Local port: blank

    Tunnel Settings
    IPv4 Tunnel Network: 10.0.100.0/24
    IPv6 Tunnel Network: blank
    IPv4 Remote network(s): 192.168.1.0/24~~,192.168.3.0/24~~
    Disable IPv6: yes

    You are attempting to create an OpenVPN route for both remote sites on both openvpn tunnels. You only need to define the remote networks that are actually remote on that connection.

    You are probably seeing all kinds of strangeness because the OpenVPN process that starts first gets both routes and the other one fails to add the routes because they already exist.

    Hey thinks that worked, after removing ,192.168.3.0/24 & ,192.168.1.0/24 from C on the two servers ping and tracert is now working and not looping. So I see what I was doing wrong after you pointed it out, think you for this Derelict.