Site-to-site loop - How to stop the loop?
-
Hey guys I have been doing some looking around to try and find a way to stop this site-to-site loop. I am a bit new to the site-to-site thing and wanting to know more about it so I set one up to test it out and so that my home network has access to my colo network both using Pfsense & Openvpn. So my issue is that when I ping a box from OTHER (C) to HOME (A) I am getting TTL expired in transit, tracert is a loop of 10.0.100.2 - 10.0.100.1 at the site OTHER (C).
Networks / Sites / Tunnels
Home (A):192.168.3.x / Openvpn tunnel:10.1.200.0/24 - Clients
Colo (B):192.168.2.x - Openvpn server
Other (C):192.168.1.x / Openvpn tunnel:10.0.100.0/24 - ClientsLayout (I know could be better ::) )
(A-192.168.3.x)==10.1.200.0/24==(B-192.168.2.x)==10.0.100.0/24==(C-192.168.1.x)Firewall Ruels
Home (A): ANY-ANY
Colo (B): ANY-ANY (The tab openvpn on this one has no settings, I was having issue where I could not get a hold of anything and after removing all settings under this tab all worked and has been like this for some time)
Other (C): ANY-ANY–---
Packet Capture from Colo (B) w/ openvpn interface 20:22:22.168581 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:22:22.182551 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48409, length 8 20:22:22.197295 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48409, length 8 20:22:22.244038 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0 20:22:22.453026 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38033, length 8 20:22:22.453039 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38033, length 8 20:22:22.683856 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48410, length 8 20:22:22.697768 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48410, length 8 20:22:22.956090 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38034, length 8 20:22:22.956100 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38034, length 8 20:22:23.168486 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:22:23.184740 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48411, length 8 20:22:23.199135 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48411, length 8 20:22:23.214819 IP 192.168.3.12.59694 > 192.168.1.55.3389: UDP, length 12 20:22:23.230688 IP 192.168.1.55.3389 > 192.168.3.12.59694: UDP, length 16 20:22:23.242777 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0 20:22:23.469242 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38035, length 8 20:22:23.469252 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38035, length 8 20:22:23.685558 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48412, length 8 20:22:23.700208 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48412, length 8 20:22:23.981996 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38036, length 8 20:22:23.982006 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38036, length 8 20:22:24.168330 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:22:24.186957 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48413, length 8 20:22:24.200678 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48413, length 8 20:22:24.242025 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0 20:22:24.501941 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38037, length 8 20:22:24.501951 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38037, length 8 20:22:24.687777 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48414, length 8 20:22:24.702551 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48414, length 8 20:22:24.774977 IP 192.168.3.2.62646 > 192.168.1.1.53: UDP, length 42 20:22:24.790620 IP 192.168.1.1.53 > 192.168.3.2.62646: UDP, length 12 20:22:25.028577 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 38038, length 8 20:22:25.028586 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 38038, length 8
Packet Capture from Other (C) w/ openvpn interface 20:20:17.966349 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:17.966358 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:17.966361 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:17.966365 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:17.966368 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 157 20:20:18.031131 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0 20:20:18.031158 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0 20:20:18.059369 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 493 20:20:18.059384 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 0 20:20:18.059889 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:18.059894 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:18.059896 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:18.059898 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:18.059900 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 1350 20:20:18.059902 IP 192.168.1.1.2486 > 192.168.3.12.57356: tcp 637 20:20:18.061946 IP 192.168.3.12.57357 > 192.168.1.1.2486: tcp 493 20:20:18.061960 IP 192.168.1.1.2486 > 192.168.3.12.57357: tcp 0 20:20:18.062120 IP 192.168.1.1.2486 > 192.168.3.12.57357: tcp 1128 20:20:18.065188 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 464 20:20:18.065206 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 0 20:20:18.065661 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350 20:20:18.065665 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350 20:20:18.065667 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350 20:20:18.065669 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350 20:20:18.065671 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 1350 20:20:18.065672 IP 192.168.1.1.2486 > 192.168.3.12.57353: tcp 838 20:20:18.113443 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0 20:20:18.114551 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0 20:20:18.115181 IP 192.168.3.12.57356 > 192.168.1.1.2486: tcp 0 20:20:18.117586 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 0 20:20:18.117608 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 0 20:20:18.118340 IP 192.168.3.12.57353 > 192.168.1.1.2486: tcp 0 20:20:18.136722 IP 192.168.3.12.57357 > 192.168.1.1.2486: tcp 0 20:20:18.185364 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37790, length 8 20:20:18.201056 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37790, length 8 20:20:18.370729 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48162, length 8 20:20:18.370739 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48162, length 8 20:20:18.532704 IP 192.168.1.55.3389 > 192.168.3.12.59695: UDP, length 16 20:20:18.705085 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37791, length 8 20:20:18.720357 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37791, length 8 20:20:18.872663 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48163, length 8 20:20:18.872673 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48163, length 8 20:20:18.876489 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:20:18.965559 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0 20:20:19.224799 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37792, length 8 20:20:19.239245 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37792, length 8 20:20:19.372774 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48164, length 8 20:20:19.372784 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48164, length 8 20:20:19.728696 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37793, length 8 20:20:19.742884 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37793, length 8 20:20:19.873835 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48165, length 8 20:20:19.873845 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48165, length 8 20:20:19.876529 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:20:19.967339 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0 20:20:20.238013 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37794, length 8 20:20:20.252919 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37794, length 8 20:20:20.374606 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48166, length 8 20:20:20.374615 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48166, length 8 20:20:20.752359 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37795, length 8 20:20:20.766346 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37795, length 8 20:20:20.875899 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48167, length 8 20:20:20.875912 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48167, length 8 20:20:20.876545 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:20:20.969120 IP 192.168.3.12.56676 > 192.168.1.55.3389: tcp 0 20:20:21.258031 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37796, length 8 20:20:21.276826 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37796, length 8 20:20:21.376922 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48168, length 8 20:20:21.376932 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48168, length 8 20:20:21.768005 IP 10.0.100.2 > 10.0.100.1: ICMP echo request, id 23039, seq 37797, length 8 20:20:21.782205 IP 10.0.100.1 > 10.0.100.2: ICMP echo reply, id 23039, seq 37797, length 8 20:20:21.876570 IP 192.168.1.55.3389 > 192.168.3.12.56676: tcp 101 20:20:21.877929 IP 10.0.100.1 > 10.0.100.2: ICMP echo request, id 15451, seq 48169, length 8 20:20:21.877938 IP 10.0.100.2 > 10.0.100.1: ICMP echo reply, id 15451, seq 48169, length 8
Tracing route from Other (C) to a box on my Home (A) network. Tracing route to 192.168.3.12 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.1.1 2 14 ms 14 ms 15 ms 10.0.100.1 3 15 ms 14 ms 15 ms 10.0.100.2 4 29 ms 28 ms 28 ms 10.0.100.1 5 29 ms 29 ms 29 ms 10.0.100.2 6 43 ms 54 ms 43 ms 10.0.100.1 7 44 ms 45 ms 44 ms 10.0.100.2 8 75 ms 58 ms 58 ms 10.0.100.1 9 57 ms 57 ms 66 ms 10.0.100.2 10 73 ms 79 ms 75 ms 10.0.100.1 11 84 ms 76 ms 80 ms 10.0.100.2 12 94 ms 94 ms 87 ms 10.0.100.1 13 88 ms 88 ms 104 ms 10.0.100.2 14 106 ms 123 ms 104 ms 10.0.100.1 15 106 ms 115 ms 104 ms 10.0.100.2 16 128 ms 118 ms 135 ms 10.0.100.1 17 118 ms 119 ms 122 ms 10.0.100.2 18 151 ms 136 ms 132 ms 10.0.100.1 19 157 ms 140 ms 131 ms 10.0.100.2 20 150 ms 160 ms 147 ms 10.0.100.1 21 157 ms 164 ms 170 ms 10.0.100.2 22 162 ms 167 ms 163 ms 10.0.100.1 23 158 ms 165 ms 162 ms 10.0.100.2 24 176 ms 182 ms 187 ms 10.0.100.1 25 177 ms 181 ms 177 ms 10.0.100.2 26 198 ms 204 ms 208 ms 10.0.100.1 27 198 ms 194 ms 195 ms 10.0.100.2 28 202 ms 212 ms 216 ms 10.0.100.1 29 203 ms 213 ms 218 ms 10.0.100.2 30 233 ms 237 ms 229 ms 10.0.100.1
Umm, I don't know what all to post so just ask!!
(I know some of this can be fixed up my ips and so on could be better but this is more for real world testing/playing around an when I get more time I will fix it all up nice 8) )
-
So you setup site to site vpn between A and B and C and B? Or A an C are just road warriors into B?
So you created a route that Tells A hey to get to C go to B? Or your just default routing it out?
-
A an C are just road warriors into B?
A an C are just road warriors into B
Just default routing it out?
Yep, I added the ip's of Home (A):192.168.3.x & Other (C):192.168.1.x into filed "IPv4 Remote network(s)" under the OpenVPN server tab on Colo (B) … Then did the same on A & C under the client tab but I left out the LAN or local network of that site and only added the remote networks. Should have I created a route??
just to add some more info.
2.3.4-RELEASE-p1 (i386)
Colo (B) the firewall and server for openvpn on a Watchguard X550eThanks for your reply johnpoz :)
-
"Yep, I added the ip's of Home (A):192.168.3.x & Other (C):192.168.1.x into filed "IPv4 Remote network(s)" "
Huh? If your server is road warrior server, why would you setup any remote networks?? If your going to setup your server as s2s then the clients should be s2s as well, not remote access..
You have pfsense at site A and B right?
https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site
-
"Yep, I added the ip's of Home (A):192.168.3.x & Other (C):192.168.1.x into filed "IPv4 Remote network(s)" "
Huh? If your server is road warrior server, why would you setup any remote networks?? If your going to setup your server as s2s then the clients should be s2s as well, not remote access..
Sorry. I must have miss understood "road warrior" (Did have my coffee yep :o an just so I know im on the same page. road warrior=mobile? ), no the server and clients are not "road warrior"…
So you setup site to site vpn between A and B and C and B?yes!? if I am reading this right this time… (look at pic in post up there^)You have pfsense at site A and B right?
Yes
https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site
I did follow this tutorial to get everything up and running, as it said to add remote networks..
Server settings: "IPv4 Remote networks : Enter the remote (Client Side) LAN here. To access more than one network, add them all here separated by a comma (e.g. 10.10.10.0/24, 192.168.10.0/24)."
Client Settings: "IPv4 Remote networks : Enter the remote (Server Side) LAN here. To access more than one network, add them all here separated by a comma (e.g. 10.10.9.0/24, 192.168.9.0/24)."
–
Screenshots of server settings and client settings.Screenshot_25 is the server side B
Screenshot_26 is client side C
-
"I did follow this tutorial to get everything up and running, a"
And did you see where the other site is also s2s and not road warrior..
according to your drawing your C site is 192.168.1.. But you have on its client setting that 192.168.1 is remote! That is not the case the networks that are remote from it or B and A, not itself.
-
And did you see where the other site is also s2s and not road warrior..
Quote what you are talking about from the tut.. By road warrior do you mean client?
according to your drawing your C site is 192.168.1.. But you have on its client setting that 192.168.1 is remote! That is not the case the networks that are remote from it or B and A, not itself.
That is a typo (C is 192.168.2.x) see Attachment.
-
"Quote what you are talking about from the tut.. By road warrior do you mean client?"
Yes the CLIENT!! You stated it was road warrior or remote access.. That is not how you do it..
[A an C are just road warriors into B
[/quote]"That is a typo (C is 192.168.2.x) see Attachment."
Your new attachment shows 192.168.1 - dude can not help you without correct info…
-
Sorry, lets start over!
I have listed all of my setting on all the box's , if I have some wrong settings or just effed up just say so lol.
A: 192.168.3.x - Home network
B: 192.168.2.x - Openvpn servers
C: 192.168.1.x - Other networkA: 192.168.3.x - Home network
General Information
Client A
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.1.200.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.1.0/24,192.168.2.0/24
Disable IPv6: yes
–--------------------------------------------------------------------------------------
B: 192.168.2.x - Openvpn servers
Server settings to (A)
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.1.200.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.3.0/24,192.168.1.0/24
Disable IPv6: yesGeneral Information
Server to other (C)
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.0.100.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.1.0/24,192.168.3.0/24
Disable IPv6: yes[[b]Why did I use two OPENVPN servers you may ask? Well I tried to use one but could not get it to work so I googled and found this video https://youtu.be/8f13lfnEKY8 and it worked. is using two OPENVPN servers wrong? ]
–---------------------------------------------------------------------------------------
C: 192.168.1.x - Other network
General Information
Client C
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.0.100.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.2.0/24,192.168.3.0/24
Disable IPv6: yesIs there more info you need johnpoz?
-
Do this:
B: 192.168.2.x - Openvpn servers
Server settings to (A)
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.1.200.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.3.0/24~~,192.168.1.0/24~~
Disable IPv6: yesServer setting to(C)
General Information
Server to other (C)
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.0.100.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.1.0/24~~,192.168.3.0/24~~
Disable IPv6: yesYou are attempting to create an OpenVPN route for both remote sites on both openvpn tunnels. You only need to define the remote networks that are actually remote on that connection.
You are probably seeing all kinds of strangeness because the OpenVPN process that starts first gets both routes and the other one fails to add the routes because they already exist.
-
Do this:
B: 192.168.2.x - Openvpn servers
Server settings to (A)
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.1.200.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.3.0/24~~,192.168.1.0/24~~
Disable IPv6: yesServer setting to(C)
General Information
Server to other (C)
Server mode: p2p (Shared Key)
Device mode: tun
Interface: WAN
Local port: blankTunnel Settings
IPv4 Tunnel Network: 10.0.100.0/24
IPv6 Tunnel Network: blank
IPv4 Remote network(s): 192.168.1.0/24~~,192.168.3.0/24~~
Disable IPv6: yesYou are attempting to create an OpenVPN route for both remote sites on both openvpn tunnels. You only need to define the remote networks that are actually remote on that connection.
You are probably seeing all kinds of strangeness because the OpenVPN process that starts first gets both routes and the other one fails to add the routes because they already exist.
Hey thinks that worked, after removing
,192.168.3.0/24&,192.168.1.0/24from C on the two servers ping and tracert is now working and not looping. So I see what I was doing wrong after you pointed it out, think you for this Derelict.