COX ISP (Rhode Island) Private Networks
-
Howdy all…
I have 3 sites. Two on Comcast (in MA) and one on COX (in RI). All sites are running pfSense 2.2.4.
The subnets are as follows:
Base (Comcast) - 10.1.2.0/24
Site 1 (Comcast) - 10.2.1.0/24
Site 2 (COX) - 10.3.1.0/24There is an IPSEC tunnel from BASE to Site 1 & another from BASE to Site 2.
I have ZERO issues between the two Comcast sites (Base and Site 1). Between the BASE site and Site 2 (COX), I've had NOTHING BUT issues...
There are 3 main problems...
1. For some reason, COX actually routes 10.x addresses... I think this is causing HUGE issues, and I can't figure out what combination of routes/firewall rules is needed to stop it from happening. What I'd love, is for all 10.x traffic to be blocked, with the exception of the 10.1.2.0/24 tunnel. PLEASE HELP.
2. The second issue I believe is related to this routing problem... The ipsec tunnel between BASE and Site 2 (COX) is completely unreliable. I think it has to do with the pfSense getting "confused" about where to route packets every once in a while... The tunnel will stay up for some length of time (never the same), and then drop out. My guess is if I can get problem 1 fixed, this one will sort itself out.
3. Lastly, I have an issue with the Synology NAS. This nas will NOT show itself across the IPSEC tunnel. My guess is, again, this is because of the private networks being exposed by COX. ALL OTHER HOSTS work across the tunnel, but not the NAS.
I'd really love some help with problem 1, and then I can start to troubleshoot problems 2 & 3. Right now, I can't even trust my routing because it doesn't matter if the tunnel is up or not, on the Site 2 side, the IP for the internal gateway of BASE (10.1.2.1) ALWAYS responds, which makes troubleshooting impossible.
Any help will be greatly appreciated.
-
Howdy all…
I have 3 sites. Two on Comcast (in MA) and one on COX (in RI). All sites are running pfSense 2.2.4.
The subnets are as follows:
Base (Comcast) - 10.1.2.0/24
Site 1 (Comcast) - 10.2.1.0/24
Site 2 (COX) - 10.3.1.0/24There is an IPSEC tunnel from BASE to Site 1 & another from BASE to Site 2.
I have ZERO issues between the two Comcast sites (Base and Site 1). Between the BASE site and Site 2 (COX), I've had NOTHING BUT issues...
There are 3 main problems...
1. For some reason, COX actually routes 10.x addresses... I think this is causing HUGE issues, and I can't figure out what combination of routes/firewall rules is needed to stop it from happening. What I'd love, is for all 10.x traffic to be blocked, with the exception of the 10.1.2.0/24 tunnel. PLEASE HELP.
What evidence can you present that shows that whatever Cox is doing with 10.x addresses has anything to do with what's going on inside your VPN tunnels?
2. The second issue I believe is related to this routing problem… The ipsec tunnel between BASE and Site 2 (COX) is completely unreliable. I think it has to do with the pfSense getting "confused" about where to route packets every once in a while... The tunnel will stay up for some length of time (never the same), and then drop out. My guess is if I can get problem 1 fixed, this one will sort itself out.
If the tunnel goes down, the Phase 2 route drops, and pfSense is going to route those networks out its default gateway. This can be blocked from egressing out your Cox WAN but it won't help bring your VPN tunnel up.
3. Lastly, I have an issue with the Synology NAS. This nas will NOT show itself across the IPSEC tunnel. My guess is, again, this is because of the private networks being exposed by COX. ALL OTHER HOSTS work across the tunnel, but not the NAS.
What do you mean "show itself?" If you're looking for some network discovery broadcast protocol to work across a routed connection you'll find nothing works.
I'd really love some help with problem 1, and then I can start to troubleshoot problems 2 & 3. Right now, I can't even trust my routing because it doesn't matter if the tunnel is up or not, on the Site 2 side, the IP for the internal gateway of BASE (10.1.2.1) ALWAYS responds, which makes troubleshooting impossible.
Any help will be greatly appreciated.
Are your Cox WAN IP addresses public/routable or RFC1918?
-
What evidence can you present that shows that whatever Cox is doing with 10.x addresses has anything to do with what's going on inside your VPN tunnels?
I don't. I'm grasping at straws. I use the exact same settings on all 3 sites, and only the COX network keeps dropping the tunnel. I will attach the log from BASE and SITE2 .
If the tunnel goes down, the Phase 2 route drops, and pfSense is going to route those networks out its default gateway. This can be blocked from egressing out your Cox WAN but it won't help bring your VPN tunnel up.
I guess I agree that it won't help with the tunnel, but it will help with troubleshooting. I've tried every instantiation of firewall rules on LAN and WAN interfaces, and I can't block 10.x traffic. It just flows right out. Help here on a firewall rule that would block all 10.x traffic (with the exception of 10.1.2.x traffic) would be helpful to me. I'm not sure why my BLOCK rules weren't weren't working. I even just tried blocking the whole 10.x/8, and that STILL didn't stop any traffic.
What do you mean "show itself?" If you're looking for some network discovery broadcast protocol to work across a routed connection you'll find nothing works.
I can't ping it. The pings and traceroutes go off into the ether. I can ping / traceroute EVERY other host, but not this one. Interestingly enough, the Synology NAS during "Router Setup", which I'm not using because we don't want it to be accessible at all from the outside, tells me that there are 2 routers on this network… The network is extremely simple, there aren't 2 routers. I have no idea where it's seeing the second router.
Are your Cox WAN IP addresses public/routable or RFC1918?
The Cox WAN IP is public/routable.
I've attached the logs and the configs of the Phase 1 and Phase 2 on both sides…
-
Create an alias called RFC1918 and include the following networks:
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16Make a floating rule: Reject, on WAN direction out, apply immediately on match, IPv4 protocol any, source any, dest RFC1918.
-
Create an alias called RFC1918 and include the following networks:
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16Make a floating rule: Reject, on WAN direction out, apply immediately on match, IPv4 protocol any, source any, dest RFC1918.
Thanks for the info. I thought I had done exactly that (although I added just the 10.x originally) and it didn't work. It seems to be working now.
In other news, I gave up on IPSEC and went over to Point-to-Point OpenVPN for all the Sites. Took me about 30 minutes to get sorted out, and everything has been up all day, no drops, very good bandwidth between everything. No complaints here. I'm going to monitor this setup and stick with OpenVPN if it works.
Thanks very much for all the help.
I still have an issue with the Synology, but I'm not completely sure what's causing it. Talk about weird… Here's what's happening with the NAS:
The NAS is at Site 2, IP Address 10.3.1.5. It's wired in directly. ALL HOSTS on my WIRED network at BASE can access it (ping, website, samba shares). My wireless clients (connected via a Ruckus Wireless 7363) can't access the NAS. They can access everything else at SITE 2, but not that ONE host... Really weird.
-
What evidence can you present that shows that whatever Cox is doing with 10.x addresses has anything to do with what's going on inside your VPN tunnels?
I don't. I'm grasping at straws. I use the exact same settings on all 3 sites, and only the COX network keeps dropping the tunnel. I will attach the log from BASE and SITE2 .
If the tunnel goes down, the Phase 2 route drops, and pfSense is going to route those networks out its default gateway. This can be blocked from egressing out your Cox WAN but it won't help bring your VPN tunnel up.
I guess I agree that it won't help with the tunnel, but it will help with troubleshooting. I've tried every instantiation of firewall rules on LAN and WAN interfaces, and I can't block 10.x traffic. It just flows right out. Help here on a firewall rule that would block all 10.x traffic (with the exception of 10.1.2.x traffic) would be helpful to me. I'm not sure why my BLOCK rules weren't weren't working. I even just tried blocking the whole 10.x/8, and that STILL didn't stop any traffic.
Make an alias including the following networks. Call it RFC1918.
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16Firewall > Rules, Floating Tab
Create a new rule
Action: Reject
Quick: Checked
Interface: Select your WAN or WANS
Direction: out
TCP/IP Version: IPv4
Protocol: any
Source: any
Destination: Single host or alias, RFC1918What do you mean "show itself?" If you're looking for some network discovery broadcast protocol to work across a routed connection you'll find nothing works.
I can't ping it. The pings and traceroutes go off into the ether. I can ping / traceroute EVERY other host, but not this one. Interestingly enough, the Synology NAS during "Router Setup", which I'm not using because we don't want it to be accessible at all from the outside, tells me that there are 2 routers on this network… The network is extremely simple, there aren't 2 routers. I have no idea where it's seeing the second router.
Sounds like you have a configuration problem on the NAS. Not sure why you're looking at the tunnel.
Are your Cox WAN IP addresses public/routable or RFC1918?
The Cox WAN IP is public/routable.
I've attached the logs and the configs of the Phase 1 and Phase 2 on both sides…
-
Make an alias including the following networks. Call it RFC1918.
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16Firewall > Rules, Floating Tab
Create a new rule
Action: Reject
Quick: Checked
Interface: Select your WAN or WANS
Direction: out
TCP/IP Version: IPv4
Protocol: any
Source: any
Destination: Single host or alias, RFC1918Thanks! Did this, and it certainly blocks the traffic now. I appreciate that.
Sounds like you have a configuration problem on the NAS. Not sure why you're looking at the tunnel.
I agree that it's likely not the tunnel. I just couldn't narrow it down any further. Now I can at least troubleshoot further. I actually think it's my WiFi access point at the BASE location.
Thanks for all the help Derelict. I really appreciate it. I'm not sure why the IPSec tunnel kept collapsing, but the OpenVPN tunnels have been rock solid for well over 24 hours now. Perhaps COX was messing with IPSec traffic.
Regardless, everything seems nice and stable. Now I just need to figure out why the NAS isn't available on the Wireless Clients, but is via the wired ones.
-
Draw your network. It'll probably be obvious if you do that.
-
Draw your network. It'll probably be obvious if you do that.
Thats the problem… Both networks are very simple... Flat networks, only one switch, both have identically configured pfSense routers.
My hard-wired machine has the ip: 10.1.2.100. My laptop (wireless) has the IP 10.1.2.101. Desktop can access the NAS, laptop can't. If I attach the laptop to the switch, and even use the same IP it works.
This is why I think it's my Wireless AP, but there's nothing special there either. It's a Rucks Wireless 7363, nothing strangely configured. I have another AP here that I can swap out and test.
