Two gateways, two subnets, one internet, subnet connectivity issue

marvosa

Check your routing table, what is the gateway of the default route? Diagnostics -> Routes. Post a screen shot if you can.

Live4soccer7

I've been using .112 instead of .254 :) My static routes on LAN1 match this.

![route table.png](/public/imported_attachments/1/route table.png)
![route table.png_thumb](/public/imported_attachments/1/route table.png_thumb)

marvosa

Can you ping 192.168.1.1? Can you ping 8.8.8.8?

Live4soccer7

I doubt it, but I'll change the configuration back right now and let you know.

Live4soccer7

Ping on both was 100% failure.

marvosa

Can you post a network map of how things are currently connected?

Also, what default gateway are your clients using on the PC's?

What happens when you ping 192.168.1.1 from PFsense itself?

Live4soccer7

What happens when you ping 192.168.1.1 from PFsense itself?

I'm checking now.

Please excuse my terrible drawing, but it is all there. Let me know if you need clarification.

IMG_1153.JPG_thumb

marvosa

Where can I input commands in pfSense?

Either from the shell via ssh or by going to Diagnostics -> Ping

Live4soccer7

Ping Results directly from pfSense. Looks good. That means something is still prohibiting a connection between the LAN and WAN on the pfsense box as I'm unable to ping from a client connected to the pfSense Subnet (192.168.5.0)

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=19.789 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=19.992 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=18.517 ms

–- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.517/19.433/19.992/0.653 ms

PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=1.897 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=3.034 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=2.398 ms

--- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.897/2.443/3.034/0.465 ms

marvosa

Run a tracert on your laptop to 8.8.8.8 and post the results.

Live4soccer7

Traceroute while I have internet access:

Traceroute has started…

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
1 192.168.5.1 (192.168.5.1) 2.032 ms 1.566 ms 3.057 ms
2 192.168.1.1 (192.168.1.1) 4.491 ms 2.755 ms 3.445 ms
3 * * *
4 WANISP.ptr.us.xo.net (WANISP) 13.697 ms 11.825 ms 15.092 ms
5 ge-0-0-2-spk-cr1.cet.com (198.202.27.65) 14.796 ms 12.241 ms 12.642 ms
6 te-1-1-sea-cr2.cet.com (198.202.26.2) 18.589 ms 20.696 ms 20.326 ms
7 six.sea01.google.com (206.81.80.17) 20.489 ms 20.139 ms 19.410 ms
8 108.170.245.113 (108.170.245.113) 20.336 ms 20.854 ms 20.953 ms
9 209.85.250.19 (209.85.250.19) 20.182 ms 20.219 ms 21.039 ms
10 google-public-dns-a.google.com (8.8.8.8) 19.162 ms 20.637 ms 20.258 ms

Let me run one when I turn NAT/Firewall off

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
1 192.168.5.1 (192.168.5.1) 1.249 ms 3.578 ms 1.490 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *

marvosa

So, that traceroute shows you hitting the LAN 2 internal interface, then routed to LAN 1 and out to the internet.

So, what do you mean when you say "traceroute while I have internet access"?

ahh… I think I know what you mean now, so once you turn off the firewall and NATing.... you can't get past the local adapter. Which tells me there's an internal routing issue.

Live4soccer7

Yes, that is correct. It hits the LAN interface on LAN2 and then dies on the spot.

Live4soccer7

Do I need a static route between the LAN interface and the WAN?

Live4soccer7

I can ping everything on LAN1 (192.168.1.0), however when I try to ping something on LAN2 I get something like this:

92 bytes from dd-wrt (192.168.1.1): Destination Host Unreachable
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 5400 4fc9 0 0000 3f 01 a41b 192.168.1.115 192.168.5.1

92 bytes from dd-wrt (192.168.1.1): Redirect Host(New addr: 192.168.1.254)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 f9f5 0 0000 3f 01 f9ee 192.168.1.115 192.168.5.1

I'm not sure what the .254 address is.

Live4soccer7

I found where the .254 was coming from. DDWRT was holding on to an old route that I set up previously in this thread. A reboot and it let go. Makes most of what I've been doing irrelevant and must go back now. Waiting for DHCP on the other end so I can connect to a PC over VNC for testing. I have not idea why VNC works over this network, but I'm glad that it is working.

Live4soccer7

I have it working! Finally. I have a several routes/rules/NATs in place and not 100% certain which allowed it to work or is doing nothing and haven't had the time to strip each one away at a time to see. I have access from both directions and internet on both sides. One strange thing is that I can not ping devices on LAN2 from LAN1, but I can connect to their web interfaces etc… I must still be missing one small key here.

On the DDWRT Router (192.168.1.1) that is hooked to the modem, I have a static route setup. Operating mode is gateway

Lan Dest: 192.168.5.0
Subnet Mask: 255.255.255.0
Gateway: 192.168.1.112

Then on pfSense (192.168.5.1)
WAN of pfsense is essentially hooked to the LAN of DDWRT
WANIP: 192.168.1.112
Gateway: 192.168.1.1
LANIP: 192.168.5.1

Static Route WAN
Destination Network: 192.168.5.0
Gateway: 192.168.1.1

Firewall/NAT are still on essentiallly. I pretty much bypassed them with the local network rules I have set.

NAT --- Outbound
Hybrid Outbound NAT
I added the following rule
Interface WAN
Source 192.168.1.0/24
Source Port *
Destination 192.168.5.0
Destination Port: *
Nat Address: WAN Address
Nat Port *
Static Port NO

Basically on the firewall rules I added any/any to WAN and LAN.

I'm sure this looks a bit ridiculous to someone with much more networking knowledge and I bet there are some major flaws as well, but it works and it is still behind the head router's firewall.

Live4soccer7

Hopefully someone is still watching this thread.

I want to go back to the beginning and take a look at something fundamental from the big picture of the design.

Essentially I wanted network 1 to still function internally if it loses connectivity to network 2 and vice versa. I also do not want local traffic on network 2 to "travel" all the way back to network 1 and then back for something that doesn't need internet. Example of streaming an IP cam locally on network2. I have limited bandwidth on the wireless bridge between the two networks and don't want to waste it on something that isn't necessary. Forget about internet connection if the two networks lose connectivity, I don't care at that point.

Have I gone about this the right way? Creating a subnet for each network and joining the subnets as I've done?

….or would have it have been better to simply have the ONE gateway in network 1 where the internet is and have pfsense essentially act as a router only and have them all on the same subnet (pfSense utilizing DHCP).

Network 1 router would function as gateway and DHCP server for let's say 192.168.1.150-.254
Network 2 router function as just a router and DHCP server for 192.168.1.1-149

Would this still allow each network to function by itself with the exception of internet and also not create redundancies in the wifi bridge traffic when devices on network 2 are communication between other devices on network 2?

I realize this thread has pretty much come full circle, but I have accomplished, mostly, what I aimed to do and have learned a lot in the process.

IMG_1154.JPG_thumb

marvosa

I have it working! Finally. I have a several routes/rules/NATs in place and not 100% certain which allowed it to work or is doing nothing and haven't had the time to strip each one away at a time to see.

Glad to hear it's working! The reason it's working is because traffic from LAN 2 is NAT'd to 192.168.1.112 as it exits the WAN interface and then routed to LAN 1's WAN interface where it's NAT'd again before hitting the internet. Since incoming traffic from LAN 2 is NAT'd to a subnet that's already in the routing table on LAN 1, it knows where to send the return traffic. Traffic sourced from LAN 1 is routed out the LAN interface and allowed thru LAN 2's firewall per the any/any rule.

Your outbound NAT rule on PFsense rule may or may not be helping the situation. Your NAT rule is translating all incoming LAN 1 traffic to the WAN IP of LAN 2. Since, LAN 2 already knows how to get to LAN 1's subnet, I doubt this rule is doing anything.

Unless there's a typo in your post about the static route on PFsense, the static route on PFsense isn't doing anything. Anything destined for 192.168.5.0/24 on LAN 2 would be handled by the switch and would never hit the firewall.

In short, it's working which is the most important thing, but because of the NATing you've lost all auditing capability.

I took some time to replicate your setup at home and got everything working as I was trying to explain it with one or two subtle differences.

My Setup
Modem -> PFsense_LAN1 -> Switch –-------->PFsense_LAN2 -> Switch -> Laptop 2 (192.168.5.2)
|
|-------> Laptop 1 (192.168.1.2)

Reproduced your subnets and connections on both sides
Statically assigned the LAN2 WAN interface with 192.168.1.254 and left the upstream gateway at "None"
Disabled the firewall and NAT on LAN2, so PFsense functions as a router only
Added a default route to 192.168.1.1 on the WAN interface of PFsense_LAN2 (System -> Routing -> Gateways) then add your gateway IP (192.168.1.1), then check the Default Gateway box
Added a static route on LAN1 for LAN2's LAN (destination 192.168.5.0/24 gateway 192.168.1.254). This step automatically adds an outbound NAT entry for 192.168.5.0/24 which allows LAN2 to get to the internet.

After the steps above, everything worked as expected:

Laptop 2 can access the internet thru LAN 1
Laptop 2 can ping Laptop 1 @ 192.168.1.2
Laptop 1 can ping Laptop 2 @ 192.168.5.2

If and when you want to try this, from what I can tell, the DDWRT side is set, you would only need to make some minor adjustments on the PFsense side to get things working.

Essentially I wanted network 1 to still function internally if it loses connectivity to network 2 and vice versa. I also do not want local traffic on network 2 to "travel" all the way back to network 1 and then back for something that doesn't need internet. Example of streaming an IP cam locally on network2. I have limited bandwidth on the wireless bridge between the two networks and don't want to waste it on something that isn't necessary. Forget about internet connection if the two networks lose connectivity, I don't care at that point.

You have the right idea. There are routers at each site, so all traffic will stay local unless explicitly destined for the other side. Also, there are DHCP servers at each site, so if there's an issue with the wireless bridge, it won't affect the LAN on either side with the exception of internet access @ LAN 2.

Have I gone about this the right way? Creating a subnet for each network and joining the subnets as I've done?

Yes, you went about it the right way, although you didn't join the subnets per se, but I know what you were trying to say :) You have separate sites directly connected by routers via a wireless bridge which is a perfectly sound design. The one caveat being ideally you want your firewall at the head end where the internet connection is, so you have more control over your network.

….or would have it have been better to simply have the ONE gateway in network 1 where the internet is and have pfsense essentially act as a router only and have them all on the same subnet (pfSense utilizing DHCP).

Having 1 gateway (DDWRT) and PFsense acting as a router is exactly what you have. Even though you have the firewall enabled on PFsense, because of the any/any rule on the WAN it's essentially acting as a router. The other option would be to replace DDWRT with PFsense at the head end and connect the wireless bridge to the switch at LAN 2. While this would work, it's not necessarily better because it leaves LAN 2 completely dependent on LAN 1's infrastructure and all the broadcast traffic from both sites would be traversing the wireless bridge and overall network performance would suffer.

Network 1 router would function as gateway and DHCP server for let's say 192.168.1.150-.254
Network 2 router function as just a router and DHCP server for 192.168.1.1-149

This wouldn't work, the subnets have to be unique at each site in a routed network. If for some reason you wanted the subnets to look similar at each site, you would have to do 192.168.1.0/25 for LAN 1 and 192.168.1.128/25 for LAN 2, but also understand that those are two different networks.

One strange thing is that I can not ping devices on LAN2 from LAN1, but I can connect to their web interfaces etc… I must still be missing one small key here.

This behavior is expected on windows machines. The windows firewall has ICMP echo reply disabled by default for traffic sourced outside of its local subnet. So, you either have to disable the windows firewall entirely or enable ICMP echo reply on every machine.

Live4soccer7

wow. I can't think you enough for the time you have invested in this. I surely hope this thread also helps out others as well.

I think I'll stick with the two subnets as it guarantees localization of the networks. I will try the tweaks you have mentioned as well. Ideally I'd like to have pfsense at the head of the network as well and I may do that in the future, but for now this will have to do.

My next project to tackle is to follow this tutorial on setting up OpenVPN with PIA: https://forum.pfsense.org/index.php?topic=76015.0
Do you foresee any issues integrating this with what's going on?