What to use for default gateway and static route
I have two sites connected by a point-to-point T1.
Internet gateway 172.16.72.254
gateway to Site B 172.16.72.49
Internet gateway 172.16.73.254 (running pfSense)
gateway to Site B 172.16.73.1
At site B if I use 172.16.73.254 as my default gateway and set a static route using 172.16.73.1 to reach site A,
none of the machines on 172.16.73.0 can reach site A unless I specifically add a static route for reaching site A
on each machine. The static route setup on the pfSense box using 172.16.73.1 seems to be overlooked.
At site B if I use 172.16.73.1 as my default gateway and set a static route using 172.16.73.254 to reach the Internet,
all the machines on 172.16.73.0 can reach site A, but for traffic to the Internet I get
"From 172.16.73.1: icmp_seq=1 Redirect Host(New nexthop: 172.16.73.254)"
which says my routing is not optimal.
It's best to set the gateway to 172.16.73.254 and add a static route on pfSense pointing to 172.16.73.1 for traffic destined to 172.16.72.0 255.255.255.0. Make sure the subnet stays right.
Do the same thing on the other side. Both sides will need to have a static route set on the Internet gateway for it to work. Otherwise, traffic will make it across the T1, but the return packets will just be sent off to the Internet and get discarded.
Thank you. That was my thinking too.
I have to look into why when I set a static route on pfSense to 172.16.73.1 for traffic to the 172.16.72.0 network that the machines on 172.16.73.0 can not ping across it. I will check that there is a static route on the other side set correctly back to the originating network.
To verify that the static route is working, you can run a traceroute(tracert windows) to something on the other network. The first hop should be pfSense and the second hop should be the 172.16.x.1 device.