Remote access one subnet to secondary subnet w/it's own gateway

hdokes

Greetings,

Have a dilemma. Need alternate access to a subnet connected to another subnet via a point to point dedicated fiber by 3rd party (AT&T) supplier. Each subnet has their respective pfSense router to independent Internet sources. We have servers at one subnet and occasionally it's Internet service drops. We have associates around the world that use services off these servers and otherwise need another method of connectivity to them when this occurs. We have been able to confirm our current configuration allows packets to get from a device on Internet A shown below to subnet (and server) D and the response is then sent back to pfSense box B. The issue is the packet gets held up at pfSense B on the return reply path and does not get beyond pfSense B to the originating device on Internet A. Shouldn't pfSense B have a reference to convert the reply back to the requesting IP on Internet A or is there a rule we may be missing?

The current rules we have on pfSense B are:

Port Forwarding: Any on WAN port 8080 to 192.168.1.74 port 8080
Outbound NAT: LAN outbound from Any to 192.168.1.74/32
Auto Generated filter rule: Pass Any via WAN from Any to 192.168.1.74 port 8080

Outbound NAT mode for pfSense B is Hybrid outbound rule generation.

_____________       _____________         172.16.0.1/24        ___________________       
| Internet A  |-----| pfSense B |---------|  Subnet C | -------| AT&T PtP  Router |--------|
--------------      -------------          -----------         --------------------        |
                                                                                           |
_____________________________Point to Point fiber__________________________________________|
|
|         ___________________       __192.168.1.0/16___        ____________        _____________  
|---------| AT&T PtP Router |------| Subnet D - Server |-------| pfSense E |-------| Internet F |
          -------------------      ---------------------       -------------       --------------

boxofrox

Heh, turns out this configuration works for hdokes and myself.

We were using curl to test the alternate path to a http server behind a reverse proxy on Subnet D and assumed the error message printed, No such site at :8080, was from curl telling us it couldn't get a response from the web server. This assumption was based, in part, on the fact that a tcpdump of the WAN port on pfSense B didn't report a response packet heading back to the client. A tcpdump of the LAN port on pfSense B did show a response packet was received.

The error message is actually from the reverse proxy because it was only configured to proxy HTTP requests to host 192.168.1.74 or to the host associated with the pfSense E WAN IP. A quick update to its configuration to add hosts for pfSense B's WAN IP and we now have a backup route when our crappy internet service for Internet F drops out for the 20th time in a day.

johnpoz

Why would you not just connect your point to point connection via a transit network to your 2 different pfsense boxes.

Your sort of connection is just a asymmetrical mess..

boxofrox

@johnpoz

I'm not familiar with the term "transit network", and what little I can find on the term suggests that it's just a network that bridges two networks, and passes traffic (packets that neither originate from or destinate to the transit network). According to that definition, the point-to-point connection is a transit network. I'm definitely misunderstanding something, because why would I connect a transit network (the point to point connection) via another transit network? Is this recommendation just a reorganization of the topology, or would it address the issue of routing return traffic back to the Internet A|F that initiated the TCP connection?

As to the reason why I would not change anything concerning to the point-to-point, it's not my equipment. It's AT&T's. It took their bureaucratic nightmare of tech's about 6 months to correctly configure their equipment when the service was installed. The install (fiber run, equipment install, and configuration) was supposed to be done within a month. Now that it's working, I'm not inclined to ask them to change anything.

I don't understand the "asymmetrical mess" generalization. With the exception of the different subnets, hdokes' diagram is symmetrical. As we're only concerned, at this time, with access to a single server on Subnet D from the Internet through either pfSense B|E, we've only got the two extra firewall rules for port forwarding and outbound NAT. Hardly constitutes a mess.

Perhaps I should clarify that for general internet access, Subnet C's default route is pfSense B, and Subnet D's default route is pfSense E. Internet-bound traffic on those subnets don't traverse the point-to-point fiber (except for this alternate route to the Subnet D server). The point-to-point allows a remote store to access the servers for POS, NAS, etc located at the primary store. Should we decide later to route internet access through the fiber, I can see how that configuration in pfSense might get messy with our current topology, although I still don't see how changing the topology avoids the need for outbound NAT rules to return traffic over the point-to-point. Maybe fewer rules are needed?

Also, keep in mind that this topology wasn't designed first, then implemented. It grew piece by piece over years. The business owner has a strict "if it ain't broke, don't touch it" policy, so new infrastructure is added in such a way that it has the least effect on existing infrastructure. We found ourselves in a not-optimal situation trying to leverage pfSense to solve an access problem within a day, and to pfSense's tremedous credit, we could and did accomplish just that.

johnpoz

Yes a transit network is a network that connections networks. Ie your point to point is a transit network.

So you should do this for example

You have asymmetrical mess because the default gateway on say a box in 192.168.1 is pfsense, but pfsense doesn't then has to send the traffic to your point to point router that has a leg in the 192.168.1 network.

BUT!!! here is where the asymmetrical problem comes in.. when the traffic comes back.. It will not go through pfsense it will just go straight to the client IP in 192.168.1.x because your point to point router has a leg in this network. So no after X mins the session will time out on the firewall on pfsense and when your client sents back an ACK to something pfsense will deny because that state is no longer open because there as no traffic, etc. And its possible your client might not even accept the return traffic because the mac s not the gateway it sent the traffic too.

You should never have asymmetrical traffic, especially when a stateful firewall is in play. So you connect your 2 locations with a transit network clients talk to their gatewas.. Pfsense says oh I know how to get to that network I send it to this 10.0.0.x address. goes down your pipe, gets to pfsense router at other location says yeah you wan to go to 172.16.0.x sure I have a network in that.. Answer comes back pfsense says oh you want to send that to 192.168 yeah I know how to get there I send it to 10.0.0.x address.. Now you have symmetrical traffic flow

boxofrox

@johnpoz said in Remote access one subnet to secondary subnet w/it's own gateway:

BUT!!! here is where the asymmetrical problem comes in… when the traffic comes back… It will not go through pfsense it will just go straight to the client IP in 192.168.1.x because your point to point router has a leg in this network. So no after X mins the session will time out on the firewall on pfsense and when your client sents back an ACK to something pfsense will deny because that state is no longer open because there as no traffic, etc. And its possible your client might not even accept the return traffic because the mac s not the gateway it sent the traffic too.

This makes perfect sense and I completely missed that scenario. Many thanks for the clarification.

You should never have asymmetrical traffic, especially when a stateful firewall is in play.

Agreed. I believe we avoid asymmetical traffic by configuring permanent routes in the clients and servers for the opposing subnet, so that traffic never uses the default gateway. It's not ideal, but We Work With What We Got (tm).

johnpoz

yes when you put hosts on a transit you have to route on them as well.. because hosts do not belong on a transit network only routers do..