Routing traffic to a remote site



  • I've been trying to get something to work for a little while and no joy from the mailing list.
    I have two separate pfSense installations, and I want to route traffic from one to a system behind the 2nd one. Our primary location has a limited number of IPs (5) and we need to host several SSL sites, so we need a unique IP for each one. I can get plenty of IPs at a remote location, and am trying to figure out some way to redirect connections from those IPs to ones at our main site.

    I can get it there just fine (over a GRE tunnel at the moment), but it doesn't look like it knows to reply back out of the remote firewall. (I tracked the packets as best I could using the packet filter). I can set up additional IPs, or even a separate subnet dedicated to this traffic at the local site if necessary. I've already tried putting it on a dedicated IP and telling it (in a firewall rule) that traffic from that IP should use the remote end of the tunnel as it's gateway. I have 3 pfSense boxes set up at the moment (4 if you count CARP failovers) but I'm a bit out of my depth at this level of networking voodoo. I'd be perfectly happy to get a support subscription if this is something that should be technically possible, I just don't have the money to spend on it and then be told it can't be done, as this is the only issue I really need assistance with at the moment.

    I'm not wedded to the idea of tunnels either, if I could get it to work, I'd be perfectly happy redirecting traffic (to port 443) from a public IP on firewall A to a public IP on firewall B.

    The frustrating thing is I've gotten both ways to work randomly (both just a simple NAT redirect from one public IP to another w/o any tunnels between firewalls, and nating over the firewall. But it's very rare, maybe 1 time in 100, and I'm at a loss for why it would work sometimes but not others.



  • I'm certainly no expert, but this looks like a job for PPP and multi-WAN and I'm pretty sure pfSense is quite capable of doing this for you.

    You might do better to move this to Routing and Multi WAN forum though, to make it a little easier for the answers to find you.



  • It's not really multi-WAN though which is why I hesitated to put it there.
    These are two different firewalls at two different physical sites. But this certainly does involve routing.



  • My knowledge of the workings of SSL is minimal. I'll discuss this solely as far as TCP and IP goes in the hope that much might be helpful.

    @palesius:

    I can get it there just fine (over a GRE tunnel at the moment), but it doesn't look like it knows to reply back out of the remote firewall.

    It is not necessary for a response to take the same path as a request UNLESS there is some sort of address change occurring. When a TCP application issues a connect request (e.g. to a web server) it expects a response from (source IP address and port) the IP address and port the request was sent to and the response needs to be directed (destination IP address and port) to the source IP address and port of the original request. If there was some address transformation (e.g. NAT) if the path between requester and responder then the response needs to go through some system which does a complementary transformation.

    Without more details I can't diagnose this.

    @palesius:

    I'm not wedded to the idea of tunnels either, if I could get it to work, I'd be perfectly happy redirecting traffic (to port 443) from a public IP on firewall A to a public IP on firewall B.

    I can't see how you could make this work. To get to firewall B over the public internet you need to redirect to an IP address that the public Internet will deliver to firewall B but you have run out of such addresses.  Consequently you need to bypass the public internet routing mechanisms by using a tunnel or a "private" network link.

    Suppose you setup a tunnel and configured firewall A to route (no NAT) a subnetwork of your "larger than needed" subnet on firewall A to firewall B. When firewall B receives a response from the routed connection request to a server what will it do (assuming the request gets that far)? The server's response will be to a public IP address (firewall A is not NATing) so firewall B will send it over the WAN interface. If firewall B is NATing between server B and WAN it will drop the response because firewall B doesn't have a record of the initial request.

    I suspect the TCP and IP part of what you are asking can be made to work provided considerable care is taken with analysing the traffic paths and configuring intermediate systems.



  • @wallabybob:

    My knowledge of the workings of SSL is minimal. I'll discuss this solely as far as TCP and IP goes in the hope that much might be helpful.

    Yes I'm hoping to make this as SSL agnostic as possible, all I want is the packets to get from point A to point B and then be able to get back to the client.

    @wallabybob:

    It is not necessary for a response to take the same path as a request UNLESS there is some sort of address change occurring. When a TCP application issues a connect request (e.g. to a web server) it expects a response from (source IP address and port) the IP address and port the request was sent to and the response needs to be directed (destination IP address and port) to the source IP address and port of the original request. If there was some address transformation (e.g. NAT) if the path between requester and responder then the response needs to go through some system which does a complementary transformation.

    Without more details I can't diagnose this.

    Well the sites would need to be reachable by users who have NAT on their end. I've tried reaching it (primarily from the browser on my phone as that was my most easily usable outside IP address that was not within one network or the other. From looking at the the packet traces it goes something like this…

    On the WAN interface of the remote site (packet capture #1)
    23:25:37.590516 IP <web client="" ip="">.2102 > <remote site="" public="" ip="">.443: tcp 0
    23:25:37.840084 IP <web client="" ip="">.33552 > <remote site="" public="" ip="">.443: tcp 0
    23:25:40.588892 IP <web client="" ip="">.2102 > <remote site="" public="" ip="">.443: tcp 0

    On the GRE interface of the remote site (this is packet capture #2 as I didn't see a way to capture traffic on more than one interface at a time)
    23:27:49.425280 IP <web client="" ip="">.4416 > <server private="" ip="">.443: tcp 0
    23:27:49.680737 IP <web client="" ip="">.21404 > <server private="" ip="">.443: tcp 0
    23:27:52.426451 IP <web client="" ip="">.4416 > <server private="" ip="">.443: tcp 0

    On the GRE interface of the local site (packet capture #1)
    19:25:41.034411 IP <web client="" ip="">.2102 > <server private="" ip="">.443: tcp 0
    19:25:41.282165 IP <web client="" ip="">.33552 > <server private="" ip="">.443: tcp 0
    19:25:44.031654 IP <web client="" ip="">.2102 > <server private="" ip="">.443: tcp 0

    On the LAN interface of the local site (this is packet capture #2 as I didn't see a way to capture traffic on more than one interface at a time)
    19:27:52.864112 IP <web client="" ip="">.4416 > <server private="" ip="">.443: tcp 0
    19:27:52.864280 IP <server private="" ip="">.443 > <web client="" ip="">.4416: tcp 0
    19:27:53.118124 IP <web client="" ip="">.21404 > <server private="" ip="">.443: tcp 0
    19:27:53.118221 IP <server private="" ip="">.443 > <web client="" ip="">.21404: tcp 0
    19:27:55.288087 IP <server private="" ip="">.443 > <web client="" ip="">.4416: tcp 0
    19:27:55.725565 IP <server private="" ip="">.443 > <web client="" ip="">.21404: tcp 0
    19:27:55.864395 IP <web client="" ip="">.4416 > <server private="" ip="">.443: tcp 0
    19:27:56.118084 IP <web client="" ip="">.21404 > <server private="" ip="">.443: tcp 0

    It seems to end here though, the lines going from the internal IP of the server back out to the web browser don't show up on the wan interface or the gre interface, the trail seems to end here.

    @wallabybob:

    I can't see how you could make this work. To get to firewall B over the public internet you need to redirect to an IP address that the public Internet will deliver to firewall B but you have run out of such addresses.  Consequently you need to bypass the public internet routing mechanisms by using a tunnel or a "private" network link.

    Suppose you setup a tunnel and configured firewall A to route (no NAT) a subnetwork of your "larger than needed" subnet on firewall A to firewall B. When firewall B receives a response from the routed connection request to a server what will it do (assuming the request gets that far)? The server's response will be to a public IP address (firewall A is not NATing) so firewall B will send it over the WAN interface. If firewall B is NATing between server B and WAN it will drop the response because firewall B doesn't have a record of the initial request.

    I suspect the TCP and IP part of what you are asking can be made to work provided considerable care is taken with analysing the traffic paths and configuring intermediate systems.

    I have one IP address to spare for this use on the public side of firewall B. My thinking in this regard was to do port translation. e.g. IP1 at the remote site would map port 443 to IP_A at the local site, but on port 8001, IP2 at the remote site would map port 443 to IP_A (same IP as IP_1 used) but at port 8002, etc…

    The thing is right now at Firewall A I have a /29, so it's not really feasible to split off a true subnet, I was hoping I could do something on an IP by IP basis. Is there a way to have it route that traffic to systems on the other side of the tunnel without NATing? What interface would I set those routes up on, WAN or GRE (No LAN interface on FW A, just WAN and GRE, and would I be routing to the public or private addresses at FW B? I fear I don't have a good enough grasp of routing to figure out what I am doing wrong here.

    BTW, thank you, this has been the most helpful reply I've received to this issue so far.</server></web></server></web></web></server></web></server></web></server></server></web></web></server></server></web></server></web></server></web></server></web></server></web></server></web></server></web></remote></web></remote></web></remote></web>



  • Sorry, I thought of this after my earlier reply.

    @palesius:

    The frustrating thing is I've gotten both ways to work randomly (both just a simple NAT redirect from one public IP to another w/o any tunnels between firewalls, and nating over the firewall. But it's very rare, maybe 1 time in 100, and I'm at a loss for why it would work sometimes but not others.

    This sounds like NAT reflection which the pfSense book describes as "a bit of a hack".

    Thanks for the additional information. I'll have to respond at a later time.



  • If the remote site that has the public IPs is set up so you route the public IPs instead of directly assigning them to WAN, you could do this with purely routing and no NAT.  The remote site's router would need a static route for the public IPs with the gateway IP being the tunnel endpoint of the local router and the local router could have an interface that directly uses the subnet for those public IPs.  The local router would likely need a firewall rule to force traffic sourced from the public IPs to use the remote site tunnel endpoint as the gateway.  The tunnel endpoints themselves could use private IPs, no public ones there would be necessary.



  • @Efonne:

    If the remote site that has the public IPs is set up so you route the public IPs instead of directly assigning them to WAN, you could do this with purely routing and no NAT.  The remote site's router would need a static route for the public IPs with the gateway IP being the tunnel endpoint of the local router and the local router could have an interface that directly uses the subnet for those public IPs.  The local router would likely need a firewall rule to force traffic sourced from the public IPs to use the remote site tunnel endpoint as the gateway.  The tunnel endpoints themselves could use private IPs, no public ones there would be necessary.

    Okay, I think I understand what you are suggesting. I though the static routes were more for telling traffic where to go, not controlling inbound traffic.

    So on the remote router I would do something like this (assuming my tunnel endpoints are 10.100.6.2 (remote) and 10.100.6.1 (local)
    Dest Network: Public IP2 (Assuming Public IP1 is reserved for the Wan interface itself)
    GW: 10.100.6.1
    Then repeat that for IP3 through IP_n.

    On the local router I would set up an interface with those same IPs.
    A few questions: what kind of interface should I be using for this, VLAN? Something else?
    And would I set it up as Public IP1 with a size of /29(assuming that is the network size at the remote site)? If so How does it know that Public IP1 does not need to be routed over the tunnel (since IP1 is the target IP for the tunnel itself)? Can I simply do that with a static route or does it need to not be part of the subnet, and if that is the case do I need to do this with a subnet that is smaller than the subnet at the remote site?

    Finally, how would I know that traffic was sourced from those public IPs (to know to redirect it out there)? Would I just need to make sure that it went to specific IPs on my local network (e.g. 10.100.10.200-10.100.10.220 would only be get traffic coming from the remote site so I could create a rule on the LAN to use 10.100.6.2 as the gateway for those IPs.


Locked