Route a Single Local IP Upstream?

calebh

I know this goes against IP routing basics, but I'm asking the question anyway in hopes of simplifying a testing project.

We have a functional network with production machines running on it. We need to make an exact copy of a virtual machine on that network, and put it in a nearly-duplicate environment with access to an existing production VM on the production network. One thought was to create a virtual pfSense firewall (on one of the VM hosts) on a different, but local, subnet, and put the VM copy behind it, so it can access the 'parent' network via NAT-ing. The 'issue' is when it tries to access the existing production VM, IP routing basics says that it doesn't need to go through the gateway because it's a local address. Would it be possible to have the virtual pfSense firewall take any traffic destined for the production VM, forward it upstream using NAT (so the upstream pfSense firewall doesn't think it's coming from the production version of the copied VM), and the upstream pfSense firewall route the traffic to the existing production VM as if it came from a different local subnet (since the virtual pfSense firewall would be on a different local subnet)?

I'm not sure how well that's explained, so here's a visual representation…
Current production routing:

+------ Production Subnet (A) --------------------+-- Production Subnet (B) --+
|                                                 |                           |
| Production Host (Y) < --- > Production Host (Z) |                           |
|                                                 |                           |
+-------------------------------------------------+---------------------------+

Desired routing for this testing project:

+------ Production Subnet (A) ----------------------+-- Production Subnet (B) --+
|                                                   |                           |
|                                                   |   +-- Test Subnet (A) --+ |
|                                                   |   |                     | |
| Production Host (Y) < --- > Production Host (Z) <-----------> Test Host (Y) | |
|                                                   |   |                     | |
|                                                   |   +---------------------+ |
|                                                   |                           |
+---------------------------------------------------+---------------------------+

… Where matching letters mean identical configurations.

Given this is possible, my first thought is to use a virtual IP on the LAN interface on the firewall for Test Subnet (A), but I'm not entirely sure what would need to be done next.

Ideas?

Note: we're hoping this is possible instead of creating a copy of Production Host (Z), because it's a database server and it's size is on the scale of TB. So...

EDIT: Solution in this post: https://forum.pfsense.org/index.php?topic=146783.msg797884#msg797884

Derelict

Yeah without network addresses, subnets, gateways, etc there's not a lot to go on there.

calebh

@Derelict:

Yeah without network addresses, subnets, gateways, etc there's not a lot to go on there.

Here's example network values for the placeholders in my previous post:

Production Subnet (A)
192.168.2.0/24
FW LAN Address: 192.168.2.1

Production Subnet (B)
192.168.3.0/24
FW LAN Address: 192.168.3.1

Test Subnet (A)
192.168.2.0/24
FW LAN Address: 192.168.2.1
FW WAN Address: 192.168.3.2

Production Host (Y)
192.168.2.2
Gateway: 192.168.2.1

Production Host (Z)
192.168.2.3
Gateway: 192.168.2.1

Test Host (Y)
192.168.2.2
Gateway: 192.168.2.1

Production Subnet (A) and Production Subnet (B) are on the same pfSense firewall. The goal is to allow Test Host (Y) to think that it's located on Production Subnet (A) and communication with Production Host (Z), but not allow it to get to anything but Production Host (Z) on Production Subnet (A).

Derelict

No because the "Test Host" machine thinks "Host Z" is on the local network so it doesn't send the traffic to the firewall at all. It thinks it is same-subnet so it just ARPs for it.

calebh

@Derelict:

No because the "Test Host" machine thinks "Host Z" is on the local network so it doesn't send the traffic to the firewall at all. It thinks it is same-subnet so it just ARPs for it.

Wouldn't giving the Test Subnet's firewall a virtual IP on the LAN interface overcome that?

Derelict

No. When Host Z has reply traffic for Test Host what happens?

And you can't have two networks on the same firewall with the same numbering.

calebh

@Derelict:

And you can't have two networks on the same firewall with the same numbering.

The Test Subnet firewall would not be connected to the two networks with the same numbering (Production Subnet A, and Test Subnet A). It would see the 'WAN' network with the "B" numbering, and the 'LAN' network/interface with the "A" numbering.

@Derelict:

No. When Host Z has reply traffic for Test Host what happens?

Given the network connectivity is as I just mentioned, I was hoping it's possible to set a static route on the Test Subnet firewall for Host Z's address that points to its upstream gateway (which happens to be the firewall that is connected to Production Subnet A. I was also hoping it would be possible for it to NAT the traffic before it sends it upstream, so that Host Z would send it back to the 'WAN' address of the Test Subnet firewall in Subnet B.

Derelict

Then it is not an IDENTICAL COPY of the host as you stated.

Even if it is just an alias (secondary address), it is still an interface with that network that matches the other interface.

You would probably need another pfSense/router that could have the identical subnet behind it and perform outbound NAT on its "WAN" to have a prayer of making this work.

calebh

@Derelict:

Then it is not an IDENTICAL COPY of the host as you stated.

Even if it is just an alias (secondary address), it is still an interface with that network that matches the other interface.

I apologize for the confusion. The hosts (production and test) are, in fact, identical. It's the two firewalls that are slightly different (see my second message about "network addresses, subnets, gateways, etc"). The alias would be on the interface connected to the subnet that the alias address is part of, thus allowing the Test Subnet firewall to accept traffic on its LAN interface destined for Host Z.

I've updated the wording on the other message to reflect these points a little clearer.

@Derelict:

You would probably need another pfSense/router that could have the identical subnet behind it and perform outbound NAT on its "WAN" to have a prayer of making this work.

I would expect that the firewall inside Production Subnet B would be the one for this purpose. I did not think of Outbound NAT settings, so that's the direction I'll work with, unless there are any other (simpler) recommendations before I finish trying that. Thanks for pointing that out!

calebh

For the reader's sake, this is what I ended up doing…

On the Production Subnets' firewall:

Add an "IP Alias" type Virtual IP on the Subnet B interface, setting the IP address set to an unused IP on that subnet
Add a port forward on the Subnet B interface (NOT the WAN interface) from the VIP to the Host Z IP

On the Test Subnet firewall:

Add an "IP Alias" type Virtual IP on the LAN interface, setting the IP address equal to that of Host Z
Add a port forward on the LAN interface (again, NOT the WAN interface) from the Host Z VIP to the upstream firewall's VIP on Subnet B

Note that this works only for protocols that are supported by pfSense's NAT-ing capabilities. Since ours was a database connection (TCP 1433), this works just fine.