Using 2 public addresses to hide a single internal IP and get replied from the correct one

[AdrianX]

Hi,

I have two public IP addresses (let's say 85.1.2.3 and 85.1.2.4) on which I want to do port forwarding for port 1111 to internal IP 192.168.1.101:1111.

The idea is that when a client goes to 85.1.2.3:1111, traffic goes to 192.168.1.101:1111 and when the reply is sent to the client, the client sees the incoming traffic as coming from 85.1.2.3. This part is easy and I have it working with a simple port forward.

However the part I'm having issues is trying to add port forwarding also for the second public ip: 85.1.2.4, same port and to the SAME internal IP. The traffic gets to the internal machine 192.168.1.101 BUT I never get a reply from it.

How can I solve this, please?

Thanks.

[viragomann]

@adrianx said in Using 2 public addresses to hide a single internal IP and get replied from the correct one:

The traffic gets to the internal machine 192.168.1.101 BUT I never get a reply from it.

Did you check that with Packet Capture on pfSense?

[AdrianX]

@viragomann I see it going out of the internal machine but I don't receive it on the client.

[AdrianX]

Ok fixed, just had to add a 1:1 NAT

[AdrianX]

But now, how I make it work with TWO public IP addresses? It works with one but if I add additional public IPs in 1:1 NAT rules then only the first one will work.

[viragomann]

@adrianx
You cannot use 1:1 NAT with two public addresses and a single internal. 1:1 means 1 public to 1 internal.

Use port forwaring instead.

Did you already add the second public IP to the WAN interface?

[AdrianX]

@viragomann Yes, I have the 2 public Ips added, let's say:

85.1.1.2
85.1.2.3

I do port forwarding from :7777 to :7777 to a machine in the network that has a load balancer. The load balancer then sends the request transparently (keeping source IP + source port) to a set of servers that reply directly to the client.

So this works great and I get the reply correctly for 1 IP, but not for 2 IPs. The problem is that for the backend servers to be able to reply directly to the client, I added a 1:1 NAT of public IP 85.1.1.2 to each of the backend servers. And hence answers from that public IP work.

But now in order to be able to reply when asking the second IP 85.1.1.3, I added another 1:1 NAT to each of the backend servers with the second IP, so I have two 1:1 NATs per backend servers (the ones replying to the client directly). But that doesn't work, only first 1:1 NAT mapping in the list works.

How should I do it so that the reply gets translated to the original public IP that was used?

Thanks.

[viragomann]

@adrianx said in Using 2 public addresses to hide a single internal IP and get replied from the correct one:

The load balancer then sends the request transparently (keeping source IP + source port) to a set of servers that reply directly to the client.

You mean, replies don't pass the load balancer?
If so, pfSense won't have states for these replies.

How should I do it so that the reply gets translated to the original public IP that was used?

That is the default behavior. If you access IP1 from outside and pfSense forward the packets to an internal device, it transaltes the source IP in replies back to IP1 when the packets leave the WAN interface.
However, this is controlled by the state table.

[AdrianX]

@viragomann Yes exactly, replies don't pass the load balancer, it's a Direct Server Reply (DSR).

It works with first public IP as I have the 1:1 NAT mappings in the servers replying to the client. And I have 4 servers that reply.

To make it work with the second IP, should I use a different set of servers to reply from the second public IP and do 1:1 NAT mappings on those? Consequently I would need a second load balancer for the second public IP port forwarding... right?

Would that be the correct way?

[viragomann]

@adrianx said in Using 2 public addresses to hide a single internal IP and get replied from the correct one:

The problem is that for the backend servers to be able to reply directly to the client, I added a 1:1 NAT of public IP 85.1.1.2 to each of the backend servers.

You added a 1:1 NAT rule to the backends themeself or on pfSense?

[AdrianX]

@viragomann I mean on pfSense, see:

🔒 Log in to view

One for each backend. This is in Firewall / NAT / 1:1. Public IP is the 198.50. The 192.168.1.213 is the server were the traffic got forwarded from the load balancer.

[johnpoz]

Ok Im a bit confused here.. Lets forget 2 public IPs for a minute.. Either I need more coffee, or I am missing something

If you send traffic from say 1.2.3.4 hitting your wan IP to a load bal 192.168.1.213, and this sends on the traffic to what? Say 192.168.1.113

If .113 responds back directly to pfsense saying I want to go to 1.2.3.4 with a SA.. How would that work? Pfsense should not allow that traffic, because there is no state..

edit: your setting up 1:1 nat on pfsense to your bankend IPs, not the load balancer? Yeah I need more coffee ;)

[viragomann]

@adrianx
So as I already mentioned, you cannot use 1:1 for that, since you have a single internal IP. There is also no need for 1:1.

I think, it should work, but instead of the 1:1 NAT rules, add port forwarding rule.
So you can add forward rule for 85.1.1.2 to 192.168.1 and a second forwarding 85.1.2.3 to 192.168.1.

The response from the backend is automatically retranslated into its origin destionation address, as already mentioned.

[viragomann]

@johnpoz
https://www.haproxy.com/blog/layer-4-load-balancing-direct-server-return-mode/
I'm not familiar with that as well. But I think it should be able.

[johnpoz]

That is ha proxy.. Did he mention he is running this through ha proxy? He is using that as a backend load bal, or on pfsense. If on pfsense why would he be setting up any port forwards or nats? Those are not used when you have ha proxy listening on wan and sending traffic.

Yeah I need more coffee ;)

[AdrianX]

@johnpoz @viragomann

Ok so here is the full picture, first a port forward from public IP port 7777 to load balancer (NGINX UDP Load Balancer, transparent mode = keep source IP + source port), see:

🔒 Log in to view

This is in Firewall / Nat / Port forward. The 192.168.1.211 is the NGINX load balancer. Then the load balancer forwards the traffic to one out of 4 backend servers, let's say that we only have 1 to simplify it, and that one is 192.168.1.213.

Then backend 192.168.1.213 gets the traffic as if it was coming directly from the client given the transparent mode from NGINX, and then replies to it, taking profit of this 1:1 NAT to translate it's IP to the public IP:

🔒 Log in to view

Makes sense? Let me know please. This works at the moment.

The problem is when using 2 public IPs.

[AdrianX]

@viragomann If I do only the port forwarding to the Load Balancer without the 1:1 to the backends, it doesn't work, and I don't get any replies from the backend servers (and they send the traffic, I checked). Neither with 1 nor 2 public IPs. But I may be missing something?

[johnpoz]

Huh.. Not sure how that would work.

Seems more like your 1:1 nat is just sending traffic to 213.. and 211 isn't getting anything?

I don't see how pfsense would allow traffic from 213, if there is no state.. If it sent traffic to 211, why would it allow return traffic from 213..

Can you show use the state table for the IPs in question.

This UDP traffic?

[AdrianX]

@johnpoz Huh I just checked and you are right, only the first packet goes to the load balancer, and the following ones go to the backend directly..... that's not what I wanted.

And yes it's UDP traffic.

Do you know how I could achieve this?

[AdrianX]

If I remove the 1:1 on the backend, everything goes into the Load balancer (correct), but the backend reply doesn't arrive to me (client).