Using 2 public addresses to hide a single internal IP and get replied from the correct one

AdrianX · Jan 13, 2021, 4:10 PM

@viragomann So with packet capture on the LAN, I see that the backend replies this:

15:32:57.557414 IP 192.168.1.213.7777 > Client.Public.IP.60428: UDP, length 15

So not using the Virtual IP. Is there a way to make it use the public IP?

viragomann · Jan 13, 2021, 4:14 PM

@adrianx
So DSR is not configured correctly on the servers.

From the linked site above:

the service VIP must be configured on a loopback interface on each backend and must not answer to ARP requests

johnpoz · Jan 13, 2021, 4:21 PM

^ exactly.

But I still don't see how that really solves a state exhaustion issue.. No matter how many IP you send to behind pfsense.. Pfsense is natting to its public IP, which has a limit of how many states it can have.

The way to solve state exhaustion issue would be to filter the traffic that is "bad" before a state is created..

viragomann · Jan 13, 2021, 4:27 PM

I'd suppose, if the backend servers are configured correctly for DSR (responding using the VIP and not responding to ARP requests) the states will be fine.
However, I've never set up something like that.

johnpoz · Jan 13, 2021, 4:42 PM

so I will get around 50000 requests from different IPs per second

He still have his public IP with states.. Doesn't matter how many IPs he sends to behind.. While sure the local boxes would have less states.. His public IP would still have the states.. at 50k a second that is going to burn through states like crazy..

I don't really see how doing something like this could solve a state exhaustion issue to be honest..

Well lets not really call them states if they are UDP... But pf tracks them like they were.. You can set an option in pf for how long these are tracked..

But yeah I believe using the VIP on these end boxes for the IP of the load balancer is how such a setup is to be done. To solve the asymmetrical flow problem.. Since pfsense will only send traffic to what it thinks is 1 IP.. And the return traffic to pfsense will be coming from that same IP. As far as pfsense knows, since its source would be the vip address.

viragomann · Jan 13, 2021, 4:59 PM

@johnpoz said in Using 2 public addresses to hide a single internal IP and get replied from the correct one:

so I will get around 50000 requests from different IPs per second

Yes, you're absolutly right. It didn't realize, that he is really having such high load and may exhausting the state table.
However, with enough memory and cpu power, increasing the state table size and shortening the state timeouts it may be doable.

AdrianX · Jan 13, 2021, 5:17 PM

@viragomann @johnpoz

Regarding the "states" for the public IP, I can modify the associated rule with the port forwarding and choose "State type" as "none", and that would solve it, no?

johnpoz · Jan 13, 2021, 5:25 PM

Hmmm?

https://docs.netgate.com/pfsense/en/latest/firewall/configure.html
login-to-view

AdrianX · Jan 13, 2021, 5:29 PM

@johnpoz Yes, that, plus also not keeping it in the outbound, no?. And how to configure UDP state expiration?

johnpoz · Jan 13, 2021, 5:34 PM

Well if you don't keep any states for the rule - it shoudn't matter.. But in the advanced section of the rule you can set the timeout option for states.. Also would need to be done on a outbound rule that matches.

I am not 100% sure if that would also pertain to what pf does for udp tracking - I would assume so..

AdrianX · Jan 13, 2021, 5:38 PM

@johnpoz So the Floating rule for the Outbound, should be something like:

login-to-view

Direction: In. State type: none

So traffic going into the LAN from the internal load balancer, intended to get towards the wan to the outside world... no? Or I'm getting it wrong?

AdrianX · Jan 13, 2021, 6:23 PM

@johnpoz Also, any idea why it still creates states even if I have the port forward rule set to "state type: none"? See:

login-to-view

(DELETED TO AVOID SHOWING PUBLIC IP)

johnpoz · Jan 13, 2021, 6:23 PM

I just assumed that none would do that - but with udp they are not actually states.. They are just tracking. So its possible those settings do not apply??

https://www.openbsd.org/faq/pf/filter.html
Keeping State for UDP
One will sometimes hear it said that "one cannot create state with UDP, as UDP is a stateless protocol!" While it is true that a UDP communication session does not have any concept of state (an explicit start and stop of communications), this does not have any impact on PF's ability to create state for a UDP session. In the case of protocols without "start" and "end" packets, PF simply keeps track of how long it has been since a matching packet has gone through. If the timeout is reached, the state is cleared. The timeout values can be set in the options section of the pf.conf file.

edit: Looks like your public IP is exposed in your state table. Are you ok with that? If not I would edit your image to obfuscate your public IP. If you need help with that just, just ask.

Are those maybe old, from before you set the none in the rule?

AdrianX · Jan 13, 2021, 6:24 PM

@johnpoz I just deleted the image.

So I did a Reset states several times, and states are created no matter what. I don't understand how to avoid that.

johnpoz · Jan 13, 2021, 7:14 PM

Not sure if possible with udp.. And have never tried it with tcp either.. It is listed as an option, but not sure on the details of that option.

We can call in maybe @Derelict he would have better understanding here of these options. I would think ;)