Slow NAT

albgen

https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html

Regarding the configs, find it here config-pfSenseEdgeRouter.localdomain-20230623180532.xml

LAN side has several machines, Windows and Linux servers.
The API is running on machine 1. There is a NAT(port forward) from wan to lan for the port of the API(7068).

Tests are done using postman.
From the same machine 1, if i issue a request for the api, it is super fast(40ms). From machine 2 which is on the same network as machine 1, i issue the same request, result is super fast.

From any laptop/computer which requests comes from the wan, more than 3 seconds of reply.

I see on the states of the firewall 2 connections from the machine outside wan. They are not closed and they are opened

The response is 0.5MBytes. So you see 2.1MB of traffic because i tried 4 times, to call the same endpoint.

SteveITS

@albgen
Machine1's gateway is the pfSense LAN IP?

Is Machine2 connecting to Machine1 through pfSense's WAN IP via NAT reflection? Or direct to Machine1 across the network?

If it's using reflection and is fast, yet connecting from WAN is slow, then I am not sure what would be going on, I don't recall running into that.

albgen

@SteveITS Correct, Machine 1 gateway is the pfSense LAN IP.
Machine 2 also has gateway the pfSense LAN IP and is on the same interface, so LAN1

viragomann

@albgen
Ensure that you've "Log packets matched from the default block rules in the ruleset" enabled and check if there are blocks in the log from machine 1.

Also there is obviously a router in front of your pfSense. Connect a computer directly to the WAN interface of pfSense and test the access via NAT from it.

albgen

@viragomann I think it is not even possible to enable the "Log packets on the default block rules in the ruleset" .
There is no router in front. It is just a masquerade of the nic: The topology is
nic with public ip->dnat(iptables -t nat -A PREROUTING --dst 65.XXX.XX.XX -j DNAT --to-destination 10.21.21.120)->wan pfsense.

viragomann

@albgen said in Slow NAT:

@viragomann I think it is not even possible to enable the "Log packets on the default block rules in the ruleset" .

Status > System Logs > Settings > Log packets matched from the default block rules in the ruleset

There is no router in front. It is just a masquerade of the nic: The topology is
nic with public ip->dnat(iptables -t nat -A PREROUTING --dst 65.XXX.XX.XX -j DNAT --to-destination 10.21.21.120)->wan pfsense.

Yeah, this is exactly what a NAT router does.

What is the sense of forwarding the whole traffic instead of having the public IP directly on pfSense?

Presumably your pfSense is virtualized. Which hypervisor? Is it installed accordingly?
Do you have other connections through pfSense, which have a low latency?

albgen

@viragomann

those logs are already enabled.
Of course it is virtualised. Check the first post

albgen

I just added a NAT on WAN2 which has a Public IP setup and yet the same slowness.

johnpoz

@albgen so you think pfsense natting the traffic is adding like 3.5 seconds to your response time?

Well easy enough to check if pfsense is doing that.. Do a sniff on your wan and on your lan at same time and send some traffic what is the delay caused by pfsense sending it on..

So example

Here is me coming from can you see me to port 23040 which is forwarded to my plex on 32400 on 192.168.9.10

So sniffing on my wan and my lan at same time with tcpdump I see when it hits my wan and when it gets sent on, see the Syn and the Syn,Acks

Lets do the math..

So syn hits my wan at 48.108772, and that is sent on to 192.168.9.100 at 48.108886 so a delay of .000114 seconds, or 0.114 ms

Now the syn,ack hits my lan at 48.109230, and sent out my wan at 48.109270 for a delay caused by pfsense of .000040 seconds..

So lets see the delay your pfsense creating - 3.5 seconds seems highly unlikely to me..

albgen

@johnpoz thanks for the tip and i did the same test.
Window on top is WAN and on the bottom is LAN. I just captured 10 packets from each interface and seems it is pretty fast so the culprit is not the NAT.

Found though two solutions but still not why it is happening.

Remove Accept-Encoding header from the http request - result is very fast.
Using a reverse proxy with https is still fast with and without the Accept-Encoding header