IPSect Site to Site (Slow Upload) - (Fast Download) issue

[AMD_infinium05]

Hello Guys,

I need help troubleshooting this problem.

• Site to Site is established
• IPSec Firewall rules are set to "Any to Any" on both ends
• Both links have 100Mbps Up/Down speed
• Both hardware has plenty of resources

SiteA
Pfsense 2.4.5 Dev
Subnet 192.168.20.0/24

SiteB
Pfsense 2.4.4 Stable
Subnet 192.168.45.0/24
Site A to Site B uploading to Qnap via Samba/CIFS the upload can only max out at 2.5mbits/sec

Test Scenarios:

1. Site A to Site B download from Qnap via Samba/CIFS the download runs betweent 70-90Mbit/sec

2. Site A to Site B remote desktop protocol to a computer in Site B file transfer via RDP is 70-90mbits/sec

3. Site B to Site A remote desktop protocol to a computer in Site A files transfers via RDP is 70-90mbits/sec

The only problem I am encountering is any computer from SiteB uploading data to Qnap Device in Site A by Samba/CIFS is very slow. Other than that everything is working normal.

I have scoured both end's pfsense for any traffic shapers, there is none. I have not configured any shapers since the deployments. I have been looking at this for over 48 hrs already and couldn't figure out whats wrong.

[P3R]

Why do you suspect pfSense and not the Qnap?

[AMD_infinium05]

Because Qnap works just fine on SiteA.
Do you have anything in mind where else to look at?

[Derelict]

@amd_infinium05 said in IPSect Site to Site (Slow Upload) - (Fast Download) issue:

The only problem I am encountering is any computer from SiteB uploading data to Qnap Device in Site A by Samba/CIFS is very slow. Other than that everything is working normal.

How are you testing?

What is the latency between the sites?

SMB is probably the slowest, chattiest, crappiest protocol you can use for WAN transfers.

[P3R]

@derelict said in IPSect Site to Site (Slow Upload) - (Fast Download) issue:

SMB is probably the slowest, chattiest, crappiest protocol you can use for WAN transfers.

The advantage with CIFS/SMB is that it's very convenient and user friendly on Windows and it's the recommended protocol on MacOS. I've been running it over an IPsec VPN to different NASes for many years now. I just tried copying a 1 TB file to and from a very (8 year) old Qnap over a symmetric 100 Mb/s connection. When writing to the NAS it had one very short drop down to 9.89 MB/s but was otherwise steadily above 10 MB/s. When reading from the Qnap it's rock steady at 10.6 MB/s with extremely small variations. In my opinion that's not a terrible protocol overhead.

I see no reason to use anything else regularly but for troubleshooting this issue it may be interesting to try something else but let's hear the answer to your questions first.

[Derelict]

Not much in the firewall could cause what he's seeing. if SMB works fine in one direction it should work as well in the other.

[AMD_infinium05]

It is not just SMB, even Https/Http browsing from SiteB to Site seems slow.
I have also tested with iPerf 3.1.3

Site B to Site A - speed maxes out at 2-3Mbits/sec
Site A to Sibe B - speed maxes out the links which is 75-95Mbits/sec

Commands used:
server: iperf.exe --server
client: iperf.exe --c <serverIP> -t 60

[AMD_infinium05]

@derelict
Hello!

The latency between sites in the tunnel is 3-6ms.

[Konstanti]

@amd_infinium05

Hey

1. if you test iperf with udp - will the speed change ?
2. what network cards are used ?

[AMD_infinium05]

@konstanti I still have to test.

all nodes uses Intel based NIC cards, workstations, pfsense boxes.
Afaik all Intel Gbe.

[Derelict]

@amd_infinium05 said in IPSect Site to Site (Slow Upload) - (Fast Download) issue:

It is not just SMB, even Https/Http browsing from SiteB to Site seems slow.
I have also tested with iPerf 3.1.3

Site B to Site A - speed maxes out at 2-3Mbits/sec
Site A to Sibe B - speed maxes out the links which is 75-95Mbits/sec

Commands used:
server: iperf.exe --server
client: iperf.exe --c <serverIP> -t 60

That conflicts with what you said here:

Site A to Site B remote desktop protocol to a computer in Site B file transfer via RDP is 70-90mbits/sec

If it is just sending to the Q-NAP that is problematic then that points to something in the Q-NAP.

Perhaps a limiter on transfers in from non-local subnets?

There really isn't anything in the firewall that would cause this unless a limiter/shaper was deliberately placed.

I assume both sites can run traditional speed tests to internet test sites at or near 100/100.

[AMD_infinium05]

@derelict I am having the same issue with iperf when Im initiating the connection from site B to any windows server on site A through the tunnel.

Im pretty sure QNAP has no problem as the workstations/servers on site A (same subnet as with qnap) can send data to qnap reaching 1gbps.

[Derelict]

@amd_infinium05 Right. Often connections from other subnets are trated differently.

I really cannot think of anything in the firewall that would cause what you are seeing unless you deliberately set a limiter. There is no checkbox to enable the issue you are seeing.

Packet capture an iperf session and see if there are retransmissions or something.

Set MSS Clamping in the advanced IPsec settings down to, say, 1300 and try again.

Try UDP iperf.

[AMD_infinium05]

@derelict thank you.

I will try udp on iperf when I get my hands on it.

I have observed that any traffic that is initiated from site a to site b gets full speed up and down through the tunnel regardless of what type of task I throw at it (rdp/samba/iperf).

Connections initiated from site B (iperf and file transfer via cifs/smb to qnap it is slow). This is really weird from my point of view.

Also I have observed that if the connection is initiated from site B, it is actually hitting the ipsec firewall rule on site A. If the connection is initiated from site A it is hitting the ipsec firewall rule in Site B. --- this is normal yes?

[Derelict]

Yes. The firewall rules on IPsec are the same as any other interface. They govern connections coming INTO that firewall on that interface.

[AMD_infinium05]

So here are my iperf tests

Site B to Site A (left window is Site B, right window is Site A)

Site B to Site A

[Derelict]

Still doesn't point at anything on the firewalls themselves.

(You have to specify a -b bandwidth flag when using UDP or it tries to send 1Mbit/sec as you saw)

[bbrendon]

How do you know its not the ISP? I swear I've seen Comcast Residential throttle all kinds of things.

[AMD_infinium05]

@bbrendon I dont know sir. I do not know where else to look at.

[AMD_infinium05]

Here are my speedtest using UDP from SiteB to SiteA

They are showing two different information.

Left: Site B (client)
Right: Site A (Server)

[AMD_infinium05]

RESOLVED!!
I have set both ends to MSS Clamping 1300 and that solved the issue.
I can now upload data to Qnap at full speed 80-90Mbps.

Wrap up thoughts?

[P3R]

Wouldn't it be better to fix what's preventing MTU discovery to work properly (your ICMP filtering perhaps)?

I've never needed MSS Clamping.

[AMD_infinium05]

@p3r ICMP filtering?

[P3R]

As far as I know MSS Clamping is a workaround to avoid MTU discovery problems. I assumed that you have some filtering in the source-destination path (ICMP was my first thought) that prevent MTU discovery.

Since throughtput was assymetric, I expected it to be fairly easy to find what was different and causing the issue at one end.