My experience with IPSEC and SMB
Hi, I read a lot about the subject with no definitive solutions, so I'm just adding my experience hoping it may be useful for someone.
I have two sites, on each one I have an AMD APU and 60/100 mbit bandwidth.
I'm using 2.4.4 with AES-NI accel enabled.
I setup an IPSEC using IKEv2 and AES128-GCM. It worked like a charm. Ping between hosts in the LANs was about 20-30 ms. RDP worked great but the first time I tried to copy a file from a server in one site to a client in the other, using Microsoft shares, it was a complete pain. I had bandwidth in the kilobits range and copying files took forever. If I used RDP to copy the file from the same server, I went full speed. So as I read on several forums I thought it was due to SMB having problems with latency.
The problem is that if I used an OpenVPN connection from a laptop in the first site, to the very same pfSense on the other end, and copied the same file from the same server, it went full speed and the latency was again in the same 20-30 ms range. So with same server and client, same latency, IPSEC was not working well, OpenVPN was. The ping test lead me to 1438 bytes MTU was set on OpenVPN, and I configured the same in the IPSEC settings on both pfSense. To make it short I tried everything from 900 to 1438 bytes with no success. For the sake of testing I used several cipher algos and enabled/disabled acceleration with no changes.
Then I tried to use IKEv1 and bum! It started working perfectly with full bandwidth usage even for SMB. I reverted to IKEv2 and again it was not working, then again to IKEv1 and it was ok.
What does this mean? I don’t really know, just wanted to share the experience and maybe help some other guy that is struggling with SMB over IPSEC.