New shiny hardware C2758 and poor OpenVPN performance. Don't know what to do.
-
Hi!
I've replaced an Atom D525 (pf 2.1.4) based 1U platform with a Supermicro A1SRi-2758F, Atom C2758, (pf 2.2.4) with big hopes to give the OpenVPN performance a boost.
But the boost didn't come through as I'd hoped. It much better than before. But my research on the C2758 did give me indications of a possibility to closer or above 100mbit/s.What I understand; Intel Quickassist and hardware support for encryption isn't implemented yet. So I'm unsure if hardware acceleration is even something to alter here. I've tried every combination under System settings and OpenVPN settings.
Encryption used doesn't make much difference. Used AES-128-CVC and BF-CBC.Both ends Internet connections are 1gbit/s and measures >800mbit from various speed-test-sites.
I copied (Windows, SMB) a large 7z file across to get a real world test of transfer.
Site-to-Site
I've got Site-to-Site which gives me up/down around 60-70mbit/s. I can push uploads to around 100-110mbit with tun-mtu 1500 and mssfix 1400 tweaks (but this i my side CPU working (multicore Xeon in Hyper-V), so I'm leaving that out)Should I be OK with the encrypted 60-70mbit/s?
Client connections
Here I get a 100-120mbit/s down and 50-70mbit/s (X220 i7).Is this what I can expect in speed from the platform?
Probably I've missed some crucial information, don't scold me! :) Ask, and I'll answer.
iperf? iperf -c y.y.y.y -P 4?Brgs,
-
Try downgrading PFsense to 2.1.5
Check this posts
https://forum.pfsense.org/index.php?topic=99536.0
https://forum.pfsense.org/index.php?topic=88758.0
-
did ya enable powerD ? set it to some different value's and reboot, see what works.
==>System: Advanced: Miscellaneous
-
"I copied (Windows, SMB) a large 7z file across to get a real world test of transfer."
And yeah that protocol is chatty as all F_ck!!! and just blows across a wan with any latency at all.. What is your RTT between these sites? This prob has nothing to do with your openvpn performance and all about 1 stream over wan with protocol that blows over a wan.
This is nothing new this has been a known issues since MS and smb and wan links.. What version are you using 2, 2.1, 3.1.1? What is your client what is your server.. How did you do the copy with robocopy, just explorer? Some application. This is dated, but goes over some things you can do to optimize smb in your applications over a long fat pipe. if site to site you might look into leveraging branch caching, etc.
http://download.microsoft.com/download/f/2/1/f2146213-4ac0-4c50-b69a-12428ff0b077/Optimizing_Applications_for_Remote_File_Access_Over_WAN.pptx
Here is even older blog article about smb and wan sucking
http://blogs.technet.com/b/neilcar/archive/2004/10/26/247903.aspxA partial solution for enterprises is deployment of wan optimization like riverbeds that help with such protocols and can do some caching, etc etc.. But if your stuck with smb over a wan that has any latency at all does not matter how FAT the pipe is.. Its prob more the smb protocol that anything else - do a iperf test between your sites using large window to make sure you can fill your pipe, do the math and what does that show you for performance?
-
Thank you all very much for your swift responses, info and suggestion!
I see now that this post got a bit coherent, I was testing stuff back and forth at the same time writing this. Hope something is understandable.
My answer right now isn't going to make you happy I think. The results though brings a smile. I'm now getting this:
Copying a 7z ~800MB with Windows File Explorer. I've come to the conclusion, if SMB is working OK over a WAN-VPN so are a lot of other things :o. "Real world" as real as I can do I quick test, that is. Many users access files over the VPN and that is with the noisy SMB protocol sadly enough. Synthetic test I've found harder to analyze. Lack of know-how and knowledge on my side probably. I'll have to look into this, maybe iperf can give me a better test.
Site-to-site
C2758-to-Xeon E3-1265L(virtual)
AES-128-CBC, Windows Server 2008/2012 and Windows 7/10
130-150Mbit/s down
80-100Mbit/s upSite-to-Client
C2758-to-i7-2640M
AES-128-CBC, Windows Server 2008/2012 and Windows 7/10, OpenVPN client, the latest export from pfSense.
140-160Mbit/s down
60-80Mbit/s upNow to the unhappy part. I really don't know which change that made a difference :-\ Sorry!
But I'll try going backwards to figure it out.
Test 1.
Info: Got some IPSEC tunnels active so was a bit worried about this one. But IPSEC worked as they should after. Have to read up more on that correlation.
Change: fast-ip-forward set to 1. Reboot.
Result: Didn't see any major improvement.Test 2.
Info: Some very unscientific tests with mtu and mssfix. Lots of googleing and searching the forums here.
Change: tun-mtu 1500;mssfix 1400; both Site-to-Site and Client. No reboot, OpenVPN restarts when you make changes and Save(?). Correct?
Result: No major impact on that either.Test 3.
Info: Chasing ghosts here. Tried something that should only affect some version of Linux kernel but some people indicated that it might help here too. https://forum.pfsense.org/index.php?topic=88758.15
Change: sndbuf 393216;rcvbuf 393216; on both side for Site-to-Site. And for Client access: sndbuf 393216;rcvbuf 393216;push "sndbuf 393216";push "rcvbuf 393216";
Results: Non that I noticed right away.Test (don't remember when in time):
Change: Maybe not a real change But checked the PowerD settings and did a Save on that page.Gave up. This was this afternoon.
Came back tonight and just for fun tried the file copy again. Wham! >100Mbit/s and the results reported above. I'm satisfied.Right now I going through the config-backup to try to figure out with diff function whats been done. ???
I'l try to come back with a more precise diag of what made real change. My suspicion right now point to the mtu and mssfix.
Does this make sense? (192.168.99.100 is a Windows 7x64 machine)
iperf-2.0.5-cygwin>iperf.exe -c 192.168.1.1 -w 204800 -P 4
–----------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 200 KByte
[ 5] local 192.168.99.100 port 37923 connected with 192.168.1.1 port 5001
[ 6] local 192.168.99.100 port 37924 connected with 192.168.1.1 port 5001
[ 4] local 192.168.99.100 port 37922 connected with 192.168.1.1 port 5001
[ 3] local 192.168.99.100 port 37921 connected with 192.168.1.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 30.4 MBytes 25.4 Mbits/sec
[ 3] 0.0-10.0 sec 30.4 MBytes 25.4 Mbits/sec
[ 6] 0.0-10.1 sec 30.4 MBytes 25.3 Mbits/sec
[ 5] 0.0-10.1 sec 30.5 MBytes 25.4 Mbits/sec
[SUM] 0.0-10.1 sec 122 MBytes 101 Mbits/secInside the tunnel Site-to-Site, ping from to the other pfsense.
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=17.117 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.273 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=17.401 ms–- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 17.117/17.264/17.401/0.116 msAnd the other way around.
PING 192.168.99.1 (192.168.99.1): 56 data bytes
64 bytes from 192.168.99.1: icmp_seq=0 ttl=64 time=16.909 ms
64 bytes from 192.168.99.1: icmp_seq=1 ttl=64 time=16.943 ms
64 bytes from 192.168.99.1: icmp_seq=2 ttl=64 time=17.069 ms--- 192.168.99.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.909/16.974/17.069/0.069 ms -
If your iperf test is inside your tunnel sure looks like your doing 101Mbps
[SUM] 0.0-10.1 sec 122 MBytes 101 Mbits/sec
If find it unlikely that you were doing SMB file copy over 17ms latency at 100mbps, with default 64k window size.. Did you change the window size? Something was chached your using smb 3.1.1 with multiple streams?
maximum throughput with a TCP window of 64 KByte and RTT of 17.0 ms <= 30.84 Mbit/sec.
-
Hi again!
Got some pictures of the SMB-transfers. It sure looks like the data is traveling, not cached. Or have got this backwards :o too?
Downloading from the server (black)
Uploading to the server (red, the black download still in the graph). Missed to show copy details. But copy speed was about 5-6MByte/s
-
So what was the SMB version used here, was it windows 10 to windows 10? This supports SMB 3.1.1, windows 8 and 2012? With over SMB 3 multichannel was introduced so you can get multiple streams and yes use way more of your pipe over a wan with latency..
-
Shuut! Missed that info.
For the above tests I used the following.
Client
OS Name: Microsoft Windows 7 Ultimate
OS Version: 6.1.7601 Service Pack 1 Build 7601Server
OS Name: Microsoft Windows Server 2008 R2 Standard
OS Version: 6.1.7601 Service Pack 1 Build 7601That gives SMB 2.1
In previous post the transfers shown was from a Windows 10 and 2012 R2
Cut and paste from previous post.https://forum.pfsense.org/index.php?topic=100020.msg557355#msg557355
Site-to-site
C2758-to-Xeon E3-1265L(virtual)
AES-128-CBC, Windows Server 2008/2012 and Windows 7/10
130-150Mbit/s down
80-100Mbit/s upSite-to-Client
C2758-to-i7-2640M
AES-128-CBC, Windows Server 2008/2012 and Windows 7/10, OpenVPN client, the latest export from pfSense.
140-160Mbit/s down
60-80Mbit/s upHere is a Windows 10 and Windows 2012R2 involved. This should be SMB 3.0.2 then.
(I attached 2 pictures)
-
I'm also running 2.2.4 (amd64) on very similar hardware (C2558) and getting very similar results to you before you had any improvement. Unfortunately i've been unable to make any headway with the suggestions in this thread. I have not tried a downgrade yet though. Does anyone know if this has any potential to be fixed in 2.3?
-
Try a UDP iperf test across the tunnel increasing -b by about 100M per attempt until you start getting significant drops.
You can also temporarily open a source-limited port on the server side and iperf between the two sites outside the tunnel.
-
So your seeing 10+ MBytes in that transfer… And you have a 100Mbits per sec pipe.. Yeah that is going to be FULL then..
Where is this issue??
-
He sez it's gig at each end.
-
Where does "diablo266" say he has 1ge, where does "diablo266" post anything about anything other than a chime in of ME TOO and what you did didn't work for me..
I didn't notice it was a different poster at first - I thought OP was still complaining, but as you see he is happy he got his 100mbps the thought he should be seeing..
-
Both ends Internet connections are 1gbit/s and measures >800mbit from various speed-test-sites.
Post 1 - You didn't specify to whom you were speaking.
-
True as I stated I thought it was same poster when first read it.. My bad…
