Performance mystery with PIA on pfsense



  • Greetings all,

    I set up PIA on my pfsense router, and everything is working as expected except for the throughput.  I'm unable to get more than about 50Mbps down through pfsense, but if I install the client locally on a machine on my network, I can get close to my full 150Mbps down.  Here's what's baffling me:  watching the CPU usage in top, I never see openvpn go over about 30% usage.  If I understand top correctly, that's 30% of one core.  I'd expect it to at least max out a CPU core if my system is simply too slow.  I've tried both BF-128 and AES-256-CBC (with and without hardware acceleration enabled) and don't see much difference.

    Here's my specs:

    CPU: AMD Sempron 2650 (dual cores, 1.45Ghz, AES-NI support)
    Motherboard:  MSI AM1I
    RAM: 4GB DDR3
    NIC:  HP NC360T PCIe x4 (Intel® 82571EB, em driver)

    Anything ring a bell?  I realize this isn't a world beating system but it should be able to do better than what I'm seeing.



  • Did you install the client locally on the same machine?

    Have you tried to find out the OpenVPN performance of the CPU?

    You could take a look here:
    https://forum.pfsense.org/index.php?topic=115673.0

    I use PIA and I have no problem to get close my full 100Mbps down with this connection log:

    Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256-bit key
    Data Channel Encrypt: Using 256-bit message hash 'SHA256' for HMAC authentication
    Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256-bit key
    Data Channel Decrypt: Using 256-bit message hash 'SHA256' for HMAC authentication
    Control Channel: TLSv1.2, cipher TLSv1 / SSLv3 DHE-RSA-AES256-SHA, RSA 4096 bit



  • Found this:  https://forum.pfsense.org/index.php?topic=88758.0  Sounds like exactly what I've got going on.



  • so did you solved just increaseing the TCP/UDP socket send and receive buffers size?

    if you want to try, these are my custom options:

    explicit-exit-notify 2;
    ifconfig-nowarn;
    tls-client;
    persist-key;
    persist-tun;
    remote-cert-tls server;
    reneg-sec 0;
    auth-nocache;
    tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384;
    fast-io;
    sndbuf 524288;
    rcvbuf 524288



  • I added

    fast-io;
    sndbuf 524288;
    rcvbuf 524288

    to my config and can get about 70Mbps down now.  I'll keep trying, but I'm traveling this week and have only remote access to my network so it's kind of difficult to really gauge the performance without other factors getting in the way.  Thanks for the advice.



  • Very good, it's a 40% increase!

    If you have not already done, you may try to activate PowerD in "Maximum" mode in System-Advanced-Miscellaneous and, if supported, to enable AES-NI in Cryptographic Hardware.

    Anyway it may be possible that's close to the limit reached by the Sempron 2650.

    To find it out, you should perform the test suggested by Ira in
    https://forum.pfsense.org/index.php?topic=105238.msg616743#msg616743

    Run from the GUI:
    openvpn –genkey --secret /tmp/secret
    and
    time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

    Then to give the execution time in seconds in real-world meaning:
    (3200 / execution_time_seconds) = Projected Maximum Performance OpenVPN in Mbps

    As you can see from what I have tested in
    https://forum.pfsense.org/index.php?topic=115673.msg642058#msg642058
    CPUs in the same class may have different performance depending on the presence AES-NI support.

    Please, let me know your benchmark.



  • The performance test with AES-NI enabled gives me a theoretical max of 92Mbps.



  • With a benchmark like that I would have expected about 100 Mbps in download.
    I regret not being able to help you more.
    The only thing I can add to the info about my settings is that I'm running the 2.3.2 stable version.
    If you will solve the issue, I'd like to read the adopted solution.
    Cheers



  • i've got 200mbps but can only seem to get 20mbps via pia

    I have a

    Intel(R) Core(TM) i5-6400 CPU @ 2.70GHz
    2 CPUs: 2 package(s) x 1 core(s)

    and the following custom options

    auth-user-pass /etc/openvpn-password.txt;
    fast-io;
    sndbuf 524288;
    rcvbuf 524288

    AES cryptographic is enabled,

    if you find a way to improve it it would be great to know

    thanks!



  • @techy82
    just out of curiosity, what PIA server are you connecting?



  • new york city



  • @techy82:

    new york city

    Never tried. I usually go through denmark or sweden and with the configuration above I easily get the limit of the line (100Mbps)



  • @mauroman33:

    I regret not being able to help you more.

    You've helped plenty. Thanks.  Once I get home from my travels and am not testing remotely I'll be able to try tweaking a few more settings.  Worst case I buy an Athlon 5350 or 5370 for a 50% + single thread improvement.



  • @mauroman33:

    @techy82:

    new york city

    Never tried. I usually go through denmark or sweden and with the configuration above I easily get the limit of the line (100Mbps)

    I'll try some different servers later and see how that goes, Thanks



  • pfSense 2.3.2. using PIA with 2 OpenVPN clients combined in one Gateway Group (PIA could not deliver coding/decoding speed with one connection).

    Get full ISP speed (500/500 Mbit) with CPU load of ~30%

    Hardware: intel i5-3450
    VPN

    • AES-256-CBC
    • SHA256
    • fast-io;
    • sndbuf 524288;
    • rcvbuf 524288
    • Hardware acceleration enabled.
    • 2 fixed (same country as client) IP adresses for PIA.

    So it should not be PIA restricted, seems CPU restricted.



  • @M_Devil:

    pfSense 2.3.2. using PIA with 2 OpenVPN clients combined in one Gateway Group (PIA could not deliver coding/decoding speed with one connection).

    Get full ISP speed (500/500 Mbit) with CPU load of ~30%

    Hardware: intel i5-3450
    VPN

    • AES-256-CBC
    • SHA256
    • fast-io;
    • sndbuf 524288;
    • rcvbuf 524288
    • Hardware acceleration enabled.
    • 2 fixed (same country as client) IP adresses for PIA.

    So it should not be PIA restricted, seems CPU restricted.

    This is interesting.
    How do you set the priority in the group? Both Tier 1 I guess.
    And what speed did you get using only one OpenVPN client?



  • Indeed, both tier 1.
    When using Blowfish (only option in the past), I could not push it above 200Mbit and unstable. By then I came up with the 2 client setup and that worked like a charm.
    Recently I switched to AES and with a quick test it seems that it could handle ISP speed also with one connection. I stick with 2 connection for stability and extra security reasons.



  • Thanks for your reply.

    I'm curious about the OpenVPN performance of various CPUs because of a future upgrade of my line and your CPU seems really interesting from my point of view.

    If you are willing, could you performed the simple OpenVPN benchmark referenced here?
    https://forum.pfsense.org/index.php?topic=105238.msg616743#msg616743 (Reply # 9 message)

    From the GUI run

    openvpn –genkey --secret / tmp / secret

    --test time openvpn-crypto --secret / tmp / secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

    Then to give the execution time in seconds in real-world meaning:
    (3200 / execution_time_seconds) = Projected Maximum Performance OpenVPN in Mbps

    My Celeron N3150 gets a value of 116 Mbps that's the same value that normally reaches during download trough a PIA client.



  • Execution time = 9.433 seconds, so Projected Maximum Performance = 339 Mbit.

    Does this represent single core performance?

    Edit: In this case it does not represent maximum performance. It could easly push 500Mbit with ~30% load.



  • As far as I know OpenVPN works in single thread, but I could be wrong… anyway your CPU is a beast!  ;)
    Thanks for letting me know.



  • Not sure if this will help, but try turning off the Hardware Crypto setting in pfSense:

    https://forum.pfsense.org/index.php?topic=115627.0



  • If OpenVPN is indeed single threaded you can try multiple clients like me.
    Looks like your Celeron has multiple cores.



  • As I remembered, OpenVPN it is not scalable:
    https://www.clearos.com/resources/documentation/clearos/content:en_us:kb_o_openvpn_performance

    I wanna say thanks to M_Devil for his tip: using multiple PIA clients I will not have the need to change my router after the line's upgrade.



  • Glad to help you. Please let us know if it worked out.



  • Of course! Thank you again.  :)



  • @M_Devil:

    pfSense 2.3.2. using PIA with 2 OpenVPN clients combined in one Gateway Group (PIA could not deliver coding/decoding speed with one connection).

    Get full ISP speed (500/500 Mbit) with CPU load of ~30%

    Hardware: intel i5-3450
    VPN

    • AES-256-CBC
    • SHA256
    • fast-io;
    • sndbuf 524288;
    • rcvbuf 524288
    • Hardware acceleration enabled.
    • 2 fixed (same country as client) IP adresses for PIA.

    So it should not be PIA restricted, seems CPU restricted.

    Could you please explain the steps you took to set this up? I'm lost on how you grouped the 2 vpn connections?

    Still learning pfsense stuff. And this would probably help others also.

    Thanks



  • First make sure you have 2 operational VPN client connections. Test both of them with firewall rules and check if you can browse pages an check the IP adres.

    After that: System -> routing -> Gateway groups. Add new gateway groep and select both VPN-client interfaces as Tier 1. Give the new gateway group a name and save it.
    Now you can select this new gateway in your firewall rules and let the traffic flow  :)

    @pigbait: Does this answer your question?



  • @M_Devil:

    First make sure you have 2 operational VPN client connections. Test both of them with firewall rules and check if you can browse pages an check the IP adres.

    After that: System -> routing -> Gateway groups. Add new gateway groep and select both VPN-client interfaces as Tier 1. Give the new gateway group a name and save it.
    Now you can select this new gateway in your firewall rules and let the traffic flow  :)

    @pigbait: Does this answer your question?

    I think I can manage  :o if not I'll keep you posted. Thanks for you time with this I appreciate it.



  • @M_Devil:

    First make sure you have 2 operational VPN client connections. Test both of them with firewall rules and check if you can browse pages an check the IP adres.

    After that: System -> routing -> Gateway groups. Add new gateway groep and select both VPN-client interfaces as Tier 1. Give the new gateway group a name and save it.
    Now you can select this new gateway in your firewall rules and let the traffic flow  :)

    @pigbait: Does this answer your question?

    Im lost in the firewall rules. I dont see the gateway group?

    thanks



  • @pigbait:

    @M_Devil:

    First make sure you have 2 operational VPN client connections. Test both of them with firewall rules and check if you can browse pages an check the IP adres.

    After that: System -> routing -> Gateway groups. Add new gateway groep and select both VPN-client interfaces as Tier 1. Give the new gateway group a name and save it.
    Now you can select this new gateway in your firewall rules and let the traffic flow  :)

    @pigbait: Does this answer your question?

    Im lost in the firewall rules. I dont see the gateway group?

    thanks

    If you can see the group you've created in Status>Gateways>Gateway Groups, you also should see it in the Advanced Options of the firewall rule you're going to modify.






  • @mauroman33:

    @pigbait:

    @M_Devil:

    First make sure you have 2 operational VPN client connections. Test both of them with firewall rules and check if you can browse pages an check the IP adres.

    After that: System -> routing -> Gateway groups. Add new gateway groep and select both VPN-client interfaces as Tier 1. Give the new gateway group a name and save it.
    Now you can select this new gateway in your firewall rules and let the traffic flow  :)

    @pigbait: Does this answer your question?

    Im lost in the firewall rules. I dont see the gateway group?

    thanks

    If you can see the group you've created in Status>Gateways>Gateway Groups, you also should see it in the Advanced Options of the firewall rule you're going to modify.

    What is the location under the firewall I want to modify that's what I don't understand. I can follow system>routing>gateway groups.  I made the group then I don't understand what I need to do or where to go in the firewall rules.. sorry I'm a complete noob…

    Also my group under status shows offline? Not sure if that's normal till the firewall rules are set.



  • @pigbait
    Have you enabled the VPN connections, one per gateway? If yes, your gateway group should be online.
    So you should go to Firewall>Rules>LAN and in the field "Gateway" of the "Advanced Options" of the pass rule that your devices are using to go out (eg "Default allow LAN IPv4 to any rule"), you should select the gateway group.



  • @mauroman33:

    With a benchmark like that I would have expected about 100 Mbps in download.
    I regret not being able to help you more.
    The only thing I can add to the info about my settings is that I'm running the 2.3.2 stable version.
    If you will solve the issue, I'd like to read the adopted solution.
    Cheers

    Back at it.  With two clients I can reach about 120Mbps in a speed test.  At that point, my CPU shows 0% idle in top.  So, I just took possession of an Athlon 5350 to replace the Sempron 2650.  That will give me four cores at 2GHz instead of two at 1.45GHz.  Additionally, I'm replacing the laptop drive with an SSD.  (not that that has any bearing on OpenVPN, just doing it while I'm taking the box down for an upgrade).  Will report back.



  • @mauroman33

    Thanks for your time. but i must be missing something and your going to start thinking I'm stupid.  :o :o :o… I have taken screen shots of what I have

    First off I setup 2 PIA vpn connections following this guide. ( https://www.privateinternetaccess.com/forum/discussion/21875/ ).... they both connect and get PIA IP addresses. afterward I take both VPN interfaces and make a group but for some reason they don't show as online... but the vpn is working as my PC's report the PIA IP addresses.

    these are the screen shots of what I think is important for you to help me with...

    Thanks for you time and patience its very well appreciated honestly... I feel bad being that annoying guy...



    ![Openvpn status.png](/public/imported_attachments/1/Openvpn status.png)
    ![Openvpn status.png_thumb](/public/imported_attachments/1/Openvpn status.png_thumb)
    ![VPN clients.png](/public/imported_attachments/1/VPN clients.png)
    ![VPN clients.png_thumb](/public/imported_attachments/1/VPN clients.png_thumb)
    ![gateway status.png](/public/imported_attachments/1/gateway status.png)
    ![gateway status.png_thumb](/public/imported_attachments/1/gateway status.png_thumb)
    ![group status.png](/public/imported_attachments/1/group status.png)
    ![group status.png_thumb](/public/imported_attachments/1/group status.png_thumb)
    ![Firewall LAN rules.png](/public/imported_attachments/1/Firewall LAN rules.png)
    ![Firewall LAN rules.png_thumb](/public/imported_attachments/1/Firewall LAN rules.png_thumb)
    ![System status.png](/public/imported_attachments/1/System status.png)
    ![System status.png_thumb](/public/imported_attachments/1/System status.png_thumb)



  • @whosmatt:

    @mauroman33:

    With a benchmark like that I would have expected about 100 Mbps in download.
    I regret not being able to help you more.
    The only thing I can add to the info about my settings is that I'm running the 2.3.2 stable version.
    If you will solve the issue, I'd like to read the adopted solution.
    Cheers

    Back at it.  With two clients I can reach about 120Mbps in a speed test.  At that point, my CPU shows 0% idle in top.  So, I just took possession of an Athlon 5350 to replace the Sempron 2650.  That will give me four cores at 2GHz instead of two at 1.45GHz.  Additionally, I'm replacing the laptop drive with an SSD.  (not that that has any bearing on OpenVPN, just doing it while I'm taking the box down for an upgrade).  Will report back.

    Ciao whosmatt,
    thanks for the report, really appreciated!
    Managing to pass from 70Mbps  to 120Mbps confirms that the M_Devil's method works and that it will save me a bit of money when I'll make the line's upgrade.  :)
    Now your situation seems definitely a CPU performance problem, so I think you made the right choice going to an Athlon 5350.
    Please keep us updated!



  • @pigbait:

    @mauroman33

    Thanks for your time. but i must be missing something and your going to start thinking I'm stupid.  :o :o :o… I have taken screen shots of what I have

    First off I setup 2 PIA vpn connections following this guide. ( https://www.privateinternetaccess.com/forum/discussion/21875/ ).... they both connect and get PIA IP addresses. afterward I take both VPN interfaces and make a group but for some reason they don't show as online... but the vpn is working as my PC's report the PIA IP addresses.

    these are the screen shots of what I think is important for you to help me with...

    Thanks for you time and patience its very well appreciated honestly... I feel bad being that annoying guy...

    Ciao pigbait,
    I compared our settings and did not find anything unusual. In the past also my gateways were offline, I solved with a new installation of pfSense. On the forum you will find some threads about it, with different solutions, you are not alone in this situation. In any case my connections had worked well and I understand that even yours are working in spite of this sort of "false positive."
    The point is if you notice a performance improvement using two aggregates client instead of only one. Let us know, please.



  • @mauroman33:

    [
    Ciao whosmatt,
    thanks for the report, really appreciated!
    Managing to pass from 70Mbps  to 120Mbps confirms that the M_Devil's method works and that it will save me a bit of money when I'll make the line's upgrade.  :)
    Now your situation seems definitely a CPU performance problem, so I think you made the right choice going to an Athlon 5350.
    Please keep us updated!
    [/quote]

    With the 5350 I can reach full speed of my connection with a single client; around 150Mbps.  Success!



  • Congratulations, well done!

    May I boring you asking to repeat the test on the first page, when you'll have the time?



  • @mauroman33:

    Congratulations, well done!

    May I boring you asking to repeat the test on the first page, when you'll have the time?

    Sure.. I get a theoretical max of 130Mbps



  • Thank you very much!