Trying to decide on hardware, IPSEC and OpenVPN server/client

[A Former User]

I am trying to figure out which hardware I need, next. My current site is running off of old hardware and my VPN speeds (via IPSEC tunnel and OpenVPN client on windows 10) never exceed 10 Mbps. It turns out the CPU in my pfsense box isn't VPN friendly.

Before I go and buy a pre-built box from the pfsense store (looking at SG-3100) I want to make sure I don't find myself in the same situation.

The main thing I'm looking for are better speeds over IPSEC VPN and OpenVPN server/client. I'd have a 3100 on both sides of the tunnel, but I'd start by buying 1 to test OpenVPN server/client. If the speeds are improved, then I'd by a second one to solve the IPSEC tunnel issue.

Can someone confirm VPN speeds with this hardware?

Thanks.

[NollipfSense]

You might have to be the Guinea Pig as not everyone with such box doing exactly what you desire to do.

[A Former User]

@NollipfSense:

You might have to be the Guinea Pig as not everyone with such box doing exactly what you desire to do.

I have a very hard time believing there aren't any 3100 owners that are not using OpenVPN and/or IPSEC. Even if that were true, I don't know why anyone would buy the hardware when someone from the netgate team could easily test this.

I'm not against building my own setup, I just don't want to do it with older computer parts that won't give me the CPU support needed for OpenVPN/IPSEC. Also, I'm referring to the traditional PC with large power supplies, I would build something or buy something pre-built as long as it doesn't throttle me to 10 Mbps when the connection is 100 Mbps or better on both sites. I'm not expecting 100 Mbps over the VPN, but maxing out at 10 Mbps is not acceptable.

[A Former User]

I just remembered that I have a small PC I built, years ago, that isn't being used, it has a Intel Core i5-3350P Ivy Bridge Quad-Core 3.1GHz CPU and two NICs. I'll have to install pfsense on here and test out OpenVPN/IPSEC speeds.

Here is a link to the specs of the CPU- https://ark.intel.com/products/69114/Intel-Core-i5-3350P-Processor-6M-Cache-up-to-3_30-GHz

Intel 64- Yes
Instruction Set- 64-bit
Intel AES New Instructions- Yes

[Guest]

It will work just fine.

[A Former User]

@johnkeates:

It will work just fine.

The 3100 will work just fine or the CPU I plan on testing with will work fine?

Edit- or both will work just fine?

[Guest]

@tdhuck:

@johnkeates:

It will work just fine.

The 3100 will work just fine or the CPU I plan on testing with will work fine?

Edit- or both will work just fine?

Both indeed. Also, if you want a better aim so you know what you'll need, we have to know your raw uplink/downlink speeds and preferred VPN speeds. De 3100 and any i5 will, however, have no problems pushing 90Mbit-ish AES VPN at least, but an i5 will probably get close to 1000Mbit if you don't set to much crypto.

[A Former User]

@johnkeates:

@tdhuck:

@johnkeates:

It will work just fine.

The 3100 will work just fine or the CPU I plan on testing with will work fine?

Edit- or both will work just fine?

Both indeed. Also, if you want a better aim so you know what you'll need, we have to know your raw uplink/downlink speeds and preferred VPN speeds. De 3100 and any i5 will, however, have no problems pushing 90Mbit-ish AES VPN at least, but an i5 will probably get close to 1000Mbit if you don't set to much crypto.

The location of the main pfsense box is ~100 Mbps down and ~30 Mbps up and the location where I'll be using OpenVPN is 100 Mbps down/up and I'll be running OpenVPN client on a windows 10 laptop.

The IPSEC tunnel will use the main pfsense box (speeds above) and the other end of the IPSEC tunnel is ~50 Mpbs down and ~15 Mbps up.

I know that a good CPU is needed for improved VPN performance, but I also thought AES-NI was needed. Is that true? If not, which is more important for better VPN performance?

Anything is better than 10 Mbps over a VPN tunnel and OpenVPN client, which is where I am at, today.

I'll be testing the spare PC with the CPU specs I posted above, hopefully today.

[Guest]

You need AES-NI and good CPU clock speeds. Two threads at the same time is good, but more than that won't actually improve performance in a noticeable manner.

OpenVPN is still single-threaded so only clock speeds help that, as well as AES-NI and other offloading functions. (but only AES-NI at this time it seems)

I think you should try the i5, if that gets you the speed you need, the only thing you'll have to think about is power consumption. If it turns out the i5 box doesn't actually use that much (40 watts or less) you can just leave it as-is depending on your power cost. If it's more, it'll get interesting to look at the 3100 in terms of optimisation of cost.

[A Former User]

@johnkeates:

You need AES-NI and good CPU clock speeds. Two threads at the same time is good, but more than that won't actually improve performance in a noticeable manner.

OpenVPN is still single-threaded so only clock speeds help that, as well as AES-NI and other offloading functions. (but only AES-NI at this time it seems)

I think you should try the i5, if that gets you the speed you need, the only thing you'll have to think about is power consumption. If it turns out the i5 box doesn't actually use that much (40 watts or less) you can just leave it as-is depending on your power cost. If it's more, it'll get interesting to look at the 3100 in terms of optimisation of cost.

I have the new box setup, but I am currently on site and won't have a chance to test OpenVPN in my normal network/setup for a couple more days. I enabled AES-NI

CPU Type Intel(R) Core(TM) i5-3350P CPU @ 3.10GHz
Current: 3100 MHz, Max: 3101 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM

Under the misc. options, I set it to, 'AES-NI CPU-based acceleration' and saved the setting.

I do plan on using this for IPSEC and OpenVPN, not one or the other, hopefully the change I made above is adequate for both.

EDIT- I was able to do some initial testing with OpenVPN on my phone. When using my phone and OpenVPN, I am not concerned with speed, I mainly use the Synology DS Camera App to view a camera at home. Before testing AES-NI enabled on this box, and using it for years on my old box w/o AES-NI, I was able to connect to OpenVPN and view the camera, everything worked great.

I enabled AES-NI (see above) on the new box and I can connect to the camera app just fine, but the video is either 10 seconds behind audio is basically live, no delay or the video freezes once I enter in the frame. I'll start walking, on camera, but I see a frozen image on my phone (disconnected from home wifi, connected to OpenVPN over cell network).

I immediately disable AES-NI, basically putting it back to the default option, and the camera image freezing issue goes away.

This is only one of my tests, I still want to test speeds when I am on a remote network (not cellular) using my laptop.

EDIT 2- Now I'm not convinced AES-NI being active is related to the issue, I took a second look at my phone and apparently I am getting poor reception, which wasn't the case, several months ago. I have re-enabled AES-NI and I'll have to wait until I am on another network to test from my laptop and from cellular, assuming I have a better cellular connection.

[A Former User]

Tested at one of the locations (via IPSEC) and it appears the tunnel is still capped at 10 Mbps down with the new CPU.

[Guest]

@tdhuck:

Tested at one of the locations (via IPSEC) and it appears the tunnel is still capped at 10 Mbps down with the new CPU.

Keep in mind that this depends on both sides of the connection. So a weak client will still limit you.

[A Former User]

@johnkeates:

@tdhuck:

Tested at one of the locations (via IPSEC) and it appears the tunnel is still capped at 10 Mbps down with the new CPU.

Keep in mind that this depends on both sides of the connection. So a weak client will still limit you.

I'm not convinced. What you say is absolutely true, but there has to be another issue, somewhere. I just disconnected the IPSEC tunnel and opened up my NAS to the internet and started to transfer a 3GB ISO file, I am still being capped at 10 Mbps w/o going through a VPN and having to worry about encryption throughput. Something seems like it isn't functioning at 100%

[Guest]

In that case, do have a different problem indeed. Make sure pfSense's interfaces are setup correctly (automatic mode etc) and check if any Link status LED's match the link speeds. If those are good, you probable have to look outside of pfSense to find the problem. Have you tried iperf yet? And packet capture to figure out if maybe a lot of trash is happening on the network?

This speed is not related to the CPU or anything like that, even a pentium 3 pulls much more bits than that.

[A Former User]

@johnkeates:

In that case, do have a different problem indeed. Make sure pfSense's interfaces are setup correctly (automatic mode etc) and check if any Link status LED's match the link speeds. If those are good, you probable have to look outside of pfSense to find the problem. Have you tried iperf yet? And packet capture to figure out if maybe a lot of trash is happening on the network?

This speed is not related to the CPU or anything like that, even a pentium 3 pulls much more bits than that.

Right, I'm convinced there is another issue since I am seeing these same issues with my other pfsense box, this rules out the interfaces, I would think, I doubt I'd have issues with interfaces on two different pfsense boxes.

My ISP equipment is a cable modem that is in bridge mode, I don't have issues getting full speeds when I am at the main network and running a speed test. Latency/ping/speeds all look normal. I stream 4k media all the time and have never seen buffering/pixelation/etc. I'm not saying that nothing needs to be checked, I am simply pointing out that there aren't any obvious issues to make me think something is wrong with the circuit.

I do think the problem is at the main connection since I experience the same 10 Mbps when I am at several different locations, two of those locations have connections of 100 Mbps or better.

I will say this, in all my tests, I am downloading files from my NAS, I guess I will start there and see if there is anything obvious. I do have two switches between my NAS box and the pfsense box, but all links should be gigabit (they were last time I checked).

EDIT- I am not physically on site at the main location (where the new pfsense install was done, yesterday), but I used SSH over the IPSEC tunnel to check the port status, everything is connected at 1000 Mbps Full Duplex. I'll see if I can run iperf from both pfsense boxes and see what that shows…

[Guest]

Also see if you can try iperf between de NAS and pfSense or another device on the same switch.

[A Former User]

@johnkeates:

Also see if you can try iperf between de NAS and pfSense or another device on the same switch.

Here are the results from iperf between the two pfsense boxes, no VPN, I opened up port 5001 on the main (new) pfsense box.

Not looking good…

---

Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 64.2 KByte (default)

[  3] local xxx.xxx.xxx.xxx port 50004 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0-10.0 sec  7.25 MBytes  6.06 Mbits/sec

[A Former User]

Here are the results when running iperf on a device connected to the main switch where the new pfsense box is located (not running off NAS). IPSEC/VPN tunnel

This is the server side:
[ ID] Interval          Transfer    Bandwidth
[  5]  0.00-1.00  sec  634 KBytes  5.18 Mbits/sec                 
[  5]  1.00-2.00  sec  1.08 MBytes  9.05 Mbits/sec                 
[  5]  2.00-3.00  sec  1.25 MBytes  10.5 Mbits/sec                 
[  5]  3.00-4.00  sec  1.38 MBytes  11.5 Mbits/sec                 
[  5]  4.00-5.00  sec  1.24 MBytes  10.4 Mbits/sec                 
[  5]  5.00-6.00  sec  1.29 MBytes  10.8 Mbits/sec                 
[  5]  6.00-7.00  sec  1.19 MBytes  9.97 Mbits/sec                 
[  5]  7.00-8.00  sec  1.28 MBytes  10.7 Mbits/sec                 
[  5]  8.00-9.00  sec  1.18 MBytes  9.92 Mbits/sec                 
[  5]  9.00-10.00  sec  1.15 MBytes  9.70 Mbits/sec                 
[  5]  10.00-10.04  sec  28.3 KBytes  6.03 Mbits/sec

---

[ ID] Interval          Transfer    Bandwidth
[  5]  0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]  0.00-10.04  sec  11.7 MBytes  9.77 Mbits/sec                  receiver

This is the client side:
[ ID] Interval          Transfer    Bandwidth      Retr  Cwnd
[  4]  0.00-1.00  sec  724 KBytes  5.93 Mbits/sec    0  45.2 KBytes     
[  4]  1.00-2.00  sec  1.10 MBytes  9.27 Mbits/sec    1  50.9 KBytes     
[  4]  2.00-3.00  sec  1.27 MBytes  10.7 Mbits/sec    0  67.9 KBytes     
[  4]  3.00-4.00  sec  1.37 MBytes  11.5 Mbits/sec    0  82.0 KBytes     
[  4]  4.00-5.00  sec  1.28 MBytes  10.7 Mbits/sec    1  65.0 KBytes     
[  4]  5.00-6.00  sec  1.29 MBytes  10.9 Mbits/sec    1  58.0 KBytes     
[  4]  6.00-7.00  sec  1.15 MBytes  9.62 Mbits/sec    1  50.9 KBytes     
[  4]  7.00-8.00  sec  1.30 MBytes  10.9 Mbits/sec    0  65.0 KBytes     
[  4]  8.00-9.00  sec  1.19 MBytes  9.95 Mbits/sec    3  59.4 KBytes     
[  4]  9.00-10.00  sec  1.14 MBytes  9.57 Mbits/sec    2  55.1 KBytes

---

[ ID] Interval          Transfer    Bandwidth      Retr
[  4]  0.00-10.00  sec  11.8 MBytes  9.90 Mbits/sec    9            sender
[  4]  0.00-10.00  sec  11.7 MBytes  9.81 Mbits/sec                  receiver

I've checked all interfaces on both pfsense boxes (via the pfsense GUI) everything is gigabit and full duplex. No errors/collisions.
I've checked all the interfaces on the switches, everything is gigabit and full duplex. No errors/collisions.

[A Former User]

I have good news and bad news.

Good news is that I am maxing out the connection at 10 Mbps on and off the VPN, on both pfsense boxes and now I know why (see bad news).

Bad news is that the ISP must have changed something or I have a problem, when I do a speed test, I get 105 Mbps down and 11 Mbps up.

Now that I know the upload is maxing at 11 Mbps, all my results are normal (see good news).

However, I have never seen cable internet, at the 100 Mbps download tier, come with 10 Mbps of upload speed. I either have an issue on the line/in the network or the ISP did in fact change their upload speeds on their packages. I am absolutely certain that my upload was more than 10 Mbps, in the past.

[Guest]

Well, now we know. Bloody ISPs and their bad uploads!  :-X