Performance mystery with PIA on pfsense
-
So I think I've been doing this right… I've got two VPN connections. I have then created an interface called PIAVPN and I put that interface into the outgoing NAT table under the WAN options... I then connected each of the VPN connections to PIAVPN under interfaces (assign) and proved that both of them are in fact working.
I then created a gateway group and put both of these interfaces into the gateway group as tier I.
I then changed my WIFI_TRUST subnet from the PIAVPN interface on the Outgoing NATt table to just "OpenVPN". I then also went into the WIFI_TRUST rules and changed the pass rules to use the new Gateway Group in Advanced Options.
At this point I then restarted all of the VPN services for double assurance.
As I then tested things from a client on the WIFI_TRUST subnet I found that all of my "what is my ip" tests came up as the VPN1 connection and the VPN2 connection never seemed to get used. I found that for some reason my console version of the speedtest.net test would no longer run reliably like it did when I was on each connection individually. I also found that when I was browsing the web that pages would take a bit longer to start loading and working. Kinda like if you had a slightly slow DNS server.
What might I be doing wrong or do I have incorrect expectations here? My understanding is that I should see my IP change on a per connection basis. Thus one "what is my ip" test should come up as Seattle and later after running a few I should see some come up as San Jose right? I also suspect that my speed tests would be the same for a single client but if I were to test on multiple clients I'd better be ale to take advantage of the CPUs and I'd get greater than the 50Mbit I get on a single core. I found that when using the gateway group its slightly lower throughput than when just linked up to the one connection direct.
Any help is greatly appreciated.
Oh and PS I'm on a D525 Atom chip (no AES-NI support dual core 1.8Ghz HT). 4GB of RAM. I can get as much as about 50Mbit with AES-128-CBC on a single thread. My connection is 120Mbit.
When I run the test mentioned on page 2 I get:
aes-256-cbc : real 56.15 : Theoretical of 56Mbit
aes-128-cbc : real 48.72 : Theoretical of 65MbitI'm actually using aes-128-cbc and I'm seeing around 50Mbit as my top speed and the CPU on the front page generally jumps to 48 or a full 50% while the speed test runs.
Thanks!
-
In short, using multiple tunnels is functionally equivalent to a multi-WAN setup, and if your tunnels are up and stable, there's no difference. There are some situations where it excels, some situations where it acts just like a single tunnel, and some situations where it breaks or at least is not ideal.
For example, your "what is my IP" tests will only ever show one of the two public IPs, and that is generally sticky in the short term but may change if you test again after a while. But it may not.
For a reliable speed test, try https://www.dslreports.com/speedtest. That will use both (or all if more than two) connections (and let you know via a proxy warning) that it did.
I have a lot of policy routing rules in my LAN ruleset to work around some of the possible issues that may occur with a multi-WAN/tunnel setup.
I'll try and respond in more detail later. It's late here.
Matt
-
you can just look at your rrd graphs to check the traffic going through each of your connections.
-
Ok then I'll do some more specific reading about multi WAN and how to set things up. I'd be very interested to learn about the shortcomings of such things and how to get around them! Then I'll also be setting up a path around the VPN and straight out for Netflix so that that still works.
Appreciate any thoughts and assistance. I use pfSense because its the best, it does what I need, but also because it lets me try things and learn!
I'll also try and check the graphs but I'm fairly certain that everything I was doing was all going out the one pipe.
Thank you again all!
-
I'm curious as to the drawbacks of gateway grouped VPN clients? I haven't noticed anything.
-
OK so I've got two VPN links and both come up. Each of these is associated with an interface… They are bolted into a gateway group both as Tier 1. I have 3 manual NAT outbound rules one for each VPN interface and one for the actual WAN. I then have my pass rule in my WIFI_TRUSTED interface set to use that gateway group.
Right after things come up it was working for a bit... I ran a couple of speed tests but didn't see any proxy warnings. Then things stopped working (I had no internet access). One of the VPN links went down. But it seems like it should have continued working with the other one. However no.
So I then added the actual WAN as Tier 2 into the gateway group. Still nothing would get through.... I then disabled both of the VPN connections and things started working through the WAN. I then deleted the WAN out of the gateway group (and expected things to stop working). However, it continued. I'd expect that it wouldn't be able to get out if the rule that I'm heading out through is locked to go out through a gateway that doesn't include the WAN...
Strange.
-
Well I now believe that I have things much closer…
I now have 3 gateway groups. 1 with both as Tier 1. Then 2 more with one as tier 1 and the other as tier 2 and vice versa.
I then created 3 firewall rules one for each of these gateways.
I also added a separate DNS server for each of the VPN links and explicitly set them to use the gateway for each VPN. This is in addition to the 8.8.8.8 and 8.8.4.4 servers that are allowed on any gateway. I'm also using 8.8.8.8 and 8.8.4.4 for the "monitor" IPs for the two VPN links.
Things come up and work quite well. If I load a what is my IP page from a few different tabs I'll see each of the IPs that I'm linking to randomly...
However after a short bit one of the links goes down and then just continually retries and never then comes back up...
I get errors in the log like these during the trouble:
write UDPv4: Permission denied (code=13)And even at one point got an error that read:
write UDPv4: No buffer space available (code=55)This box has 4GB of RAM and right now only shows 13% usage. If I go in and change the IP address of the sever that is no longer connecting it works again... But only for a short while.
~Brett
-
What packages are you running?
-
Here's the packages listing from the dashboard of what I have running and version numbers.
 -
Try disabling pfBNG & Snort (also clear out the snort2c table) and see if it works for you.
It's possible that one of those is misconfigured, or configured in such a way that it conflicts with your VPN.
-
My guess is that a rule in snort is flagging your VPN traffic. So when you change the servers IP address, it works for a short period before it triggers your snort rule and is blocked again.
Again, make sure you clear your snort2c table after disabling snort. If you can identify snort as the problem then you can reenable it as an IDS only until you identify the rule(s) generating false positives on your VPN and remove them.
-
Ah yes great idea! I didn't think to look there being that a single VPN link never triggered it and I could go for days without issue but Snort can be a fickle one from time to time… I've got too much work going on tonight and the next day to play with it but I'll certainly have time over the rainy weekend to give this a go. Thanks for the ideas!
~Brett
-
Aaaand its working!!!
Got around the Snort issue by adding my VPN server IPs to the Snort Whitelist. I have found that using IPs makes life easier than using PIA's server URL's and even PIA's tech support recommends doing so.
I've got all my interfaces updated to use the 3 gateway groups that I've created (its my understanding that I need to have 3 gateway groups in order for failover to work properly). Anyway once that was all done and working I tested Netflix and Amazon Prime and they were failing because I was through a VPN. I then created some aliases for these two networks in pfBlocker and then created some forward around rules and that is now all working great. Then created the rules to allow my local networks to still see one another once I deleted the rules that allowed routing before hand. After that a bit of clean up to allow remote VPN connections into the house to go back out through the VPN except for Netflix and Amazon AWS. A bit of testing and all seems to do what I'd expect.
Now to wait a few days and see if it all remains but now that its been working for over an hour I'm thinking all will remain.
THANK YOU ALL SO MUCH!
-
I'm glad you got it all working, your configuration is very similar to mine!
-
Just thought I'd chime in and say I resolved a similar issue by disabling
1:2200073 SURICATA IPv4 invalid checksum
It was blocking PIA.