IPSec Issues 2.2.3 and 2.2.4
-
So yeah, there is something seriously wrong with builds 2.2.3 and 2.2.4 with respect to IPsec.
Rolled back to 2.2.2, and my throughput goes back to maxing out the circuit (50 down/3up). I've attached a screenshot proving this.
From a "just work" on every version of pfSense with these SG boxes sold on the pfsense store, what options do I have here? Do I submit a ticket of some sort? It's easily to reproduce.
With regards,
-
No, only a aes128-CCM16 (nothing GCM). Otherwise just AES variants, 3DES. Is GCM all that and a bag of chips? I'm not familiar with it.
No difference with AES-NI disabled, if anything, a bit slower but same behavior (that was the first thing I tested on the new 2.2.4).
AES-CCM isn't a great mode for IPSec. In fact, the only support I can find in the FreeBSD kernel for it is in the wireless code, so I'm confused how you've configured to use it. (AES-CCM gets used a lot in 802.11.)
If you don't want to use AES-GCM, have you tried AES-CBC-128 with HMAC-SHA1, because that's the bog-standard "best practice" until you get concerned with the strength of SHA1 and a 128-bit key length.
In face, I can't find any support for using AES-CCM in the IPSec subsystem in FreeBSD. Here are the auth and encryption tokens that 'setkey' will recognize. These are copy-pasta straight for the source code.
/* authentication alogorithm */
hmac-md5
hmac-sha1
keyed-md5
keyed-sha1
hmac-sha2-256
hmac-sha2-384
hmac-sha2-512
hmac-ripemd160
aes-xcbc-mac
tcp-md5
null/* encryption alogorithm */
des-cbc
3des-cbc
null
simple
blowfish-cbc
cast128-cbc
des-deriv
des-32iv
rijndael-cbc
aes-ctr
camellia-cbcNot can I find any support in the GUI for AES-CCM.
BTW, the only modes registered with the AES-NI module are:
AES-CBC
AES-ICM
AES-GCM
AES-GHASH (128, 192, 256 bit)
AES-XTSThat said, AES-NI isn't going to help much for modes with a separate HMAC (basically all but AES-GCM) because the pass over the packet with the HMAC will dominate the time to encode/decode the packet before transmit/reception.
This is why AES-GCM is a 'win' with AES-NI.
I have ZERO doubt that 3DES is slower than AES.
please send the output of "ipsec statusall". I don't suggest posting it here in the forum. Since you purchased these from the pfSense store, you have support. Open a ticket. If it's a bug that we've somehow missed, then I'll ensure that you don't "use" that ticket.
-
SG-2220 (yes, they do exist, C2358 2 cores @ 1.7GHz) at home.
C2758 (8 cores @ 2.4GHz) as VPN gateway at work.
Both running pfSense software version 2.2.41Gbps link from home, 1Gbps link at work, what happens between those two is good, but not ideal.
Jims-MacBook-Pro:~ jim$ ping -c 3 nfs4
PING nfs4.pfmechanics.com (172.27.32.4): 56 data bytes
64 bytes from 172.27.32.4: icmp_seq=0 ttl=61 time=4.352 ms
64 bytes from 172.27.32.4: icmp_seq=1 ttl=61 time=4.434 ms
64 bytes from 172.27.32.4: icmp_seq=2 ttl=61 time=4.860 ms–- nfs4.pfmechanics.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.352/4.549/4.860/0.223 ms
Jims-MacBook-Pro:~ jim$ ssh nfs4
Last login: Sat Aug 1 15:48:30 2015 from 172.21.0.26
FreeBSD 10.1-RELEASE-p5 (GENERIC) #0: Tue Jan 27 08:55:07 UTC 2015[jim@nfs4 ~]$ rm testfile
[jim@nfs4 ~]$ dd if=/dev/random of=testfile bs=1k count=200k
204800+0 records in
204800+0 records out
209715200 bytes transferred in 6.281192 secs (33387802 bytes/sec)
[jim@nfs4 ~]$ ls -l testfile
-rw-r–r-- 1 jim netgate 209715200 Aug 1 15:49 testfile
[jim@nfs4 ~]$ exit
logout
Connection to nfs4 closed.
Jims-MacBook-Pro:~ jim$ scp nfs4:testfile /tmp/testfile
testfile 100% 200MB 22.2MB/s 00:09
Jims-MacBook-Pro:~ jim$ tcsh
[Jims-MacBook-Pro:~] jim% repeat 10 sftp nfs4:testfile /dev/null
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 25.0MB/s 00:08
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 25.0MB/s 00:08
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 16.7MB/s 00:12
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 16.7MB/s 00:12
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 18.2MB/s 00:11
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 25.0MB/s 00:08
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 28.6MB/s 00:07
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 25.0MB/s 00:08
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 25.0MB/s 00:08
Connected to nfs4.
Fetching /usr/home/jim/testfile to /dev/null
/usr/home/jim/testfile 100% 200MB 25.0MB/s 00:08
[Jims-MacBook-Pro:~] jim%
 -
And a little longer test
[jim@nfs4 ~]$ dd if=/dev/random of=testfile bs=1k count=2000k
2048000+0 records in
2048000+0 records out
2097152000 bytes transferred in 64.266291 secs (32632224 bytes/sec)
[jim@nfs4 ~]$ ls -l testfile
-rw-r–r-- 1 jim netgate 2097152000 Aug 1 16:10 testfile
[jim@nfs4 ~]$ exit
 -
and, now that I've recovered the nuc from last night's 1hr+ power hit at work…
Note that running across a LAN is faster, but no VPN.
jim@nucatwork:~ % sudo scp jim@nfs4:testfile /usr/local/www/apache24/data/
Password for jim@nfs4:
testfile 100% 2000MB 87.0MB/s 00:23
jim@nucatwork:~ %
 -
@jwt:
AES-CCM isn't a great mode for IPSec. In fact, the only support I can find in the FreeBSD kernel for it is in the wireless code, so I'm confused how you've configured to use it. (AES-CCM gets used a lot in 802.11.)
I didn't, likely my post wasn't clear. Those are options on the other end of the tunnel, a Palo Alto Networks 3000 series. The response was to let CMB know what other options I have available to try. Although I appreciate your long response!
I sent an email to the support from the store asking how to use that support ticket, but I have not yet heard back. That was last week Monday when I sent it. I'd love to report back that I got great support on this hardware.
-
I don't doubt you are getting that on different hardware. I'm getting a lot better on some of my old home built hardware from the scrap heap. But not an apples to apples comparison.
Once I rolled back to 2.2.2 I'm getting reasonable performance from the tunnel. With nothing else changing except moving to 2.2.3 or 2.2.4 the tunnel fails to pass traffic and passes it terribly slow respectively. Not exactly "just works".
-
I don't doubt you are getting that on different hardware. I'm getting a lot better on some of my old home built hardware from the scrap heap. But not an apples to apples comparison.
It's pretty close, actually. I'm quite familiar with the SG-4860. If anything, the 2220 is slower, and that was the point. It's really straight-forward to get > 200Mbps using AES-GCM with AES-NI.
If I'd wanted to quote lab performance, I've seen > 1.5Gbps using fairly modern Xeons. But the SG-2220 is slower than what you're using.
Once I rolled back to 2.2.2 I'm getting reasonable performance from the tunnel. With nothing else changing except moving to 2.2.3 or 2.2.4 the tunnel fails to pass traffic and passes it terribly slow respectively. Not exactly "just works".
Have you turned off AES-NI?
-
@jwt:
AES-CCM isn't a great mode for IPSec. In fact, the only support I can find in the FreeBSD kernel for it is in the wireless code, so I'm confused how you've configured to use it. (AES-CCM gets used a lot in 802.11.)
I didn't, likely my post wasn't clear. Those are options on the other end of the tunnel, a Palo Alto Networks 3000 series. The response was to let CMB know what other options I have available to try. Although I appreciate your long response!
OK, so send the output of "ipsec statusall", as requested. Then we'll know what we're dealing with.
I sent an email to the support from the store asking how to use that support ticket, but I have not yet heard back. That was last week Monday when I sent it. I'd love to report back that I got great support on this hardware.
I've forwarded an internal request to see what happened here.
-
statusall on 2.2.2 and 2.2.4? Or just one?
-
both would be interesting, but 2.2.2 would be OK
-
More information in there then I'm willing to post publicly, so I've PM'd it to you.
This was on 2.2.2 working the way it should.
-
There's something to this, I'm working on narrowing it down. I also grabbed your support ticket, will reply back there with an update before I wrap up for the day today.
-
Thanks Chris.
Nice to be validated. ;D I'm a newbie on these forums, but I'm not a newbie with networks.
With regards,
-
I too am experiencing issues with IPSec 2.2.3 and 2.2.4. My tunnel is fast, stays up for a couple hours and then just disconnects…. Not the same issue as what others are reporting, but I have three tunnels connecting to my IPSec 2.2.3 instance, (two far ends are 2.2.3 and one is 2.2.4) The 2.2.4 does not stay healthly for more than 4 hours.. deleting both instances at both ends are recreating brings everything back up for another 4 hours and then the tunnel dies again.
-
What kind of hardware?
What kind of tunnel?
CMB is working on my issue for a couple of weeks now but I haven't heard anything recently. I got the impression it was an upstream problem.
-
The firmware version of Intel NICs can play a significant role in these types of problems. I have had identical issues on a LAN. Take a look at that if you would.
afrojoe: I would update your 2.2.3 concentrator to 2.2.4 if you can. I am curious, are these pfSense HW or hand rolled? If hand rolled what are the NICs?
/M
-
It's one of the official pfsense units sold directly from pfsense which will not have issues like firmware incompatibility.
-
Hi rain!
The issue with Intel firmware is not necessarily incompatibility but rather the Intel developers seem to like to "play around" with different things for various customers in releases, some of this is latency on WAN links. I was one of the first WISPS in the 90s and it drove us nuts.
I have an SG-2440 that is working flawlessly. I have a ticket open regarding the issues of different clients in mobile configurations. CMB is very busy on many fronts. Being he is AWESOME, we should help every way we can to narrow down issues.
I would also not rule out an ISP related issue. Do you have the same provider on both ends? AND are all of your endpoints pfSense HW? TIA
/M
-
Not and ISP issue, same hardware on two different providers behaves the same, also on the same provider. Different hardware on the same two different providers work without issue, also on the same provider.
Quality of circuits is outstanding in all my remote locations. I'm using a hub spoke model, with a pair of Palo Alto 3000 series as the hub. Multiple spokes, all pfSense. Any pfsense running 2.2.2 has no issues (AES-256). All running 2.2.4 work fine except the pfsense official hardware firewall from the store.
I have no issues other than with this one firewall hardware. All other factors I can remove, have been removed.
CMB has I think all the details he's asked for, but I'm sure if he needs more he'll ask.
And trust that I have nothing but respect for CMB and the team at ESF. I honestly believe that pfsense is the best platform for perimeter security out there, commercial or not. The only reason I use PAN as my hub is because of executive concerns around an open source platform doing all security between all subnets, local and remote.
I'm just in an awkward position. I promised the CEO of the company that I would get him the best of the best, rather than what I usually build using spare parts, and I looked like an amateur after 2.2.2. All the technical reasons aside, he sees me handing him a black box that doesn't work as I told him it would. Meanwhile an old grinder under the desk supports 10-20 people on a regular basis and never blips.
The only reason I bought the pfsense branded hardware was because I read these forums regularly, and I see pfsense experts brag all the time about their bulletproof hardware from the pfsense store. I wanted to be one of those too because quite frankly although I have good good success with old hardware, one day I'm sure that might end (given the end user problems on these forums). :)
I'm grateful, honest!
Cheers,