Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IPSec Issues 2.2.3 and 2.2.4

    Scheduled Pinned Locked Moved IPsec
    32 Posts 5 Posters 5.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      cmb
      last edited by

      TSO and LRO are global to the system, not a per-tunnel config.

      I thought from your earlier description you had an old system of some sort that you were swapping between at the same location and same config as the 4860, is that not the case? The ones running at other locations wouldn't be relevant.

      In that case it's safe to downgrade to 2.2.2 for testing that circumstance. Can use the manual update under System>Firmware with:
      https://files.pfsense.org/mirror/updates/old/pfSense-Full-Update-2.2.2-RELEASE-amd64.tgz

      It will complain that your config revision is newer, but that's OK in this specific case.

      1 Reply Last reply Reply Quote 0
      • R
        rain
        last edited by

        That's correct, the Core 2 system is at my house, so is the 4860. I've been swapping back and forth between those two systems just to validate that there is nothing upstream or network related.

        The third system is in the other city.

        I'll try 3DES tonight, failing that, I'll put the 2.2.2 on.

        1 Reply Last reply Reply Quote 0
        • R
          rain
          last edited by

          Sorry, didn't get to testing last night, will try tonight.

          1 Reply Last reply Reply Quote 0
          • R
            rain
            last edited by

            Tested 3DES with and without MSS clamping, even worse throughput than AES, about 4.7Mbit/sec.

            Tried AES-256, get about 9.6Mbit/sec.

            Rolling back to 2.2.2 right now.

            1 Reply Last reply Reply Quote 0
            • R
              rain
              last edited by

              So yeah, there is something seriously wrong with builds 2.2.3 and 2.2.4 with respect to IPsec.

              Rolled back to 2.2.2, and my throughput goes back to maxing out the circuit (50 down/3up). I've attached a screenshot proving this.

              From a "just work" on every version of pfSense with these SG boxes sold on the pfsense store, what options do I have here? Do I submit a ticket of some sort? It's easily to reproduce.

              With regards,

              at222.JPG_thumb
              at222.JPG

              1 Reply Last reply Reply Quote 0
              • J
                jwt Netgate
                last edited by

                @rain:

                No, only a aes128-CCM16 (nothing GCM). Otherwise just AES variants, 3DES. Is GCM all that and a bag of chips? I'm not familiar with it.

                No difference with AES-NI disabled, if anything, a bit slower but same behavior (that was the first thing I tested on the new 2.2.4).

                AES-CCM isn't a great mode for IPSec.  In fact, the only support I can find in the FreeBSD kernel for it is in the wireless code, so I'm confused how you've configured to use it.  (AES-CCM gets used a lot in 802.11.)

                If you don't want to use AES-GCM, have you tried AES-CBC-128 with HMAC-SHA1, because that's the bog-standard "best practice" until you get concerned with the strength of SHA1 and a 128-bit key length.

                In face, I can't find any support for using AES-CCM in the IPSec subsystem in FreeBSD.  Here are the auth and encryption tokens that 'setkey' will recognize.  These are copy-pasta straight for the source code.

                /* authentication alogorithm */
                hmac-md5
                hmac-sha1
                keyed-md5
                keyed-sha1
                hmac-sha2-256
                hmac-sha2-384
                hmac-sha2-512
                hmac-ripemd160
                aes-xcbc-mac
                tcp-md5
                null

                /* encryption alogorithm */
                des-cbc
                3des-cbc
                null
                simple
                blowfish-cbc
                cast128-cbc
                des-deriv
                des-32iv
                rijndael-cbc
                aes-ctr
                camellia-cbc

                Not can I find any support in the GUI for AES-CCM.

                BTW, the only modes registered with the AES-NI module are:
                AES-CBC
                AES-ICM
                AES-GCM
                AES-GHASH (128, 192, 256 bit)
                AES-XTS

                That said, AES-NI isn't going to help much for modes with a separate HMAC (basically all but AES-GCM) because the pass over the packet with the HMAC will dominate the time to encode/decode the packet before transmit/reception.

                This is why AES-GCM is a 'win' with AES-NI.

                I have ZERO doubt that 3DES is slower than AES.

                please send the output of "ipsec statusall".  I don't suggest posting it here in the forum.  Since you purchased these from the pfSense store, you have support.  Open a ticket.  If it's a bug that we've somehow missed, then I'll ensure that you don't "use" that ticket.

                1 Reply Last reply Reply Quote 0
                • J
                  jwt Netgate
                  last edited by

                  SG-2220 (yes, they do exist, C2358 2 cores @ 1.7GHz) at home.
                  C2758 (8 cores @ 2.4GHz) as VPN gateway at work.
                  Both running pfSense software version 2.2.4

                  1Gbps link from home, 1Gbps link at work, what happens between those two is good, but not ideal.

                  Jims-MacBook-Pro:~ jim$ ping -c 3 nfs4
                  PING nfs4.pfmechanics.com (172.27.32.4): 56 data bytes
                  64 bytes from 172.27.32.4: icmp_seq=0 ttl=61 time=4.352 ms
                  64 bytes from 172.27.32.4: icmp_seq=1 ttl=61 time=4.434 ms
                  64 bytes from 172.27.32.4: icmp_seq=2 ttl=61 time=4.860 ms

                  –- nfs4.pfmechanics.com ping statistics ---
                  3 packets transmitted, 3 packets received, 0.0% packet loss
                  round-trip min/avg/max/stddev = 4.352/4.549/4.860/0.223 ms
                  Jims-MacBook-Pro:~ jim$ ssh nfs4
                  Last login: Sat Aug  1 15:48:30 2015 from 172.21.0.26
                  FreeBSD 10.1-RELEASE-p5 (GENERIC) #0: Tue Jan 27 08:55:07 UTC 2015

                  [jim@nfs4 ~]$ rm testfile
                  [jim@nfs4 ~]$ dd if=/dev/random of=testfile bs=1k count=200k
                  204800+0 records in
                  204800+0 records out
                  209715200 bytes transferred in 6.281192 secs (33387802 bytes/sec)
                  [jim@nfs4 ~]$ ls -l testfile
                  -rw-r–r--  1 jim  netgate  209715200 Aug  1 15:49 testfile
                  [jim@nfs4 ~]$ exit
                  logout
                  Connection to nfs4 closed.
                  Jims-MacBook-Pro:~ jim$ scp nfs4:testfile /tmp/testfile
                  testfile                                          100%  200MB  22.2MB/s  00:09   
                  Jims-MacBook-Pro:~ jim$ tcsh
                  [Jims-MacBook-Pro:~] jim% repeat 10 sftp nfs4:testfile /dev/null
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  25.0MB/s  00:08   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  25.0MB/s  00:08   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  16.7MB/s  00:12   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  16.7MB/s  00:12   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  18.2MB/s  00:11   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  25.0MB/s  00:08   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  28.6MB/s  00:07   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  25.0MB/s  00:08   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  25.0MB/s  00:08   
                  Connected to nfs4.
                  Fetching /usr/home/jim/testfile to /dev/null
                  /usr/home/jim/testfile                            100%  200MB  25.0MB/s  00:08   
                  [Jims-MacBook-Pro:~] jim%

                  ![Screen Shot 2015-08-01 at 4.06.37 PM.png](/public/imported_attachments/1/Screen Shot 2015-08-01 at 4.06.37 PM.png)
                  ![Screen Shot 2015-08-01 at 4.06.37 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-08-01 at 4.06.37 PM.png_thumb)

                  1 Reply Last reply Reply Quote 0
                  • J
                    jwt Netgate
                    last edited by

                    And a little longer test
                    [jim@nfs4 ~]$ dd if=/dev/random of=testfile bs=1k count=2000k
                    2048000+0 records in
                    2048000+0 records out
                    2097152000 bytes transferred in 64.266291 secs (32632224 bytes/sec)
                    [jim@nfs4 ~]$ ls -l testfile
                    -rw-r–r--  1 jim  netgate  2097152000 Aug  1 16:10 testfile
                    [jim@nfs4 ~]$ exit

                    ![Screen Shot 2015-08-01 at 4.21.22 PM.png](/public/imported_attachments/1/Screen Shot 2015-08-01 at 4.21.22 PM.png)
                    ![Screen Shot 2015-08-01 at 4.21.22 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-08-01 at 4.21.22 PM.png_thumb)

                    1 Reply Last reply Reply Quote 0
                    • J
                      jwt Netgate
                      last edited by

                      and, now that I've recovered the nuc from last night's 1hr+ power hit at work…

                      Note that running across a LAN is faster, but no VPN.
                      jim@nucatwork:~ % sudo scp jim@nfs4:testfile /usr/local/www/apache24/data/
                      Password for jim@nfs4:
                      testfile                                          100% 2000MB  87.0MB/s  00:23   
                      jim@nucatwork:~ %

                      ![Screen Shot 2015-08-01 at 5.10.57 PM.png](/public/imported_attachments/1/Screen Shot 2015-08-01 at 5.10.57 PM.png)
                      ![Screen Shot 2015-08-01 at 5.10.57 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-08-01 at 5.10.57 PM.png_thumb)

                      1 Reply Last reply Reply Quote 0
                      • R
                        rain
                        last edited by

                        @jwt:

                        AES-CCM isn't a great mode for IPSec.  In fact, the only support I can find in the FreeBSD kernel for it is in the wireless code, so I'm confused how you've configured to use it.  (AES-CCM gets used a lot in 802.11.)

                        I didn't, likely my post wasn't clear. Those are options on the other end of the tunnel, a Palo Alto Networks 3000 series. The response was to let CMB know what other options I have available to try. Although I appreciate your long response!

                        I sent an email to the support from the store asking how to use that support ticket, but I have not yet heard back. That was last week Monday when I sent it. I'd love to report back that I got great support on this hardware.

                        1 Reply Last reply Reply Quote 0
                        • R
                          rain
                          last edited by

                          I don't doubt you are getting that on different hardware. I'm getting a lot better on some of my old home built hardware from the scrap heap. But not an apples to apples comparison.

                          Once I rolled back to 2.2.2 I'm getting reasonable performance from the tunnel. With nothing else changing except moving to 2.2.3 or 2.2.4 the tunnel fails to pass traffic and passes it terribly slow respectively. Not exactly "just works".

                          1 Reply Last reply Reply Quote 0
                          • J
                            jwt Netgate
                            last edited by

                            @rain:

                            I don't doubt you are getting that on different hardware. I'm getting a lot better on some of my old home built hardware from the scrap heap. But not an apples to apples comparison.

                            It's pretty close, actually.  I'm quite familiar with the SG-4860.  If anything, the 2220 is slower, and that was the point.  It's really straight-forward to get > 200Mbps using AES-GCM with AES-NI.

                            If I'd wanted to quote lab performance, I've seen > 1.5Gbps using fairly modern Xeons.  But the SG-2220 is slower than what you're using.

                            @rain:

                            Once I rolled back to 2.2.2 I'm getting reasonable performance from the tunnel. With nothing else changing except moving to 2.2.3 or 2.2.4 the tunnel fails to pass traffic and passes it terribly slow respectively. Not exactly "just works".

                            Have you turned off AES-NI?

                            1 Reply Last reply Reply Quote 0
                            • J
                              jwt Netgate
                              last edited by

                              @rain:

                              @jwt:

                              AES-CCM isn't a great mode for IPSec.  In fact, the only support I can find in the FreeBSD kernel for it is in the wireless code, so I'm confused how you've configured to use it.  (AES-CCM gets used a lot in 802.11.)

                              I didn't, likely my post wasn't clear. Those are options on the other end of the tunnel, a Palo Alto Networks 3000 series. The response was to let CMB know what other options I have available to try. Although I appreciate your long response!

                              OK, so send the output of "ipsec statusall", as requested.  Then we'll know what we're dealing with.

                              @rain:

                              I sent an email to the support from the store asking how to use that support ticket, but I have not yet heard back. That was last week Monday when I sent it. I'd love to report back that I got great support on this hardware.

                              I've forwarded an internal request to see what happened here.

                              1 Reply Last reply Reply Quote 0
                              • R
                                rain
                                last edited by

                                statusall on 2.2.2 and 2.2.4? Or just one?

                                1 Reply Last reply Reply Quote 0
                                • J
                                  jwt Netgate
                                  last edited by

                                  both would be interesting, but 2.2.2 would be OK

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    rain
                                    last edited by

                                    More information in there then I'm willing to post publicly, so I've PM'd it to you.

                                    This was on 2.2.2 working the way it should.

                                    1 Reply Last reply Reply Quote 0
                                    • C
                                      cmb
                                      last edited by

                                      There's something to this, I'm working on narrowing it down. I also grabbed your support ticket, will reply back there with an update before I wrap up for the day today.

                                      1 Reply Last reply Reply Quote 0
                                      • R
                                        rain
                                        last edited by

                                        Thanks Chris.

                                        Nice to be validated.  ;D I'm a newbie on these forums, but I'm not a newbie with networks.

                                        With regards,

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          afrojoe
                                          last edited by

                                          I too am experiencing issues with IPSec 2.2.3 and 2.2.4.  My tunnel is fast, stays up for a couple hours and then just disconnects…. Not the same issue as what others are reporting, but I have three tunnels connecting to my IPSec 2.2.3 instance, (two far ends are 2.2.3 and one is 2.2.4)  The 2.2.4 does not stay healthly for more than 4 hours.. deleting both instances at both ends are recreating brings everything back up for another 4 hours and then the tunnel dies again.

                                          1 Reply Last reply Reply Quote 0
                                          • R
                                            rain
                                            last edited by

                                            What kind of hardware?

                                            What kind of tunnel?

                                            CMB is working on my issue for a couple of weeks now but I haven't heard anything recently. I got the impression it was an upstream problem.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.