Is the ubsec crypto hardware support broken?



  • I am beginning to think that the ubsec crypto engine for OpenSSL is broken on pfSense 2.1 and has been so for some years.

    I have just installed a BCM5821 crypto accelerator and I just cannot get OpenSSL to stop using the software crypto. My experience is almost identical to user 'frontsidebus' https://forum.pfsense.org/index.php/topic,41332.msg214036.html#msg214036

    I have managed to get cryptotest to use the ubsec engine and ubsecstats confirms that it is being used. OpenSSL just ignores it and I suspect that everyone that has a Broadcom accelerator in pfSense is blissfully unaware that OpenSSL falls back to using the motherboard software crypto instead of using the crypto hardware.

    The only thing I can think of to try to fix this is to build a separate FreeBSD 8.3 box and download and compile the same version of OpenSSL that pfSense uses on it. Hardware crypto support used to be built in so I am going to try to bypass the dynamic engine altogether and compile OpenSSL with a static version of ubsec instead. If I can get it working on FreeBSD 8.3 I will try installing a copy of the fresh compilation on pfSense.

    If someone has managed to get the ubsec dynamic engine working, please let me know. I'm not expecting that anyone has been successful.



  • I have been spending a few hours here and there on this and I think I have answered my own question.
    Yes, hardware crypto support for ubsec cards (and probably others) does appear to be broken in pfSense 2.1 .

    I haven't finished my work yet but I have had a breakthrough of sorts that I thought was worth mentioning if anyone was currently officially working on OpenSSL for the next release.

    On my freshly created CF card containing 32-bit nano pfSense 2.1 there are two versions of OpenSSL present, 0.9.8y and 1.0.1e!
    They are in different directory structures but path search precedence often calls the old version in preference to the new one depending on what you are doing.

    Before I found the second installation, I managed to successfully compile and link 0.9.8y with static support for the following engines:-

    dynamic
    4758cca
    aep
    atalla
    cswift
    chil
    nuron
    sureware
    ubsec

    Unfortunately, cryptodev and padlock were missing, but I have since found the sources elsewhere in the tarball. The OpenSSL 0.9.8y tarball is in a bit of a mess with outdated READMEs referencing files that have since been renamed and relocated in different folders. It is taking time sorting this out and I plan to feed my corrections back to the OpenSSL project if they are still relevant. I haven't looked at the latest sources yet but I think I just as well use the latest stable release now. I am confident that with attention to detail (reading all of the READMEs very carefully) OpenSSL can be built successfully to use both static and dynamic engines.

    I'm not sure what to do now. I have been slowly creating a 'fix' script to remove 0.9.8y and replace it with links to the new locations of 1.0.1e where appropriate. If anyone has used a hard coded reference elsewhere in pfSense, using links might not break it. I am going to proceed with fixing my own installation but I don't know anything about packaging a patch for pfSense for others to use.

    If the person or team responsible for OpenSSL on pfSense are reading this, I would prefer that they pick this up rather than me stepping on anyone else's toes. Alternatively, point me in the direction of the info for creating an official patch.





  • Thanks. That at least explains why there are two versions, but still doesn't explain why the ubsec engine support appears to be broken in both the FreeBSD 8.3 distribution (0.9.8y) and the pfSense 2.1 installation (1.0.1e).

    I'm going to take a break now, but tomorrow I will download the latest sources and transfer a built version of openssl laden with static engines to my test pfSense box.

    I'm going to strip out all of the old versions of openssl and see what happens to ssh. I'm finding it hard to believe that ssh doesn't work with 1.0.1e.


  • Rebel Alliance Developer Netgate

    OpenSSL need not support the card directly, so long as the OS/Kernel do, and it hooks into the crypto framework.

    See what it does for the ALIX glxsb:

    [2.1.1-PRERELEASE][admin@pfsense.localdomain]/root(1): /usr/local/bin/openssl engine -t -c
    (cryptodev) BSD cryptodev engine
     [RSA, DSA, DH]
         [ available ]
    (dynamic) Dynamic engine loading support
         [ unavailable ]
    (padlock) VIA PadLock: not supported
         [ unavailable ]
    [2.1.1-PRERELEASE][admin@pfsense.localdomain]/root(2): kldload glxsb
    [2.1.1-PRERELEASE][admin@pfsense.localdomain]/root(3): /usr/local/bin/openssl engine -t -c
    (cryptodev) BSD cryptodev engine
     [RSA, DSA, DH, AES-128-CBC]
         [ available ]
    (dynamic) Dynamic engine loading support
         [ unavailable ]
    (padlock) VIA PadLock: not supported
         [ unavailable ]
    

    Note the extra AES-128-CBC on the cryptodev line.



  • Thanks. I am beginning to realise that there is more than one way of doing this and not all of them are fully implemented.

    Using the command kldstat -v | more I can see that the kernel supports directly three crypto accelerators that I recognise.

    Id Refs Address    Size    Name
    1    8 0xc0400000 1083d78  kernel (/boot/kernel/kernel)
            Contains modules:
                    Id Name
                263 pci/ubsec
                126 pci/hifn
                238 pci/safe

    It also shows these:-

    412 cryptodev
        413 nexus/cryptosoft
                420 nexus/padlock

    The only loadable kernel modules that I can see are in /boot/kernel are:-

    aesni.ko
    glxsb.ko

    I have ubsec_load="YES" in /boot/loader.conf but I'm not sure if it is required for the kernel or the loadable kernel module (that I don't have on my pfSense box).

    My regular FreeBSD 8.3 installation on my development PC has /boot/kernel/ubsec.ko and I don't have a BCM95821 installed in it yet.

    I ran two OpenSSL benchmarks today on my test pfSense machine using:-

    ubsecstats
    /usr/local/bin/openssl speed
    ubsecstats
    /usr/local/bin/openssl speed -engine cryptodev
    ubsecstats

    The benchmarks are almost the same with only a small variation in results with and without cryptodev.
    Each ubsecstats report shows absolutely no packets going in or out of ubsec according to this tool.

    However, if I run a benchmark using cryptotest I do get some variation.

    ubsecstats before running what I think is a non accelerated benchmark.

    ubsecstats
    input 1901731920 bytes 1070795 packets
    output 1901731920 bytes 1070795 packets
    invalid 0 badsession 0 badflags 0
    nodesc 0 badalg 0 nomem 0 queuefull 0
    dmaerr 0 mcrerr 0 nodmafree 0
    lenmismatch 0 skipmisatch 0 iovmisalined 0
    noirq 0 unaligned 0 nomap 0 noload 0 nomcl 0
    totbatch 0 maxbatch 0
    maxqueue 1 maxqchip 1 mcr1full 0
    rng 2345258 modexp 0 moexpcrt 0

    cryptotest 10 1024
      0.003 sec,      20  3des crypts,    1024 bytes,  6530612 byte/sec,    49.8 Mb/sec

    ubsecstats run after the first test to see if packets have increased, 10 encrypts and 10 decrypts going through ubsec will show an increase of 20. I was expecting no increase but clearly there are 20 more which shows ubsec processed them according to this tool.

    ubsecstats
    input 1901767440 bytes 1070815 packets
    output 1901767440 bytes 1070815 packets
    invalid 0 badsession 0 badflags 0
    nodesc 0 badalg 0 nomem 0 queuefull 0
    dmaerr 0 mcrerr 0 nodmafree 0
    lenmismatch 0 skipmisatch 0 iovmisalined 0
    noirq 0 unaligned 0 nomap 0 noload 0 nomcl 0
    totbatch 0 maxbatch 0
    maxqueue 1 maxqchip 1 mcr1full 0
    rng 2349431 modexp 0 moexpcrt 0

    cryptotest -d ubsec 10 1024
      0.003 sec,      20  3des crypts,    1024 bytes,  6854083 byte/sec,    52.3 Mb/sec

    ubsecstats
    input 1901802960 bytes 1070835 packets
    output 1901802960 bytes 1070835 packets
    invalid 0 badsession 0 badflags 0
    nodesc 0 badalg 0 nomem 0 queuefull 0
    dmaerr 0 mcrerr 0 nodmafree 0
    lenmismatch 0 skipmisatch 0 iovmisalined 0
    noirq 0 unaligned 0 nomap 0 noload 0 nomcl 0
    totbatch 0 maxbatch 0
    maxqueue 1 maxqchip 1 mcr1full 0
    rng 2353246 modexp 0 moexpcrt 0

    I'm confused! It appears ubsec is being used by cryptotest automatically, presumably this is by the kernel but openssl benchmark appears to ignore it.


  • Rebel Alliance Developer Netgate

    OpenSSL will automatically pick an accelerator if you choose a cipher for which it advertises support. Check the output of the command I showed in my previous message.

    Since ubsec is in the kernel (no need to load it) so long as the card is present the results will not be valid. You'll have to pull the card, run the "without" tests, put the card in, and then run tests that will use the card.



  • This is with the BCM board installed…

    /usr/local/bin/openssl engine -t -c
    (cryptodev) BSD cryptodev engine
    [RSA, DSA, DH, DES-CBC, DES-EDE3-CBC]
        [ available ]
    (dynamic) Dynamic engine loading support
        [ unavailable ]
    (padlock) VIA PadLock: not supported
        [ unavailable ]

    I will pull the board tomorrow and repeat the test.


  • Rebel Alliance Developer Netgate

    Also try using -elapsed on the OpenSSL tests.  See the second sheet here for better info on crypto testing, though it's written about AESNI, you can apply the same tactics here for testing.



  • Thanks. Will do.



  • With the BCM board removed…

    /usr/local/bin/openssl engine -t -c

    (cryptodev) BSD cryptodev engine
    [RSA, DSA, DH]
        [ available ]
    (dynamic) Dynamic engine loading support
        [ unavailable ]
    (padlock) VIA PadLock: not supported
        [ unavailable ]

    I cannot use cryptotest for a comparison as the output above shows that without the BCM board I don't have a compatible algorithm where algorithm is one of:
        des 3des (default) blowfish cast skipjack
        aes (aka rijndael) aes192 aes256 arc4

    So using openssl instead for comparisons, testing without the BCM board first.
    openssl speed

    sign    verify    sign/s verify/s
    rsa  512 bits 0.001150s 0.000122s    869.4  8214.2
    rsa 1024 bits 0.005187s 0.000296s    192.8  3377.4
    rsa 2048 bits 0.031142s 0.000894s    32.1  1119.2
    rsa 4096 bits 0.196700s 0.003057s      5.1    327.1
                      sign    verify    sign/s verify/s
    dsa  512 bits 0.000893s 0.000970s  1119.7  1031.2
    dsa 1024 bits 0.002461s 0.002825s    406.3    354.0
    dsa 2048 bits 0.008048s 0.009546s    124.3    104.8

    With the BCM board installed.
    openssl speed -engine cryptodev

    sign    verify    sign/s verify/s
    rsa  512 bits 0.000350s 0.000038s  2854.3  26445.7
    rsa 1024 bits 0.000555s 0.000053s  1801.6  18825.8
    rsa 2048 bits 0.001310s 0.000107s    763.5  9315.1
    rsa 4096 bits 0.647222s 0.003194s      1.5    313.1
                      sign    verify    sign/s verify/s
    dsa  512 bits 0.000389s 0.000415s  2571.9  2410.9
    dsa 1024 bits 0.000584s 0.000511s  1711.0  1955.9
    dsa 2048 bits 0.000760s 0.000684s  1316.0  1461.9

    From my calculations there is a worthwhile performance increase using 512, 1024, 2048 certs, but a performance degradation using 4096 certs. 2048 bits is almost 24 times faster on signs and more than 8 times faster on verify. The BCM board was also tested in a regular 32 bit 33Mhz PCI slot when it should be in a 64 bit 66Mhz slot. CPU is 2.6Ghz Celeron.

    rsa 512 bits   3.28      3.22
    rsa 1024 bits   9.34      5.57
    rsa 2048 bits 23.79 8.32
    rsa 4096 bits 0.29        0.96

    dsa 512 bits   2.30        2.34
    dsa 1024 bits   4.21        5.53
    dsa 2048 bits 10.59 13.95

    So my conclusion is that the ubsec kernel module is working and I will be changing all my rsa and dsa certificates to 2048 bits. I will also be looking for a pair of Xeon based systems with PCI-X slots now.