Went physical to virtual, AES is having no effect on OpenVPN performance



  • So I went ahead and virtualized pfsense (in ESXi 6, with NICs passedthrough) and everything as far as I can tell functions flawlessly. However, I am noticing a strange problem with OpenVPN performance post the P-to-V. Here are the specs for servers.

    Server 1:
    Lenovo RS140
    E3-1225v3
    2x 8GB DDR3 ECC UDIMM
    X520-DA2

    Server 2:
    ASRock EP2C602-4L/D16
    2x E5-2670
    16x 8GB DDR3 ECC RDIMM

    I have OpenVPN server set up on both, and Server 2 is set as a client to connect to 1 as site-to-site. Server 1 has a 150/150 link, and Server 2 is 200/20. Prior to virtualizing, I was able to pull stuff to Server 2 side at about link speed of Server 1. After virtualizing, it's now stuck at about ~40Mb. I looked around on pfsense forums, and the only options that could be related to OpenVPN performance is aesni.ko on/off, cryptodev on/off and ip.fastforwarding=0/1. I've tried combinations of all 3, and it's having zero effect on the performance. CPU load during transfers is about 3-4% on Server 1 and about 10% on Server 2. I also confirmed that on CLI level, both servers seems to be seeing AES support from CPU properly, and loading the aesni.ko module is indeed allowing the proper ciphers to be available to cryptodev. I was even able to run commands to measure the encryption performancing using AES-128-CBC. I'm not really understanding what else could be causing this performance cap.