IPSec bandwidth between two sites cycles between high and low

  • Hello, I have two sites each with a 1Gbps symmetric link connected over a low-latency (< 3ms) IPSEC VPN. Hardware is as follows:

    A. Intel(R) Atom(TM) CPU C3858 @ 2.00GHz (16 GB RAM) - AES-NI enabled
    B. Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz (16 GB RAM) - AES-NI enabled

    I'm noticing a strange bandwidth pattern that I can't quite figure out. Throughput between sites peaks at ~100MB/s, then suddenly drops to ~50 MB/s, and climbs back up. Then repeats.

    IPSec Bandwidth

    When I download a file directly via Public IP from one site to the other, I achieve 110 MB/s. QoS, IDS/IPS are not enabled.

    Any ideas on what this could be?