Xeon-D integrated Quickassist, up to four 10Gb launching, possibly called 15x3N



  • It looks like they are finally going to integrate Quickassist into the Xeon D chips which were launched in 2015 and had additional variants released in 2016  They also will include up to four 10Gbe NICs now which will be great for router/networking purposes and power/cost savings:
    https://newsroom.intel.com/newsroom/wp-content/uploads/sites/11/2017/02/Xeon-D-Atom-C-QAT-25GBE-Fact-Sheet.pdf
    https://www.servethehome.com/new-intel-xeon-d-networking-chips/
    http://www.anandtech.com/show/11152/intel-announces-xeon-d1500-network-series-socs-with-quickassist-four-10-gbe-ports

    I posted about the 15x8 variants which had the DPDK but turned out to need a separate QAT accelerator for that functionality which really defeated the cost benefits of the Xeon-D versus the Atom Rangely chips having Quickassist built-in.  Here are posts about that and confirming Intel QAT still required the separate expensive accelerator when you read the Intel fine print:
    https://forum.pfsense.org/index.php?topic=103211.0
    https://forum.pfsense.org/index.php?topic=110783.0
    https://forum.pfsense.org/index.php?topic=108255.30

    On Intel's site it is showing the Xeon-D 1513N unlisted MSRP, 1523N $256 MSRP, 1533N $590 MSRP, 1543N unlisted MSRP, and 1553N $855 MSRP launching Q3 '17 so I believe those are the newer variants since there they haven't announced any other new variants.
    http://ark.intel.com/products/series/87041/Intel-Xeon-Processor-D-Family

    Keep in mind that these are BGA SoC so you're probably looking at $100-$250 more for the actual chip soldered on the motherboard with the PHY part of the NICs based on how much the other Xeon-D chips cost.  Supermicro is probably the best bet to offer these since I haven't seen any other manufacturers carrying the 15x8 variants.
    https://www.supermicro.com/products/motherboard/Xeon3000/#1667

    I suppose it is also worth mentioning that these use Broadwell architecture, although that is still 14nm like Skylake/Kaby Lake/Coffee Lake.  It really isn't a factor since these have special features for other purposes which aren't really relevant to those architecture improvements.  They may not even update these until they go to a smaller lithography with Cannon Lake for better power efficiency since they are meant to be lower cost lower powered networking and storage oriented server chips.

    I found the best sources whenever they get around to be carried retail were these two sites:
    http://wiredzone.com/
    http://www.serversdirect.com/
    I'd appreciate any other good places to get these Xeon-D chips if you do know of better ones.  I think most are probably being ordered by enterprise customers through their B2B distributors in bulk so it is harder to find them individually retail.

    It took about six months or so for the previous ones to show up where us commoners could order them.  I imagine most of the initial offerings are sucked up by major data center customers.  Hopefully, we will hear more soon about specific boards for these and it won't take too long to actually be able to buy one.  I never ended up getting the Xeon-D based on the costs, availability, and lack of QAT, but if these get around to being offered at a reasonable price, I will get the 4 or 6 core variant.



  • I'm curious what you think QAT will do for you.



  • For me personally, probably mostly VPN, SSL, and 802.1x improvements.  For the deployments, I might consider it for in place of really expensive enterprise stuff where they have in some cases multiple VPNs, SSL, VDI/VM, compression, and other things going on with gigabit fiber uplinks, I imagine it would make a significant difference versus having to pay for expensive hardware to handle that.  PFSense supports it in their OS and hardware so I don't really see a problem with trying to get the most features for the least amount of money.  If the costs don't end up being much different than the existing variants, then why not?  Also, for my personal setup, I'd like to get as close to 1Gb routing with QOS enabled if possible which any instruction sets that more quickly handle traffic and router functions will help with along with the additional power of these chips versus the older Rangeley chips.



  • QAT isn't going to give you multigigabit openvpn, and you can already hot the performance target now with relatively cheap hardware if you use ipsec. Hence my curiosity: people seem to treat QAT more as a piece of magic than a piece of technology.



  • I'm not treating it as magic.  Just one more thing that PFSense supports which in addition to other things like AES-NI or VT-x will help improve performance dependent on use case.  I agree for my personal use it doesn't really have much application, and I didn't think the cost of a QAT adapter was remotely justifiable.  I'm sure in some enterprise cases it helps with huge loads which is why they can justify getting the QAT adapter.  My main purpose for using it myself would be to test its performance for other uses.  In this case with QAT being integrated into the SoC like it was on Rangeley, assuming the costs aren't much different, what is the harm?  It's just one more tool in the server hardware chest, and it will be at a lower power/price point than before.

    Also, having 4 10Gb links in the SoC would be great for lower power and costs when you already have a use for that many routeable uplinks or subnets.



  • QAT won't do what you think it will do. QAT+DPDK will definitely help with plain L2 and some L3, but it won't do anything brilliant for your crypto.
    I think the best way to describe it, is that it will allow FreeBSD (and thus pfSense) to push more packets per second.


  • Netgate

    @VAMike:

    QAT isn't going to give you multigigabit openvpn,

    Not without a rewrite.

    and you can already hot the performance target now qwith relatively cheap hardware if you use ipsec.

    Wrong. Not without a heavy rewrite, which is 3.0.

    Do you have ANY actual experience with the subject, or are you yet another armchair quarterback?



  • @jwt:

    @VAMike:

    and you can already hit the performance target now with relatively cheap hardware if you use ipsec.

    Wrong. Not without a heavy rewrite, which is 3.0.

    Do you have ANY actual experience with the subject, or are you yet another armchair quarterback?

    Bidirectional gigabit ipsec isn't too hard with aes-gcm on Linux. I honestly haven't tried it on pfsense but if it can't hit that it's not because the hardware can't do it. Higher speeds, pushing toward and beyond 10gbps are a whole different story, but the context was gigabit wan.

    It's also worth pointing out that "cheap" also depends on context, and the perspective of an oem looking at a bom is different from a consumer because the pricing of bare boards using Intel's communication chips series has been gruesome.



  • At first MOC9, nice and fine thread and articles you was collecting, I am also looking forward to that 3rd generation of
    Intel Xeon D-15xx platform likes many other too.

    It looks like they are finally going to integrate Quickassist into the Xeon D chips which were launched in 2015 and had additional variants released in 2016  They also will include up to four 10Gbe NICs now which will be great for router/networking purposes and power/cost savings:

    The main impact is right to see all is present there this time!

    • QAT
    • AES-NI
    • DPDK & SPDK will be both matching at or to this new platform or SoC

    It took about six months or so for the previous ones to show up where us commoners could order them.  I imagine most of the initial offerings are sucked up by major data center customers.  Hopefully, we will hear more soon about specific boards for these and it won't take too long to actually be able to buy one.  I never ended up getting the Xeon-D based on the costs, availability, and lack of QAT, but if these get around to being offered at a reasonable price, I will get the 4 or 6 core variant.

    I was only one step before buying a SYS-E300-D8 platform from Supermicro, but then I read an article at serverthehome.com
    about the third generation of this Intel Xeon D-15xx platform and will wait until that will launched.

    I'd appreciate any other good places to get these Xeon-D chips if you do know of better ones.  I think most are probably being ordered by enterprise customers through their B2B distributors in bulk so it is harder to find them individually retail.

    Here in Germany you will be able to get boards of the 1st and 2nd generation, bare bones and spare parts from several shops:

    and others.

    I'm curious what you think QAT will do for you.

    That can be on two different cases:

    • On the first it can be, but if there will be many Linux and BSD based servers in a network that are using that QAT
      adapters, pfSense could be a bottleneck but with that adapters from Netgate and if QAT is active working in pfSense
      this could be driving around.
    • And at the second if AES-NI is speeding up only the IPsec VPN, it could be that if on both VPN ends
      pfSense is working, the entire traffic could be compressed by QAT that is running through that tunnel,
      then it works direct. Surely this will be not for all customers and users interesting and perhaps it is not
      hitting your personal interest or willing or needs, but interesting enough to follow that thinking way for
      many others.

    QAT isn't going to give you multigigabit openvpn,

    For sure this might be, but if it will be present on both ends of another VPN such IPsec
    it could compress the data through that tunnel more then now, for sure not interesting
    for OpenVPN users.

    and you can already hot the performance target now with relatively cheap hardware if you use ipsec.

    Someone named gonzopancho was telling at reddit that he was able to push ~470 MBit/s over a 1 GBit/s symmetric
    internet line, using a SG-4860 and AES-NI, why cheap and which cheap hardware will do that?

    Hence my curiosity: people seem to treat QAT more as a piece of magic than a piece of technology.

    I think I know what you are want to tell us here, but please accept that other users are keeping an watching eye on
    that feature or option likes QAT, perhaps for future versions of pfSense and not yet, what is wrong with that?

    QAT won't do what you think it will do.

    It is good for compression data, and I personally expect anything else from it.

    QAT+DPDK will definitely help with plain L2 and some L3, but it won't do anything brilliant for your crypto.

    If you will be able to receive 3x more TCP/IP packets as now, on the same hardware we are talking about, DPDK can be
    a nice feature or be interesting for many of us. For sure there will be many different methods to archive more throughput
    and yes it is also pending on what services or protocols are in usage, I have no problems with that to accept, but if
    there will be sometimes something likes netmap- fwrd, try-forward or fast-fwrd I would be happy to use it or
    give it a try.

    All in all if there will be a totally new written pfSense code or version such version 3.0 is announced to be, it might be
    really interesting to see how it is working and using the one or other new feature, perhaps running on all present cpu
    cores or using HT on top of this, a cpu multicore written igb(4) driver or perhaps multicore using PPPoE part in pfSense.

    But to come closer to other points what can be, or what will be nice to see or watching out for, is this not allowed to
    talk about? Is a forum not the real point of a software such pfSense is? Fir sure under the right topic and perhaps in
    general discussion part of this forum, but why not?

    @VAMike
    Please don´t get me wrong here, also perhaps based on my poor English language skills, but there are different
    groups of users and customers and yes they have also different wishes, needs, expectations and they are all
    looking perhaps let us say for total different new features, options and things they want to know or see or get
    in pfSense.



  • @BlueKobold:

    Someone named gonzopancho was telling at reddit that he was able to push ~470 MBit/s over a 1 GBit/s symmetric
    internet line, using a SG-4860 and AES-NI, why cheap and which cheap hardware will do that?

    An SG-4860 has a specific purpose, it's not a CPU monster. The AES-NI implementation on the silvermonts is pretty bad. If you're looking specifically for AES-NI performance, it's not the right platform to talk about. (It has other advantages, I have some silvermont based systems myself that work very well, but don't compare apples and oranges.) In most of the cases where a rangeley number was impressive, it was impressive in that it could hit that number with a 15W (or whatever, depends on the chip) TDP, not in absolute terms. If you have a specific thermal or power budget that's important, if you don't you it isn't.

    I think I know what you are want to tell us here, but please accept that other users are keeping an watching eye on
    that feature or option likes QAT, perhaps for future versions of pfSense and not yet, what is wrong with that?

    Nothing's wrong with that, and I can hardly tell people not to do what they want. But I'd like people reading along to know that it's not a magic bullet and that they should carefully consider both their requirements and the available hardware, and make an informed decision about whether future QAT products should affect their current purchasing decisions. There have been a lot of people who have been very confused over the past few years that their QAT rangeleys aren't openvpn monsters, because so much of the initial coverage of the silvermont platforms was so misleading (mostly because, like the current coverage of the D series QAT, they just say "it has QAT to make crypto fast" and lack real, contextualized numbers).

    It is good for compression data, and I personally expect anything else from it.

    That is excellent if you have a workload that benefits from that. I can't personally think of cases where I have a large volume of compressible data going over the wire, but that is certainly why site specific requirements are so important.

    But to come closer to other points what can be, or what will be nice to see or watching out for, is this not allowed to
    talk about?

    I'm very confused about who's being prevented from talking about what. I guess it's just one of those perennial pastimes on the net that someone has to be oppressed? Or are you trying to prevent me from talking? It's super unclear.

    @MOC9:

    My main purpose for using it myself would be to test its performance for other uses.

    You need to be very, very careful about this because "QAT" is a marketing term which covers some very different (and incompatible) technologies. It's very likely that conclusions you draw from one implementation will be misleading if applied to a different implementation.

    In this case with QAT being integrated into the SoC like it was on Rangeley, assuming the costs aren't much different, what is the harm?

    Everything's a tradeoff. In the case of rangeley, to get the QAT you took ~10-20% hit on CPU clock (no turbo) and paid ~20-30% more for the equivalent CPU in order to get the QAT. If you got good use over the past 4 years from rangeley QAT then that was a good tradeoff. If you didn't get much use from the QAT, it was a bad tradeoff. (This is based on the RRP of the chips themselves, some very, very strange things happened to the pricing once they were soldered to boards and hit the marketplace; I blame intel's parts availability for a lot of that, but that's something to be concerned about with any new stuff as well–especially watching the fiasco they've made of the denverton release.)

    n.b., none of this is at all relevant for people buying an appliance like the netgear SG's–in that case you're essentially buying a black box with guarantees from the company, not spec'ing a cpu from a long list at retail (and the retail component costs don't translate directly to what it costs to build the appliance). The context of my comments is the guy who buys a c2758 board because he read that it's good and has QAT and doesn't understand why its openvpn performance is inferior to an i3 at a quarter of the cost. (And the other benefits of rangeley, like long-term parts availability, are completely irrelevant for the guy buying one board at retail but important for someone designing an appliance.) It's quite possible for something to be a great part for someone building an appliance and a stupid waste of money for someone buying a board at retail, because the markets are so different.

    Don't get me wrong, there are definitely situations where I'd heartily recommend a QAT solution--it's just that articles with QAT cheerleading very rarely get into the details of when and how it might be useful.



  • @VAMike
    After reading your last post it is now more clear to me why you are complaining in each thread against the presence of QAT
    in pfSense hardware. For sure all peoples and users who where thinking to get a rocket fast OpenVPN machine based on the
    presence of QAT in their new hardware will be fairly a bitt disappointed, but each CPU core can hold or drive a OpenVPN tunnel
    and yes this is never real multi-cpu core usage but better to let run all the tunnels over one single CPU core alone.

    ….and paid ~20-30% more for the equivalent CPU in order to get the QAT.

    For sure that is right, but if I am looking at the Netgate or pfSense site, it must be something why they are
    staying to use this Intel Xeon D-15xx and QAT based hardware, or am I wrong with that and I was mislead
    only by my own? But to read then something like that thread here and you can get all in one platform was
    let me thinking "this must my next hardware platform for pfSense for sure"! But often there will be also a
    second feeling that tell you is it right or is it wrong? And if someone opens a thread such this here, I feel
    once more again that this could or must be the right road to walk on. Who knows?

    I'm very confused about who's being prevented from talking about what.

    This was only pointed to the circumstance that each forum thread about QAT and pfSense I was watching, you were
    against that or I was thinking you were speaking against the presence of QAT in that or this hardware. Nothing more!

    I guess it's just one of those perennial pastimes on the net that someone has to be oppressed?

    From my point of view it was more in that direction that even if someone or more were talking about QAT you were
    running against this "wall" or argument that this will be a nice to have thing. But as said once more again after your
    last post this is now more clear and acceptable.

    Or are you trying to prevent me from talking? It's super unclear

    I will never do something like this, not to you and to no one else here and everywhere! I am only a guest here!
    I think mostly peoples could misunderstood things based on my poor English language skills.


Log in to reply