• Hi everyone,

    I'm looking into finding a build for my WAN connection which is 1000d/200u over PPPoE. I've spent tons of time browsing this forum (and others) and am looking for help to decide which path to pursue.

    1. network

    First, with respect to network is clear that Intel NICs are highly recommended. However there are a variety of them out there and am unclear which one is worth it.
    There are is the dual 1000 PT, available second hand  or the variety of i210/211/217/218/219 or 82574 & co. The 1000 PT is a server controller, somewhat old and requires a PCIe (not available on all mATX boards) while the latter seem to be part of desktop controllers, typically on-board and with some impressive low watts.
    There are all kind of posts about the features of one vs the other (rx/tx  queues, multiple vs single IRQs, etc…) but I'm unclear which one offers better reliability and performance.

    Is there any true advantage of using one over the other? The Intel ARK doesn't seem to be conclusive.

    2. cpu

    PPPoE is taking on the CPU. I see a lot of small, attractive boards with Atom based CPUs (the name has changed but the architecture remained the same) such as J1900 or N3150. While low power, they seem unfit to handle gigabit PPPoE, especially since currently the encapsulation is still single-core. i3 sounds like a more appropriate candidate (sure an i5 would be even better but the cost as well as the TDP will increase significantly).

    3. availability

    Last but not least, as somebody who's based in Eastern Europe, grabbing a hold of the actual hardware can be sometimes a challenge. In practice this means mainstream products (consumer) as oppose to exotic hardware (looking at you Supermicro) simply based on availability.

    Thanks,


  • I have the same setup u are looking for.
    works just great.

    Supermicro A1SRi-2758F + 8gb ram ECC + ssd + dedicated pciE gigabit controller for pppoe.


  • Supermicro A1SRi-2758F + 8gb ram ECC + ssd + dedicated pciE gigabit controller for pppoe.

    What is the dedicated PCIe GB LAN controller for PPPoE in your case?


  • Hi louf,

    Not sure if you came across my post: https://forum.pfsense.org/index.php?topic=104282.msg587496

    From my research and experiments, you need an older system that uses an "em" driver card not an "igb" card. https://kdemaria.wordpress.com/2015/04/22/how-to-configure-pfsense-2-2-2-for-centurylink-gigabit-seattle-edition/#comment-300

    You might have bad luck with PPPoE and gigabit with an igb card because of this issue: https://redmine.pfsense.org/issues/4821

    I recently bought and tested a SG-2440 from Pfsense in addition to self-built machine and had the same issue. The SG-2440 is going to the office where there is no PPPoE so no worries there.

    I posted a bounty for $500 to fix the underlying issue - https://forum.pfsense.org/index.php?topic=108280.msg603145#msg603145


  • You might have bad luck with PPPoE and gigabit with an igb card because of this issue: https://redmine.pfsense.org/issues/4821

    PPPoE is only using one CPU cpre at the WAN interface actual, and I think personally it is more
    pointed to that circumstance that some CPUs would not be strong enough to route 1 GBit/s at
    the WAN interface. Prerhaps it might be also driver related, but this problems was set to low
    and it would reinspected if FreeBSD version 11 is on the its way or some times before.

    I recently bought and tested a SG-2440 from Pfsense in addition to self-built machine and had the same issue. The SG-2440 is going to the office where there is no PPPoE so no worries there.

    So if the SG-2440 is used then in the office, and there will be the same igb(4) driver in the game and
    you are reporting there will be then no problems anymore, the driver is in my opinion not really so hard
    involved as you were thinking. Could this be?

    I posted a bounty for $500 to fix the underlying issue - https://forum.pfsense.org/index.php?topic=108280.msg603145#msg603145

    This is not really necessary as I see it, because there was told something about a patch that is available
    so why you would spend some coins on this then? There is a patch suggested there.

    Translation:
    –---------------------------------------------------------------------------------------------------------
    assigned to Jim Thompson
    priority was changed from Normal to Low
    as target was changed from version 2.3 to future
    Fixing this likely requires an in-kernel RSS (Toeplitz) implementation. Such a thing is coming for
    FreeBSD (Adrian is working on it for the upper layers of the stack), but it's going to be a while
    before it's ready to interface to netisr.

    Priority dropped to "low". Will review when we're based on 11.

    So I really think it is better to change the single CPU core usage on the PPPoE part more than patching
    the driver igb(4) or recode it. Perhaps with the next version of pfSense the version 2.3 and netmap-fwd
    you will be able to route a full GBit/s at the WAN port, so please rethink or think over about the made
    bounty you opened.


  • @BlueKobold:

    Supermicro A1SRi-2758F + 8gb ram ECC + ssd + dedicated pciE gigabit controller for pppoe.

    What is the dedicated PCIe GB LAN controller for PPPoE in your case?

    Intel Gigabit CT EXPI9301CTBLK


  • @nikkon:

    @BlueKobold:

    Supermicro A1SRi-2758F + 8gb ram ECC + ssd + dedicated pciE gigabit controller for pppoe.

    What is the dedicated PCIe GB LAN controller for PPPoE in your case?

    Intel Gigabit CT EXPI9301CTBLK

    … So you're not using the nice Intel i354 NICs that are onboard and instead decided to use a 82574L (which is a entry-server part at best, and better described as desktop chip)?


  • yeah…i decided to use this cz after testing with the wan in igb0 -> first lan on the mobo the performance was lower.still looking for an alternative on pciE.
    need to change it soon...but i have no much to chooce from


  • @nikkon:

    @BlueKobold:

    Supermicro A1SRi-2758F + 8gb ram ECC + ssd + dedicated pciE gigabit controller for pppoe.

    What is the dedicated PCIe GB LAN controller for PPPoE in your case?

    Intel Gigabit CT EXPI9301CTBLK

    Hmmm… you get gigabit pppoe with this?


  • i only get ~550 Mb


  • the limitation is on my wan interface.not the system specs


  • the limitation is on my wan interface.not the system specs

    Hm, and why it is so? Do you think that with another NIC you will recive more throughput than now?

    i only get ~550 Mb

    What is the entire or full speed of your Internet connection?

    Did you try out high up or narrow down the mbuf size?
    Did you try out enabling PowerD (hi adaptive or adative mode)?

    Ok if there will also Snort, Squid, ClamAV and other packets are installed and working
    it all narrows down also the entire throughput as I will assume.


  • isp provides 1000 Mbps
    it works for real to max 980 Mb
    http://www.rcs-rds.ro/internet-digi-net/fiberlink?t=internet-fix&pachet=digi_net_fiberlink_1000
    yes i have snort, dansguardian, squid on top


  • it works for real to max 980 Mb

    This is really good! Plus the TCP/IP overhead you will be easily sorted with real 1 GBit/s, that is really nice!

    yes i have snort, dansguardian, squid on top

    I am pretty sure that this three packets are shorten the entire WAN throughput and that it is not really
    owed to your Intel NIC at the WAN port, it is more that the packets "eating" much CPU power.


  • i'm gonna try using a Intel i350-T4 adaptor for wan and see if there is any difference…i expect it to be.
    CPU looks verry low usage....always....even when i run the python script for speed test...it goes to max 35% per core.


  • As per my signature below, I get 717Mbps from the same provider, with Supermicro C2758, and using one of the onboard ports as wan.
    I don't get 980 with this hardware.
    I only get around 980 Mbps with their CPE router provided, instead of pfSense, but I don't want to use that.


  • i'm gonna try using a Intel i350-T4 adaptor for wan and see if there is any difference…i expect it to be.

    Really? Is there a so great difference between the Intel i354 onBoard NICs and the Intel i350-T4?
    Both seems to be pretty new and also server grade hardware, or am I wrong with this?

    As per my signature below, I get 717Mbps from the same provider, with Supermicro C2758, and using one of the onboard ports as wan.

    Perhaps you get more or a higher throughput pending on that you are not using on top of pfSense
    all of the following packets like Snort, Squid and DansGuardian? Only perhaps I mean.

    I don't get 980 with this hardware.

    This is a little bit odd or curious because it is the same ISP and perhaps the same Internet connection with
    1 GBit/s of speed. And besides getting 980 MBit/s + count the TCP/IP overhead on top  might be a real
    1 GBit/s line that is delivered to you.

    I only get around 980 Mbps with their CPE router provided, instead of pfSense, but I don't want to use that.

    But ok this routers are doing the whole work in silicon by using an ASIC or FPGA and normally
    there will be also no firewall rules in that game. And running on this router some stuff likes
    IDS, HTTP Proxy and AVscan it would never be able  to reach the full 1 GBit/s too I would imagine.


  • It must be that damn single-core pppoe bug, that limits my speed through pfSense. When at 717Mbps, usage is at about 13% - which corresponds to the load of only one single CPU core (out of 8 cores) on C2758.
    No Snort, Squid and DansGuardian, but about 10 vlans behind it, and 3 site-to-site OpenVPNs, +some road warriors.


  • @robi:

    It must be that damn single-core pppoe bug, that limits my speed through pfSense. When at 717Mbps, usage is at about 13% - which corresponds to the load of only one single CPU core (out of 8 cores) on C2758.
    No Snort, Squid and DansGuardian, but about 10 vlans behind it, and 3 site-to-site OpenVPNs, +some road warriors.

    Yes it is!
    I got the same speed without Snort, dansguardian and squid or pfblokerNG.

    I was tested also an ASA 5506 and the max i got was 325Mb in the same line.
    for some reason…on my onbord controller i got slower speed.Will keep the i350-T4 pciE for wan and the other 4 for lan.
    2x for lan --> connected to my distribution switch (microtik CCR1009-8G-1S-PC)
    2x for NAS --> connect to NAS lagg

    I may use some ports from the i350-T4 to connect my ESXi Server...maybe...

    btw: do we know for sure that 2.3 will solve the PPPoE issue ?


  • It must be that damn single-core pppoe bug,

    If they get solved this I would imagine 70 % of all users will be happy.

    and 3 site-to-site OpenVPNs, +some road warriors.

    As I am informed each OpenVPN tunnel is using one CPU core. So this could also
    narrow down the entire throughput a bit more as we could imagine. Or am I wrong with this.

    btw: do we know for sure that 2.3 will solve the PPPoE issue?

    Yep if so it many customers would be sorted right at one touch. I was also lurking on
    the new Xeon D-15x8 network accelerated platforms, but they are not fully launched till
    today so I have to wait longer. But it is likes it is, their is the NVMe M.2 SSD the problem
    to get this SSD type working flawless as reported here in the forum.


  • @BlueKobold:

    It must be that damn single-core pppoe bug,

    If they get solved this I would imagine 70 % of all users will be happy.

    Agree…

    @BlueKobold:

    and 3 site-to-site OpenVPNs, +some road warriors.

    As I am informed each OpenVPN tunnel is using one CPU core. So this could also
    narrow down the entire throughput a bit more as we could imagine. Or am I wrong with this.

    Well I don't really see which process goes to which core, but I really hope OpenVPN procresses don't stick all to the same single core as pppoe… The operating system should take care to distribute different processes to different available cores of the CPU.
    So far I didn't have problems with this, as traffic inside these tunnels is only limited to inter-site intranet traffic, which is minimal.

    @BlueKobold:

    btw: do we know for sure that 2.3 will solve the PPPoE issue?

    Yep if so it many customers would be sorted right at one touch.

    As far as I read the bug reports, they are not so optimistic. Where did you see a clear statement that this is going to be fixed in pfSense v2.3???


  • As far as I read the bug reports, they are not so optimistic. Where did you see a clear statement that this is going to be fixed in pfSense v2.3???

    There is not clear statement out! @nikkon was asking for what we can be sure on this that it will be
    solved out in the version 2.3 and I was answering if they get it to work, many users will be sorted with
    one touch.


  • @nikkon @robi I have RDS as well. Too bad PPPoE is an issue since one of the reasons for moving to it was to get away from the proprietary wifi routers and their 'magic'/'hardware' NAT without sacrificing speed. I wonder if there are any PPPoE bridges out there - use the ISP box to handle PPPoE and pfsense for routing (maybe through some virtual lan or such)?

    @nikkon where in RO did you get supermicro? I don't see a lot of options.

    @sudonim interesting comment about em vs igb driver. Initially I went with em but after reading the comments about it being old and igb being newly written and supported, I decided to go for the respective Intel NICs, basically 82575 and 82576 - on paper they look a lot beefier than 82571. Can you expand on why em is better than igb? Thanks!


  • I got my both supermicro mobo's from Elko
    First one was an J1900. Both ware planned for pfsense :)


  • How big was the CPU difference after the upgrade?


  • Same goes to this thread here..
    Would you run iperf test first ?
    So that at least you roughly know what your hardware is capable of.

    Some good ref on how user test his hardware so that he understand what exactly his hardware is capable of:
    https://forums.openvpn.net/topic15861.html


  • sory for my late asnwer :(
    C2758 is much faster than J1900, firs of all is octo-core vs quad, still if you don't use openvpn/ipsec only the difference in pfsense is not much…notable but not so big.In my case, with pppoe cpu matters when i use the full bandwith downloading stuff.

  • Rebel Alliance

    You could go completely overboard with a Supermicro X10SRM-F and a Xeon E5-1620 v3 ;)

    I think this setup could even handle surricata-inline on 2.3 at 1GBe. Will try as soon as the X10SRM with 10GBe is available.


  • @Perforado:

    You could go completely overboard with a Supermicro X10SRM-F and a Xeon E5-1620 v3 ;)

    I think this setup could even handle surricata-inline on 2.3 at 1GBe. Will try as soon as the X10SRM with 10GBe is available.

    Suricata will be much more easy to handle!!! it's multithreading application.On the other side Snort is single and doesn't use the advantage of multi core CPU's
    no clue if suricata will support PPPoE ?


  • Is using em driver better than igb for pppoe connections?


  • have no clue…does the driver matters?


  • See this previous post (has some nice links):
    @sudonim:

    From my research and experiments, you need an older system that uses an "em" driver card not an "igb" card. https://kdemaria.wordpress.com/2015/04/22/how-to-configure-pfsense-2-2-2-for-centurylink-gigabit-seattle-edition/#comment-300

    You might have bad luck with PPPoE and gigabit with an igb card because of this issue: https://redmine.pfsense.org/issues/4821