PPPoE with igb on APU2



  • Hello people,

    After having pfsense running as my router virutally for nearly over a year I thought is was time to upgrade a little bit.
    So I bought an APU2. At the same time I changed from ISP, my new ISP uses PPPoE instead of normal routing.
    After installing everything I found out that my newly installed APU couldn't reach the speed that the ISP Fritzbox could reach. (With less cpu power and less ram)

    So I started googling. And it seems that my Intel NIC (i210AT) in combination with Freebsd and the igb driver couldn't handle PPPoE traffic that well.
    There also was a fix out, but the link was death.

    I hope you guys can help me a little bit because I'm starting to get a little desperate.

    Details:

    • APU2B4 with i210AT Intel NIC using igb

    • XS4ALL 500mbit Fiber connection, currently reaching maximum 300mbit Down and around 450mbit up. (Fritzbox is doing 490/490)

    • It's not the same as this issue: https://redmine.pfsense.org/issues/4821
      Because it uses all of its queues

    dev.igb.0.queue0.tx_packets: 107933
    dev.igb.0.queue0.rx_packets: 1662157
    dev.igb.0.queue1.tx_packets: 80902
    dev.igb.0.queue1.rx_packets: 1627566
    dev.igb.0.queue2.tx_packets: 85285
    dev.igb.0.queue2.rx_packets: 16160
    dev.igb.0.queue3.tx_packets: 328231
    dev.igb.0.queue3.rx_packets: 88380
    dev.igb.1.queue0.tx_packets: 1584851
    dev.igb.1.queue0.rx_packets: 35034
    dev.igb.1.queue1.tx_packets: 1622082
    dev.igb.1.queue1.rx_packets: 14893
    dev.igb.1.queue2.tx_packets: 26441
    dev.igb.1.queue2.rx_packets: 28279
    dev.igb.1.queue3.tx_packets: 73658
    dev.igb.1.queue3.rx_packets: 323706
    
    
    • I found this peace on the freebsd wiki but the link in it is broken:

      Unfortunately, RSS is usually capable of hashing IPv4 and IPv4 traffic (L3+L4). All other traffic like PPPoE or MPLS or .. is usually received by queue 0.
      This is bad, but even worse is that e1000 (and maybe others) unconditionally sets flowid to 0 effectively causing later hashing (by netisr, of flowtable, or lagg, or ..) to be skipped (patches for: igb).
      Patch adds dev.XXX.Y.generate_flowid sysctl which needs to be set to 0 on links with non-ip traffic.



  • i will try to upgrade apu's bios

    i have the apu2c4 and my PPPoE works great



  • Hello people,

    Hey,

    After having pfsense running as my router virutally for nearly over a year I thought is was time to upgrade a little bit.

    Ok but would you please tell us first some points to be clear to give you the right answers.

    • How fast is your Internet speed?
    • From where location you are? (US,CA,UK,DE,….)

    So I bought an APU2. At the same time I changed from ISP, my new ISP uses PPPoE instead of normal routing.

    First trap you are sitting in! Together with PPPoE, pfSense is using and running the whole WAN
    part only over one single CPU core and virtually mostly on all cores available in the system!

    After installing everything I found out that my newly installed APU couldn't reach the speed that the ISP Fritzbox could reach. (With less cpu power and less ram)

    The whole workload what the CPU and RAM from the AVM FB is doing isn´t like the one from pfSense
    this at first, the Fritz!OS is Linux based and the entire workload is supported by an ASIC/FPGA inside
    of the AVM FB and the pfSense firewall is performing out also firewall rules on top and this is not the
    really same thing you are talking about here. pfSense is a x86 based firewall that will be using much
    more power then the normal routers from the consumer segment!

    So I started googling. And it seems that my Intel NIC (i210AT) in combination with Freebsd and the igb driver couldn't handle PPPoE traffic that well.

    It is not really a igb(4) driver issue and FreeBSD, because in one place we run a Xeon E3-1240v3
    together with the Intel i210AT NICs and are reaching nearly 1 GBit/s throughput, it is something
    around ~940 MBit/s + the TCP/IP overhead + SPI & NAT & working out or passing through the
    firewall rules it might be a real 1 GBit/s routet over the WAN interface.

    There also was a fix out, but the link was death.

    Stronger CPU and higher CPU clock frequency is here needed and not only 1GHz, it is also
    shown and announced on the pfSense hardware website, please read the part about the throughput
    that can be awaited and what hardware is needed for that to reach this amount of throughput.
    pfSense hardware
    > 501+ Mbps - Multiple cores at > 2.0GHz are required. Server class hardware with PCI-e network adapters.

    I hope you guys can help me a little bit because I'm starting to get a little desperate.

    No one can make your hardware faster, but some things can be done that the hardware will be running
    more flawless, perhaps!

    Details:
    APU2B4 with i210AT Intel NIC using igb
    XS4ALL 500mbit Fiber connection, currently reaching maximum 300mbit Down and around 450mbit up. (Fritzbox is doing 490/490)
    It's not the same as this issue: https://redmine.pfsense.org/issues/4821
    Because it uses all of its queues

    No, it is because for >500 MBit/s it must be or should be a 2,0GHz CPU likes shown on the
    pfSense website. You can try out to high up the mbuf size that more data will be able to rest
    in that queues but with no guarantee and you must be careful, because you might be ending
    otherwise in a booting loop, so it is wise to do a configuration backup of that.

    • update the BIOS of the APU2B4
    • do a fresh and full install on a mSATA or SSD drive
    • only configure the WAN and LAN part
    • do a speed test from one PC to another PC with iPerf or NetIO
      Please connect a small switch in front of the WAN Port and then place the PC with the iPerf server there
    • do the test and post us the results

    By the way do you using the AVM FB as a pure modem in the so called bridge mode or is it acting as
    a fully router doing SPI & NAT in front of the pfSense box?



  • Ok but would you please tell us first some points to be clear to give you the right answers.

    • How fast is your Internet speed?
    • From where location you are? (US,CA,UK,DE,….)

    To answer your first questions:
    My internet speed is 500 up and 500 down.
    My location is The Netherlands.

    It is not really a igb(4) driver issue and FreeBSD, because in one place we run a Xeon E3-1240v3
    together with the Intel i210AT NICs and are reaching nearly 1 GBit/s throughput, it is something
    around ~940 MBit/s + the TCP/IP overhead + SPI & NAT & working out or passing through the
    firewall rules it might be a real 1 GBit/s routet over the WAN interface.

    Do you run it with PPPoE? Because apperently it seems that PPPoE traffic isn't seen as IP traffic.

    • update the BIOS of the APU2B4
    • do a fresh and full install on a mSATA or SSD drive
    • only configure the WAN and LAN part
    • do a speed test from one PC to another PC with iPerf or NetIO
      Please connect a small switch in front of the WAN Port and then place the PC with the iPerf server there
    • do the test and post us the results

    I'll try that later today. Only one problem. To test the wan interface I can't place a switch with pc in front of it. Because I can't test the PPPoE speed that way.

    Stronger CPU and higher CPU clock frequency is here needed and not only 1GHz, it is also
    shown and announced on the pfSense hardware website, please read the part about the throughput
    that can be awaited and what hardware is needed for that to reach this amount of throughput.
    pfSense hardware

    501+ Mbps - Multiple cores at > 2.0GHz are required. Server class hardware with PCI-e network adapters.

    The CPU is barely used…

    By the way do you using the AVM FB as a pure modem in the so called bridge mode or is it acting as
    a fully router doing SPI & NAT in front of the pfSense box?

    The fritzbox isn't used



  • Do you run it with PPPoE? Because apperently it seems that PPPoE traffic isn't seen as IP traffic.

    Its not the traffic we are talking about, it is the number of CPU cores that will be used in each case.

    • together with PPPoE only one cpu core is used for the WAN interface
    • Not using PPPoE and all CPU core will be in usage for the WAN interface
      So if PPPoE is single CPU core threaded you get perhaps lower throughput at the WAN interface, but you might
      be thinking that this is pending on other circumstances or points!

    The fritzbox isn't used

    So there is then a modem in front of the WAN interface from the pfSense ?



  • Its not the traffic we are talking about, it is the number of CPU cores that will be used in each case.

    • together with PPPoE only one cpu core is used for the WAN interface
    • Not using PPPoE and all CPU core will be in usage for the WAN interface
      So if PPPoE is single CPU core threaded you get perhaps lower throughput at the WAN interface, but you might
      be thinking that this is pending on other circumstances or points!

    So you say that because of the low cpu speed my throughput is lower on the wan side. But why is my upload speed correct?

    I did the testing with a freshley installed APU and only lan en wan configured this is the result:

    Update:

    So there is then a modem in front of the WAN interface from the pfSense ?

    No only an NTU they call it here. It's just an Fiber to Ethernet converter.

    When I place PfSense behing the FritzBox: