Netgate 3100 unable to saturate gigabit WAN

njacobs

I've recently upgraded my FTTP connection from 160 Mbps down / 30 Mbps up to 1000 Mbps down / 115 Mbps up. My ISP hosts a LibreSpeed speed test site on the same network that my connection is terminated on and I consistently achieve ~900-950 Mbps with the ONT connected directly to a host machine, however when connected via my Netgate 3100 I'm only seeing ~685-700 Mbps.

This post was incredibly helpful, and from what I can tell I think I'm CPU limited whilst downloading?

last pid:  2912;  load averages:  0.27,  0.20,  0.13                                          up 1+02:03:45  14:09:06
143 threads:   4 running, 118 sleeping, 21 waiting
CPU:  0.0% user,  0.0% nice,  0.8% system, 58.8% interrupt, 40.4% idle
Mem: 24M Active, 56M Inact, 179M Wired, 83M Buf, 1718M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        -92    -     0B   176K CPU1     1  12:49  97.37% [intr{mpic0: mvneta2}]
   10 root        155 ki31     0B    16K RUN      0  25.4H  72.38% [idle{idle: cpu0}]
   11 root        -92    -     0B   176K WAIT     0   3:57  24.54% [intr{mpic0: mvneta1}]
   10 root        155 ki31     0B    16K RUN      1  25.4H   4.43% [idle{idle: cpu1}]

I think I've got a fairly straightforward setup:

WAN (PPPoE) on mvneta2
WAN2 (4G modem for failover) on mvneta0
LAN on mvneta1

All my hosts are connected to a switch via mvneta1.

I have a small number of VLANs configured on the LAN with DHCP and DNS and have firewall rules to block traffic between them, and I've got a few packages installed but nothing doing packet inspection, etc.

Am I just hitting the limits of the 3100?

Thanks

Nathan

rcoleman-netgate

@njacobs Back up the config, reset to default, configure a single network (default LAN is fine for this, in fact ideal) and see if the speed remains.

If the speed gets back closer to 900-950 then it's the config you're running that's shaving off overhead.

stephenw10

You are probably just seeing the PPPoE overhead there though. You are hitting a CPU limit on one core there. The normal PPPoE restrictions don't really apply on the 3100 because it doesn't use multi-queue NICs anyway. However there is still overhead because PPPoE using netgraph and all traffic has to go through it. The SG-3100 is capable of Gigabit throughput with that but will be reduced with PPPoE.

Steve

njacobs

@stephenw10 Forgive my ignorance, but if it's PPPoE overhead wouldn't I see similar speeds with the ONT connected directly to my host machine? Or are you saying it's the overhead that's maxing the CPU?

stephenw10

Yes, the CPU overhead. All traffic has to pass through netgraph and the single CPU core has to do that.

Steve

njacobs

@stephenw10 Thanks for the clarification.

I'll certainly test after a reset to default, but I do want the ability to run with my current configuration. Can I expect a 4100 to achieve gigabit WAN speeds with PPPoE?

stephenw10

Yes, or very close to it. See:
https://forum.netgate.com/topic/172448/1g-pppoe-cpu-bottleneck-on-3100

Steve

njacobs

@stephenw10 Thanks for that, really helpful.

Before upgrading to a 4100 (or dare I say it, a 6100...) I thought I'd try offloading the PPPoE work to my ISP supplied router by using it as a gateway in front of the 3100. With this configuration I'm now seeing ~800-850 Mbps with CPU load peaking ~84%.

stephenw10

That sounds about right. It will pass 1G given the latency and traffic but a bit less than that in real conditions is expected.

Steve

njacobs

Just sharing that I upgraded to a 6100 and the results were disappointing. I've started a new thread rather than continue this one.

Nathan

njacobs

Resolved.