Download at full speed then got packet loss

stephenw10

Ok those rules and queues look correct.

Do you actually see a reduction in available bandwidth when they are applied?

The key is to have them set slightly below the available speed to that all the queuing is done in pfSense where it can control what is passed and when. You might try setting the values lower to be sure.

Steve

Lip

@stephenw10

The Vigor outputs the following values, in my thread above you can see my limits. I would say the limiter works?

But look at these spikes on cmd:

And this is what my ping looks like when there is only little traffic on the line:

Rico

Bad network card / drivers like Realtek?

-Rico

Lip

@rico

Hi Rico,

unfortunately you are right. However, I followed the following recommendation regarding the drivers.

https://forums.serverbuilds.net/t/guide-resolve-realtek-nic-stability-issues-on-freebsd-pfsense-2-4-4-2-4-5-2-5-0-opnsense-use-2-5gb-realtek/3555

My hardware is the following:
ZBOX PRO CI329 nano
https://www.zotac.com/product/mini_pcs/zbox-pro-ci329-nano

Rico

When it comes to PPPoE, the problems get even worse with Realtek.
Can you put your WAN in DHCP or Static mode (run double NAT) for testing?

-Rico

Lip

@rico
Unfortunately, I can't change that. It would be extremely difficult. Is there another option or do I have to buy new hardware?

stephenw10

What are you pinging in that example?
Do you see the same variation to all external IPs?

Steve

Rico

For my opinion you've bought the wrong device to run pfSense. That ZBOX is ~250 bucks as barebone with 2 Realtek NICs...makes my heart bleeding.
300/50(40) Mbps looks like Telekom Germany, refund that box if possible (14 day window) and get some serious pfSense gear.

-Rico

Lip

@stephenw10
I ping on www.google.de.
It doesn't matter what I ping.

Lip

@rico

Yes it is a telecom line.
It is really difficult to get good hardware for the PFsense in Germany that is energy efficient.

What kind of hardware would you suggest?

stephenw10

Mmm, the Celeron N4100 is not that fast. It's possible you're hitting something there though you are seeing the full bandwidth. I would expect it to pass 300Mbops without issues though. Maybe it's stuck at the lowest speed.

Try running at the CLI top -aSH whilst testing. See if one core at loaded 100%.

Steve

Lip

@stephenw10

Here is a screenshot, is the CPU the problem?

stephenw10

Nope. All 4 cores are at least 73% idle. That's only 224Mbps though.

I'm still not sure you're actually limiting that traffic though.

What does the speed test result show when you disable the Limiters? vs with the Limiters?

Steve

Lip

@stephenw10

Active Limiter

inactive Limiter with States Reset

stephenw10

Hmm, well that looks OK. Maybe more than FQ_Codel can deal with then.

I don't actually see any packet loss there though, are you still seeing that?

Lip

@stephenw10

There are always packet losses at full speed downloads. Worse is that the ping rises from 5ms to 50. Then you get problems with Voip

stephenw10

Mmm, the latency is unavoidable to a certain extent though I would not expect packet loss.

You can try setting a lower limit for all other traffic and passing VoIP traffic outside the Limiters to reserve bandwidth for it.

Steve

AndyRH

@stephenw10 said in Download at full speed then got packet loss:

Celeron N4100

More for my own education than anything, why is the general theme that the HW pfSense is running on is the problem and not the ISP with what looks to be a full link?

While Realtek NICs are not the best, the speed is only 25% of what the NIC should be able to do, so it is not stressed. I would not expect a unstressed NIC causing a problem.

I run on an i3 540 with 4 port Intel 1Gb NIC and even with a speed test running @ 980Mbps and file copies crossing the other NICs I see no change in pings and a CPU getting close to 30%. The CPU in the problem system is generations newer and should be faster per cycle than mine.

Lip

@andyrh
Unfortunately I can not say what the problem is. I only know that if there is no full load traffic on the line, there is no packet loss. I also think that my hardware should be sufficient.

stephenw10

@andyrh said in Download at full speed then got packet loss:

why is the general theme that the HW pfSense is running on is the problem and not the ISP with what looks to be a full link?

It isn't, not here at least. As I said there I would expect that CPU to pass 300Mbps with ease and the output from top showed that to be true.
I would always expect to see some increase in latency when you use more WAN bandwidth but not packet loss as we're seeing here.
I would not expect to see either when loading the CPU with traffic between other interfaces. Unless you are maxing at least one core completely.

Steve