Flooding logs with fq_codel_enqueue over limit

IsaacFL · Oct 12, 2020, 4:18 PM

For the last few days starting Oct 8, my system log has been flooded with log entries about fq_codel. I was sending the logs to an external server and within this 9 minute interval was over 50Mbyte of logs sent.

Oct 08 23:43:08 kernel fq_codel_enqueue over limit
Oct 08 23:43:08 kernel fq_codel_enqueue maxidx = 9649
Oct 08 23:43:08 kernel fq_codel_enqueue over limit
Oct 08 23:43:08 kernel fq_codel_enqueue maxidx = 9649
Oct 08 23:43:08 kernel fq_codel_enqueue over limit
Oct 08 23:43:08 kernel fq_codel_enqueue maxidx = 9649
Oct 08 23:43:08 kernel fq_codel_enqueue over limit
Oct 08 23:43:08 kernel fq_codel_enqueue maxidx = 9649
Oct 08 23:43:08 kernel fq_codel_enqueue over limit
Oct 08 23:43:08 kernel fq_codel_enqueue maxidx = 9649
.
.
.
Oct 08 23:51:53 kernel fq_codel_enqueue maxidx = 15649
Oct 08 23:51:53 kernel fq_codel_enqueue over limit
Oct 08 23:51:53 kernel fq_codel_enqueue maxidx = 15649
Oct 08 23:51:53 kernel fq_codel_enqueue over limit

What could cause this?

I tried just rebooting, but when I checked last night I noticed it was in the logs again:

Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit
Oct 11 22:37:09	kernel		fq_codel_enqueue maxidx = 383
Oct 11 22:37:09	kernel		fq_codel_enqueue over limit

Version:
2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLE

Installed Packages:
acme 0.6.8_3
Avahi 2.1_1
nut 2.7.4_7
pfBlockerNG-devel 2.2.5_36

tman222 · Oct 13, 2020, 6:18 PM

@IsaacFL - do you a large amount of traffic or connections (states) any given time? I would try to increase the FQ-Codel "limit" parameter and see if that helps.

IsaacFL · Oct 13, 2020, 9:11 PM

@tman222

I don't think there is any large amount of traffic. Both times it seemed to happen after we were all in bed.

Also I have been using the traffic shaper in this configuration for a while with no issue.

What I am thinking is that I recently have been trying out pfblockerng and I set it up to use Floating rules. So my theory right now, is that the default was to put the pfblockerng rules at the top, and maybe they were interfering with the floating rules I had for the traffic shaper.

pfBlocker has an option to put its rules after, so I am trying that and checking the logs periodically. It makes since that the traffic shaper rules should be at the top, so maybe that was it.

bobbenheim · Oct 14, 2020, 6:53 PM

You can try Setting Queue Management Algorithm, at the limiter and the child queue, to Tail Drop. Might also be because of a low limit value compared to flows if you have changed those from default, in that case increase the value of limit.

IsaacFL · Oct 15, 2020, 12:46 AM

@bobbenheim - I double checked and both limiter and child queue are already set to tail drop.

Here are my settings (120M in, 12M out):

target 5
interval 100
quantum 300
limit 10240
flows 20480

I think I got them from one of the posts here. It has been working ok with these since 2.4.5 came out.

I still think that I had caused a problem with the pfblockerng floating rules located before my limiter floating rules. I have reversed the order and so far the log entries have not shown again.