Netgate 6100 - Significant Interface Interrupt Rates

RobbieTT · Apr 24, 2023, 1:46 PM

The systat -vmstat 1 probably adds some detail.

Normal background tasks only:

login-to-view

Maximum download bandwidth:

login-to-view

Idle:

login-to-view

Regards.

️

stephenw10 · Apr 25, 2023, 6:40 PM

Those don't seem like particularly high interrupt numbers. What are you comparing them to? Have they recently increased?

RobbieTT · Apr 25, 2023, 7:49 PM

@stephenw10

Compared to previous routers they at a rate an order of magnitude greater than I have seen. Even then you could usual attribute a cause such as CPU, RAM or interface at or close to saturation. In this case the router is ticking along with cycles, RAM and bandwidth to spare.

Of course, my experience means little if this is what is expected. Before I posted I did search around and I could only find similar examples when there was a fault, driver issue or a NIC that didn't play well with others - including on this forum with a non-Intel NIC.

The rate varies in direct proportion to the flow of packets, probably a sign that all is well, as you suggest. But the rate still seems* excessive.

️

*Due acknowledgement that ISO 80000-1 has yet to formally define the standard unit of measure for the universally recognised value of 'seems' or 'seems like'

stephenw10 · Apr 25, 2023, 9:12 PM

Oh, sorry I was looking at the wrong screen shot there! Hmm, yeah that does seem high.

What sort of throughput are you passing when you're seeing that?

RobbieTT · Apr 26, 2023, 7:47 AM

@stephenw10
The values shown on the interfaces page are cumulative, so a couple of days after the first screenshot the interrupt values are now:

LAN: 84601554 (172/s)
ONT: 86153166 (176/s)

As you can see, the rate of interrupts has also increased since the first screenshot, probably due to increased traffic during the working week.

The throughput on the ONT (WAN) connection is capped by my ISP to 930/115Mbps. I rarely ever approach the ISP's download figure but I do hit the upload limit a few times a day. Of course, this is on an interface that is capable of 2.5GbE.

The WAN connection itself is unremarkable, with a nice flat trace:

login-to-view

The throughput on the LAN (LAN +1 VLAN) rarely goes above 1GbE and no more than ~1.6GbE on the 10GbE SFP+ DAC link.

If you can point me at a better source of numbers for your diagnostics I can provide them (I am still new to pfSense).

Regards.

️

heper · Apr 26, 2023, 8:07 AM

@robbiett
what is the problem really?
cpu interrupts are a normal thing?

one of the 6100 i run:

LAN Interface (lan, ix0)
Status
up 
MAC Address
90:ec:77:1d:17:cf 
IPv4 Address
192.168.55.1 
Subnet mask IPv4
255.255.255.0 
IPv6 Link Local
fe80::92ec:77ff:fe1d:17cf%ix0 
MTU
1500 
Media
10Gbase-Twinax <full-duplex,rxpause,txpause> 
Plugged
SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail) 
Vendor
ZyXEL PN: DAC10G-3M SN: S1302 DATE: 2019-11-29 
In/out packets
0/0 (0 B/0 B) 
In/out packets (pass)
0/0 (0 B/0 B) 
In/out packets (block)
0/0 (0 B/0 B) 
In/out errors
614/0 
Collisions
0 
Interrupts
2018328426 (1970/s)

RobbieTT · Apr 26, 2023, 8:44 AM

@heper said in Netgate 6100 - Significant Interface Interrupt Rates:

@robbiett
what is the problem really?
cpu interrupts are a normal thing?

I'm not quite sure how to answer that. If you are pointing at my apparent lack of knowledge then please correct me. I've already posted:

@robbiett said:

Of course, my experience means little if this is what is expected. Before I posted I did search around and I could only find similar examples when there was a fault, driver issue or a NIC that didn't play well with others

...which gives you free rein to poke holes in my knowledge; I already think I forget more things on a given day than I re-learn.

If it is a more general query then I guess I have got used to interrupt moderation being in play for packets, especially when compared to how we viewed things 10 or 20 years ago. With NICs employing interrupt moderation (turning packets into bundles) other factors now have more of an impact on the numbers, including errant state changes, error interrupts, or even excessive 'housekeeping' events.

️

heper · Apr 26, 2023, 9:05 AM

@robbiett said in Netgate 6100 - Significant Interface Interrupt Rates:

I'm not quite sure how to answer that. If you are pointing at my apparent lack of knowledge then please correct me.

well do you experience performance/stability issues ?
if yes -> there is a problem
if no -> there is no problem

me personally don't give a shit about any metrics / stats / counters as long as things work as expected.

about interrupt moderation:
the increase of crappy clients&software that generate broadcast/multicast "storms" don't compare well to 10-20 years ago.

10 years ago when you looked at a rack from a distance you could spot a broadcast-storm by eye.
These days 'normal' operation looks like a broadcast-storm 24/7 :) :)
These days you need to enable broadcast-control at the switch-level even at small sites with only a few 100's of wifi-clients

stephenw10 · Apr 26, 2023, 1:14 PM

Rate in 100-200/s range don't seem like anything unusual. Those numbers you had showing >10K/s are what I would say is unusual but not if that's when the firewall is under full load. Unless it's actually dropping packets it's not a problem.

RobbieTT · Apr 26, 2023, 1:16 PM

@heper

Yes, I have interface stability issues, some of which will be removed on the next update, albeit with a bit of a kludge of a fix. Parsing out additional issues that may or not be related is difficult when it impacts stability, can trigger reboots or require multiple crypto handshakes to enable a connection.

Ignoring logs is no way to resolve issues or help pfSense's development. (I know you put it a bit more crudely than that.)

Regarding interrupt moderation, it does not mean what you think it means.

Are you trying to help me, as it has become a little unclear?

️

RobbieTT · Apr 26, 2023, 1:28 PM

@stephenw10 said in Netgate 6100 - Significant Interface Interrupt Rates:

Rate in 100-200/s range don't seem like anything unusual. Those numbers you had showing >10K/s are what I would say is unusual but not if that's when the firewall is under full load. Unless it's actually dropping packets it's not a problem.

Thanks Steve, welcome news and means it may not be linked to IPv6 / PPPoE etc. At worst I see about 1.5% packet loss when the issues strike but most of the time the PL is zero.

Even when the bandwidth is at ISP limits the router should (and appears to have) plenty of resources left on the table. My packages are not that demanding and the first bottleneck to hit would be the single-core BSD issue.

Thanks for looking at this for me.

️

heper · Apr 27, 2023, 11:43 AM

@robbiett said in Netgate 6100 - Significant Interface Interrupt Rates:

Are you trying to help me, as it has become a little unclear?

less and less when i read things like above.

anyhow, what i tried to convey is that the # of interrupts seems to be in line with what i find "normal" on my sites with a NG6100 device:

300mbit wan bandwidth + unknown amount of inter-vlan traffic:
login-to-view

1.2gbit wan bandwidth + unknown amount of inter-vlan traffic:
login-to-view

so, for me 70k interrupts does not cause any issues. it could be different for you, but it could also be that you have different issues that are not related to interrupts at all.