Netgate 6100 - Significant Interface Interrupt Rates
-
Compared to previous routers they at a rate an order of magnitude greater than I have seen. Even then you could usual attribute a cause such as CPU, RAM or interface at or close to saturation. In this case the router is ticking along with cycles, RAM and bandwidth to spare.
Of course, my experience means little if this is what is expected. Before I posted I did search around and I could only find similar examples when there was a fault, driver issue or a NIC that didn't play well with others - including on this forum with a non-Intel NIC.
The rate varies in direct proportion to the flow of packets, probably a sign that all is well, as you suggest. But the rate still seems* excessive.
️
*Due acknowledgement that ISO 80000-1 has yet to formally define the standard unit of measure for the universally recognised value of 'seems' or 'seems like'
-
Oh, sorry I was looking at the wrong screen shot there! Hmm, yeah that does seem high.
What sort of throughput are you passing when you're seeing that?
-
@stephenw10
The values shown on the interfaces page are cumulative, so a couple of days after the first screenshot the interrupt values are now:LAN: 84601554 (172/s)
ONT: 86153166 (176/s)As you can see, the rate of interrupts has also increased since the first screenshot, probably due to increased traffic during the working week.
The throughput on the ONT (WAN) connection is capped by my ISP to 930/115Mbps. I rarely ever approach the ISP's download figure but I do hit the upload limit a few times a day. Of course, this is on an interface that is capable of 2.5GbE.
The WAN connection itself is unremarkable, with a nice flat trace:
The throughput on the LAN (LAN +1 VLAN) rarely goes above 1GbE and no more than ~1.6GbE on the 10GbE SFP+ DAC link.
If you can point me at a better source of numbers for your diagnostics I can provide them (I am still new to pfSense).
Regards.
️ -
@robbiett
what is the problem really?
cpu interrupts are a normal thing?one of the 6100 i run:
LAN Interface (lan, ix0) Status up MAC Address 90:ec:77:1d:17:cf IPv4 Address 192.168.55.1 Subnet mask IPv4 255.255.255.0 IPv6 Link Local fe80::92ec:77ff:fe1d:17cf%ix0 MTU 1500 Media 10Gbase-Twinax <full-duplex,rxpause,txpause> Plugged SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail) Vendor ZyXEL PN: DAC10G-3M SN: S1302 DATE: 2019-11-29 In/out packets 0/0 (0 B/0 B) In/out packets (pass) 0/0 (0 B/0 B) In/out packets (block) 0/0 (0 B/0 B) In/out errors 614/0 Collisions 0 Interrupts 2018328426 (1970/s) -
@heper said in Netgate 6100 - Significant Interface Interrupt Rates:
@robbiett
what is the problem really?
cpu interrupts are a normal thing?I'm not quite sure how to answer that. If you are pointing at my apparent lack of knowledge then please correct me. I've already posted:
@robbiett said:
Of course, my experience means little if this is what is expected. Before I posted I did search around and I could only find similar examples when there was a fault, driver issue or a NIC that didn't play well with others
...which gives you free rein to poke holes in my knowledge; I already think I forget more things on a given day than I re-learn.
If it is a more general query then I guess I have got used to interrupt moderation being in play for packets, especially when compared to how we viewed things 10 or 20 years ago. With NICs employing interrupt moderation (turning packets into bundles) other factors now have more of an impact on the numbers, including errant state changes, error interrupts, or even excessive 'housekeeping' events.
️ -
@robbiett said in Netgate 6100 - Significant Interface Interrupt Rates:
I'm not quite sure how to answer that. If you are pointing at my apparent lack of knowledge then please correct me.
well do you experience performance/stability issues ?
if yes -> there is a problem
if no -> there is no problemme personally don't give a shit about any metrics / stats / counters as long as things work as expected.
about interrupt moderation:
the increase of crappy clients&software that generate broadcast/multicast "storms" don't compare well to 10-20 years ago.10 years ago when you looked at a rack from a distance you could spot a broadcast-storm by eye.
These days 'normal' operation looks like a broadcast-storm 24/7 :) :)
These days you need to enable broadcast-control at the switch-level even at small sites with only a few 100's of wifi-clients -
Rate in 100-200/s range don't seem like anything unusual. Those numbers you had showing >10K/s are what I would say is unusual but not if that's when the firewall is under full load. Unless it's actually dropping packets it's not a problem.
-
Yes, I have interface stability issues, some of which will be removed on the next update, albeit with a bit of a kludge of a fix. Parsing out additional issues that may or not be related is difficult when it impacts stability, can trigger reboots or require multiple crypto handshakes to enable a connection.
Ignoring logs is no way to resolve issues or help pfSense's development. (I know you put it a bit more crudely than that.)
Regarding interrupt moderation, it does not mean what you think it means.
Are you trying to help me, as it has become a little unclear?
️ -
@stephenw10 said in Netgate 6100 - Significant Interface Interrupt Rates:
Rate in 100-200/s range don't seem like anything unusual. Those numbers you had showing >10K/s are what I would say is unusual but not if that's when the firewall is under full load. Unless it's actually dropping packets it's not a problem.
Thanks Steve, welcome news and means it may not be linked to IPv6 / PPPoE etc. At worst I see about 1.5% packet loss when the issues strike but most of the time the PL is zero.
Even when the bandwidth is at ISP limits the router should (and appears to have) plenty of resources left on the table. My packages are not that demanding and the first bottleneck to hit would be the single-core BSD issue.
Thanks for looking at this for me.
️ -
@robbiett said in Netgate 6100 - Significant Interface Interrupt Rates:
Are you trying to help me, as it has become a little unclear?
less and less when i read things like above.
anyhow, what i tried to convey is that the # of interrupts seems to be in line with what i find "normal" on my sites with a NG6100 device:
300mbit wan bandwidth + unknown amount of inter-vlan traffic:
1.2gbit wan bandwidth + unknown amount of inter-vlan traffic:
so, for me 70k interrupts does not cause any issues. it could be different for you, but it could also be that you have different issues that are not related to interrupts at all.
-
8 8ayM referenced this topic on
