• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Netgate 6100 - Significant Interface Interrupt Rates

Official Netgateยฎ Hardware
3
13
3.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R
    RobbieTT
    last edited by Apr 24, 2023, 1:15 PM

    According to the pfSense GUI I am experiencing an excessive amount of interface interrupts.

    For context I am on pfSense 23.01 release with all applicable patches applied. I have 2 physical interfaces in use:

    1 x LAN connection (ix1), 10 GbE, SFP+ DAC to UniFi switch (trunking 1 x LAN and 1 x VLAN)

    1 x ONT connection (igc3), 2.5 GbE RJ45 connections running at 1 GbE to an ONT, containing my WAN PPPoE link to ISP

    The interrupts on both physical interfaces are at a rate exceeding 160/sec:

    ๐Ÿ”’ Log in to view

    The CPU load is insignificant, no memory issues, nothing of note in 'Top' and maximum throughput is well below my ISP cap (940/115 mbps).

    As a PPPoE user I am aware of the 1-core BSD issue and the current issue with IPv6 and PPPoE interface race-condition issue (not at play here).

    I caught this primarily to the unexpected retries when trying to establish a PPPoE connection. On previous routers this was always a first-attempt connect, as you would expect for a first-hop link. It can take my 6100 up to 9 attempts, but more typically 4 or 5 attempts, to establish the full PPPoE handshake. Once established the link is rock-solid.

    I did find by searching that these issues were often seen with non-Intel NICs but as this is on a all-Intel Netgate device this does not seem relevant.

    Anyone have any ideas?

    โ˜•๏ธ

    1 Reply Last reply Reply Quote 0
    • R
      RobbieTT
      last edited by Apr 24, 2023, 1:46 PM

      The systat -vmstat 1 probably adds some detail.

      Normal background tasks only:

      ๐Ÿ”’ Log in to view

      Maximum download bandwidth:

      ๐Ÿ”’ Log in to view

      Idle:

      ๐Ÿ”’ Log in to view

      Regards.

      โ˜•๏ธ

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Apr 25, 2023, 6:40 PM

        Those don't seem like particularly high interrupt numbers. What are you comparing them to? Have they recently increased?

        R 1 Reply Last reply Apr 25, 2023, 7:49 PM Reply Quote 0
        • R
          RobbieTT @stephenw10
          last edited by Apr 25, 2023, 7:49 PM

          @stephenw10

          Compared to previous routers they at a rate an order of magnitude greater than I have seen. Even then you could usual attribute a cause such as CPU, RAM or interface at or close to saturation. In this case the router is ticking along with cycles, RAM and bandwidth to spare.

          Of course, my experience means little if this is what is expected. Before I posted I did search around and I could only find similar examples when there was a fault, driver issue or a NIC that didn't play well with others - including on this forum with a non-Intel NIC.

          The rate varies in direct proportion to the flow of packets, probably a sign that all is well, as you suggest. But the rate still seems* excessive.

          โ˜•๏ธ


          *Due acknowledgement that ISO 80000-1 has yet to formally define the standard unit of measure for the universally recognised value of 'seems' or 'seems like'

          1 Reply Last reply Reply Quote 0
          • S
            stephenw10 Netgate Administrator
            last edited by Apr 25, 2023, 9:12 PM

            Oh, sorry I was looking at the wrong screen shot there! Hmm, yeah that does seem high.

            What sort of throughput are you passing when you're seeing that?

            R 1 Reply Last reply Apr 26, 2023, 7:47 AM Reply Quote 0
            • R
              RobbieTT @stephenw10
              last edited by Apr 26, 2023, 7:47 AM

              @stephenw10
              The values shown on the interfaces page are cumulative, so a couple of days after the first screenshot the interrupt values are now:

              LAN: 84601554 (172/s)
              ONT: 86153166 (176/s)

              As you can see, the rate of interrupts has also increased since the first screenshot, probably due to increased traffic during the working week.

              The throughput on the ONT (WAN) connection is capped by my ISP to 930/115Mbps. I rarely ever approach the ISP's download figure but I do hit the upload limit a few times a day. Of course, this is on an interface that is capable of 2.5GbE.

              The WAN connection itself is unremarkable, with a nice flat trace:

              ๐Ÿ”’ Log in to view

              The throughput on the LAN (LAN +1 VLAN) rarely goes above 1GbE and no more than ~1.6GbE on the 10GbE SFP+ DAC link.

              If you can point me at a better source of numbers for your diagnostics I can provide them (I am still new to pfSense).

              Regards.

              โ˜•๏ธ

              H 1 Reply Last reply Apr 26, 2023, 8:07 AM Reply Quote 0
              • H
                heper @RobbieTT
                last edited by Apr 26, 2023, 8:07 AM

                @robbiett
                what is the problem really?
                cpu interrupts are a normal thing?

                one of the 6100 i run:

                LAN Interface (lan, ix0)
                Status
                up 
                MAC Address
                90:ec:77:1d:17:cf 
                IPv4 Address
                192.168.55.1 
                Subnet mask IPv4
                255.255.255.0 
                IPv6 Link Local
                fe80::92ec:77ff:fe1d:17cf%ix0 
                MTU
                1500 
                Media
                10Gbase-Twinax <full-duplex,rxpause,txpause> 
                Plugged
                SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail) 
                Vendor
                ZyXEL PN: DAC10G-3M SN: S1302 DATE: 2019-11-29 
                In/out packets
                0/0 (0 B/0 B) 
                In/out packets (pass)
                0/0 (0 B/0 B) 
                In/out packets (block)
                0/0 (0 B/0 B) 
                In/out errors
                614/0 
                Collisions
                0 
                Interrupts
                2018328426 (1970/s) 
                
                R 1 Reply Last reply Apr 26, 2023, 8:44 AM Reply Quote 0
                • R
                  RobbieTT @heper
                  last edited by Apr 26, 2023, 8:44 AM

                  @heper said in Netgate 6100 - Significant Interface Interrupt Rates:

                  @robbiett
                  what is the problem really?
                  cpu interrupts are a normal thing?

                  I'm not quite sure how to answer that. If you are pointing at my apparent lack of knowledge then please correct me. I've already posted:

                  @robbiett said:

                  Of course, my experience means little if this is what is expected. Before I posted I did search around and I could only find similar examples when there was a fault, driver issue or a NIC that didn't play well with others

                  ...which gives you free rein to poke holes in my knowledge; I already think I forget more things on a given day than I re-learn.

                  If it is a more general query then I guess I have got used to interrupt moderation being in play for packets, especially when compared to how we viewed things 10 or 20 years ago. With NICs employing interrupt moderation (turning packets into bundles) other factors now have more of an impact on the numbers, including errant state changes, error interrupts, or even excessive 'housekeeping' events.

                  โ˜•๏ธ

                  1 Reply Last reply Reply Quote 0
                  • H
                    heper
                    last edited by heper Apr 26, 2023, 9:05 AM Apr 26, 2023, 9:03 AM

                    @robbiett said in Netgate 6100 - Significant Interface Interrupt Rates:

                    I'm not quite sure how to answer that. If you are pointing at my apparent lack of knowledge then please correct me.

                    well do you experience performance/stability issues ?
                    if yes -> there is a problem
                    if no -> there is no problem

                    me personally don't give a shit about any metrics / stats / counters as long as things work as expected.

                    about interrupt moderation:
                    the increase of crappy clients&software that generate broadcast/multicast "storms" don't compare well to 10-20 years ago.

                    10 years ago when you looked at a rack from a distance you could spot a broadcast-storm by eye.
                    These days 'normal' operation looks like a broadcast-storm 24/7 :) :)
                    These days you need to enable broadcast-control at the switch-level even at small sites with only a few 100's of wifi-clients

                    R 1 Reply Last reply Apr 26, 2023, 1:16 PM Reply Quote 0
                    • S
                      stephenw10 Netgate Administrator
                      last edited by Apr 26, 2023, 1:14 PM

                      Rate in 100-200/s range don't seem like anything unusual. Those numbers you had showing >10K/s are what I would say is unusual but not if that's when the firewall is under full load. Unless it's actually dropping packets it's not a problem.

                      R 1 Reply Last reply Apr 26, 2023, 1:28 PM Reply Quote 0
                      • R
                        RobbieTT @heper
                        last edited by Apr 26, 2023, 1:16 PM

                        @heper

                        Yes, I have interface stability issues, some of which will be removed on the next update, albeit with a bit of a kludge of a fix. Parsing out additional issues that may or not be related is difficult when it impacts stability, can trigger reboots or require multiple crypto handshakes to enable a connection.

                        Ignoring logs is no way to resolve issues or help pfSense's development. (I know you put it a bit more crudely than that.)

                        Regarding interrupt moderation, it does not mean what you think it means.

                        Are you trying to help me, as it has become a little unclear?

                        โ˜•๏ธ

                        H 1 Reply Last reply Apr 27, 2023, 11:42 AM Reply Quote 0
                        • R
                          RobbieTT @stephenw10
                          last edited by Apr 26, 2023, 1:28 PM

                          @stephenw10 said in Netgate 6100 - Significant Interface Interrupt Rates:

                          Rate in 100-200/s range don't seem like anything unusual. Those numbers you had showing >10K/s are what I would say is unusual but not if that's when the firewall is under full load. Unless it's actually dropping packets it's not a problem.

                          Thanks Steve, welcome news and means it may not be linked to IPv6 / PPPoE etc. At worst I see about 1.5% packet loss when the issues strike but most of the time the PL is zero.

                          Even when the bandwidth is at ISP limits the router should (and appears to have) plenty of resources left on the table. My packages are not that demanding and the first bottleneck to hit would be the single-core BSD issue.

                          Thanks for looking at this for me. ๐Ÿ‘

                          โ˜•๏ธ

                          1 Reply Last reply Reply Quote 1
                          • H
                            heper @RobbieTT
                            last edited by heper Apr 27, 2023, 11:43 AM Apr 27, 2023, 11:42 AM

                            @robbiett said in Netgate 6100 - Significant Interface Interrupt Rates:

                            Are you trying to help me, as it has become a little unclear?

                            less and less when i read things like above.

                            anyhow, what i tried to convey is that the # of interrupts seems to be in line with what i find "normal" on my sites with a NG6100 device:

                            300mbit wan bandwidth + unknown amount of inter-vlan traffic:
                            ๐Ÿ”’ Log in to view

                            1.2gbit wan bandwidth + unknown amount of inter-vlan traffic:
                            ๐Ÿ”’ Log in to view

                            so, for me 70k interrupts does not cause any issues. it could be different for you, but it could also be that you have different issues that are not related to interrupts at all.

                            1 Reply Last reply Reply Quote 2
                            • 8 8ayM referenced this topic on Mar 20, 2024, 9:34 PM
                            10 out of 13
                            • First post
                              10/13
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.