netisr running close to 100% on a single core
-
Hi,
We noticed a problem we are not sure how to tackle. One of our host initiated quite a big download (2.5 TB) from the web. We have ntop-ng running on PFsense, and after about an hour we started to receive notifications of failing ICMP (Ping) checks to our servers from firewall. We started to get packet loss.
I looked at system activity and noticed software interrupt service (netisr) running close to 100% on a single core. We were getting drops in our IP queue, but I changed the
net.inet.ip.intr_queue_maxlenand that seems to solved the packet loss problem. (https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#ip-input-queue-intr-queue)But we are still getting ICMP problem during and netisr is still using only one thread. I tried changing the net.isr.bindthreads, net.isr.maxthreads and packet dispatching policy to hybrid or deferred, but nothing has helped. Can you help us with this issue?
Our WAN is not PPPoE.
Platform: Netgate 6100
Software Version: pfSense Plus 24.11 -
What sort of throughput does the download create? What is the WAN bandwidth?
For a single TCP connection you could end up with a single core being the limit. But I wouldn't expect that to prevent the firewall responding to pings unless it's also filling one of the links.
-
@Gustas said in netisr running close to 100% on a single core:
One of our host initiated quite a big download (2.5 TB) from the web. We have ntop-ng running on PFsense, and after about an hour we started to receive notifications of failing ICMP (Ping) checks to our servers from firewall. We started to get packet loss.
Do you have both WAN and LAN enabled as Monitored Interfaces in ntopng by chance?
-
said in netisr running close to 100% on a single core:
@Gustas said in netisr running close to 100% on a single core:
One of our host initiated quite a big download (2.5 TB) from the web. We have ntop-ng running on PFsense, and after about an hour we started to receive notifications of failing ICMP (Ping) checks to our servers from firewall. We started to get packet loss.
Do you have both WAN and LAN enabled as Monitored Interfaces in ntopng by chance?
Yes, we do. Can that be the issue?
-
@stephenw10 said in netisr running close to 100% on a single core:
What sort of throughput does the download create? What is the WAN bandwidth?
For a single TCP connection you could end up with a single core being the limit. But I wouldn't expect that to prevent the firewall responding to pings unless it's also filling one of the links.
Download throughput was on average at 90 MB/s. Our WAN is at 2.5 Gbps.
What about rss queueing, why is everything being sent to a single core?
-
@Gustas said in netisr running close to 100% on a single core:
Do you have both WAN and LAN enabled as Monitored Interfaces in ntopng by chance?
Yes, we do. Can that be the issue?
Certainly a contributor. There is a caution in the pfSense ntopng package when selection interfaces to monitor that says "It is generally not recommended to monitor WAN interfaces." At a minimum, it will double your load. You should remove any WAN interfaces from the list of Monitored Interfaces.
Also, if you have any form of active discovery enabled inside ntopng itself, be sure to turn that off as well.
-
@dennypage said in netisr running close to 100% on a single core:
@Gustas said in netisr running close to 100% on a single core:
Do you have both WAN and LAN enabled as Monitored Interfaces in ntopng by chance?
Yes, we do. Can that be the issue?
Certainly a contributor. There is a caution in the pfSense ntopng package when selection interfaces to monitor that says "It is generally not recommended to monitor WAN interfaces." At a minimum, it will double your load. You should remove any WAN interfaces from the list of Monitored Interfaces.
Also, if you have any form of active discovery enabled inside ntopng itself, be sure to turn that off as well.
Sorry, I just checked and monitoring in ntop is configured only for internal interfaces, WAN is not being monitored. Sorry for misleading you.
