Should NICs share IRQs or not?
-
The received wisdom on the forum seems to be that each NIC should be assigned its own IRQ; no other devices should share those IRQs.
http://forum.pfsense.org/index.php/topic,3196.msg19312.html#msg19312
IRQ sharing is not really a good thing.http://forum.pfsense.org/index.php/topic,3687.msg22557.html#msg22557
Find out the IRQ being assigned to those cards. If its the same try manually assigned them different IRQ in the bios, or move the cards around in different slots. IRQ sharing could be causing your problem.
<<However, Henning Brauer's 2002 white paper on tuning OpenBSD suggests that sharing the same IRQ among NICs will improve performance:
http://www.pantz.org/os/openbsd/runningandtunningopenbsd.shtml
IRQ
Every interrupt requires a context switch, from the process running when the IRQ took place, to the interrupt handler. As a number of things must be done upon entering the interrupt handler, a large quantity of interrupts can result in excess time required for context switching. One non-obvious way to reduce this load is to share interrupts between the network adapters, something permitted on the PCI bus. As many people are not even aware of the the possibility of interrupt sharing, and the benefits are not obvious, let's look at this a little closer.
…
In a busy environment, when many devices are needing service, saving these context switches can significantly improve performance by permitting the processor to spend more time processing data, rather than switching between tasks. In fact, in a very high load situation, it may be desireable to switch the adapters and drivers from an interrupt driven mode to a polling mode...
<<So what's the real story?
-
We use FreeBSD, not OpenBSD. PF was ported to FreeBSD. Anyhow I have found that shared IRQ's can have negative results with FreeBSD.
-
We use FreeBSD, not OpenBSD. PF was ported to FreeBSD.
<<Obviously.
Henning's paper just happens to focus on OpenBSD network tuning. Shouldn't the NIC/IRQ-sharing issue transcend operating systems?
http://www.wormulon.net/files/pub/FreeBSD_Network_Tuning_-_slides.pdf
IRQ sharing is mentioned as a performance tuning strategy (for FreeBSD), especially when combined with device polling.http://www.ussg.iu.edu/hypermail/linux/kernel/0110.0/0714.html
Sharing irq between unrelated devices is probably evil in all cases,
but for identical devices like multiple NICs, the shared irq results in
one irq call, followed by polling the devices connected, which can be
lower overhead than servicing N interrupts on a multi-NIC system.
<<More from Henning:
http://www.security-express.com/archives/openbsd/2003-06/1176.htmlAnyhow I have found that shared IRQ's can have negative results with FreeBSD.
Thanks. I appreciate your experience in this area. I'll set up separate IRQs for my NICs.
-
Well I think that collective wisdom is that sharing IRQs is really bad, but the Henning article is quite interesting. After reading it, I can understand that it might be possible that sharing IRQs with polling might improve performance (though my knowledge of hardware level stuff is abysmal).
Did you run any tests?
-
Haven't had a chance to run any tests yet.
-
Experience shows FreeBSD and shared IRQ's will result in problems. I've seen it myself. There are many, many threads on here and in the list archives of people having issues because of shared IRQ's.
Now, that's not typically NIC's sharing IRQ's amongst each other. I don't think I've ever seen that. Usually it's a USB controller or IDE controller or some other piece of hardware sharing with a NIC.
The FreeBSD list archives have posts from people much more knowledgeable of FreeBSD than anybody on this forum saying it'll result in reduced performance. Again, that was never in the context of NIC's sharing with NIC's though.