Best way to handle a high interrupt rate
-
I'm using pfSense 2.1-release. I'm using a Dell R420 with an Intel E5-2470 CPU @ 2.30GHz. I have an instance that is pretty busy, 400+Mbps and 80K+pps inbound to the wan port. Last evening, it hit over 500Mbps and at least 120Kpps inbound to the wan port - at that point the CPU went to about 100% with the intr process taking the bulk of the CPU. At the same time we had a large number of states, around 1.2M - I suspect we had a fair amount of state turn-over too. I'm using Intel nics that use the igb driver (4 port card). Wondering if there are ideal settings in sysctl and/or loader.conf that might allow this system to handle a higher packet rate. Without going into tons of detail, here are the basics of what I have configured:
hw.igb.max_interrupt_rate="30000"
hw.igb.num_queues="8"
hw.igb.rxd=4096
hw.igb.txd=4096
hw.igb.rx_process_limit=600Interrupt rate currently looking like this (problem not occurring):
vmstat -i
interrupt total rate
irq9: acpi0 2 0
irq20: atapci0 105434 2
irq22: ehci1 100042 2
irq23: ehci0 168822 3
cpu0: timer 97225973 1999
irq256: igb0:que 0 364864094 7505
irq257: igb0:que 1 368656391 7583
irq258: igb0:que 2 358627740 7377
irq259: igb0:que 3 365491790 7518
irq260: igb0:que 4 362224363 7451
irq261: igb0:que 5 366677567 7542
irq262: igb0:que 6 365450117 7517
irq263: igb0:que 7 367536882 7560
irq264: igb0:link 2 0
irq265: igb1:que 0 58476 1
irq266: igb1:que 1 26564 0
irq267: igb1:que 2 13787 0
irq268: igb1:que 3 13579 0
irq269: igb1:que 4 45337 0
irq270: igb1:que 5 25653 0
irq271: igb1:que 6 11149 0
irq272: igb1:que 7 10420 0
irq273: igb1:link 2 0
irq274: igb2:que 0 368420183 7578
irq275: igb2:que 1 378268338 7781
irq276: igb2:que 2 367532547 7560
irq277: igb2:que 3 375227273 7718
irq278: igb2:que 4 365938763 7527
irq279: igb2:que 5 376144055 7737
irq280: igb2:que 6 374729460 7708
irq281: igb2:que 7 376743590 7749
irq282: igb2:link 2 0
irq291: igb3:link 1 0
cpu1: timer 97205774 1999
cpu7: timer 97205773 1999
cpu5: timer 97205771 1999
cpu2: timer 97205773 1999
cpu4: timer 97205773 1999
cpu6: timer 97205773 1999
cpu3: timer 97205773 1999
Total 6680778808 137424I'm wondering if setting hw.igb.hdr_split could help, I have a mix of packet sizes but something like 30% are small (<255 bytes).
Also wondering if increasing hw.igb.rx_process_limit may help, even going as far as unlimited (-1 setting).
I noticed these sysctl settings:
net.inet.ip.intr_queue_maxlen: 1000
net.inet.ip.intr_queue_drops: 5588360
net.route.netisr_maxqlen: 256Seems like increasing net.inet.ip.intr_queue_maxlen could help, not too familiar with this and have not tried tweaking it.
I've tried tweaking the number of rx queues (hw.igb.max_queues) in the past, 8 has given the best performance thus far.
I would appreciate any input.
Thanks,
Steve