Web Spider, lack of scaling with pfsense
-
Hi guys, posting here to see if anyone can help me out a bit.
I run a web spider which due to software limitation can only have a certain amount of workers open at once. The way around it is to have multiple instances open at the same time on the same machine. 1 single node can average about 15Mbps, the issue is when i open a second node i average about 20Mbps, and any subsequent more nodes i open i still only get about 20Mbps of traffic overall. The node statistics seem to confirm this as well. I have a 100Mbps up/down line, and torrenting or using speedtest shows 95Mbps real available.
My Hardware
Dell 1950
Dual Intel E5410 @2.33Ghz
8GB Ram
Dual port Broadcom onboard for LAN side
Dual port intel 82576 chipset for WANI've ran netstat -ihw 1 in a shell and no errors were ever recorded, and the packets per second don't seem to scale either.
vmstat -i originally pointed that the IRQ for my intel card was using the same IRQ and many other internal hardware. changing the IRQ has given minimal but measurable improvements.
I'm fairly baffled after doing a lot of reading, any help would greatly appreciated.
-
You might want to NOT use the Broadcom onboard garbage at all.
-
I have tried using the intel card for both LAN and WAN, and separate intel cards, one for LAN and one for WAN without any effect.