High interrupts.

[houkouonchi]

Ok so I originally was running linux but pppoe was using near 100% CPU after tweaking it to use synch and some other options which was limiting my speed. I switched to pfsense.

The pfsense box is:
1.7 Ghz Celeron
512 MB of ram
intel onboard gigabit
realtek PCI gigabit.

The problem is I am approaching 0% idle when downloading/upload at fast speeds.

CPU:  0.0% user,  0.3% nice, 37.4% system, 61.2% interrupt,  1.0% idle
Mem: 211M Active, 78M Inact, 72M Wired, 73M Cache, 59M Buf, 48M Free

It looks like its the interrupt which is really killing me. This is what I get downloading at around 9 megabytes/sec. I tried enabling polling which greatly reduced CPU and interrupts but with polling my max speeds are around 8 megabytes/sec. Here is what the top of top looks like with oplling downloading at its max (around 8 megabytes/sec):

CPU:  0.4% user,  0.0% nice,  0.7% system, 27.3% interrupt, 71.5% idle
Mem: 211M Active, 78M Inact, 72M Wired, 73M Cache, 59M Buf, 48M Free

So all the values are much lower but now I am getting lower speeds?

Anyway I took of polling since its lowering my speeds. I think this might be due to the realtek NIC. I am thinking of trying to find another intel one (if I can get it cheaply).

Without polling downloading via WAN (realtek) limiting it to 5 megabytes/sec gives me:

CPU:  0.7% user,  0.0% nice, 17.6% system, 40.4% interrupt, 41.2% idle
Mem: 244M Active, 78M Inact, 72M Wired, 49M Cache, 59M Buf, 39M Free

The same as above except downloading via LAN (intel) also 5 Mbytes/sec:

CPU:  1.1% user,  0.0% nice,  7.1% system, 29.3% interrupt, 62.4% idle
Mem: 226M Active, 78M Inact, 72M Wired, 60M Cache, 59M Buf, 47M Free

Downloading Downloading a file behind the NAT limited to 5 Mbytes/sec (5 mbytes/sec realtek down 5 Mbytes/sec intel up) gives me:

CPU:  0.0% user,  0.0% nice, 16.9% system, 40.8% interrupt, 42.3% idle
Mem: 244M Active, 78M Inact, 72M Wired, 48M Cache, 59M Buf, 40M Free

So… Do you think buying another intel NIC will allow me to get 100/100 and reduce the interrupts? Or am I wasting my money and I should just buy a more powerful machine?

[houkouonchi]

Well I went ahead and bought an intel gigabit PCI NIC. Sadly it cost more than the computer did. Anyway it did seem to help a little but I am only getting about 500 kilobytes/sec faster than I got before. I was hoping to get a little bit faster. Interesting enough interrupt is now like 0% but now the system cpu% is high. There is a lot more idle with the changed NIC now and I am getting slightly faster speed.

Downloading at around 10 megabytes/sec I am getting:

CPU:  0.0% user,  0.0% nice, 84.0% system,  0.4% interrupt, 15.7% idle
Mem: 31M Active, 13M Inact, 29M Wired, 132K Cache, 20M Buf, 409M Free

It does look like upload is acting a lot better though now. It looks like I am getting better uplaod speed than download:

I am quite impresssed with that upload rate considering I tested by sshing into my server 5000 miles away from and wget'd the file from a server running on a machine behind the NAT. Not bad at all considering the distance and the fact it was a single stream.

Here is what CPU/interrupt looked like during the upload:

CPU:  1.5% user,  0.0% nice, 58.6% system,  0.4% interrupt, 39.6% idle
Mem: 30M Active, 13M Inact, 30M Wired, 132K Cache, 20M Buf, 408M Free

It kind of surprises me that uploading is taking a lot less resources than downloading. I guess I can try testing the download again later tonight just to see if it was soemthing with the ISP and it being prime-night time.

[houkouonchi]

Well. I ended up going back to linux and maxing out at around 80 megabits. Unfortunately i started having a very similar problem to what I originally had where certain sites and stuff wouldn't work. It was related the MTU. I originally had it set at 1492 and later changing it to 1454 fixed the issue but then it started having problems even at 1454. Certain sites and stuff wouldn't load. I need the connection to be reliable and linux was slightly lower latency so I switched back to linux.