WebGUI dying under heavy load (Internal Server Error 500 etc.) ?

SatireWolf

No offense, but a 266 mHz box with cheap cheasy realtek adapters being expected to handle a 64 byte packet storm at 30 mB/s is rediculous. The system is meant for 5 mBit and less WAN use. When in doubt, overbuild, don't underbuild your firewalls. You don't have the luxury of super optimized TCP offloading and handling by the NIC's unless you're using nice Intel Server grade PCI-X gigabit adapters (which I do when I can). The real issue of PPS is the fact that you are memory / copy throttled. You can only make a dozen or so memory copies so fast per packet before you run out of internal machine bandwidth. It's not so much GHz as much as memory latency, bus latency, and overhead on the packet processing. If you have specialized packet processors that do nothing but offload TCP headers and minimize copy processes you have something more in line with a Cisco / Juniper networks box.

Just speaking from personal experience here, if you want balls to the wall performance with 64 byte packets, overbuild the crap out of the box with an opteron processor or two and really fast low latency memory and a PCI-X TCP offloading nic. I've pushed over 100 mBit of 64 byte packets this way.

But yeah, the FreeBSD guys are going to laugh and say, get a bigger box.

The only way to fix this behavior at all is to study how many copy's happen between packet input and output, then reduce that number. Or otherwise implement some packet storm throttling / protection to damp down an attack.

But yeah, why would you expect a 266 mHz box to perform at 30 mBit? Cisco puts 166-266 mHz processors in their T1 grade routers with full TCP offload and only rate them for 1-2 mBit connections. Just FYI.

You won't ever reach overload conditions on a 2 mBit pipe. If you're trying to push a 10 mBit pipe you need a bigger box, period.

JesperDJ

Hi.. i´m dealing with the same "problem" and therefore i ask..:

How much hardware do you need to run +80MBit/s througput?

Im getting max. 10MBit/s with my pIII 933/512MB mem and a 3Com 3C982 dual NIC

If i try the m0n0wall with the same HW the result is 22MBit/s ??

hoba

@JesperDJ:

Hi.. i´m dealing with the same "problem" and therefore i ask..:

How much hardware do you need to run +80MBit/s througput?

Im getting max. 10MBit/s with my pIII 933/512MB mem and a 3Com 3C982 dual NIC

If i try the m0n0wall with the same HW the result is 22MBit/s ??

Something is wrong here. I'm able to get about 87 mbit/s throughput from one of my C3 1GHz LAN->WAN in factory default configs. If I remove the pfSense I only get about 2 mbit/s more and that is with crappy viarhine onboard nics. Your system should push much more.

lightning

Actually, the throughput should not be measured in Mbits/s but in packets/s (pps), like in routers.

This will be putting better light on the true performance, as the packet size usually matters less than the packet handling overhead.

Typically, with lots of 1500 bytes packets one way, and few 64 bytes ack packets the other way, if you are well below wirespeed, you can take the Mbits/s figure and divide it roughly by 1500*8 to get the pps throughput. e.g. 22 Mbits/s = 1'830 pps.

The true throughput performances are usually made with minimum size packets of 64 bytes, and I would be curious to see some of such performance figures :)

Please don't take this as critisism, pfSense is great work and runs great :)

JesperDJ

hmm wierd..

in bone stock config i get 45MBit/s with m0n0 and 22 with pfSense..

Any ideas?

My HW is P3 933CPU on MSI mainboard, 512MB PC133 SD, and a 3Com Dual NIC 3C982, well, it is the same with 2 Realtek 8139C…

sullrich

FreebSD 6.1 is much slower than 4.11. This is the reason that m0n0wall was hesitant to switch initially. Why they want to switch now is beyond me because its going to slow down every installation.

I would suggest going back to m0n0wall.

JesperDJ

yea but still the m0n0wall isnt doing the job either.. i should be able to get +80MBit througput on my existing hardware…

and the pfSense has some of the features i need

sullrich

Well then it sounds like you need beefier hardware then if you want to stick with pfSense.

JesperDJ

hmm.. ???

hoba wrote:

"Something is wrong here. I'm able to get about 87 mbit/s throughput from one of my C3 1GHz LAN->WAN in factory default configs. If I remove the pfSense I only get about 2 mbit/s more and that is with crappy viarhine onboard nics. Your system should push much more."

sullrich

I can push 40 megabit with a 500 mhz box so I somewhat agree with Holger.

What the exact problem is with your hardware is hard to predict, however.

JesperDJ

hmm yes.. im am trying with the default config, så it must be the HW.. im gonna try another machine tommorow then..

Thanks so far :)

JesperDJ

Ok.. the results with the new HW

HW: P4 2,5Ghz, 256MB DDR2, Onboard GB NICs

Config: 100% default

pfSense:
Max speed = 60MBit/s
Payload = 100%
System unstabel!
WebGUI = unreachable

m0n0wall:
Max speed = 85MBit/s (the switch couldt handle more, trying another GB Switch tommorrow)
Payload = 30%
System very stabel!
WebGUI = 100% functioning

seems 2 me that pfSense has a problem?

sullrich

You are not specifying which type of NICS. Please specify your NIC types.

databeestje

So far I have been able to push ~400Mbps through a dual port Intel GigE in a P3 1Ghz and 22Mbits+ through a via rhine(vr0)/3com 2000-T (sk0) combination on a Via Eden 933.

Then again, I have not been trying to access the webgui.
A p3 1Ghz doing 400Mbits is about ~55% interrupt. So that one doesn't die from load.

The Via with 22Mbits and a traffic shaper was pretty loaded but the queues page at 22Mbits was still accessible.

One point of thought though. This testing was performed somewhere in the pre RC1-ish days.
I have not ran benchmarks one rc2 and above yet.

Something may indeed be amiss.

JesperDJ

@sullrich, actually that isnt important in this case, i have tryied with 3Coms, Realteks & Onboards both 100MBits and 1GBits.. the difference is less than 2MBit/s when im using pfSense..

But if you need the exect type of NICs, please let me know.

The interrupt is 99/100% when pfSense is doing 60MBit/s on my actual system.. wich was; P4 2,5Ghz, 256MB DDR2

and perhaps your (databeestje) results where on some RC1-ish, but in my oppinion the newer version shouldnt be slower?

sullrich

This is a known problem. I've posted to the freebsd lists but nobody has responded yet.

JesperDJ

I have now tryied with a different setup:

pfSense testet with:

Motherboard: MSI MS-6723
CPU P3 933Mhz
Memory: 256MB SD DDR 400
NICs: Realtek 8139

Now i was able to get aproxx 96MBit (i guess the NICs couldnt handle more…)
Interrupt load was 83%.. leaving the system 100% stable and accessable..

maybe the other HW wasnt 100% compatible with pfSense.. or?

is there any recommandations wich HW to use ie. wich chipset, cpu, mem, nics etc?

rsw686

Thats interesting you and databeestje both had good throughput on a P3 system. Your P4 box didn't work as well. Wonder if its the processor or the NIC cards.

sullrich

That is VERY interesting that you are able to saturate the ethernet and access the system with different nics. That makes me wonder.

JesperDJ

@sullrich, i didnt understand your answer?