High Latency on WAN links. Server Downsized?

[fourm101]

Hi,
First of all, the configuration of my network :
2 LAN Interfaces, 2 WAN Interfaces. The WAN interfaces are configured in FailOver.

The first WAN Interface, twice a day for a few hours, is getting High Latency (200ms, then 500ms and sometimes 1,5s).
When the latency is too high, I have some packet loss (around 10%) and pfSense switch to the other WAN interface which had a latency of 20-30ms. After a while, the latency on the other link is getting high also…

As test, when the latency was high, I unplugged the WAN interface and directly connect a computer on it to test the latency directly. And the latency was all right (20-30ms and not 1s with pfsense 2 minutes before).

So I suspect that my server is undersized for what I ask but the CPU charge is all right and the Memory also...

I'm using PfSense 2.0.1-RELEASE on a Dual Pentium 3 (yeah...) and 2GB of RAM.
And that's what I use on my configuration :
More than 150 computers, 80 rules on Firewall (maybe more), DHCP, DNS Forwarder, Squid, SquidGard, LightSquid, CaptivePortal...

Is that too much for the machine?

Thank you for your help.