Heavy CPU load?

Burken

I have a problem.. my CPU usage is to high…

I have 100Mbit/s download and 50Mbit/s upload.
In my router i use 3x Intel Pro 1000 MT Dual port

It seems that my CPU is my bottleneck...

I use Load balancing over 5 connections...
when i download torrents my CPU gets to high and stops the speed. it cant even make graphs over it...

Here is my CPU usage:

and here is my LAN speed:

And here is phpsysinfo:
http://pfsense.burken.biz:800/phpsysinfo/

Pentium 4 2.20GHz…
1GB ram...

Do i have to upgrade? or is something wrong?

marsboer

You have the same cards as I have and I tried to say something about it in this thread:
http://forum.pfsense.org/index.php/topic,16236.0.html

In the end I just installed Debian from scratch, turned off interrupt moderation (for minimum packet latency) (InterruptThrottleRate=0) and just use Linux without any kind of GUI for the firewall.
In Linux I use iptables (FW+NAT), dnsmasq (DNS+DHCP) and pppoeconf (ppp connection).
Don't use load balancing and traffic shaping yet, but I will implement a traffic shaping ruleset later.

The Intel NICs just won't shine on pfSense. I got much better performance from my card when using Linux. I don't know if this is a FreeBSD vs Linux issue or an em (FreeBSD) vs e1000e (linux) driver issue.

The problems with the current implementation are:
Much too high CPU usage, higher latency AND lot less throughput (must be running full duplex testing to show the difference, as pfSense easily maxes out the gigabit in one direction only with the kind of bulk traffic iperf generates). The throughput difference between pfSense and Debian firewall running on my computer is in the 600Mbit/s range when maxing the gigabit in both directions. Since you've got 3 of these cards with 6 interfaces total I guess your problems would ble noticed a lot sooner.

I get about 880/280Mbit/s with pfSense, and with my newly set up Debian fw I get 930/850 Mbit/s when running full duplex testing. Quite large difference.

Burken

oh my god.. not the answer i wanted to hear..

i looked on this site:

http://doc.pfsense.org/index.php/Hardware_requirements

the page says:
"Intel Pro/100 and Pro/1000 cards tend to be the best performing and most reliable on pfsense. Cheap cards like those containing Realtek chipsets (FreeBSD rl driver) are very poor performers in comparison. "

Thats why i buyd Intel Pro/1000….

:(

jbcorwin

Would the drivers by chance be open source?

–

Edit, I ask this wondering if they can be integrated on the BSD side to reflect the changes on the Linux side. Then again I show my lack of knowledge asking this question ;)

wallabybob

Here's something you could try to see if you can reduce the CPU utilisation in times of high traffic load.

Enable polling (In web GUI, System -> Advanced in section Miscellaneous, check the box labelled Use device polling). With polling enabled, the NICs have interrupts disabled and once a clock interrupt (current every millisecond) the device driver is called to for each interface to process received frames and frames whose transmission has completed. This tends to increase latency somewhat but has been observed to considerably reduce CPU time in some circumstances. I believe polling can be enabled and disabled without rebooting.

tehtrk

@Burken:

oh my god.. not the answer i wanted to hear..

i looked on this site:

http://doc.pfsense.org/index.php/Hardware_requirements

the page says:
"Intel Pro/100 and Pro/1000 cards tend to be the best performing and most reliable on pfsense. Cheap cards like those containing Realtek chipsets (FreeBSD rl driver) are very poor performers in comparison. "

Thats why i buyd Intel Pro/1000….

:(

Same here… what gives? My boxes aren't in production yet so I still have time to fix this issue, but not too much longer.

Any word on whether polling has helped?

Also, see http://forum.pfsense.org/index.php/topic,16236.0.html

marsboer

@wallabybob:

Here's something you could try to see if you can reduce the CPU utilisation in times of high traffic load.

Enable polling (In web GUI, System -> Advanced in section Miscellaneous, check the box labelled Use device polling). With polling enabled, the NICs have interrupts disabled and once a clock interrupt (current every millisecond) the device driver is called to for each interface to process received frames and frames whose transmission has completed. This tends to increase latency somewhat but has been observed to considerably reduce CPU time in some circumstances. I believe polling can be enabled and disabled without rebooting.

Device polling could be used to reduce CPU, yes, but my performance numbers with Linux is actually with no interrupt moderation at all, and certainly not using device polling.
So even if device polling reduces CPU usage, it doesn't fix the underlying problem.

With the dynamic interrupt moderation algorithm enabled in Linux the performance numbers are even higher, with just a tad more latency (still not pfSense levels though), but reduced CPU usage compared to running with InterruptThrottleRate=0,0 to get the lowest possible packet latency as I do. (Interrupt Moderation and Device Polling is not the same thing, by the way)

But we shouldn't make this a huge problem as the performance is still way good enough for almost everyone, but the current implementation is clearly lacking in maximum performance compared to the Linux implementation for the ones with firewalls that really get hit by huge amounts of traffic (maybe not so huge with complex scenarios using load balancing, shaping, large ruleset, vpn and so on)

charliem

In this post, it seems that FreeBSD 6.4 was OK, but there was a regression in 7.1 with the em driver:
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/performance/2009-04/msg00004.html

On the linux side, e1000e supports only the pci-e intel nics, while the e1000 supports the whole family. Interrupt moderation is only in the newer e1000e driver.

That said, it's all open source, and intel does supply drivers for FreeBSD:
http://downloadcenter.intel.com/T8Clearance.aspx?url=/17509/eng/em-6.9.8.tar.gz&agr=Y&ProductID=880&DwnldID=17509&lang=eng
but I haven't compared the intel drivers to what's in the FreeBSD tree. Maybe ping freebsd@intel.com to see if interrupt moderation can be added?

regards, …....... Charlie

Burken

Hello,

The "device polling" option did not help me.
I get 80Mbit/s MAX…

80Mbit/s = 100% CPU-load...

What NICs works best with pfsense then?

tell me and i will throw those shitty 3x "Intel PRO/1000 MT Dual Port Server Adapter" out the window!

My old D-link router for €20 gave me more throughput...

Perry

I was thinking. Your running on a pci bus not a pci-x bus correct?
1. I don't seem to remember seeing any post with 5 LB and p2p being used before. So before you throw them out it would be interesting to see the load when downloading a dvd distro with a download manager like getright.
2. Under System -> Advanced
Change Firewall Optimization Options to aggressive and set Firewall Maximum States to 500000
3. Will the load change if it was 5 different clients using 1 wan each.

Basically I tent to believe the bottleneck could be related to the pci bus, pps or slbd.

Eugene

According to graphs lion's share of load comes from 'system', why don't you give us

top
```that you get during the test.
slbd is known for its bad manners to create cpu-load.
try```
killall-9 slbd
```  and repeat (or during) the test. I do not believe that it's Intel's driver problem.

Burken

Hi guys!
Thx for trying to help me out..

Yes i only have PCI… but why all the CPU usage then?
I now have aggressive mode + 500000 states.
no change there..

then i did Eugenes tip.

No change…. :(

When the top were taken the total speed was ~88Mbit/s (torrents)

I need to find a fast server to do the getright/FTP test..

Perry

You could try with systat -vm 1 and others http://www.acmesecurity.org/~thiago/public/freebsd/FreeBSD_Bottleneck_Detection.pdf

I need to find a fast server to do the getright/FTP test..

For Ubuntu there lot's of location. AFAIR In Getright split the file into 5 segment and use file mirror to search for different locations.

Cry Havok

Torrents are generally a good way of doing bandwidth testing. Grab yourself the DVD of your chosen Linux distro (or a randomly selected one).

wallabybob

Your top output shows an unexpected (by me) large number of dhclient processes using an unexpectedly large amount of CPU time.

On my system:

uptime

7:22AM up 43 days, 10:39, 2 users, load averages: 0.31, 0.31, 0.26

ps ax | grep dhclient

335 ?? Is 0:00.51 dhclient: rl0 (dhclient)
284 con- I 0:00.10 dhclient: rl0 [priv] (dhclient)

On my system there are two dhclient processes which in 43 days haven't even used a second of CPU time between them while in yours (uptime of over 55 days) you have at least 7 dhclient processes which have each used at least 130 MINUTES of cpu time and all 7 are in the RUN state. On my system only the WAN interface (rl0) acquires an IP address by DHCP.

How many interfaces should be trying to acquire an IP address by DHCP?

Why are so many dhcp clients all in the run state? (Are your leases expiring every milli-second? :) )

Are there any log files which would give a hint as to why the DHCP clients are so busy?

Burken

its true it seems like _dhcp (dhclient) that makes the CPU load?

I have 5 NICS on DHCP… and DHCP-Server on the 6 interface...

i get alots entrys like this:
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em4
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em3
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em5
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em2

but i know why...:

They are all on the same VLAN…
:/

can this be the problem?

Eugene

May I ask you about the reason you have five WAN interfaces? with one ISP… :-\

wallabybob

@Burken:

I have 5 NICS on DHCP… and DHCP-Server on the 6 interface...

Its not clear to me what this means. I guess you are saying you have most (or all) of your interfaces serving DHCP addresses AND requesting DHCP addresses from another DHCP server. This is not a good idea. Your DHCP server interfaces should have static (fixed) IP addresses.

@Burken:

i get alots entrys like this:
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em4
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em3
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em5
May 28 19:51:02 kernel: arp: 85.226.120.1 is on em1 but got reply from 00:03:a0:3b:80:00 on em2

Your network topology and/or address assignments are messed up. 85.226.120.1 is accessible on multiple interfaces, it should be accessible over only one interface (unless you have bridged interfaces, but then why would you have a switch?) And printing these messages repeatedly will be another consumer of CPU time.

What are you trying to accomplish with this configuration? At first sight it appears overly complex.

marsboer

@Burken:

can this be the problem?

Yep. This is most likely your problem as the DHCP processes shouldn't be using any CPU at all.
It still doesn't solve the em-problems, but that's probably not what's limiting you with that massive CPU-usage from DHCP.
Probably your problem is solved by ensuring that the DHCP-server is not running on the WAN-interfaces as it seems that you are actually running DHCP-server on those in addition to the LAN-interface.
This should be a configurable setting.

Burken

My ISP gives me 10Mbit/s for every IP we use.
Max 5 IP-addresses

Thats why i use five NIC's to get my five IPs.

So with one IP 10Mbit/s with two 20Mbit/s.. and five 50Mbit/s

Okay?

My ISP will never give me static IPs
Always DHCP…

here is how it works:

em0/LAN Static 192.168.1.111. And runs DHCP Server for LAN clients.
em1/WAN Dynamic DHCP Client
em2/WAN1 Dynamic DHCP Client
em3/WAN2 Dynamic DHCP Client
em4/WAN3 Dynamic DHCP Client
em5/WAN4 Dynamic DHCP Client

If i do killall dhclient

My CPU usage get low. But pfsens stop working after a while.....

So what is wrong ?:(

Okay... kill dhclinet works... but the firewall dies so i have to restart it after a while....

chpalmer

All your ports share the same switch?

Ive never had good luck when I had two dhcp servers (your pfsense lan and your isp's modem) on the same switch…

Can you move your lan to another switch to rule that possible issue out?

Burken

I use VLANs.
So its physically one switch but inside they are different.

You can read about it here: http://en.wikipedia.org/wiki/VLAN

chpalmer

I know what you are trying to do so I guess Ill ask outright…

Have you ruled out a misconfiguration on your switch as the root cause of your problem?

What else have you tried in your troubleshooting process?

Start with the basics and add one element at a time until you can reproduce the result.

Your setup while innovative is not typical.

Good Luck!

Burken

There is nothing wrong with the Switch. As you can se here the Vlan settings is so simple.

You guys just helpt me to see that it is wrong with the dhclient.

what is wrong with my setup thats makes it non typical?

What else can i do to troubleshoot? i have killd dhclient and everyhing works fine..

Cry Havok

Most people either have:

Multiple interfaces, connecting to different ISPs
Multiple static IPs (possibly with one dynamic), on a single interface

It's very uncommon to have a single ISP, with multiple dynamic IPs across the same subnet on multiple interfaces, particularly using a single VLAN capable switch to separate WAN(s) and LAN.

Burken

I have tested with a second GS724T so VLAN works. Thats not the problem..

No1 else have problem with dhclient CPU usage?

Eugene

I am afraid your problem is network design. Everything esle is the result of this problem.

wallabybob

I'm trying to understand your configuration rather better because I also think its unusual. I take it you have 5 "WAN" interfaces from your ISP purely to get additional bandwidth.

From what you have displayed about your switch it looks as if you MIGHT be purely using its "VLAN" capability to segment the ports so as to isolate one group of ports ("LAN") from another group of ports ("WAN"). Correct?

Apparently 6 of the 24 switch ports are in the "WAN" LAN. Of those 6 ports 5 go to pfSense interfaces em1 through em5. From your network diagram your sixth port goes to your ISP but what does it actually connect to? Is there is a web page (in English) describing it or holding a pointer to a downloadable manual or datasheet? I'm guessing that its something that will allow up to 5 systems to connect to it, each able DHCP request an address and that these addresses are all on the same IP subnet. I've not come across anything like this that would assign additional bandwidth on the WAN (Internet) side with each additional IP address assigned. If we can find out a bit more about the equipment that connects you to the ISP we may be able to help solve your configuration problem.

The fact that you have 5 pfSense interfaces on the same LAN is a configuration error unless they are bridged. (Each interface should be on its own distinct IP subnet.) And why would you bridge them in pfSense when they are connected to a switch?

Burken

Eugene:
what is wrong with my network design then? im open for change. U just want my 50mbit and not 10mbit. thats the reason i installed pfsense.

wallabybob:
One of the biggest ISP's in sweden gives homes Fibre to the house and after that
one RJ45 contact in the wall. We get five public IP's. The download speed is 100Mbit. The upload speed is limited to 10Mbit for every IP we get. The advertised is "100/10". The reason we are geting 10Mbit/IP is just poor restrictions from there side.

And yes. 6 ports are "WAN" u can call them "WANswitch" and the other one "LANswitch" the 2 VLANS never get in contact. Everyting has to through pfsense.

The sixth port is from the RJ45 Connector in the wall.

The reason im using this network design is i got help to. I asked here and got told that i can't make virtual interfaces in freebsd. If i use virtualization i can bridge them easy and get new MAC for every virtual NIC.. But in this case we come up to the conclusion that we cudent make virtual NIC's.

My ISP don't like to say whats behind the walls… im using:
www.bredbandsbolaget.se
They doesn't even have a webpage en english.. but if there is something you guys need to know. I will be happy to call them and ask.

WAN (em1)

IP address  	85.226.121.133  
Subnet mask 255.255.248.0
Gateway 85.226.120.1 

ISP DNS servers  
195.54.122.199
81.26.227.3
195.54.122.204
81.26.228.3

WAN1 (em2)
IP address  	 85.226.122.10  
Subnet mask 	255.255.248.0
Gateway 	85.226.120.1 

WAN2 (em3)
IP address  	 85.226.122.11  
Subnet mask 	255.255.248.0
Gateway 	85.226.120.1 

WAN3 (em4)

IP address  	 85.226.122.20  
Subnet mask 	255.255.248.0
Gateway 	85.226.120.1 

WAN4 (em5)

IP address  	 85.226.122.23  
Subnet mask 	255.255.248.0
Gateway 	85.226.120.1

All ips are in the same subnet.

I can undertstand that you guys dont like vlans. So here is without VLAN configurations:

Still looks stupid and unusual? :(

Perry

Hmm to make load balancing work you have to have different gateways on wan's , do you have that?

If not I wonder if a esxi server could be used so no additional hardware would be needed.

Burken

Cry Havok

Do you have to be actively uploading from all 5 IPs? Can you simply have them allocated to you, or does there have to be a device using those IPs?

The simple I can see are:

Have the IPs allocated, don't use them
Allocate 4 of them to another device that you don't use
Insert simple firewall/routers between pfSense and the Internet connection, each with a different LAN subnet

As for what is wrong with your design:
@wallabybob:

The fact that you have 5 pfSense interfaces on the same LAN is a configuration error unless they are bridged. (Each interface should be on its own distinct IP subnet.) And why would you bridge them in pfSense when they are connected to a switch?

Burken

I can remove four of the uplinks… Then only use one WAN...
then i can check if dhclient still uses that much CPU?

Eugene

I am sorry probably I am stupid but I still can not understand.
Your provider gives you one RJ-45 cable and 5 public IPs belonging to the same subnet. It allows you to download at 100Mb/s and upload with 20Mb/s per IP. And (what is most interesting) you have to acquire all 5 public IPs through DHCP.
Please tell me that I am wrong.
If everything above is correct I am afraid you can not use all 5 IPs without having 5 routers.
I would ask provider to provide me with 1 public IP and allow me to upload at 100 Mb/s paying the same price as you do now for 5 IPs.

PS: we love VLANs!

blewis

I must agree that the provider is giving you such high bandwidth w/ such a setup is to prevent people from aggregating the bandwidth. You can use separate gateways for each wan. Several years ago, I've seen it done on a clarckconnect setup, until they started charging a subscription for such features. They just package a bunch of other people's "hard work/ingenuity" into a very nice gui and charge quite a bit for. I believe it's based on Centos/Redhat/Linux. It's quite reliable, I've been told. You might wanna check it out.