High MBUF usage
-
Hi netgate brain Trust,
I have a pfsense router directly installed on a HP prodesk computer running my small community wireless ISP business. Recently the router crashed and stopped passing traffic, and I couldn't find the cause in any of the log files. Had to pull the power as couldn't even log in to GUI. I have noticed the MBUF usage is very high, around 70% of its maximum (32768). I have 1 Chelsio card and 2 Intel cards. If this value gets too High, is that likely to crash the pfsense router? I have seen other posts suggesting I increase the cluster value in the conf file as below:
kern.ipc.nmbclusters="1000000"
Anyone else had this issue?
Thanks
-
J jimp moved this topic from Problems Installing or Upgrading pfSense Software on
-
What pfSense version is that?
Did you have a crash report after rebooting?
Anything in the system logs showing nmbufs exhausted?
Steve
-
@stephenw10 Hi there,
No I looked for a crash report but couldn't find one. All the logs started from the reboot so not very helpful. Unless I'm looking in the wrong place? What should the MFUB percentage usually be in normal operation? I am on the latest stable version.
Cheers
-
I've also got another wierd problem, everyday for the past week, my WAN in throughput has been through the roof, so much so that it's maxing out my connection. Its not continuous, but it does it for a while then goes back to normal. No clients on the network are using all this bandwidth as the other graphs for the interfaces are normal, there is no explanation as to where this data is going. Could this be an internal service thats doing this? I can't find out where to look to find out what process is using a huge amount of bandwidth, see attached photo.
-
@richvalleywirelessisp said in High MBUF usage:
Could this be an internal service thats doing this?
Very unlikely. Not anything legitimate at least. There's nothing that would download data like that for no reason.
Anything pulling in that sort of data is going to be using significant CPU if it really is a process on the firewall. Check the system activity while it's happening.
It's also possible it's data being pushed to the firewall that's not going anywhere. Do you have firewall rules on WAN allowing incoming connections?
What pfSense version is that? What packages do you have installed?
Steve
-
@stephenw10 Hi Steve,
When this occurs yes there is High cpu usage but the data doesn't seem to be going through any other interface except the incoming WAN, which is why I was suspicious of an internal service. However I can't find a way of locating what is using all this bandwidth. I am on the latest version of pfsense (2.6.0).
I do have some rules regarding incoming connections on my WAN as I have public IPs from my ISP. Maybe I'm allowing traffic I shouldn't be which then can't go anywhere because of my other network rules (my internal networks are segregated).
I have attached a list of running services. How do I check the bandwidth / traffic usage of internal services? This appears to have started about 2 weeks ago.
Thanks
-
You should check Diag > System Activity inn the gui or run `top -HaSP' at the command line.
Are you running services behind that firewall that need to be publicly accessible? If not you shouldn't have any rules open on WAN.
-
@stephenw10 yes I do have some incoming WAN rules in place for specific public IP's running services. However none of those showed unusually high data totals. By process of elimination I actually think it was my Squid proxy server service downloading and caching updates. Only the internal services have unrestricted access to the full 2GB bandwidth, all other users and clients are either locked down by the 1GB ports or bandwidth limited further downstream. I'm still concerned about the high MBUF usage (hovers around 65-75% most days) but I will apply that line of code in to the config from the first post and reboot, see if that solves it. I don't want the pfsense machine crashing all the time!
Thanks for your help so far!
-
@richvalleywirelessisp said in High MBUF usage:
yes I do have some incoming WAN rules in place for specific public IP's running services.
If no PPPoE is in usage you may be able to set up the queue amount and size also! Pending on the used CPU
and the CPU cores & HTs. One core = one WAN queue.However none of those showed unusually high data totals. By process of elimination I actually think it was
my Squid proxy server service downloading and
caching updates.What amount of RAM disk (Squid) you were setting up?
Please have a look for that and high it up if enough
free RAM is in the system. ClamAV running?Only the internal services have unrestricted access to
the full 2GB bandwidth, all other users and clients are either locked down by the 1GB ports or bandwidth
limited further downstream.Perhaps it will be nice to get in touch with one or two
10 GBit/s ports to the DMZ and LAN switch breaking bottlenecks here too?I'm still concerned about the high MBUF usage
(hovers around 65-75% most days) but I will
apply that line of code in to the config from
the first post and reboot, see if that solves it.How many RAM ist installed in that system?
I don't want the pfsense machine crashing all the time!
Then perhaps you may be having a overview to the entire
system and not only the one point as today? Are servers
on your site you provide your clients with? Have a look
on the state tables too please. -
Hi Dobby
*@dobby_ said in High MBUF usage:
If no PPPoE is in usage you may be able to set up the queue amount and size also! Pending on the used CPU
and the CPU cores & HTs. One core = one WAN queue.*No PPPOE just DHCP - the normal CPU usage is very low <2%
*> What amount of RAM disk (Squid) you were setting up?
Please have a look for that and high it up if enough
free RAM is in the system. ClamAV running?*No RAM disk for squid but have a large HDD dedicated to cache and DNS requests (around 400GB). 8GB RAM (using about 20%) and 8GB swap space.
Not using CLAM AV, only Snort and Squid Proxy, and DNS resolver. This is a DMZ/Perimeter gateway set up to provide public IPs to other routers and function as a DNS server and web caching system.*> Perhaps it will be nice to get in touch with one or two
10 GBit/s ports to the DMZ and LAN switch breaking bottlenecks here too?*
The pfsense machine has both 10GB WAN link to the fibre line plus a 10GB link to the core network.
Then perhaps you may be having a overview to the entire
system and not only the one point as today? Are servers
on your site you provide your clients with? Have a look
on the state tables too please.State tables are sitting about 1%. So no problems there. I think the issue is still with the MBUF usage.
Thanks
-
@richvalleywirelessisp said in High MBUF usage:
8GB RAM (using about 20%)
Then it makes perhaps sense (step-by-step) high up
the amount and the size of the mbufer. You have to
watch over the RAM (how much is in usage) after
setting the tuneables and after all is fine "weighted"
or tuned you should write this tune ables into the
"/boot/loader.conf.local" file, to persist the next
updates/upgrades. -
@richvalleywirelessisp Not here, I have often wondered why there is so much fuss around mbuf's as even on my busiest devices they remain extremely low, what type of usage tends to increase mbuf usage? If you find the answer to that question it might help you out aside from the obvious of increasing the limit.
