[Solved] Wire memory slowly increasing
I have been seeing my wire memory usage slowly increase more than usual over the past month. I'm trying to understand why and I'm worried it's not gonna stop till it starts overflowing into the swap territory. I'm now at about 60% wire memory (Close to 75% total) and I have 8 GB of memory. That's a lot of memory usage!
The network usage is pretty minimal in our office. The traffic on average is around 550 kbps. Average packets per second is around 500. I haven't notice any jumps or changes in those numbers during this past month when the wire memory is increasing. Looking at the 3 month trend, the wire memory is typically creeping up but this past month it's going up at a much faster rate. Any thoughts on why that's happening?
I have two Intel EXPI9301CT gigabit desktop adapters installed.
The rest of my system info is attached. I also attached the a 3 month memory graph and system activity from shell.
Suricata in legacy mode
![System info.JPG](/public/imported_attachments/1/System info.JPG)
![System info.JPG_thumb](/public/imported_attachments/1/System info.JPG_thumb)
![Memory usage.JPG](/public/imported_attachments/1/Memory usage.JPG)
![Memory usage.JPG_thumb](/public/imported_attachments/1/Memory usage.JPG_thumb)
![system activity.JPG](/public/imported_attachments/1/system activity.JPG)
![system activity.JPG_thumb](/public/imported_attachments/1/system activity.JPG_thumb)
JKnott last edited by
What is "wire memory"? The only thing that comes to mind is an old memory technology that stored data on ferrous wire. Or perhaps ferrite core memory, in which the cores were woven into a core plane. I have some of that here.
You are running several heavy packages, and squid is notorious for eating RAM.
Shell in and run:
squidclient -h YOUR_LAN_IP -p 3128 mgr:info
and check the Resource usage for squid and Memory accounted for sections.
Thanks Kom, that is very useful info on squid.
That command didn't seem to give me anything helpful. Below is what it returned. I used the IP of the LAN interface of the pfsense box.
In any case, I saw one of the recommendation in that link was to lower the cache_mem aka memory cache size. I had it set to 1024 MB since I have a lot of memory. Maybe I was too ambitious with that so I lowered it back to the default 64 MB. I also lowered the Maximum object size in RAM back to the default 256 kB. I had that set to 1024 kB. I also cleared the local cache, but it hasn't made any impact on memory usage, at least not any immediate impact. If anything, wire memory went up a tad. I guess I'll keep an eye on it. Maybe it needs a reboot?
HTTP/1.1 403 Forbidden Server: squid Mime-Version: 1.0 Date: Wed, 02 May 2018 17:34:06 GMT Content-Type: text/html;charset=utf-8 Content-Length: 3470 X-Squid-Error: ERR_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en X-Cache: MISS from pfsense X-Cache-Lookup: NONE from pfsense:3128 Via: 1.1 pfsense (squid) Connection: close <title>ERROR: The requested URL could not be retrieved</title> # ERROR ## The requested URL could not be retrieved * * * The following error was encountered while trying to retrieve the URL: [cache_object://172.16.21.2/info](cache_object://172.16.21.2/info) > **Access Denied.** Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect. Your cache administrator is [email@example.com](mailto:firstname.lastname@example.org?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20pfsense%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Wed,%2002%20May%202018%2017%3A34%3A06%20GMT%0D%0A%0D%0AClientIP%3A%20172.16.21.2%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Finfo%20HTTP%2F1.0%0AUser-Agent%3A%20squidclient%2F3.5.27%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20close%0D%0AHost%3A%20172.16.21.2%0D%0A%0D%0A%0D%0A). * * * Generated Wed, 02 May 2018 17:34:06 GMT by pfsense (squid)
This is a common error.
You need to go to Services - Squid proxy server - Local Cache - Squid Cache General Settings - External Cache Managers.
Set it to the LAN IP squid listens on as well as 127.0.0.1 like this:
Replace 10.10.4.1 with your LAN IP. Save it and then try the squidclient command again.
Awesome! That did the trick. This looks more useful, but I don't know what it really means. The section on analyzing the cache manager output doesn't seem to apply to the version I'm using. Here is the new output. I think the total memory accounted for looks low. Only 11 MB.
HTTP/1.1 200 OK Server: squid Mime-Version: 1.0 Date: Wed, 02 May 2018 19:13:59 GMT Content-Type: text/plain;charset=utf-8 Expires: Wed, 02 May 2018 19:13:59 GMT Last-Modified: Wed, 02 May 2018 19:13:59 GMT X-Cache: MISS from pfsense X-Cache-Lookup: MISS from pfsense:3128 Via: 1.1 pfsense (squid) Connection: close Squid Object Cache: Version 3.5.27 Build Info: Service Name: squid Start Time: Wed, 02 May 2018 17:31:05 GMT Current Time: Wed, 02 May 2018 19:13:59 GMT Connection information for squid: Number of clients accessing cache: 16 Number of HTTP requests received: 3247 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 31.6 Average ICP messages per minute since start: 0.0 Select loop called: 517484 times, 11.931 ms avg Cache information for squid: Hits as % of all requests: 5min: 0.4%, 60min: 2.3% Hits as % of bytes sent: 5min: 0.2%, 60min: 0.3% Memory hits as % of hit requests: 5min: 100.0%, 60min: 37.0% Disk hits as % of hit requests: 5min: 0.0%, 60min: 2.2% Storage Swap size: 5714 KB Storage Swap capacity: 0.1% used, 99.9% free Storage Mem size: 2680 KB Storage Mem capacity: 4.1% used, 95.9% free Mean Object Size: 55.48 KB Requests given to unlinkd: 6 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 54.42904 57.44813 Cache Misses: 0.05951 0.06640 Cache Hits: 0.00000 0.00091 Near Hits: 0.00000 0.00000 Not-Modified Replies: 0.00000 0.00091 DNS Lookups: 0.02336 0.02562 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 6173.953 seconds CPU Time: 16.255 seconds CPU Usage: 0.26% CPU Usage, 5 minute avg: 0.21% CPU Usage, 60 minute avg: 0.22% Maximum Resident Size: 266352 KB Page faults with physical i/o: 9 Memory accounted for: Total accounted: 11164 KB memPoolAlloc calls: 726383 memPoolFree calls: 748846 File descriptor usage for squid: Maximum number of file descriptors: 233370 Largest file desc currently in use: 215 Number of file desc currently in use: 179 Files queued for open: 0 Available number of file descriptors: 233191 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 155 StoreEntries 150 StoreEntries with MemObjects 149 Hot Object Cache Items 103 on-disk objects
"Wired memory" is a BSD specific term for kernel memory that is "wired in place" and can't be swapped out by the memory management. What you're seeing might be normal depending on the circumstances, unless your system runs out of memory and starts swapping heavily there might be no problem to fix.
Squid only appears to be using ~256 MB.
Oh, based on your top(1) output you are using ZFS and the ARC cache of ZFS is using 3465 MBs of wired memory. Looks totally normal to me. ARC cache is auto tuned/scaled based on the available memory so you don't really have to worry about it. The rest of the wired memory is used by other kernel data structures that can't be swapped out.
Thanks kps, nothing to worry about is what I want to hear. I'll keep an eye on it.
Thanks Kom, I think squid may not be an issue then. The changes I made to squid didn't affect the memory use, so that kind of hinted at that too. I'm hoping it's nothing to worry about. I'll worry when swap starts being used. Thanks for the info on squid anyway. I'm glad I learned something new today.
If it ever becomes a problem there's a system tunable that you can set to limit the amount of memory allocated for the ARC cache. Place this in /boot/loader.conf.local (assuming you want the limit to be at 2048 MBs).
Great! Thanks. I'll keep that in mind if it does become a problem.