[Solved] Wire memory slowly increasing



  • I have been seeing my wire memory usage slowly increase more than usual over the past month. I'm trying to understand why and I'm worried it's not gonna stop till it starts overflowing into the swap territory. I'm now at about 60% wire memory (Close to 75% total) and I have 8 GB of memory. That's a lot of memory usage!

    The network usage is pretty minimal in our office. The traffic on average is around 550 kbps. Average packets per second is around 500. I haven't notice any jumps or changes in those numbers during this past month when the wire memory is increasing. Looking at the 3 month trend, the wire memory is typically creeping up but this past month it's going up at a much faster rate. Any thoughts on why that's happening?

    I have two Intel EXPI9301CT gigabit desktop adapters installed.
    The rest of my system info is attached. I also attached the a 3 month memory graph and system activity from shell.

    Packages running:
    Squid
    ClamAV
    Suricata in legacy mode
    Pfblocker
    Syslog
    NUT
    OpenVPN
    ![System info.JPG](/public/imported_attachments/1/System info.JPG)
    ![System info.JPG_thumb](/public/imported_attachments/1/System info.JPG_thumb)
    ![Memory usage.JPG](/public/imported_attachments/1/Memory usage.JPG)
    ![Memory usage.JPG_thumb](/public/imported_attachments/1/Memory usage.JPG_thumb)
    ![system activity.JPG](/public/imported_attachments/1/system activity.JPG)
    ![system activity.JPG_thumb](/public/imported_attachments/1/system activity.JPG_thumb)



  • What is "wire memory"?  The only thing that comes to mind is an old memory technology that stored data on ferrous wire. Or perhaps ferrite core memory, in which the cores were woven into a core plane.  I have some of that here.



  • You are running several heavy packages, and squid is notorious for eating RAM.

    http://www.comfsm.fm/computing/squid/FAQ-8.html

    Shell in and run:

    squidclient -h YOUR_LAN_IP -p 3128 mgr:info
    

    and check the Resource usage for squid and Memory accounted for sections.



  • Thanks Kom, that is very useful info on squid.

    That command didn't seem to give me anything helpful. Below is what it returned. I used the IP of the LAN interface of the pfsense box.

    In any case, I saw one of the recommendation in that link was to lower the cache_mem aka memory cache size. I had it set to 1024 MB since I have a lot of memory. Maybe I was too ambitious with that so I lowered it back to the default 64 MB. I also lowered the Maximum object size in RAM back to the default 256 kB. I had that set to 1024 kB. I also cleared the local cache, but it hasn't made any impact on memory usage, at least not any immediate impact. If anything, wire memory went up a tad. I guess I'll keep an eye on it. Maybe it needs a reboot?

    HTTP/1.1 403 Forbidden
    Server: squid
    Mime-Version: 1.0
    Date: Wed, 02 May 2018 17:34:06 GMT
    Content-Type: text/html;charset=utf-8
    Content-Length: 3470
    X-Squid-Error: ERR_ACCESS_DENIED 0
    Vary: Accept-Language
    Content-Language: en
    X-Cache: MISS from pfsense
    X-Cache-Lookup: NONE from pfsense:3128
    Via: 1.1 pfsense (squid)
    Connection: close
    
    <title>ERROR: The requested URL could not be retrieved</title>
    
    # ERROR
    
    ## The requested URL could not be retrieved
    
    * * *
    
    The following error was encountered while trying to retrieve the URL: [cache_object://172.16.21.2/info](cache_object://172.16.21.2/info)
    
    > **Access Denied.**
    
    Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.
    
    Your cache administrator is [raffi@telebyteusa.com](mailto:raffi@telebyteusa.com?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20pfsense%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Wed,%2002%20May%202018%2017%3A34%3A06%20GMT%0D%0A%0D%0AClientIP%3A%20172.16.21.2%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Finfo%20HTTP%2F1.0%0AUser-Agent%3A%20squidclient%2F3.5.27%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20close%0D%0AHost%3A%20172.16.21.2%0D%0A%0D%0A%0D%0A).
    
    * * *
    
    Generated Wed, 02 May 2018 17:34:06 GMT by pfsense (squid)
    
    


  • This is a common error.

    You need to go to Services - Squid proxy server - Local Cache - Squid Cache General Settings - External Cache Managers.

    Set it to the LAN IP squid listens on as well as 127.0.0.1 like this:

    10.10.4.1;127.0.0.1

    Replace 10.10.4.1 with your LAN IP.  Save it and then try the squidclient command again.



  • Awesome! That did the trick. This looks more useful, but I don't know what it really means. The section on analyzing the cache manager output doesn't seem to apply to the version I'm using. Here is the new output. I think the total memory accounted for looks low. Only 11 MB.

    HTTP/1.1 200 OK
    Server: squid
    Mime-Version: 1.0
    Date: Wed, 02 May 2018 19:13:59 GMT
    Content-Type: text/plain;charset=utf-8
    Expires: Wed, 02 May 2018 19:13:59 GMT
    Last-Modified: Wed, 02 May 2018 19:13:59 GMT
    X-Cache: MISS from pfsense
    X-Cache-Lookup: MISS from pfsense:3128
    Via: 1.1 pfsense (squid)
    Connection: close
    
    Squid Object Cache: Version 3.5.27
    Build Info:
    Service Name: squid
    Start Time:     Wed, 02 May 2018 17:31:05 GMT
    Current Time:   Wed, 02 May 2018 19:13:59 GMT
    Connection information for squid:
            Number of clients accessing cache:      16
            Number of HTTP requests received:       3247
            Number of ICP messages received:        0
            Number of ICP messages sent:    0
            Number of queued ICP replies:   0
            Number of HTCP messages received:       0
            Number of HTCP messages sent:   0
            Request failure ratio:   0.00
            Average HTTP requests per minute since start:   31.6
            Average ICP messages per minute since start:    0.0
            Select loop called: 517484 times, 11.931 ms avg
    Cache information for squid:
            Hits as % of all requests:      5min: 0.4%, 60min: 2.3%
            Hits as % of bytes sent:        5min: 0.2%, 60min: 0.3%
            Memory hits as % of hit requests:       5min: 100.0%, 60min: 37.0%
            Disk hits as % of hit requests: 5min: 0.0%, 60min: 2.2%
            Storage Swap size:      5714 KB
            Storage Swap capacity:   0.1% used, 99.9% free
            Storage Mem size:       2680 KB
            Storage Mem capacity:    4.1% used, 95.9% free
            Mean Object Size:       55.48 KB
            Requests given to unlinkd:      6
    Median Service Times (seconds)  5 min    60 min:
            HTTP Requests (All):  54.42904 57.44813
            Cache Misses:          0.05951  0.06640
            Cache Hits:            0.00000  0.00091
            Near Hits:             0.00000  0.00000
            Not-Modified Replies:  0.00000  0.00091
            DNS Lookups:           0.02336  0.02562
            ICP Queries:           0.00000  0.00000
    Resource usage for squid:
            UP Time:        6173.953 seconds
            CPU Time:       16.255 seconds
            CPU Usage:      0.26%
            CPU Usage, 5 minute avg:        0.21%
            CPU Usage, 60 minute avg:       0.22%
            Maximum Resident Size: 266352 KB
            Page faults with physical i/o: 9
    Memory accounted for:
            Total accounted:        11164 KB
            memPoolAlloc calls:    726383
            memPoolFree calls:     748846
    File descriptor usage for squid:
            Maximum number of file descriptors:   233370
            Largest file desc currently in use:    215
            Number of file desc currently in use:  179
            Files queued for open:                   0
            Available number of file descriptors: 233191
            Reserved number of file descriptors:   100
            Store Disk files open:                   0
    Internal Data Structures:
               155 StoreEntries
               150 StoreEntries with MemObjects
               149 Hot Object Cache Items
               103 on-disk objects
    
    


  • "Wired memory" is a BSD specific term for kernel memory that is "wired in place" and can't be swapped out by the memory management. What you're seeing might be normal depending on the circumstances, unless your system runs out of memory and starts swapping heavily there might be no problem to fix.



  • Squid only appears to be using ~256 MB.



  • Oh, based on your top(1) output you are using ZFS and the ARC cache of ZFS is using 3465 MBs of wired memory. Looks totally normal to me. ARC cache is auto tuned/scaled based on the available memory so you don't really have to worry about it. The rest of the wired memory is used by other kernel data structures that can't be swapped out.



  • Thanks kps, nothing to worry about is what I want to hear. I'll keep an eye on it.

    Thanks Kom, I think squid may not be an issue then. The changes I made to squid didn't affect the memory use, so that kind of hinted at that too. I'm hoping it's nothing to worry about. I'll worry when swap starts being used. Thanks for the info on squid anyway. I'm glad I learned something new today.



  • If it ever becomes a problem there's a system tunable that you can set to limit the amount of memory allocated for the ARC cache. Place this in /boot/loader.conf.local (assuming you want the limit to be at 2048 MBs).

    
    vfs.zfs.arc_max=2048M
    
    


  • Great! Thanks. I'll keep that in mind if it does become a problem.


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy