Huge difference between real ram vs avail ram.



  • good day all,
    Let me first say that I love pfsense. I just started using it and have found it to be an excellent project and product. Please forgive me if this question is silly, but recently I added some packages to my install (snort, squid, squidguard) and just out of curiosity I was checking my memory usage (I have 2x1GB sticks in the machine) and noticed that the output of;
    'dmesg | grep memory' shows

    $ dmesg | grep memory
    real memory  = 2112749568 (2014 MB)
    avail memory = 1520783360 (1450 MB)

    Can anyone tell me why I am seeing a +500MB difference? Is it taking into account the memory I am currently using, I assume it is not since the output of 'top' is

    Mem: 344M Active, 14M Inact, 75M Wired, 20K Cache, 112M Buf, 1022M Free
    Swap: 4096M Total, 4096M Free

    And that also does not seem to add up to ~2GB. I know it will not be exactly 2048MB but ~500MB difference seems excessive.

    Can anyone help me out on this? Thanks in advance for your help. Any assistance would be greatly appreciated.



  • The BIOS typically uses a few pages of RAM for its data structures (including ACPI data). If you have an "on motherboard" video adapter, that will also typically use a few MB as "video ram".

    If I recall correctly, real memory is what the BIOS tells the OS is available for the OS to use (amount of RAM less what BIOS requires and video RAM) while available memory is real memory less memory required to manage memory (the page tables and page use data structure and such). If this is correct, then the difference between the amount of RAM in the system and real memory will be pretty much independent of the amount of RAM in the system while the difference between real memory and avail memory will grow as the amount of RAM in the system grows (more pages to manage).

    Its a while since I looked into this (and that was FreeBSD 6.x). I think the kernel heap is allowed to grow to a certain size which (by default) depends on the amount of RAM. Page tables to map the kernel heap are preallocated on startup, probably to significantly simplify the expansion of this area.



  • @wallabybob:

    The BIOS typically uses a few pages of RAM for its data structures (including ACPI data). If you have an "on motherboard" video adapter, that will also typically use a few MB as "video ram".

    If I recall correctly, real memory is what the BIOS tells the OS is available for the OS to use (amount of RAM less what BIOS requires and video RAM) while available memory is real memory less memory required to manage memory (the page tables and page use data structure and such). If this is correct, then the difference between the amount of RAM in the system and real memory will be pretty much independent of the amount of RAM in the system while the difference between real memory and avail memory will grow as the amount of RAM in the system grows (more pages to manage).

    Its a while since I looked into this (and that was FreeBSD 6.x). I think the kernel heap is allowed to grow to a certain size which (by default) depends on the amount of RAM. Page tables to map the kernel heap are preallocated on startup, probably to significantly simplify the expansion of this area.

    Thank you for your response… though I am not sure I understand all of it :). If I understand it correctly, your saying the kernel is allowed to preallocated page tables for its own use? something like caching? Is 500 MB reasonable in that regard? i mean if BIOS reports say 2014MB, then this is real memory, but is it supposed to use +500MB to manage 2014MB? and if so, then is it really only managing the avail memory? I am just trying to understand if it is actually using all the ram? I know that question is a bit of an oversimplification, but is that "unavailable" ram being utilized as ram or for kernel functions? is it designed that way? and is that specific to freebsd? Again thank you for your response, and I apologize if any of my questions don't make sense. I am just trying to get a better understanding of this concept.



  • My previous response was meant to be qualitative rather than quantitative, indicative rather than exhaustive.

    I don't currently have access to FreeBSD sources on a local system so it would be quite time consuming to do the necessary trolling through the source files to get detailed answers to your questions.

    Here's a bit of quantitative speculation to help illustrate why a fair bit of memory might be needed to "manage" 2GB. A common thing an operating system wants to do is allocate and free pages. Pages can be in a number of different states: free, locked in memory (can't be swapped or paged, example: kernel code, kernel data, active i/o buffers), swappable/pageable etc. We might want to keep some flags about pages (example: a page is shared by a number of users, shared data or code) and a reference count (of sharers) to avoid "freeing" a page which has been released but is still in use by other sharers. So, for each page we might want a data structure with a couple of pointers (to allow quick insertion and removal from doubly linked lists of free pages, locked pages etc, (2 x 4 bytes), flags (say 32 bits), reference counter (say 32 bits). Thats 16 bytes. In a kernel for multiprocessor systems we might also want to add a lock to prevent multiple CPUs trying to change characteristics of the same page at the same time. We might also want to add some debugging information, say 32 bytes all up.

    On a system with 2GB and a page size of 4kB there are 512K pages. So an array of page structures (one element per page) will occupy 325121024 bytes = 1610241024 bytes (16MB).

    Suppose the kernel is allowed to grow to 1GB and it is decided to preallocate the page map pages for the kernel to avoid the complication of having an attempt to grow the kernel fail. The preallocated page tables require a little more than 1MB of RAM. (Note this is not the same as preallocating the 1GB kernel space.)

    @houms:

    Thank you for your response… though I am not sure I understand all of it :). If I understand it correctly, your saying the kernel is allowed to preallocated page tables for its own use?

    Yes.

    something like caching?

    yes (f preallocation is like caching)

    Is 500 MB reasonable in that regard?

    Don't know. I haven't done an exhaustive calculation.

    i mean if BIOS reports say 2014MB, then this is real memory, but is it supposed to use +500MB to manage 2014MB?

    See previous answer.

    and if so, then is it really only managing the avail memory?

    My list of examples of the likely use of that 500MB memory was not meant to be an exhaustive list.

    I am just trying to understand if it is actually using all the ram? I know that question is a bit of an oversimplification, but is that "unavailable" ram being utilized as ram or for kernel functions?

    If FreeBSD swaps or pages its probably using all the available RAM. If it is not swapping and not paging its probably not using all the available RAM.

    The default FreeBSD kernel build in FreeBSD 6.x allowed for a maximum kernel size of 1GB. (I don't know about pfSense nor about FreeBSD 7.x.) If the same applied to pfSense in basic firewall applications there would be little point in having much more than 1GB RAM because the basic firewall application generally does most of its work in the kernel. Of course some applications run sometimes so it would be good to have a bit more than 1GB. In a lot of cases, 2GB will be the next practical step up from 1GB. If the firewall runs applications that use a lot of RAM (snort is reportedly one of those) then additional RAM could be useful.


Log in to reply