Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Quality RRD data collection on WAN apparently stopped nearly a year ago

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 5 Posters 5.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      kejianshi
      last edited by

      When mine stopped, I deleted RRD data and it started working fine again  (Probably not the fix you were hoping for?)

      1 Reply Last reply Reply Quote 0
      • T
        torontob
        last edited by

        @kejianshi:

        When mine stopped, I deleted RRD data and it started working fine again  (Probably not the fix you were hoping for?)

        Thanks - Is there a way to safely delete other than RESET?
        This is annoying. I appreciate any other feedback.

        1 Reply Last reply Reply Quote 0
        • W
          wallabybob
          last edited by

          @torontob:

          I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues

          When the kernel goes killing "random" processes because there is no swap space file corruption is quite likely. I suspect you will probably need to get the swap space issue fixed to resolve this, but the feedback so far suggests "not until 2.2".

          Does the offending system run the nano-BSD variant?

          1 Reply Last reply Reply Quote 0
          • T
            torontob
            last edited by

            Sorry, what you mean "not until 2.2"? Is this a known issue?
            I downloaded a snapshot of 2.1-RC1 and installed. No changes to base operating system.

            Thanks,

            1 Reply Last reply Reply Quote 0
            • D
              doktornotor Banned
              last edited by

              @torontob:

              I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues but here are the logs:

              There is no swap used on nanobsd. You simply are running out of RAM => trying to do too much on an inadequate HW.

              1 Reply Last reply Reply Quote 0
              • W
                wallabybob
                last edited by

                @torontob:

                Sorry, what you mean "not until 2.2"? Is this a known issue?

                See discussion in http://forum.pfsense.org/index.php/topic,66188.0.html in particular see reply #2 in which ermal says

                There is a plan hopefully to be ready fro 2.2 to make this a non-issue.

                I don't know that fixing the "out of swap space" issue will fix your RRD issue but I do think it is likely that your RRD issue is caused by the kernel killing processes (due to running out of swap space) and that leaving one or more RRD data files "corrupt".

                1 Reply Last reply Reply Quote 0
                • T
                  torontob
                  last edited by

                  @doktornotor:

                  @torontob:

                  I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues but here are the logs:

                  There is no swap used on nanobsd. You simply are running out of RAM => trying to do too much on an inadequate HW.

                  Guys, I am using THE standard hardware - Alix2D13 - and I have only 30 SIP phones on pfSense. I have put much much much more on this hardware and it hasn't run out of RAM ever. I would like to clarify two things though and I think we are getting closer:

                  I see that it's not a swap issue (wallabybob) that is causing RRD graphs issue but killing processes due to that may be. So, what is my next step in diagnosing?

                  1 Reply Last reply Reply Quote 0
                  • D
                    doktornotor Banned
                    last edited by

                    @torontob:

                    I see that it's not a swap issue (wallabybob) that is causing RRD graphs issue but killing processes due to that may be. So, what is my next step in diagnosing?

                    Dude, that message tells you that you have run out of RAM. Period.

                    1 Reply Last reply Reply Quote 0
                    • P
                      phil.davis
                      last edited by 

                      Sep 7 00:12:38	kernel: pid 91802 (php), uid 0, was killed: out of swap space
Sep 7 00:12:40	kernel: pid 43574 (php), uid 0, was killed: out of swap space
Sep 7 00:12:42	kernel: pid 93676 (php), uid 0, was killed: out of swap space

                      I just want to clarify to make sure it is understood. On nanoBSD there is no swap partition or file, i.e. swap space is zero. Therefore when it says "out of swap space" it really means "I needed some swap space, but there is none" =  "out of real memory".
                      Some people (including me with some configs) have been having this issue on 2.1 with 256MB RAM systems. It seems to be a transient thing, for me particularly when WAN links go up/down and a few OpenVPN links reestablish at pretty much the same time. A lot of processes fire up initialising/restarting various things and for those few seconds there is not enough real memory for it all. By using the new System Watchdog package, things normally recover because after a minute System Services Watchdog wakes up and restarts important things that got killed.

                      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                      1 Reply Last reply Reply Quote 0
                      • T
                        torontob
                        last edited by

                        @phil.davis:

                        Sep 7 00:12:38	kernel: pid 91802 (php), uid 0, was killed: out of swap space
Sep 7 00:12:40	kernel: pid 43574 (php), uid 0, was killed: out of swap space
Sep 7 00:12:42	kernel: pid 93676 (php), uid 0, was killed: out of swap space

                        I just want to clarify to make sure it is understood. On nanoBSD there is no swap partition or file, i.e. swap space is zero. Therefore when it says "out of swap space" it really means "I needed some swap space, but there is none" =  "out of real memory".
                        Some people (including me with some configs) have been having this issue on 2.1 with 256MB RAM systems. It seems to be a transient thing, for me particularly when WAN links go up/down and a few OpenVPN links reestablish at pretty much the same time. A lot of processes fire up initialising/restarting various things and for those few seconds there is not enough real memory for it all. By using the new System Watchdog package, things normally recover because after a minute System Watchdog wakes up and restarts important things that got killed.

                        Thanks for the clarification. So, I meant I have not done any modifications to cause swap issues. I do have couple OpenVPN links and do use DSL. So, if that becomes a problem and that is not a problem on 2.0 (which it is not for my other boxes) then it should be treated as a bug. Maybe tune down everything or create a delay for services to start?

                        Also, I see Services Watchdog - not System Watchdog - is that what you were referring to? Never used it though. Is that a stable package? (shows beta). I am wondering if that in itself creates more overhead sending me into a vicious circle :)

                        1 Reply Last reply Reply Quote 0
                        • W
                          wallabybob
                          last edited by

                          @torontob:

                          Also, I see Services Watchdog - not System Watchdog - is that what you were referring to? Never used it though. Is that a stable package? (shows beta). I am wondering if that in itself creates more overhead sending me into a vicious circle :)

                          And I can't see how Services Watchdog will help if RRD data is getting corrupted because processes are killed due to swap space exhaustion.

                          1 Reply Last reply Reply Quote 0
                          • P
                            phil.davis
                            last edited by

                            I meant Services Watchdog - I fixed the text in the previous.
                            Services Watchdog works well for what it does, a simple check every minute of what is running or missing. Of course, it sometimes happens to run at the same time as some down/up event on WAN. So then it can try to start things that are already in the process of starting by the link up/down scripts, and also use memory itself when memory is short. Services Watchdog is really intended to restart services that should never go missing but genuinely happen to crash (program crashes, divide by zero…). To recover nicely from service exits due to low memory it needs more parameters so it can wait a bit longer if the service is down, then only start it if it has been missing for some time - avoiding race conditions when other code is in the process of getting a WAN link into the up state again.

                            We have made code to "slow down" the boot process also - https://github.com/pfsense/pfsense/pull/798 - this helps ensure that critical things start OK during the boot. As Ermal comments, it is a "kludge" - using a heuristic that waits for free memory to stabilize in order to infer that the thing just started has initialised and stabilized, and it is time to move on in the bootup process. This "kludge" does nothing for memory problems due to WAN down/up etc after the system is booted. But for us, at least it gets us a functional system at boot with site-to-site OpenVPN. That means we can login remotely and are not stuck with doing telephone support to someone in a remote office after a power failure.

                            My Alix 256MB systems typically run with memory usage on the dashboard at 45% to 55%. As long as all the physical links and OpenVPN links are working nicely, nothing restarts and there is plenty of memory. So these Alix systems can run with configs such as like I have: 2 WANs, 3 LANs (on VLAN switch), OpenVPN server for site-to-site links with 9 client offices connecting, OpenVPN server for Road Warrior with a handful of client connections, DHCP, DNS, only Cron, Sudo, Service Watchdog and OpenVPN Client Export packages. It just needs some serialization of real-time event handling so that the free memory is used in a nicely controlled manner when recovering from events.

                            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                            1 Reply Last reply Reply Quote 0
                            • P
                              phil.davis
                              last edited by

                              @wallabybob:

                              @torontob:

                              Also, I see Services Watchdog - not System Watchdog - is that what you were referring to? Never used it though. Is that a stable package? (shows beta). I am wondering if that in itself creates more overhead sending me into a vicious circle :)

                              And I can't see how Services Watchdog will help if RRD data is getting corrupted because processes are killed due to swap space exhaustion.

                              Correct - if the only problem is the original post about corrupted RRD data, then that can happen whenever the "killed" happens randomly during the RRD data writing process, leaving part-updated corrupt data files. To fix that, the system has to be fixed so that "killed" never happens.
                              Services Watchdog or other recovery methods are only useful to get processes running again, as long as their associated data files are in good shape.

                              As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                              If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                              1 Reply Last reply Reply Quote 0
                              • T
                                torontob
                                last edited by

                                Thanks wallabybob for input.
                                Since the topic changed a bit, I might as well ask this question. From time to time, I see that OpenVPN is not trying to re-establish a connection. Why is that? why is there a need for something like Watchdog and why won't vital services just automatically restart?

                                Thanks,

                                1 Reply Last reply Reply Quote 0
                                • P
                                  phil.davis
                                  last edited by

                                  If a critical service process (dhcp, dns, OpenVPNs…) exits abnormally (system problem like out of swap space, program problem like divide by zero) there is no controlling script that called it that then gets the error code, and can loop around to try and run the program again. These services are forked off into independent processes by the bootup scripts.
                                  There is nothing else built into the system that monitors AND restarts them automatically. Yes, there are dashboard displays, but they don't take automatic action. And in any case, what if nobody has a dashboard running.
                                  So JimP has kindly made Service Watchdog - in a perfect world it would not be needed, just like a real watchdog.

                                  OpenVPN: in my experience the OpenVPN code is very good at trying forever to connect and eventually connecting once the underlying physical links and internet is up and working. I expect that an OpenVPN client would only stop trying to establish a connection if the process has actually crashed, which I have only seen happen because of "killed: out of swap space".

                                  As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                  If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.