Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Quality RRD data collection on WAN apparently stopped nearly a year ago

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 5 Posters 5.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      torontob
      last edited by

      I have to reset RRD graphs to get either of WAN_PPPoE Quality or Traffic every few days on a 2.1-RC1. This is really annoying now. I looked at your findings and it seems to not be my issue as I have the target DSL PPPoE gateway in my apinger.conf file. Here are my files:

      $ ls -l /var/db/rrd/
total 15352
-rw-r--r--  1 nobody  wheel   47608 Sep  8 11:16 PBXVPN_VPNV4-quality.rrd
-rw-r--r--  1 nobody  wheel   47608 Sep  8 11:16 VPNINBOUND_VPNV4-quality.rrd
-rw-r--r--  1 nobody  wheel   47608 Sep  8 11:16 WAN_PPPOE-quality.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 ipsec-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 ipsec-traffic.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 lan-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 lan-traffic.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 opt1-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 opt1-traffic.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 opt2-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 opt2-traffic.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 opt3-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 opt3-traffic.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 ovpns2-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 ovpns2-traffic.rrd
-rw-r--r--  1 nobody  wheel   49632 Sep  7 00:13 ovpns2-vpnusers.rrd
-rw-r--r--  1 nobody  wheel  588376 Sep  7 00:13 system-mbuf.rrd
-rw-r--r--  1 nobody  wheel  735104 Sep  7 00:13 system-memory.rrd
-rw-r--r--  1 nobody  wheel  245888 Sep  7 00:13 system-processor.rrd
-rw-r--r--  1 nobody  wheel  245888 Sep  7 00:13 system-states.rrd
-rw-r--r--  1 root    wheel    8799 Sep  7 00:13 updaterrd.sh
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 wan-packets.rrd
-rw-r--r--  1 nobody  wheel  393080 Sep  7 00:13 wan-traffic.rrd

      I think this is a serious issue with PPPoE? I have seen this on all boxes with RC1 since last month :(

      Any thoughts?

      Thanks,

      1 Reply Last reply Reply Quote 0
      • W
        wallabybob
        last edited by

        Looking at the timestamps on your RRD files I suspect RRD collection stopped around midnight 7-Sep for a number of data sets including LAN packet and LAN traffic. What happened around then? system reboot due to power failure causing file corruption?

        I presume the more current timestamps on your WAN related files are because you "reset" them. What do you do to "reset" them?

        1 Reply Last reply Reply Quote 0
        • T
          torontob
          last edited by

          I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues but here are the logs:

          Sep 7 00:12:16	check_reload_status: updating dyndns WAN_PPPOE
Sep 7 00:12:16	check_reload_status: Restarting ipsec tunnels
Sep 7 00:12:16	check_reload_status: Restarting OpenVPN tunnels/interfaces
Sep 7 00:12:16	check_reload_status: Reloading filter
Sep 7 00:12:24	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_PPPOE.
Sep 7 00:12:24	php: rc.openvpn: OpenVPN: Resync server2 OVEPN-inbound
Sep 7 00:12:24	php: rc.dyndns.update: phpDynDNS (rbpbxcam102.qbooksonline.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 7 00:12:25	kernel: ovpns2: link state changed to DOWN
Sep 7 00:12:25	php: rc.openvpn: OpenVPN: Resync client1 PBX-VPN
Sep 7 00:12:26	kernel: ovpns2: link state changed to UP
Sep 7 00:12:26	kernel: ovpnc1: link state changed to DOWN
Sep 7 00:12:26	check_reload_status: rc.newwanip starting ovpns2
Sep 7 00:12:28	php: rc.filter_configure_sync: Could not find IPv4 gateway for interface (opt2).
Sep 7 00:12:30	kernel: ovpnc1: link state changed to UP
Sep 7 00:12:30	check_reload_status: rc.newwanip starting ovpnc1
Sep 7 00:12:35	kernel: pid 80615 (php), uid 0, was killed: out of swap space
Sep 7 00:12:36	php: rc.newwanip: rc.newwanip: Informational is starting ovpns2.
Sep 7 00:12:36	php: rc.newwanip: rc.newwanip: on (IP address: 172.16.50.1) (interface: opt3) (real interface: ovpns2).
Sep 7 00:12:38	kernel: pid 91802 (php), uid 0, was killed: out of swap space
Sep 7 00:12:40	kernel: pid 43574 (php), uid 0, was killed: out of swap space
Sep 7 00:12:42	kernel: pid 93676 (php), uid 0, was killed: out of swap space
Sep 7 00:12:42	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_PPPOE.
Sep 7 00:12:42	php: rc.openvpn: OpenVPN: Resync server2 OVEPN-inbound
Sep 7 00:12:43	php: rc.dyndns.update: phpDynDNS (rbpbxcam102.qbooksonline.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 7 00:12:44	kernel: ovpns2: link state changed to DOWN
Sep 7 00:12:44	check_reload_status: Reloading filter
Sep 7 00:12:44	php: rc.openvpn: OpenVPN: Resync client1 PBX-VPN
Sep 7 00:12:45	kernel: ovpns2: link state changed to UP
Sep 7 00:12:45	php: rc.newwanip: rc.newwanip: Informational is starting ovpnc1.
Sep 7 00:12:45	php: rc.newwanip: rc.newwanip: on (IP address: 172.20.20.10) (interface: opt2) (real interface: ovpnc1).
Sep 7 00:12:45	kernel: ovpnc1: link state changed to DOWN
Sep 7 00:12:45	check_reload_status: rc.newwanip starting ovpns2
Sep 7 00:12:49	kernel: ovpnc1: link state changed to UP
Sep 7 00:12:49	check_reload_status: rc.newwanip starting ovpnc1
Sep 7 00:12:50	php: rc.newwanip: Creating rrd update script
Sep 7 00:12:52	php: rc.newwanip: rc.newwanip: Informational is starting ovpns2.
Sep 7 00:12:52	php: rc.newwanip: rc.newwanip: on (IP address: 172.16.50.1) (interface: opt3) (real interface: ovpns2).
Sep 7 00:12:53	php: rc.newwanip: pfSense package system has detected an ip change 172.20.20.10 -> 172.20.20.10 ... Restarting packages.
Sep 7 00:12:53	check_reload_status: Starting packages
Sep 7 00:12:58	php: rc.newwanip: rc.newwanip: Informational is starting ovpnc1.
Sep 7 00:12:58	php: rc.newwanip: rc.newwanip: on (IP address: 172.20.20.10) (interface: opt2) (real interface: ovpnc1).
Sep 7 00:12:58	check_reload_status: updating dyndns PBXVPN_VPNV4
Sep 7 00:12:58	check_reload_status: Restarting ipsec tunnels
Sep 7 00:12:58	check_reload_status: Restarting OpenVPN tunnels/interfaces
Sep 7 00:12:59	php: rc.newwanip: Creating rrd update script
Sep 7 00:13:01	kernel: pid 3336 (php), uid 0, was killed: out of swap space
Sep 7 00:13:02	php: rc.newwanip: pfSense package system has detected an ip change 172.16.50.1 -> 172.16.50.1 ... Restarting packages.
Sep 7 00:13:04	php: rc.newwanip: Creating rrd update script
Sep 7 00:13:06	kernel: pid 31988 (php), uid 0, was killed: out of swap space
Sep 7 00:13:07	php: rc.newwanip: pfSense package system has detected an ip change 172.20.20.10 -> 172.20.20.10 ... Restarting packages.
Sep 7 00:13:11	kernel: pid 34761 (php), uid 0, was killed: out of swap space
Sep 7 00:13:13	kernel: pid 47671 (php), uid 0, was killed: out of swap space
Sep 7 00:13:15	php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use PBXVPN_VPNV4.
Sep 7 00:13:21	php: rc.start_packages: Restarting/Starting all packages.
Sep 7 00:13:21	kernel: pid 65238 (php), uid 0, was killed: out of swap space
Sep 7 00:13:24	php: rc.start_packages: Restarting/Starting all packages.
Sep 7 00:13:24	php: rc.start_packages: The command '/usr/local/etc/rc.d/darkstat.sh stop' returned exit code '1', the output was 'No matching processes were found'
Sep 7 00:13:27	php: rc.start_packages: You should specify an interface for bandwidthd to listen on. Exiting.

          Also, I reset by browsing to RRD Settings page and reset data there.

          Thanks

          1 Reply Last reply Reply Quote 0
          • K
            kejianshi
            last edited by

            When mine stopped, I deleted RRD data and it started working fine again  (Probably not the fix you were hoping for?)

            1 Reply Last reply Reply Quote 0
            • T
              torontob
              last edited by

              @kejianshi:

              When mine stopped, I deleted RRD data and it started working fine again  (Probably not the fix you were hoping for?)

              Thanks - Is there a way to safely delete other than RESET?
              This is annoying. I appreciate any other feedback.

              1 Reply Last reply Reply Quote 0
              • W
                wallabybob
                last edited by

                @torontob:

                I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues

                When the kernel goes killing "random" processes because there is no swap space file corruption is quite likely. I suspect you will probably need to get the swap space issue fixed to resolve this, but the feedback so far suggests "not until 2.2".

                Does the offending system run the nano-BSD variant?

                1 Reply Last reply Reply Quote 0
                • T
                  torontob
                  last edited by

                  Sorry, what you mean "not until 2.2"? Is this a known issue?
                  I downloaded a snapshot of 2.1-RC1 and installed. No changes to base operating system.

                  Thanks,

                  1 Reply Last reply Reply Quote 0
                  • D
                    doktornotor Banned
                    last edited by

                    @torontob:

                    I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues but here are the logs:

                    There is no swap used on nanobsd. You simply are running out of RAM => trying to do too much on an inadequate HW.

                    1 Reply Last reply Reply Quote 0
                    • W
                      wallabybob
                      last edited by

                      @torontob:

                      Sorry, what you mean "not until 2.2"? Is this a known issue?

                      See discussion in http://forum.pfsense.org/index.php/topic,66188.0.html in particular see reply #2 in which ermal says

                      There is a plan hopefully to be ready fro 2.2 to make this a non-issue.

                      I don't know that fixing the "out of swap space" issue will fix your RRD issue but I do think it is likely that your RRD issue is caused by the kernel killing processes (due to running out of swap space) and that leaving one or more RRD data files "corrupt".

                      1 Reply Last reply Reply Quote 0
                      • T
                        torontob
                        last edited by

                        @doktornotor:

                        @torontob:

                        I have a2GB CF card installed so not sure why swap space run out and not sure if that is the cause of the RRD issues but here are the logs:

                        There is no swap used on nanobsd. You simply are running out of RAM => trying to do too much on an inadequate HW.

                        Guys, I am using THE standard hardware - Alix2D13 - and I have only 30 SIP phones on pfSense. I have put much much much more on this hardware and it hasn't run out of RAM ever. I would like to clarify two things though and I think we are getting closer:

                        I see that it's not a swap issue (wallabybob) that is causing RRD graphs issue but killing processes due to that may be. So, what is my next step in diagnosing?

                        1 Reply Last reply Reply Quote 0
                        • D
                          doktornotor Banned
                          last edited by

                          @torontob:

                          I see that it's not a swap issue (wallabybob) that is causing RRD graphs issue but killing processes due to that may be. So, what is my next step in diagnosing?

                          Dude, that message tells you that you have run out of RAM. Period.

                          1 Reply Last reply Reply Quote 0
                          • P
                            phil.davis
                            last edited by 

                            Sep 7 00:12:38	kernel: pid 91802 (php), uid 0, was killed: out of swap space
Sep 7 00:12:40	kernel: pid 43574 (php), uid 0, was killed: out of swap space
Sep 7 00:12:42	kernel: pid 93676 (php), uid 0, was killed: out of swap space

                            I just want to clarify to make sure it is understood. On nanoBSD there is no swap partition or file, i.e. swap space is zero. Therefore when it says "out of swap space" it really means "I needed some swap space, but there is none" =  "out of real memory".
                            Some people (including me with some configs) have been having this issue on 2.1 with 256MB RAM systems. It seems to be a transient thing, for me particularly when WAN links go up/down and a few OpenVPN links reestablish at pretty much the same time. A lot of processes fire up initialising/restarting various things and for those few seconds there is not enough real memory for it all. By using the new System Watchdog package, things normally recover because after a minute System Services Watchdog wakes up and restarts important things that got killed.

                            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                            1 Reply Last reply Reply Quote 0
                            • T
                              torontob
                              last edited by

                              @phil.davis:

                              Sep 7 00:12:38	kernel: pid 91802 (php), uid 0, was killed: out of swap space
Sep 7 00:12:40	kernel: pid 43574 (php), uid 0, was killed: out of swap space
Sep 7 00:12:42	kernel: pid 93676 (php), uid 0, was killed: out of swap space

                              I just want to clarify to make sure it is understood. On nanoBSD there is no swap partition or file, i.e. swap space is zero. Therefore when it says "out of swap space" it really means "I needed some swap space, but there is none" =  "out of real memory".
                              Some people (including me with some configs) have been having this issue on 2.1 with 256MB RAM systems. It seems to be a transient thing, for me particularly when WAN links go up/down and a few OpenVPN links reestablish at pretty much the same time. A lot of processes fire up initialising/restarting various things and for those few seconds there is not enough real memory for it all. By using the new System Watchdog package, things normally recover because after a minute System Watchdog wakes up and restarts important things that got killed.

                              Thanks for the clarification. So, I meant I have not done any modifications to cause swap issues. I do have couple OpenVPN links and do use DSL. So, if that becomes a problem and that is not a problem on 2.0 (which it is not for my other boxes) then it should be treated as a bug. Maybe tune down everything or create a delay for services to start?

                              Also, I see Services Watchdog - not System Watchdog - is that what you were referring to? Never used it though. Is that a stable package? (shows beta). I am wondering if that in itself creates more overhead sending me into a vicious circle :)

                              1 Reply Last reply Reply Quote 0
                              • W
                                wallabybob
                                last edited by

                                @torontob:

                                Also, I see Services Watchdog - not System Watchdog - is that what you were referring to? Never used it though. Is that a stable package? (shows beta). I am wondering if that in itself creates more overhead sending me into a vicious circle :)

                                And I can't see how Services Watchdog will help if RRD data is getting corrupted because processes are killed due to swap space exhaustion.

                                1 Reply Last reply Reply Quote 0
                                • P
                                  phil.davis
                                  last edited by

                                  I meant Services Watchdog - I fixed the text in the previous.
                                  Services Watchdog works well for what it does, a simple check every minute of what is running or missing. Of course, it sometimes happens to run at the same time as some down/up event on WAN. So then it can try to start things that are already in the process of starting by the link up/down scripts, and also use memory itself when memory is short. Services Watchdog is really intended to restart services that should never go missing but genuinely happen to crash (program crashes, divide by zero…). To recover nicely from service exits due to low memory it needs more parameters so it can wait a bit longer if the service is down, then only start it if it has been missing for some time - avoiding race conditions when other code is in the process of getting a WAN link into the up state again.

                                  We have made code to "slow down" the boot process also - https://github.com/pfsense/pfsense/pull/798 - this helps ensure that critical things start OK during the boot. As Ermal comments, it is a "kludge" - using a heuristic that waits for free memory to stabilize in order to infer that the thing just started has initialised and stabilized, and it is time to move on in the bootup process. This "kludge" does nothing for memory problems due to WAN down/up etc after the system is booted. But for us, at least it gets us a functional system at boot with site-to-site OpenVPN. That means we can login remotely and are not stuck with doing telephone support to someone in a remote office after a power failure.

                                  My Alix 256MB systems typically run with memory usage on the dashboard at 45% to 55%. As long as all the physical links and OpenVPN links are working nicely, nothing restarts and there is plenty of memory. So these Alix systems can run with configs such as like I have: 2 WANs, 3 LANs (on VLAN switch), OpenVPN server for site-to-site links with 9 client offices connecting, OpenVPN server for Road Warrior with a handful of client connections, DHCP, DNS, only Cron, Sudo, Service Watchdog and OpenVPN Client Export packages. It just needs some serialization of real-time event handling so that the free memory is used in a nicely controlled manner when recovering from events.

                                  As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                  If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                  1 Reply Last reply Reply Quote 0
                                  • P
                                    phil.davis
                                    last edited by

                                    @wallabybob:

                                    @torontob:

                                    Also, I see Services Watchdog - not System Watchdog - is that what you were referring to? Never used it though. Is that a stable package? (shows beta). I am wondering if that in itself creates more overhead sending me into a vicious circle :)

                                    And I can't see how Services Watchdog will help if RRD data is getting corrupted because processes are killed due to swap space exhaustion.

                                    Correct - if the only problem is the original post about corrupted RRD data, then that can happen whenever the "killed" happens randomly during the RRD data writing process, leaving part-updated corrupt data files. To fix that, the system has to be fixed so that "killed" never happens.
                                    Services Watchdog or other recovery methods are only useful to get processes running again, as long as their associated data files are in good shape.

                                    As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                    If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                    1 Reply Last reply Reply Quote 0
                                    • T
                                      torontob
                                      last edited by

                                      Thanks wallabybob for input.
                                      Since the topic changed a bit, I might as well ask this question. From time to time, I see that OpenVPN is not trying to re-establish a connection. Why is that? why is there a need for something like Watchdog and why won't vital services just automatically restart?

                                      Thanks,

                                      1 Reply Last reply Reply Quote 0
                                      • P
                                        phil.davis
                                        last edited by

                                        If a critical service process (dhcp, dns, OpenVPNs…) exits abnormally (system problem like out of swap space, program problem like divide by zero) there is no controlling script that called it that then gets the error code, and can loop around to try and run the program again. These services are forked off into independent processes by the bootup scripts.
                                        There is nothing else built into the system that monitors AND restarts them automatically. Yes, there are dashboard displays, but they don't take automatic action. And in any case, what if nobody has a dashboard running.
                                        So JimP has kindly made Service Watchdog - in a perfect world it would not be needed, just like a real watchdog.

                                        OpenVPN: in my experience the OpenVPN code is very good at trying forever to connect and eventually connecting once the underlying physical links and internet is up and working. I expect that an OpenVPN client would only stop trying to establish a connection if the process has actually crashed, which I have only seen happen because of "killed: out of swap space".

                                        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.