Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Bandwidthd issues?

    Scheduled Pinned Locked Moved Traffic Monitoring
    60 Posts 17 Posters 18.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      charliem
      last edited by

      Take a look at /usr/local/pkg/bandwidthd.inc; it writes both bandwidthd.sh and bandwidthd.conf

      1 Reply Last reply Reply Quote 0
      • P
        phil.davis
        last edited by

        AFAIK it is normal to have 4 bandwidthd processes:

        ps aux | grep bandwidthd
root       41074  0.0  2.4 15748  5576  -  S     9:41PM  0:00.02 /var/bandwidthd/bandwidthd
root       41237  0.0  2.4 15748  5588  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       41425  0.0  2.4 15748  5576  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       41449  0.0  2.4 15748  5576  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       44012  0.0  0.9 10396  1952  1  S+    9:42PM  0:00.01 grep bandwidthd

        I think they are related to the recording of daily, weekly, monthly and yearly data/graphs. Each updates data/graphs at different intervals.

        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

        1 Reply Last reply Reply Quote 0
        • arrmoA
          arrmo
          last edited by

          Hmmm … at least when logging to an external database (PostgreSQL) this causes problems -> I have confirmed multiple entries for the same points in time, and the daily totals in PostgreSQL are 2x what they should be (due to 2 processes running).

          Thoughts?

          Thanks!

          1 Reply Last reply Reply Quote 0
          • arrmoA
            arrmo
            last edited by

            Could it be this is why we're seeing different results (working / not working)? I think you're letting bandwidthd generate "local" info, but I'm logging to an external database? Just thinking out loud, trying to figure it out.

            It is also interesting that the path to your bandwidth process is different - this looks to be processing the data (as it's inside var, right?), but mine is the rc.d service / daemon? I'm just trying to stop more than one of those existing, as I am getting double entries in the database (not a good thing).

            Thoughts?

            Thanks very much!

            1 Reply Last reply Reply Quote 0
            • C
              charliem
              last edited by

              Hmm, I have 8 running:

              [2.2-RC][root@pfsense.localdomain]/root: ps axfw | grep bandwidthd
              93615  -  S        0:05.36 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              93674  -  S        0:04.71 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              93823  -  S        0:04.30 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              94051  -  S        0:04.26 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              94636  -  S        0:05.40 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              94844  -  S        0:04.75 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              95011  -  S        0:04.31 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
              95137  -  S        0:04.26 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd

              Mine is a full 64bit install, with both "generate CDF" and "recover CDF" enabled, but no PostgreSQL logging enabled.  Bandwidthd runs without issues for me for the past few months, but I haven't tried to reconcile traffic reported by bandwidthd to that reported by the rrd graphs in pfSense.  Bandwidthd would be more helpful for me if it could track IPv6 traffic, so I don't use it much.

              The nano and full pfSense installs are treated differently by bandwidthd: check /usr/local/pkg/bandwidthd.inc starting around line 302.  The nano config for the $rc['start'] stanza runs from lines 315 to 348, while the stanza for full install is lines 351-353.  So, I think your different command lines are expected (maybe not bug-free :) but expected).

              1 Reply Last reply Reply Quote 0
              • arrmoA
                arrmo
                last edited by

                Thanks for the awesome pointer! That does help. I admit, I'm not sure how these packages and files fit together - stupidity on my part!

                I modified /usr/local/pkg/bandwidthd.inc (locally), and I now avoid the double processes … so one of my problems is resolved. But this also helped me to see the conditions to cause php-fpm to die - more below,

                If I go in to /usr/local/etc/rc.d, I can start and stop bandwidthd (and with my change, I only get one copy of the process now). But I also found the conditions to cause php-fpm to die. If I do a Save from the GUI under bandwidthd (which updates /usr/local/etc/rc.d/bandwidth.sh!), then after this if I run bandwidthd.sh, telling it to stop -> php-fpm dies. If I restart php-fpm (manually) ... I can start or stop bandwidth as much as I want, php-fpm never dies again. Only after another Save (and regeneration of bandwidth.sh), then stopping bandwidthd (the first time) kills php-fpm. I also tried this by just manually stopping bandwidthd, by executing /usr/bin/killall bandwidthd ... and this definitely kills php-fpm. If I restart php-fpm, I can't kill php-fpm again (with starts or stops of bandwidthd) ... until I do a Save from the GUI again, then stopping bandwidthd kills php-fpm.

                Does this make sense? It seems odd to me, but it is very repeatable.

                Thoughts?

                Thanks again for the help!

                1 Reply Last reply Reply Quote 0
                • arrmoA
                  arrmo
                  last edited by

                  BTW (and here comes my stupid, uneducated question … :() - how can doing a killall to bandwidthd kill other processes (php-fpm, but also ntopng)? It really does seem to kill them - but I can't understand why. Figuring this out really is the root cause.

                  Thanks!!!

                  1 Reply Last reply Reply Quote 0
                  • P
                    phil.davis
                    last edited by

                    My home system had 8 bandwidthd processes, for some unknown reason - I guess this is one of your problems with it starting twice under some conditions.
                    I did the killall command by hand from the command line to see if that would also break php-fpm, and put "-v" so it would tell me what it thinks it killed:

                    [2.2-RC][root@testoffice-rt-01.xxx]/usr/local/etc/rc.d: ps aux | grep bandwidthd
root    17460   0.0  2.7 15748  6072  -  S     8:29AM    0:00.45 /var/bandwidthd/bandwidthd
root    17871   0.0  2.6 15748  6004  -  S     8:29AM    0:00.33 /var/bandwidthd/bandwidthd
root    18178   0.0  2.5 15748  5804  -  S     8:29AM    0:00.10 /var/bandwidthd/bandwidthd
root    18334   0.0  2.5 15748  5808  -  S     8:29AM    0:00.05 /var/bandwidthd/bandwidthd
root    18587   0.0  2.7 15748  6072  -  S     8:29AM    0:00.45 /var/bandwidthd/bandwidthd
root    18876   0.0  2.6 15748  5988  -  S     8:29AM    0:00.33 /var/bandwidthd/bandwidthd
root    18962   0.0  2.5 15748  5804  -  S     8:29AM    0:00.10 /var/bandwidthd/bandwidthd
root    19024   0.0  2.5 15748  5808  -  S     8:29AM    0:00.06 /var/bandwidthd/bandwidthd
root    77011   0.0  0.9 10396  1960  0  S+    8:48AM    0:00.01 grep bandwidthd
[2.2-RC][root@testoffice-rt-01.xxx]/usr/local/etc/rc.d: /usr/bin/killall -v bandwidthd
kill -TERM 19024
kill -TERM 18962
kill -TERM 18876
kill -TERM 18587
kill -TERM 18334
kill -TERM 18178
kill -TERM 17871
kill -TERM 17460

                    All worked as expected, and my php-fpm and webGUI still works.
                    Then I did a few save on the Bandwidthd webGUI page. No problem there either, 4 old processes go away, 4 new ones start.
                    This system is using local bandwidthd data. I will try in a while with "Log data to a PostgreSQL database" option.

                    As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                    If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                    1 Reply Last reply Reply Quote 0
                    • P
                      phil.davis
                      last edited by

                      (which updates /usr/local/etc/rc.d/bandwidth.sh!)

                      Various scripts an conf files in pfSense are generated from the GUI and startup code, like this one. Once you discover exactly what needs to be changed in the script, then we can change the PHP code to generate the script correctly.

                      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                      1 Reply Last reply Reply Quote 0
                      • arrmoA
                        arrmo
                        last edited by

                        Thanks! And you're right, just need to figure out the dependencies - trying to follow the bread crumbs … ;)

                        This may be nano vs full, local vs. Postgre related - I admit, not quite sure. I did try killall, with -v like you noted. Here is what I get ...

                        1. No Save done first (watching pfp-fpm, ntopng, bandwidthd),

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep php-fpm
                        root    11169  0.0  1.3 236688  51316  -  I    7:53AM    0:00.05 php-fpm: pool lighty (php-fpm)
                        root    45513  0.0  0.9 228364  34508  -  Ss    1:29PM    0:00.47 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep ntopng
                        root    32603  0.1  1.5 183216  55908  -  Ss    4:26PM    3:44.38 /usr/local/bin/ntopng -s -e -i bge0 –dns-mode 1 --local-networks 192.168
                        root    29103  0.0  0.1  24072  4844  -  I    4:26PM    0:09.84 redis-server: /usr/pbi/ntopng-amd64/local/bin/redis-server *:6379 (redis-

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep bandwidthd
                        root    10913  0.0  0.2  55728  8844  0  S    1:38PM    0:01.81 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd

                        => killall -v bandwidthd,
                        [2.2-RC][root@pfSense.home]/root: killall -v bandwidthd
                        kill -TERM 10913

                        Result: Only bandwidthd seems to be killed (checked by command above again, all running but bandwidthd).

                        1. Save done first (watching pfp-fpm, ntopng, bandwidthd),

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep bandwidthd
                        root    55458  0.0  0.2  55728  8312  -  S    8:00AM    0:00.00 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd

                        [2.2-RC][root@pfSense.home]/root: killall -v bandwidthd
                        kill -TERM 55458

                        But, see the following …

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep php-fpm
                        root    78190  0.0  0.1  18900  2400  1  S+    8:01AM    0:00.00 grep php-fpm

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep ntopng
                        root    93951  0.0  0.1  18900  2400  1  S+    8:01AM    0:00.00 grep ntopng

                        [2.2-RC][root@pfSense.home]/root: ps -aux | grep bandwidthd
                        root    94123  0.0  0.1  18900  2408  1  S+    8:01AM    0:00.00 grep bandwidthd

                        Result: All three killed, php-fpm, ntopng and bandwidthd … and perhaps others, the GUI is obvious (it's down), and I just happened to stumble on to ntopng ... :(.

                        Yell if there are other things I should try. Thanks!

                        1 Reply Last reply Reply Quote 0
                        • arrmoA
                          arrmo
                          last edited by

                          Just and FYI, but I had an issue yesterday where I saved OpenVPN settings to restart the server (with a settings change) … and it killed the GUI. So this may be a bit deeper than just Bandwidthd.

                          1 Reply Last reply Reply Quote 0
                          • arrmoA
                            arrmo
                            last edited by

                            Hi,

                            OK, I think this problem goes deeper than bandwidthd … :(. I think it's a more generic issue, and bandwidthd is just how I stumbled on to it - here is why I say this (feel free to tell me I'm full of it!),

                            I was getting OpenVPN set up on my pfSense box, made one setting change, and hit save -> GUI dead again! And several other services were killed in the process ... ntopng, bandwidthd, and I also noticed that OpenVPN didn't come up. I left my client in a loop, trying to connect to OpenVPN ... no luck, until I restarted php-fpm -> and this also brought OpenVPN back up!

                            So there seems to be some sort of interaction here between these services / settings (and a bit bigger issue I fear). So I still have the 2x Bandwidthd issue (that I have a patch for myself that works), but this other issue with php-fpm and other services.

                            Thoughts?

                            Thanks!

                            1 Reply Last reply Reply Quote 0
                            • T
                              tojaktoty
                              last edited by

                              I also get '503 Service not available' and no ability to access the webgui on a new install of 2.2RC amd64 nano built on Jan 7, 2015 after adding bandwidthd. No other packages added. Only running traffic shaper on lan. Nothing else special.

                              The pfsense box is a thousand miles away and appears it is still working otherwise. reggie14 posted earlier in page 2 of this thread that he restarted his pfsense box and the system powered back on with bandwidthd off and the system came back online working normal. Brrm posted that after reboot everything seemed to be working.

                              Does anyone else have any experience with rebooting pfsense with this bandwidthd issue and being able to get back into a functioning system? Or has anyone experienced worse problems after rebooting?

                              I'm trying to assess the probability and risk of remotely rebooting pfsense and being able to get back into the webgui of a working system or if pfsense regresses with more issues after a reboot.

                              Thanks

                              1 Reply Last reply Reply Quote 0
                              • D
                                doktornotor Banned
                                last edited by

                                Well you can remove the broken package from  pfSense Developer Shell (option 12), IIRC. Then restart webConfigurator (or possibly PHP-FPM if things are really messed up).

                                1 Reply Last reply Reply Quote 0
                                • arrmoA
                                  arrmo
                                  last edited by

                                  Hi,

                                  Not sure this is related to just this package. Was making some OpenVPN settings changes yesterday - Saving there also broken php-fpm (confirmed several times). I think this is a bit deeper issue … :(.

                                  Thanks!

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    cmb
                                    last edited by

                                    It's definitely specific to just this package. Changing OpenVPN will restart packages, which is probably why that'd trigger it. I'm still not sure how or why it triggers any problem along those lines, haven't been able to replicate that and not sure why it seems to be so easy for some.

                                    1 Reply Last reply Reply Quote 0
                                    • arrmoA
                                      arrmo
                                      last edited by

                                      Completely agreed, and I understand your pain. Can't fix a problem you can't duplicate!

                                      Is there any way for me to try to "monitor" what is happening when I do this? Not sure if there is a way to generically turn up logging levels, to try to debug it. Willing to do what I can to help, but I can't figure it out either … :(.

                                      Thanks!

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        tojaktoty
                                        last edited by

                                        @doktornotor:

                                        Well you can remove the broken package from  pfSense Developer Shell (option 12), IIRC. Then restart webConfigurator (or possibly PHP-FPM if things are really messed up).

                                        What commands would need be run in developer shell option 12 in order to remove bandwidthd and restart webConfigurator and PHP-FPM?

                                        I setup a test vm pfsense, ran 'vi /config/config.xml', edited bandwidthd package to off,  ':wq!' won't let me save.

                                        Read the developer shell help info and https://doc.pfsense.org/index.php/Using_the_PHP_pfSense_Shell
                                        I can't figure out how to remove the package in the developer shell. I've searched the forums and haven't found a clue.

                                        Thanks

                                        1 Reply Last reply Reply Quote 0
                                        • arrmoA
                                          arrmo
                                          last edited by

                                          Hi,

                                          I have a patch that works for my startup issues (multiple copies of Bandwidthd writing to PostgreSQL), and it is installed in System Patches (in pfSense) … but it's not getting applied after Bandwidthd installs (on an upgrade) / before it starts. Is there a way to make this happen?

                                          Not sure if others want the patch, I can post it if desired.

                                          Thanks!

                                          1 Reply Last reply Reply Quote 0
                                          • H
                                            HellMind
                                            last edited by

                                            Is pfsense abandoned?
                                            Why nobody removes bandwithd from pkg list

                                            Also why theres no way to fix the webgui unavailable?

                                            Its a know issue.
                                            Its easy, pfsense webgui fails if bandwidthd is enabled.

                                            WHAT CAN A USER WITHOUT KNOWLEDGE ABOUT PFSENSER OR FREEBSD DO?

                                            I hate to see pfsense isn't ready AGAIN to be used.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.