Bandwidthd issues?

arrmo

BTW (and here comes my stupid, uneducated question … :() - how can doing a killall to bandwidthd kill other processes (php-fpm, but also ntopng)? It really does seem to kill them - but I can't understand why. Figuring this out really is the root cause.

Thanks!!!

phil.davis

My home system had 8 bandwidthd processes, for some unknown reason - I guess this is one of your problems with it starting twice under some conditions.
I did the killall command by hand from the command line to see if that would also break php-fpm, and put "-v" so it would tell me what it thinks it killed:

[2.2-RC][root@testoffice-rt-01.xxx]/usr/local/etc/rc.d: ps aux | grep bandwidthd
root    17460   0.0  2.7 15748  6072  -  S     8:29AM    0:00.45 /var/bandwidthd/bandwidthd
root    17871   0.0  2.6 15748  6004  -  S     8:29AM    0:00.33 /var/bandwidthd/bandwidthd
root    18178   0.0  2.5 15748  5804  -  S     8:29AM    0:00.10 /var/bandwidthd/bandwidthd
root    18334   0.0  2.5 15748  5808  -  S     8:29AM    0:00.05 /var/bandwidthd/bandwidthd
root    18587   0.0  2.7 15748  6072  -  S     8:29AM    0:00.45 /var/bandwidthd/bandwidthd
root    18876   0.0  2.6 15748  5988  -  S     8:29AM    0:00.33 /var/bandwidthd/bandwidthd
root    18962   0.0  2.5 15748  5804  -  S     8:29AM    0:00.10 /var/bandwidthd/bandwidthd
root    19024   0.0  2.5 15748  5808  -  S     8:29AM    0:00.06 /var/bandwidthd/bandwidthd
root    77011   0.0  0.9 10396  1960  0  S+    8:48AM    0:00.01 grep bandwidthd
[2.2-RC][root@testoffice-rt-01.xxx]/usr/local/etc/rc.d: /usr/bin/killall -v bandwidthd
kill -TERM 19024
kill -TERM 18962
kill -TERM 18876
kill -TERM 18587
kill -TERM 18334
kill -TERM 18178
kill -TERM 17871
kill -TERM 17460

All worked as expected, and my php-fpm and webGUI still works.
Then I did a few save on the Bandwidthd webGUI page. No problem there either, 4 old processes go away, 4 new ones start.
This system is using local bandwidthd data. I will try in a while with "Log data to a PostgreSQL database" option.

phil.davis

(which updates /usr/local/etc/rc.d/bandwidth.sh!)

Various scripts an conf files in pfSense are generated from the GUI and startup code, like this one. Once you discover exactly what needs to be changed in the script, then we can change the PHP code to generate the script correctly.

arrmo

Thanks! And you're right, just need to figure out the dependencies - trying to follow the bread crumbs … ;)

This may be nano vs full, local vs. Postgre related - I admit, not quite sure. I did try killall, with -v like you noted. Here is what I get ...

No Save done first (watching pfp-fpm, ntopng, bandwidthd),

[2.2-RC][root@pfSense.home]/root: ps -aux | grep php-fpm
root 11169 0.0 1.3 236688 51316 - I 7:53AM 0:00.05 php-fpm: pool lighty (php-fpm)
root 45513 0.0 0.9 228364 34508 - Ss 1:29PM 0:00.47 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)

[2.2-RC][root@pfSense.home]/root: ps -aux | grep ntopng
root 32603 0.1 1.5 183216 55908 - Ss 4:26PM 3:44.38 /usr/local/bin/ntopng -s -e -i bge0 –dns-mode 1 --local-networks 192.168
root 29103 0.0 0.1 24072 4844 - I 4:26PM 0:09.84 redis-server: /usr/pbi/ntopng-amd64/local/bin/redis-server *:6379 (redis-

[2.2-RC][root@pfSense.home]/root: ps -aux | grep bandwidthd
root 10913 0.0 0.2 55728 8844 0 S 1:38PM 0:01.81 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd

=> killall -v bandwidthd,
[2.2-RC][root@pfSense.home]/root: killall -v bandwidthd
kill -TERM 10913

Result: Only bandwidthd seems to be killed (checked by command above again, all running but bandwidthd).

Save done first (watching pfp-fpm, ntopng, bandwidthd),

[2.2-RC][root@pfSense.home]/root: ps -aux | grep bandwidthd
root 55458 0.0 0.2 55728 8312 - S 8:00AM 0:00.00 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd

[2.2-RC][root@pfSense.home]/root: killall -v bandwidthd
kill -TERM 55458

But, see the following …

[2.2-RC][root@pfSense.home]/root: ps -aux | grep php-fpm
root 78190 0.0 0.1 18900 2400 1 S+ 8:01AM 0:00.00 grep php-fpm

[2.2-RC][root@pfSense.home]/root: ps -aux | grep ntopng
root 93951 0.0 0.1 18900 2400 1 S+ 8:01AM 0:00.00 grep ntopng

[2.2-RC][root@pfSense.home]/root: ps -aux | grep bandwidthd
root 94123 0.0 0.1 18900 2408 1 S+ 8:01AM 0:00.00 grep bandwidthd

Result: All three killed, php-fpm, ntopng and bandwidthd … and perhaps others, the GUI is obvious (it's down), and I just happened to stumble on to ntopng ... :(.

Yell if there are other things I should try. Thanks!

arrmo

Just and FYI, but I had an issue yesterday where I saved OpenVPN settings to restart the server (with a settings change) … and it killed the GUI. So this may be a bit deeper than just Bandwidthd.

arrmo

Hi,

OK, I think this problem goes deeper than bandwidthd … :(. I think it's a more generic issue, and bandwidthd is just how I stumbled on to it - here is why I say this (feel free to tell me I'm full of it!),

I was getting OpenVPN set up on my pfSense box, made one setting change, and hit save -> GUI dead again! And several other services were killed in the process ... ntopng, bandwidthd, and I also noticed that OpenVPN didn't come up. I left my client in a loop, trying to connect to OpenVPN ... no luck, until I restarted php-fpm -> and this also brought OpenVPN back up!

So there seems to be some sort of interaction here between these services / settings (and a bit bigger issue I fear). So I still have the 2x Bandwidthd issue (that I have a patch for myself that works), but this other issue with php-fpm and other services.

Thoughts?

Thanks!

tojaktoty

I also get '503 Service not available' and no ability to access the webgui on a new install of 2.2RC amd64 nano built on Jan 7, 2015 after adding bandwidthd. No other packages added. Only running traffic shaper on lan. Nothing else special.

The pfsense box is a thousand miles away and appears it is still working otherwise. reggie14 posted earlier in page 2 of this thread that he restarted his pfsense box and the system powered back on with bandwidthd off and the system came back online working normal. Brrm posted that after reboot everything seemed to be working.

Does anyone else have any experience with rebooting pfsense with this bandwidthd issue and being able to get back into a functioning system? Or has anyone experienced worse problems after rebooting?

I'm trying to assess the probability and risk of remotely rebooting pfsense and being able to get back into the webgui of a working system or if pfsense regresses with more issues after a reboot.

Thanks

doktornotor

Well you can remove the broken package from pfSense Developer Shell (option 12), IIRC. Then restart webConfigurator (or possibly PHP-FPM if things are really messed up).

arrmo

Hi,

Not sure this is related to just this package. Was making some OpenVPN settings changes yesterday - Saving there also broken php-fpm (confirmed several times). I think this is a bit deeper issue … :(.

Thanks!

cmb

It's definitely specific to just this package. Changing OpenVPN will restart packages, which is probably why that'd trigger it. I'm still not sure how or why it triggers any problem along those lines, haven't been able to replicate that and not sure why it seems to be so easy for some.

arrmo

Completely agreed, and I understand your pain. Can't fix a problem you can't duplicate!

Is there any way for me to try to "monitor" what is happening when I do this? Not sure if there is a way to generically turn up logging levels, to try to debug it. Willing to do what I can to help, but I can't figure it out either … :(.

Thanks!

tojaktoty

@doktornotor:

Well you can remove the broken package from pfSense Developer Shell (option 12), IIRC. Then restart webConfigurator (or possibly PHP-FPM if things are really messed up).

What commands would need be run in developer shell option 12 in order to remove bandwidthd and restart webConfigurator and PHP-FPM?

I setup a test vm pfsense, ran 'vi /config/config.xml', edited bandwidthd package to off, ':wq!' won't let me save.

Read the developer shell help info and https://doc.pfsense.org/index.php/Using_the_PHP_pfSense_Shell
I can't figure out how to remove the package in the developer shell. I've searched the forums and haven't found a clue.

Thanks

arrmo

Hi,

I have a patch that works for my startup issues (multiple copies of Bandwidthd writing to PostgreSQL), and it is installed in System Patches (in pfSense) … but it's not getting applied after Bandwidthd installs (on an upgrade) / before it starts. Is there a way to make this happen?

Not sure if others want the patch, I can post it if desired.

Thanks!

HellMind

Is pfsense abandoned?
Why nobody removes bandwithd from pkg list

Also why theres no way to fix the webgui unavailable?

Its a know issue.
Its easy, pfsense webgui fails if bandwidthd is enabled.

WHAT CAN A USER WITHOUT KNOWLEDGE ABOUT PFSENSER OR FREEBSD DO?

I hate to see pfsense isn't ready AGAIN to be used.

tompeeters

I 'm having the same issues with bandwithd and would like to help on a constructive way to solve this.

Running release 2.2.1

Installed package
Going to Services >>> bandwithd >>> enable >>> save >>> GUI crash
Restarting PHP-FPM from shell to fix GUI

config.xml:

		 <bandwidthd><config><enable>on</enable>
				<active_interface>opt6</active_interface>
				<subnets_custom>192.168.128.0/21</subnets_custom>
				 <skipintervals><graphcutoff>512</graphcutoff>
				 <promiscuous><outputcdf><recovercdf><outputpostgresql><postgresqlhost><postgresqldatabase><postgresqlusername><postgresqlpassword><sensorid><filter><drawgraphs><meta_refresh><graph_log_info></graph_log_info></meta_refresh></drawgraphs></filter></sensorid></postgresqlpassword></postgresqlusername></postgresqldatabase></postgresqlhost></outputpostgresql></recovercdf></outputcdf></promiscuous></skipintervals></config></bandwidthd>

system.log:

Mar 24 10:13:32 firewall php-fpm[60155]: /pkg_mgr_install.php: Beginning package installation for bandwidthd .
Mar 24 10:13:33 firewall check_reload_status: Syncing firewall
Mar 24 10:13:45 firewall check_reload_status: Syncing firewall
Mar 24 10:13:45 firewall php-fpm[60155]: /pkg_mgr_install.php: Successfully installed package: bandwidthd.
Mar 24 10:13:46 firewall check_reload_status: Reloading filter
Mar 24 10:14:01 firewall check_reload_status: Syncing firewall
Mar 24 10:14:01 firewall php-fpm[16579]: /pkg_edit.php: The command '/usr/local/etc/rc.d/bandwidthd.sh stop' returned exit code '1', the output was 'No matching processes were found' 
Mar 24 10:14:01 firewall bandwidthd: Monitoring subnet 255.255.255.252 with netmask 255.255.255.252
Mar 24 10:14:01 firewall bandwidthd: Monitoring subnet 255.255.248.0 with netmask 255.255.248.0
Mar 24 10:14:01 firewall bandwidthd: Opening em1_vlan70
Mar 24 10:14:01 firewall bandwidthd: Packet Encoding: Ethernet
Mar 24 10:14:29 firewall check_reload_status: Syncing firewall
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.2562) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: unix:/var/run/php-fpm.socket 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.3346) response not received, request sent: 1434 on socket: unix:/var/run/php-fpm.socket for /pkg_edit.php?, closing connection 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.3587) all handlers for /ifstats.php?if=em5_vlan131 on .php are down. 
Mar 24 10:14:30 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:31 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:32 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket 
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.3587) all handlers for /ifstats.php?if=em4_vlan99 on .php are down. 
Mar 24 10:14:33 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:34 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:35 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket 
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.3587) all handlers for /ifstats.php?if=em2_vlan98 on .php are down. 
Mar 24 10:14:36 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:37 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:38 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:38 firewall lighttpd[42020]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket 
Mar 24 10:14:38 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:38 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:39 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:39 firewall rc.php-fpm_restart[85880]: >>> Restarting php-fpm

The logging "check_reload_status: Could not connect to /var/run/php-fpm.socket" errors disappear when I disable bandwithd

Feel free to ask any other thing to test.

heper

@HellMind:

Is pfsense abandoned?
Why nobody removes bandwithd from pkg list

Also why theres no way to fix the webgui unavailable?

Its a know issue.
Its easy, pfsense webgui fails if bandwidthd is enabled.

WHAT CAN A USER WITHOUT KNOWLEDGE ABOUT PFSENSER OR FREEBSD DO?

I hate to see pfsense isn't ready AGAIN to be used.

yes because there are some packages that are not maintained, pfsense is dead and is unuseable.
You could volunteer and maintain the bandwidthd package and fix it's issues.

@tompeeters:

I 'm having the same issues with bandwithd and would like to help on a constructive way to solve this.

yes, constructive ways FTW :))
there is an open bug on this issue, please add all your relevant debug info on it … then perhaps this'll be resolved more quickly.
https://redmine.pfsense.org/issues/3986

jimp

bandwidthd has not been removed from the list of packages because bandwidthd still works fine for many people without errors – the cause of these errors is not yet known and if we could reliably replicate/repeat the errors, we could attempt to locate a fix. I demonstrated bandwidthd during the Gold Member Hangout this past Friday and it worked fine. I installed it, set it up, it recorded data, showed graphs, etc. No problems whatsoever. I could even remove and re-add the package and it was still OK.

That was on a full install of pfSense 2.2.1.

To make any sort of progress we'll need to find out what aspect of your system(s) lead to the crash. Getting info about the device make/model, type of install (full, nanobsd, what size, etc), amount of RAM, other packages you have installed, what features are in use, etc. Anything that could lead to a clue about what is causing it to crash.

tompeeters

Running FULL 2.2.1 RELEASE on this hardware: https://www.applianceshop.eu/sense-quad-core-gen2-ssd.html (no virtual appliance).

Not running any other packages besides bandwithd.

All nic's are configured in trunk, which handle in total 20 interfaces.

We are using the following modules:

Firewall + aliases + NAT
DHCP Relay
2 OpenVPN Servers s2s, connected to others sites which also run pfsense

dmesg output: http://pastebin.com/ckZWRkq1

system.log output: http://pastebin.com/nn5DM29a

romainp

Hi all!
I too have the issue.
I use bandwidthd with postgresql. I will post some details about what I found to date.

Romain

arrmo

FYI, still seeing this issue also - on 2.2.2-RELEASE (amd64).

What info would help out here? More than happy to provide it.

Thanks!