Bandwidthd issues?

arrmo

Hi,

Not sure this is related to just this package. Was making some OpenVPN settings changes yesterday - Saving there also broken php-fpm (confirmed several times). I think this is a bit deeper issue … :(.

Thanks!

cmb

It's definitely specific to just this package. Changing OpenVPN will restart packages, which is probably why that'd trigger it. I'm still not sure how or why it triggers any problem along those lines, haven't been able to replicate that and not sure why it seems to be so easy for some.

arrmo

Completely agreed, and I understand your pain. Can't fix a problem you can't duplicate!

Is there any way for me to try to "monitor" what is happening when I do this? Not sure if there is a way to generically turn up logging levels, to try to debug it. Willing to do what I can to help, but I can't figure it out either … :(.

Thanks!

tojaktoty

@doktornotor:

Well you can remove the broken package from pfSense Developer Shell (option 12), IIRC. Then restart webConfigurator (or possibly PHP-FPM if things are really messed up).

What commands would need be run in developer shell option 12 in order to remove bandwidthd and restart webConfigurator and PHP-FPM?

I setup a test vm pfsense, ran 'vi /config/config.xml', edited bandwidthd package to off, ':wq!' won't let me save.

Read the developer shell help info and https://doc.pfsense.org/index.php/Using_the_PHP_pfSense_Shell
I can't figure out how to remove the package in the developer shell. I've searched the forums and haven't found a clue.

Thanks

arrmo

Hi,

I have a patch that works for my startup issues (multiple copies of Bandwidthd writing to PostgreSQL), and it is installed in System Patches (in pfSense) … but it's not getting applied after Bandwidthd installs (on an upgrade) / before it starts. Is there a way to make this happen?

Not sure if others want the patch, I can post it if desired.

Thanks!

HellMind

Is pfsense abandoned?
Why nobody removes bandwithd from pkg list

Also why theres no way to fix the webgui unavailable?

Its a know issue.
Its easy, pfsense webgui fails if bandwidthd is enabled.

WHAT CAN A USER WITHOUT KNOWLEDGE ABOUT PFSENSER OR FREEBSD DO?

I hate to see pfsense isn't ready AGAIN to be used.

tompeeters

I 'm having the same issues with bandwithd and would like to help on a constructive way to solve this.

Running release 2.2.1

Installed package
Going to Services >>> bandwithd >>> enable >>> save >>> GUI crash
Restarting PHP-FPM from shell to fix GUI

config.xml:

		 <bandwidthd><config><enable>on</enable>
				<active_interface>opt6</active_interface>
				<subnets_custom>192.168.128.0/21</subnets_custom>
				 <skipintervals><graphcutoff>512</graphcutoff>
				 <promiscuous><outputcdf><recovercdf><outputpostgresql><postgresqlhost><postgresqldatabase><postgresqlusername><postgresqlpassword><sensorid><filter><drawgraphs><meta_refresh><graph_log_info></graph_log_info></meta_refresh></drawgraphs></filter></sensorid></postgresqlpassword></postgresqlusername></postgresqldatabase></postgresqlhost></outputpostgresql></recovercdf></outputcdf></promiscuous></skipintervals></config></bandwidthd>

system.log:

Mar 24 10:13:32 firewall php-fpm[60155]: /pkg_mgr_install.php: Beginning package installation for bandwidthd .
Mar 24 10:13:33 firewall check_reload_status: Syncing firewall
Mar 24 10:13:45 firewall check_reload_status: Syncing firewall
Mar 24 10:13:45 firewall php-fpm[60155]: /pkg_mgr_install.php: Successfully installed package: bandwidthd.
Mar 24 10:13:46 firewall check_reload_status: Reloading filter
Mar 24 10:14:01 firewall check_reload_status: Syncing firewall
Mar 24 10:14:01 firewall php-fpm[16579]: /pkg_edit.php: The command '/usr/local/etc/rc.d/bandwidthd.sh stop' returned exit code '1', the output was 'No matching processes were found' 
Mar 24 10:14:01 firewall bandwidthd: Monitoring subnet 255.255.255.252 with netmask 255.255.255.252
Mar 24 10:14:01 firewall bandwidthd: Monitoring subnet 255.255.248.0 with netmask 255.255.248.0
Mar 24 10:14:01 firewall bandwidthd: Opening em1_vlan70
Mar 24 10:14:01 firewall bandwidthd: Packet Encoding: Ethernet
Mar 24 10:14:29 firewall check_reload_status: Syncing firewall
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.2562) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: unix:/var/run/php-fpm.socket 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.3346) response not received, request sent: 1434 on socket: unix:/var/run/php-fpm.socket for /pkg_edit.php?, closing connection 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:29 firewall lighttpd[42020]: (mod_fastcgi.c.3587) all handlers for /ifstats.php?if=em5_vlan131 on .php are down. 
Mar 24 10:14:30 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:31 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:32 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket 
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:32 firewall lighttpd[42020]: (mod_fastcgi.c.3587) all handlers for /ifstats.php?if=em4_vlan99 on .php are down. 
Mar 24 10:14:33 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:34 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:35 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket 
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:35 firewall lighttpd[42020]: (mod_fastcgi.c.3587) all handlers for /ifstats.php?if=em2_vlan98 on .php are down. 
Mar 24 10:14:36 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:37 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:38 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:38 firewall lighttpd[42020]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket 
Mar 24 10:14:38 firewall lighttpd[42020]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket 
Mar 24 10:14:38 firewall lighttpd[42020]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1 
Mar 24 10:14:39 firewall check_reload_status: Could not connect to /var/run/php-fpm.socket
Mar 24 10:14:39 firewall rc.php-fpm_restart[85880]: >>> Restarting php-fpm

The logging "check_reload_status: Could not connect to /var/run/php-fpm.socket" errors disappear when I disable bandwithd

Feel free to ask any other thing to test.

heper

@HellMind:

Is pfsense abandoned?
Why nobody removes bandwithd from pkg list

Also why theres no way to fix the webgui unavailable?

Its a know issue.
Its easy, pfsense webgui fails if bandwidthd is enabled.

WHAT CAN A USER WITHOUT KNOWLEDGE ABOUT PFSENSER OR FREEBSD DO?

I hate to see pfsense isn't ready AGAIN to be used.

yes because there are some packages that are not maintained, pfsense is dead and is unuseable.
You could volunteer and maintain the bandwidthd package and fix it's issues.

@tompeeters:

I 'm having the same issues with bandwithd and would like to help on a constructive way to solve this.

yes, constructive ways FTW :))
there is an open bug on this issue, please add all your relevant debug info on it … then perhaps this'll be resolved more quickly.
https://redmine.pfsense.org/issues/3986

jimp

bandwidthd has not been removed from the list of packages because bandwidthd still works fine for many people without errors – the cause of these errors is not yet known and if we could reliably replicate/repeat the errors, we could attempt to locate a fix. I demonstrated bandwidthd during the Gold Member Hangout this past Friday and it worked fine. I installed it, set it up, it recorded data, showed graphs, etc. No problems whatsoever. I could even remove and re-add the package and it was still OK.

That was on a full install of pfSense 2.2.1.

To make any sort of progress we'll need to find out what aspect of your system(s) lead to the crash. Getting info about the device make/model, type of install (full, nanobsd, what size, etc), amount of RAM, other packages you have installed, what features are in use, etc. Anything that could lead to a clue about what is causing it to crash.

tompeeters

Running FULL 2.2.1 RELEASE on this hardware: https://www.applianceshop.eu/sense-quad-core-gen2-ssd.html (no virtual appliance).

Not running any other packages besides bandwithd.

All nic's are configured in trunk, which handle in total 20 interfaces.

We are using the following modules:

Firewall + aliases + NAT
DHCP Relay
2 OpenVPN Servers s2s, connected to others sites which also run pfsense

dmesg output: http://pastebin.com/ckZWRkq1

system.log output: http://pastebin.com/nn5DM29a

romainp

Hi all!
I too have the issue.
I use bandwidthd with postgresql. I will post some details about what I found to date.

Romain

arrmo

FYI, still seeing this issue also - on 2.2.2-RELEASE (amd64).

What info would help out here? More than happy to provide it.

Thanks!

forceps

I have this issue too.

I am running bandwidthd with postgresql on two pfsense 2.2.4 system. It crashes the Captive Portal after running for 1 one 2 days, the webgui is still okay though. I tried start from a fresh pfsense then copy the bandwidthd folder from another pfsense, put it under /opt, edit config and start bandwidthd from console, it is running, but still crashes Captive Portal.

arrmo

FYI, still seeing this in 2.2.6 … :(.

Fresh install, reboot -> multiple copies of bandwidthd running. When I kill them, it also kills other services, like the Web GUI.

Thoughts?

Thanks!

superwutze

hello!

i installed bandwidthd recently an quickly ran into the problems described here, so i reinstalled my pfsense-box today from scratch and it crashed immediately upon package reinstall.
my box is a pc-engines apu and i installed the embedded-serial-4g-amd64 image. then i restored the config file ond the following message appeared on the serial terminal:


One moment please, reinstalling packages...
 >>> Trying to fetch package info... Done.
 >>> List of packages to reinstall: bandwidthd
ERROR: can't find installed pbi (bandwidthd-2.0.1_6-amd64)

Removing package...
Starting package deletion for bandwidthd-2.0.1_6-amd64...done.
Removing bandwidthd components...
Configuration... done.
done.
Beginning package installation for bandwidthd . 100%
Installing bandwidthd and its dependencies. 100% 100%Enter full pathname of shell or RETURN for /bin/sh: 
#

the firewall seems to work as expected but it never signalled to be ready with it's little tune. the webgui is running and shows the 'packages are being reinstalled' sign, the cpu load is above 50%.
bandwidthd is configured to write to an external postgres-db only and it is bound to collect data on a bridge-interface, maybe that is important.
trying to reboot from the webgui plays the tune, then the webgui crashes but the system doesn't reboot. rebooting via ssh works.
after a reboot the system seems to work without a problem (there are 2 bandwidthd processes) but trying to restart bandwidthd kills php-fpm immediately. after restarting php-fpm via ssh bandwidthd is stopped and after starting there is just one process. cpu-load is below 10%.
i'm glad the system at least boots again after the fresh install. before the reinstall php-fpm got killed during startup and i never got a dhcp address on wan, so i had to connect via ssh to lan and restart php-fpm.

i hope to help the mysteries of this problem a little, bandwidthd seems to be a very usefull program. feel free to ask for more details!

regards,
andy