Bandwidthd issues?

cmb

The sad truth is, this install from the Pfsense library has been borked for TWO years!!!!

That's not the truth, it's one of the most widely-used packages. I'm not aware of any issues in versions prior to 2.2, the things discussed in this thread and the one where this was originally posted were specific to 2.2 back in the alpha days. It's working fine in 2.2 now as well.

arrmo

Hi,

Still trying to get this working. It sort of works … but breaks the GUI (I can work around that), but also creates two copies of bandwidthd on upgrade (that's more painful). Is there a way to fully de-install, so I can try a re-install? It could be my machine, as others don't see this, but not sure what all to remove to clean it up.

Thanks!

arrmo

Hi,

FYI - did a complete uninstall of bandwidthd, then reinstalled. All looked good, then I rebooted … :(. Again I had two copies of bandwidthd running, and it broke my GUI.

So the reboot after seems critical (to demonstrate the problem).

Thanks.

cmb

Anyone who can replicate the issue with bandwidthd breaking the web interface, is there anything unusual about your setup? Seems it's easy for some to replicate, but most of us, including myself with at least a handful of 2.2 systems running bandwidthd, have never been able to replicate. Could someone share a config backup from a system that exhibits the issue?

arrmo

Yep, I can get that to you - but is there an easy way to remove any sensitive information?

Thanks!

reggie14

For what its worth, installing bandwidthd seemed to bork my webGUI too. More specifically, after installation I started configuring it (i.e., selecting the LAN interface and identifying the subnet), I hit save and that's when the webGUI went down.

I restarted the pfsense box and managed to get back in. bandwidthd isn't running, and I don't plan to enable it for fear of not being able to get back in next time it goes down.

My setup isn't hugely special. I'm running 2.2-RC (amd64), Jan 02 build, with packages apinger, darkstat and snort.

Brrm

@reggie14:

For what its worth, installing bandwidthd seemed to bork my webGUI too. More specifically, after installation I started configuring it (i.e., selecting the LAN interface and identifying the subnet), I hit save and that's when the webGUI went down.

I restarted the pfsense box and managed to get back in. bandwidthd isn't running, and I don't plan to enable it for fear of not being able to get back in next time it goes down.

My setup isn't hugely special. I'm running 2.2-RC (amd64), Jan 02 build, with packages apinger, darkstat and snort.

Same thing happened to me. But after a reboot it's running without issues.

arrmo

Hi,

Do you have bandwidthd running on boot (i.e. is it enabled)? That may be your "fix" - below is why I say this …

I tried a few cases,

Upgrade my release - breaks on reboot, and 2 copies of bandwidthd are running.
Reboot, with bandwidthd enabled. Again, 2 copies are running, and I can see this in the log,
Jan 5 13:24:37 pfSense bandwidthd: Monitoring subnet 255.255.255.0 with netmask 255.255.255.0
Jan 5 13:24:37 pfSense bandwidthd: Monitoring subnet 255.255.255.0 with netmask 255.255.255.0
Jan 5 13:24:37 pfSense bandwidthd: Opening bge0
Jan 5 13:24:37 pfSense bandwidthd: Packet Encoding: Ethernet
Jan 5 13:24:37 pfSense bandwidthd: Opening bge0
Jan 5 13:24:37 pfSense bandwidthd: Packet Encoding: Ethernet
Disable bandwidthd, reboot ... then all is good, GUI doesn't break. So bandwidthd seems to be the culprit. I then manually started bandwidthd (from the GUI ... enable and save). Only a single copy runs (log below), and GUI stays up,
Jan 6 06:58:51 pfSense bandwidthd: Monitoring subnet 255.255.255.0 with netmask 255.255.255.0
Jan 6 06:58:51 pfSense bandwidthd: Opening bge0
Jan 6 06:58:51 pfSense bandwidthd: Packet Encoding: Ethernet

So it seems that reboot with bandwidhd is the issue ... and 2 copies of bandwidthd are started for some reason (confirmed by ps aux, and also in the logs). Any idea why this happens (and how to fix it)?

Thanks!

arrmo

Hi,

OK, perhaps another interesting finding here (that I admit, I stumbled on to accidentally … ;)).

In the case where things break, I see the following in the logs. I did make this happen once with manual (command line) restarts of bandwidthd, but can't seem to make it happen again ... :(.

Jan 6 09:28:11 lighttpd[27258]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket
Jan 6 09:28:11 lighttpd[27258]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
Jan 6 09:28:14 lighttpd[27258]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket

Followed by,
sshlockout[43240]: sshlockout/webConfigurator v3.0 starting up

Thoughts? I think the sshlockout may be what is causing this, but why is it happening?

Thanks!

arrmo

Hi,

OK, a bit more here - hoping to get some thoughts on this … ;).

It seems that the GUI is broken when I stop the two operating bandwidthd processes after boot (more on this below). When I boot up, there are two bandwidthd processes ... does anyone else see this? I did an uninstall / reinstall, get the same thing. This itself doesn't kill the GUI, it's when I ssh in and stop bandwidthd ... then the GUI breaks.

If I restart php-fpm, then after that I can can stop and start bandwidthd as much as I want - the GUI stays up.

Thoughts?

Thanks!

arrmo

And one more thing … stopping bandwidthd also kills ntopng (not just the GUI / php-fpm). Very odd ... :(

arrmo

Hi,

I have a change I want to try (locally), but it seems that files inside /usr/local/etc/rc.d get recreated on boot - and I admit, I can't find the source file (in text format at least … ;)). If anyone has any pointers I'd appreciate it, just trying to debug.

Thanks!

arrmo

Hi,

Hoping someone else is smarter than me here (that wouldn't be difficult … :(). I want to change /usr/local/etc/rc.d/bandwidthd.sh as follows,

Current:
rc_start() {
cd /usr/pbi/bandwidthd-amd64/local/bandwidthd
LD_LIBRARY_PATH=/usr/pbi/bandwidthd-amd64/local/lib /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
cd -
}

Updated:
rc_start() {
cd /usr/pbi/bandwidthd-amd64/local/bandwidthd
if [ -z "ps auxw | grep "[/]usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd"|awk '{print $2}'" ];then
LD_LIBRARY_PATH=/usr/pbi/bandwidthd-amd64/local/lib /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
fi
cd -
}

This is just to avoid multiple copies of bandwidthd from being started (as I see all the time). But I can't figure out how /usr/local/etc/rc.d/bandwidthd.sh is getting generated on boot.

Help?!?!

Thanks very much!

charliem

Take a look at /usr/local/pkg/bandwidthd.inc; it writes both bandwidthd.sh and bandwidthd.conf

phil.davis

AFAIK it is normal to have 4 bandwidthd processes:

ps aux | grep bandwidthd
root       41074  0.0  2.4 15748  5576  -  S     9:41PM  0:00.02 /var/bandwidthd/bandwidthd
root       41237  0.0  2.4 15748  5588  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       41425  0.0  2.4 15748  5576  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       41449  0.0  2.4 15748  5576  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       44012  0.0  0.9 10396  1952  1  S+    9:42PM  0:00.01 grep bandwidthd

I think they are related to the recording of daily, weekly, monthly and yearly data/graphs. Each updates data/graphs at different intervals.

arrmo

Hmmm … at least when logging to an external database (PostgreSQL) this causes problems -> I have confirmed multiple entries for the same points in time, and the daily totals in PostgreSQL are 2x what they should be (due to 2 processes running).

Thoughts?

Thanks!

arrmo

Could it be this is why we're seeing different results (working / not working)? I think you're letting bandwidthd generate "local" info, but I'm logging to an external database? Just thinking out loud, trying to figure it out.

It is also interesting that the path to your bandwidth process is different - this looks to be processing the data (as it's inside var, right?), but mine is the rc.d service / daemon? I'm just trying to stop more than one of those existing, as I am getting double entries in the database (not a good thing).

Thoughts?

Thanks very much!

charliem

Hmm, I have 8 running:

[2.2-RC][root@pfsense.localdomain]/root: ps axfw | grep bandwidthd
93615 - S 0:05.36 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
93674 - S 0:04.71 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
93823 - S 0:04.30 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
94051 - S 0:04.26 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
94636 - S 0:05.40 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
94844 - S 0:04.75 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
95011 - S 0:04.31 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
95137 - S 0:04.26 /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd

Mine is a full 64bit install, with both "generate CDF" and "recover CDF" enabled, but no PostgreSQL logging enabled. Bandwidthd runs without issues for me for the past few months, but I haven't tried to reconcile traffic reported by bandwidthd to that reported by the rrd graphs in pfSense. Bandwidthd would be more helpful for me if it could track IPv6 traffic, so I don't use it much.

The nano and full pfSense installs are treated differently by bandwidthd: check /usr/local/pkg/bandwidthd.inc starting around line 302. The nano config for the $rc['start'] stanza runs from lines 315 to 348, while the stanza for full install is lines 351-353. So, I think your different command lines are expected (maybe not bug-free :) but expected).

arrmo

Thanks for the awesome pointer! That does help. I admit, I'm not sure how these packages and files fit together - stupidity on my part!

I modified /usr/local/pkg/bandwidthd.inc (locally), and I now avoid the double processes … so one of my problems is resolved. But this also helped me to see the conditions to cause php-fpm to die - more below,

If I go in to /usr/local/etc/rc.d, I can start and stop bandwidthd (and with my change, I only get one copy of the process now). But I also found the conditions to cause php-fpm to die. If I do a Save from the GUI under bandwidthd (which updates /usr/local/etc/rc.d/bandwidth.sh!), then after this if I run bandwidthd.sh, telling it to stop -> php-fpm dies. If I restart php-fpm (manually) ... I can start or stop bandwidth as much as I want, php-fpm never dies again. Only after another Save (and regeneration of bandwidth.sh), then stopping bandwidthd (the first time) kills php-fpm. I also tried this by just manually stopping bandwidthd, by executing /usr/bin/killall bandwidthd ... and this definitely kills php-fpm. If I restart php-fpm, I can't kill php-fpm again (with starts or stops of bandwidthd) ... until I do a Save from the GUI again, then stopping bandwidthd kills php-fpm.

Does this make sense? It seems odd to me, but it is very repeatable.

Thoughts?

Thanks again for the help!

arrmo

BTW (and here comes my stupid, uneducated question … :() - how can doing a killall to bandwidthd kill other processes (php-fpm, but also ntopng)? It really does seem to kill them - but I can't understand why. Figuring this out really is the root cause.

Thanks!!!