Bandwidthd issues?

charliem

The patch mentioned near the start of this thread has been applied already, you don't need to edit anything. It fixed bandwidthd so that it would start up correctly with a full install version of pfSense. (Prior to that, it only started on embedded versions).

Problems mentioned later in the thread, such as bandwidthd causing php-fpm (and thus the webgui) to fail, have not been addressed, tracked down or patched. Patches welcome.

Note that the later issues do not apply universally; ie, I continue to use bandwidthd without issue on 64bit 2.2Beta

arrmo

Hi,

I admit, still a bit confused the fact that some installations see this, others don't. Any thoughts on what to check?

Thanks!

cmb

I split part of the old thread into this thread, leaving here only the part referencing something that might still be an issue for some.

@arrmo:

I admit, still a bit confused the fact that some installations see this, others don't. Any thoughts on what to check?

I'm not seeing this either, with bandwidthd running on a handful of different systems on latest 2.2 snapshot.

If you're still seeing this, make sure you're on the latest 2.2 snapshot and can still replicate, and please provide the following info:

which platform, full or nano? (not sure? If you have a Diagnostics>Nanobsd menu, it's nano. otherwise it's full)
32 or 64 bit? (not sure? look for i386 or amd64 on dashboard)
what other packages do you have installed?
have you installed anything manually at the command line via 'pkg'?
is there anything about your setup or configuration that's atypical? anything really out of the ordinary you're doing

cmb

@lshantz:

The sad truth is, this install from the Pfsense library has been borked for TWO years!!!!

That's not the truth, it's one of the most widely-used packages. I'm not aware of any issues in versions prior to 2.2, the things discussed in this thread and the one where this was originally posted were specific to 2.2 back in the alpha days. It's working fine in 2.2 now as well.

arrmo

Hi,

Still trying to get this working. It sort of works … but breaks the GUI (I can work around that), but also creates two copies of bandwidthd on upgrade (that's more painful). Is there a way to fully de-install, so I can try a re-install? It could be my machine, as others don't see this, but not sure what all to remove to clean it up.

Thanks!

arrmo

Hi,

FYI - did a complete uninstall of bandwidthd, then reinstalled. All looked good, then I rebooted … :(. Again I had two copies of bandwidthd running, and it broke my GUI.

So the reboot after seems critical (to demonstrate the problem).

Thanks.

cmb

Anyone who can replicate the issue with bandwidthd breaking the web interface, is there anything unusual about your setup? Seems it's easy for some to replicate, but most of us, including myself with at least a handful of 2.2 systems running bandwidthd, have never been able to replicate. Could someone share a config backup from a system that exhibits the issue?

arrmo

Yep, I can get that to you - but is there an easy way to remove any sensitive information?

Thanks!

reggie14

For what its worth, installing bandwidthd seemed to bork my webGUI too. More specifically, after installation I started configuring it (i.e., selecting the LAN interface and identifying the subnet), I hit save and that's when the webGUI went down.

I restarted the pfsense box and managed to get back in. bandwidthd isn't running, and I don't plan to enable it for fear of not being able to get back in next time it goes down.

My setup isn't hugely special. I'm running 2.2-RC (amd64), Jan 02 build, with packages apinger, darkstat and snort.

Brrm

@reggie14:

For what its worth, installing bandwidthd seemed to bork my webGUI too. More specifically, after installation I started configuring it (i.e., selecting the LAN interface and identifying the subnet), I hit save and that's when the webGUI went down.

I restarted the pfsense box and managed to get back in. bandwidthd isn't running, and I don't plan to enable it for fear of not being able to get back in next time it goes down.

My setup isn't hugely special. I'm running 2.2-RC (amd64), Jan 02 build, with packages apinger, darkstat and snort.

Same thing happened to me. But after a reboot it's running without issues.

arrmo

Hi,

Do you have bandwidthd running on boot (i.e. is it enabled)? That may be your "fix" - below is why I say this …

I tried a few cases,

Upgrade my release - breaks on reboot, and 2 copies of bandwidthd are running.
Reboot, with bandwidthd enabled. Again, 2 copies are running, and I can see this in the log,
Jan 5 13:24:37 pfSense bandwidthd: Monitoring subnet 255.255.255.0 with netmask 255.255.255.0
Jan 5 13:24:37 pfSense bandwidthd: Monitoring subnet 255.255.255.0 with netmask 255.255.255.0
Jan 5 13:24:37 pfSense bandwidthd: Opening bge0
Jan 5 13:24:37 pfSense bandwidthd: Packet Encoding: Ethernet
Jan 5 13:24:37 pfSense bandwidthd: Opening bge0
Jan 5 13:24:37 pfSense bandwidthd: Packet Encoding: Ethernet
Disable bandwidthd, reboot ... then all is good, GUI doesn't break. So bandwidthd seems to be the culprit. I then manually started bandwidthd (from the GUI ... enable and save). Only a single copy runs (log below), and GUI stays up,
Jan 6 06:58:51 pfSense bandwidthd: Monitoring subnet 255.255.255.0 with netmask 255.255.255.0
Jan 6 06:58:51 pfSense bandwidthd: Opening bge0
Jan 6 06:58:51 pfSense bandwidthd: Packet Encoding: Ethernet

So it seems that reboot with bandwidhd is the issue ... and 2 copies of bandwidthd are started for some reason (confirmed by ps aux, and also in the logs). Any idea why this happens (and how to fix it)?

Thanks!

arrmo

Hi,

OK, perhaps another interesting finding here (that I admit, I stumbled on to accidentally … ;)).

In the case where things break, I see the following in the logs. I did make this happen once with manual (command line) restarts of bandwidthd, but can't seem to make it happen again ... :(.

Jan 6 09:28:11 lighttpd[27258]: (mod_fastcgi.c.1754) connect failed: No such file or directory on unix:/var/run/php-fpm.socket
Jan 6 09:28:11 lighttpd[27258]: (mod_fastcgi.c.3021) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
Jan 6 09:28:14 lighttpd[27258]: (mod_fastcgi.c.2848) fcgi-server re-enabled: unix:/var/run/php-fpm.socket

Followed by,
sshlockout[43240]: sshlockout/webConfigurator v3.0 starting up

Thoughts? I think the sshlockout may be what is causing this, but why is it happening?

Thanks!

arrmo

Hi,

OK, a bit more here - hoping to get some thoughts on this … ;).

It seems that the GUI is broken when I stop the two operating bandwidthd processes after boot (more on this below). When I boot up, there are two bandwidthd processes ... does anyone else see this? I did an uninstall / reinstall, get the same thing. This itself doesn't kill the GUI, it's when I ssh in and stop bandwidthd ... then the GUI breaks.

If I restart php-fpm, then after that I can can stop and start bandwidthd as much as I want - the GUI stays up.

Thoughts?

Thanks!

arrmo

And one more thing … stopping bandwidthd also kills ntopng (not just the GUI / php-fpm). Very odd ... :(

arrmo

Hi,

I have a change I want to try (locally), but it seems that files inside /usr/local/etc/rc.d get recreated on boot - and I admit, I can't find the source file (in text format at least … ;)). If anyone has any pointers I'd appreciate it, just trying to debug.

Thanks!

arrmo

Hi,

Hoping someone else is smarter than me here (that wouldn't be difficult … :(). I want to change /usr/local/etc/rc.d/bandwidthd.sh as follows,

Current:
rc_start() {
cd /usr/pbi/bandwidthd-amd64/local/bandwidthd
LD_LIBRARY_PATH=/usr/pbi/bandwidthd-amd64/local/lib /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
cd -
}

Updated:
rc_start() {
cd /usr/pbi/bandwidthd-amd64/local/bandwidthd
if [ -z "ps auxw | grep "[/]usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd"|awk '{print $2}'" ];then
LD_LIBRARY_PATH=/usr/pbi/bandwidthd-amd64/local/lib /usr/pbi/bandwidthd-amd64/local/bandwidthd/bandwidthd
fi
cd -
}

This is just to avoid multiple copies of bandwidthd from being started (as I see all the time). But I can't figure out how /usr/local/etc/rc.d/bandwidthd.sh is getting generated on boot.

Help?!?!

Thanks very much!

charliem

Take a look at /usr/local/pkg/bandwidthd.inc; it writes both bandwidthd.sh and bandwidthd.conf

phil.davis

AFAIK it is normal to have 4 bandwidthd processes:

ps aux | grep bandwidthd
root       41074  0.0  2.4 15748  5576  -  S     9:41PM  0:00.02 /var/bandwidthd/bandwidthd
root       41237  0.0  2.4 15748  5588  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       41425  0.0  2.4 15748  5576  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       41449  0.0  2.4 15748  5576  -  S     9:41PM  0:00.01 /var/bandwidthd/bandwidthd
root       44012  0.0  0.9 10396  1952  1  S+    9:42PM  0:00.01 grep bandwidthd

I think they are related to the recording of daily, weekly, monthly and yearly data/graphs. Each updates data/graphs at different intervals.

arrmo

Hmmm … at least when logging to an external database (PostgreSQL) this causes problems -> I have confirmed multiple entries for the same points in time, and the daily totals in PostgreSQL are 2x what they should be (due to 2 processes running).

Thoughts?

Thanks!

arrmo

Could it be this is why we're seeing different results (working / not working)? I think you're letting bandwidthd generate "local" info, but I'm logging to an external database? Just thinking out loud, trying to figure it out.

It is also interesting that the path to your bandwidth process is different - this looks to be processing the data (as it's inside var, right?), but mine is the rc.d service / daemon? I'm just trying to stop more than one of those existing, as I am getting double entries in the database (not a good thing).

Thoughts?

Thanks very much!