Dozens of netstat commands

[Stewart]

I feel as if I've asked or seen this before but I can't find the resolution.  I have a router that's running at 100%.  If I run top-aSH a see dozens of

netstat -I re1 -nWb -f link
and
netstat -I re2 -nWb -f link

each taking a few percent all culminating with a load in excess of 175.  Any idea what might be going on?  We rebooted it to get functioning again but it wont last but a couple of days at the most.

Thanks!

[johnpoz]

What packages do you have installed - I just looked on my pfsense and not seeing any such processes/commands running

[doktornotor]

I don't think it's so much packages like ridiculously outdated stone-age pfSense version.

https://github.com/pfsense/pfsense/blob/RELENG_1_2/usr/local/www/ifstats.php

[cmb]

That would be a 1.x version. Upgrade.

[Stewart]

The version is 2.1.4.  It's not the most current but it's not that old.

2.1.4-RELEASE (amd64)
built on Fri Jun 20 15:48:47 EDT 2014

Packages are:
Bandwidthd
Cron
HAVP
mailreport
nmap
notes
pfblocker
snort
squid
suidguard
sudo

[David_W]

2.1.x might only be one minor version back, but is based around much older underpinnings than 2.2.x - FreeBSD 8.3 as opposed to FreeBSD 10.1.

2.1 is a discontinued branch based on an operating system that reached end of life some time ago. There is unlikely to be any interest in debugging any problems. 2.2.5 should be released in the next few days and you are strongly recommended to back up your configuration and upgrade to 2.2.5..

It is important to keep security software up to date. I am pretty sure the version of Snort available for 2.1.x has stopped getting VRT updates.

[cmb]

Maybe that's coming from a package then? Not seeing that command in any current packages either. The only instance of it in base code is what doktornotor linked, which hasn't existed since 1.x versions. Custom code on the box?

[doktornotor]

There's no such nonsense in any package I've seen. Look at the crontab perhaps. (And yeah, Snort is completely gone from 2.1.x branch)

[Stewart]

So, this is the part where I have egg on my face.

We have a script on these APU1Cs that activates the LEDs on the front.  I have a few dozen units out and for whatever reason this only randomly occurs every now and then.  I still can't find my post about this from some months back but it's essentially the same thing.  It's happened twice now with all the units I have out.  It confuses me since it shows in different ways.

The script is called blinkled.sh but it doesn't show up in top or ps.  I would expect if that script was the problem I would see it listed but it never is.  In top it seems to show up as this netstat command since that is how it polls the interfaces and in ps it shows up as tcsh (I guess because that's the shell we had to use as bash wouldn't work).  The script is started at the top of each hour and updates every 5 seconds.  When called, it runs the /usr/bin/kilall tcsh command to clear out any running instances.  For whatever reason, the killall command will stop killing the tcsh process and these just keep getting called over and over again and they all run concurrently and eventually take up all processing power.  It may work for days, weeks, or months just fine before the killall tcsh command doesn't do anything anymore when called from the scripts.  We have units that have been out over a year and still don't have this problem.  The only solution so far is just to put the killall tcsh command somewhere else in the script.  No idea why that solves it as it still gets processed at essentially the same time (right at the start to kill the previous process before calling the new one.)

tl'dr - A script we created was out of control.  pfSense and its packages are just fine.

The reason we are on the older version is that we send out very specific builds of the routers to incorporate the features we want to all of our clients.  For instance, via a series of scripts we are able to run snort, squid, and havp perfectly fine on these units.  The process only writes about 250MB each month to the cards, or about 75% of the provisioned space each year (4GB provisioned using 8GB and 16GB cards).  That should give them over 10 years of life.  While packages such as snort may not get updates since the OS was EoL on the 2.1.4 back in August, they still load up with slightly older rule sets and still offer strong protection until they get the new 2.2.4 image we are rolling out.

Thanks everyone for your contributions.  Let me know if you have anything else to add (like why it works on some and craps out after a few months on others).