pkg-static update still using 100% cpu! Unacceptable!

KOM

@RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

I'm just about ready to throw this thing in the trash and go buy a firewall solution that actually works.

Send it to me. I'll even pay the freight.

Instead of venting in a year-old thread, why not try to troubleshoot the problem?

Did you try rebooting the unit and then restoring again?

Did you notice if there are any messages that might help debug this, such as which package is being installed?

Is there anything in the System log about the issue?

How many packages do you have installed? Worst case, could you restore without package data and then install them manually?

RedDelPaPa

@KOM I've tried everything I know how to do.

I've rebooted, power cycled, restored at least 3 times. Nothing has worked.

Couldn't find anything on the internet that matches my exact problem.

There are only 2 critical packages that I need installed. PF blocker and Suricata. Suricata absolutely refuses to install. It just sits there stuck with the CPU maxed out.

I only have 5 packages installed. The other 3 are ntopng, mail report, and open vpn client config export. They all finally reinstalled.

I don't know how to restore without restoring the package data, and I don't see any log files anywhere that indicate why this thing is broken.

I'm sorry. I liked this pfsense box at first, but it has really soured my opinion of it as of last night. I've been up the whole night fighting with it. There is no reason in the world a piece of production hardware/software should endlessly max out a cpu just because it can't check for updates correctly.

Gertjan

I just hit the "Install suricata" button :

It installed :

>>> Installing pfSense-pkg-suricata... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 14 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	pfSense-pkg-suricata: 4.1.5 [pfSense]
	suricata: 4.1.5 [pfSense]
	libyaml: 0.1.6_2 [pfSense]
	nss: 3.39 [pfSense]
	nspr: 4.20 [pfSense]
	libpcap: 1.8.1 [pfSense]
	libnet: 1.1.6_5,1 [pfSense]
	py27-yaml: 5.1 [pfSense]
	jansson: 2.11 [pfSense]
	hyperscan: 4.6.0 [pfSense]
	hiredis: 0.13.3 [pfSense]
	barnyard2: 1.13_1 [pfSense]
	broccoli: 1.97,1 [pfSense]
	GeoIP: 1.6.12 [pfSense]

Number of packages to be installed: 14

The process will require 38 MiB more space.
7 MiB to be downloaded.
[1/13] Fetching pfSense-pkg-suricata-4.1.5.txz: .......... done
[2/13] Fetching suricata-4.1.5.txz: .......... done
[3/13] Fetching libyaml-0.1.6_2.txz: ......... done
[4/13] Fetching nss-3.39.txz: .......... done
[5/13] Fetching nspr-4.20.txz: .......... done
[6/13] Fetching libpcap-1.8.1.txz: .......... done
[7/13] Fetching libnet-1.1.6_5,1.txz: .......... done
[8/13] Fetching py27-yaml-5.1.txz: .......... done
[9/13] Fetching jansson-2.11.txz: ..... done
[10/13] Fetching hyperscan-4.6.0.txz: .......... done
[11/13] Fetching hiredis-0.13.3.txz: .......... done
[12/13] Fetching barnyard2-1.13_1.txz: .......... done
[13/13] Fetching broccoli-1.97,1.txz: .......... done
Checking integrity... done (0 conflicting)
[1/14] Installing nspr-4.20...
[1/14] Extracting nspr-4.20: .......... done
[2/14] Installing GeoIP-1.6.12...
[2/14] Extracting GeoIP-1.6.12: .......... done
[3/14] Installing libyaml-0.1.6_2...
[3/14] Extracting libyaml-0.1.6_2: ......... done
[4/14] Installing nss-3.39...
[4/14] Extracting nss-3.39: .......... done
[5/14] Installing libpcap-1.8.1...
[5/14] Extracting libpcap-1.8.1: .......... done
[6/14] Installing libnet-1.1.6_5,1...
[6/14] Extracting libnet-1.1.6_5,1: .......... done
[7/14] Installing py27-yaml-5.1...
[7/14] Extracting py27-yaml-5.1: .......... done
[8/14] Installing jansson-2.11...
[8/14] Extracting jansson-2.11: .......... done
[9/14] Installing hyperscan-4.6.0...
[9/14] Extracting hyperscan-4.6.0: .......... done
[10/14] Installing hiredis-0.13.3...
[10/14] Extracting hiredis-0.13.3: .......... done
[11/14] Installing broccoli-1.97,1...
[11/14] Extracting broccoli-1.97,1: .......... done
[12/14] Installing suricata-4.1.5...
[12/14] Extracting suricata-4.1.5: .......... done
[13/14] Installing barnyard2-1.13_1...
[13/14] Extracting barnyard2-1.13_1: ...... done
[14/14] Installing pfSense-pkg-suricata-4.1.5...
[14/14] Extracting pfSense-pkg-suricata-4.1.5: .......... done
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...   
  Setting up initial configuration.
  Setting package version in configuration file.
done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
Message from GeoIP-1.6.12:

GeoIP does not ship with the actual data files. You must download
them yourself! To obtain the free database, run:
# /usr/local/bin/geoipupdate.sh
Message from suricata-4.1.5:

===========================================================================

If you want to run Suricata in IDS mode, add to /etc/rc.conf:

	suricata_enable="YES"
	suricata_interface="<if>"

NOTE: Declaring suricata_interface is MANDATORY for Suricata in IDS Mode.

However, if you want to run Suricata in Inline IPS Mode in divert(4) mode,
add to /etc/rc.conf:

	suricata_enable="YES"
	suricata_divertport="8000"

NOTE:
	Suricata won't start in IDS mode without an interface configured.
	Therefore if you omit suricata_interface from rc.conf, FreeBSD's
	rc.d/suricata will automatically try to start Suricata in IPS Mode
	(on divert port 8000, by default).

Alternatively, if you want to run Suricata in Inline IPS Mode in high-speed
netmap(4) mode, add to /etc/rc.conf:

	suricata_enable="YES"
	suricata_netmap="YES"

NOTE:
	Suricata requires additional interface settings in the configuration
	file to run in netmap(4) mode.

RULES: Suricata IDS/IPS Engine comes without rules by default. You should
add rules by yourself and set an updating strategy. To do so, please visit:

 http://www.openinfosecfoundation.org/documentation/rules.html
 http://www.openinfosecfoundation.org/documentation/emerging-threats.html

You may want to try BPF in zerocopy mode to test performance improvements:

	sysctl -w net.bpf.zerocopy_enable=1

Don't forget to add net.bpf.zerocopy_enable=1 to /etc/sysctl.conf

===========================================================================
Message from barnyard2-1.13_1:

Read the notes in the barnyard2.conf file for how to configure
/usr/local/etc/barnyard2.conf after installation.  For addtional information
see the Securixlive FAQ at http://www.securixlive.com/barnyard2/faq.php.

In order to enable barnyard2 to start on boot, you must edit /etc/rc.conf
with the appropriate flags, etc.  See the FreeBSD Handbook for syntax:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-rcng.html

For the various options available, type % barnyard2 -h after install or read
the options in the startup script - in /usr/local/etc/rc.d.

Barnyard2 can process unified2 files from snort or suricata.  It can also
interact with snortsam firewall rules as well as the sguil-sensor. Those
ports must be installed separately if you wish to use them.

************************************************************************
>>> Cleaning up cache... done.
Success

Please note : I did not set up the package - neither do I have set up info in my config (that can reak / make / make work / do something else ....). I'm not an Suricata expert, don't know where it used for - and so I'm not going to use it.

Right now, the package server at 'pfSense' end (Netgate) is up, the Internet works, my system is ok, so I could install it.

What I do know : If a package does not install :

The DNS are not default any more - may be broken (happens a lot).
The package maintenance database is 'broken' => many posts on this forum explain how to repair that. Start going to the console, and do an 'update' from there.
More ... see forum.

Also : very important : I'm using

2.4.4-RELEASE-p3 (amd64)
built on Wed May 15 18:53:44 EDT 2019
FreeBSD 11.2-RELEASE-p10

jimp

Have you tried doing a fresh firmware install/recovery?

If the package databases are corrupt, a simple config restore or factory reset won't help that.

If you need the recovery image, it's freely available by contacting support at https://go.netgate.com

Once you have a fresh installation, then restore your configuration.

RedDelPaPa

@jimp I haven't tried that yet. I just did a factory reset and pfblocker is still installed. What the hell? This thing is screwed up beyond belief and I have no idea how it got like this. How do I get that image to do a REAL factory restore?

jimp

Sounds like all the symptoms point to either the package database being corrupt, or something in the filesystem being corrupted. A reinstall is the next best thing to try.

If you go to https://go.netgate.com and open a request, the support crew will send you a link to the image.

RedDelPaPa

@jimp I opened a ticket. Hope they're quick about it.

I thought a factory reset would restore it to a fresh pfsense install. Apparently not. This thing is totally screwed. Upon factory reset, it walks me through the initial setup wizard. I input all my info and hit save. It doesn't save it. The thing reboots and everything I entered is gone.

RedDelPaPa

@jimp I'm totally lost! I downloaded the image, but how the heck am I supposed to get it loaded on the box? I have a little USB cable in the box. What am I supposed to do with that? Do I just write the image to a usb stick and reboot it or what?

jimp

You'll use a USB stick. Did they not give you a pointer to the instructions? They are on the documentation site:

https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

RedDelPaPa

@jimp yes they sent the link, but all i have is the usb cable. I don't have a serial cable

RedDelPaPa

@jimp Now apparently i need a driver for my com port in order to use putty? This is a disaster.

RedDelPaPa

@jimp I can't get the com port driver installed on my windows server 2008 box. Tried to open a putty session and get only a blank screen. I need help!

jimp

The USB cable is (effectively) a serial console cable for these devices. Directions for the driver are all on the doc site if you need them. If you get hung up, send a message back to the support crew, they'll help you through the process.

RedDelPaPa

@jimp I know that, i can't get the appropriate driver installed for my com port so i can putty into the pfsense box. I need help! The support guys haven't offered me any more help beyond sending me the image link.

bmeeks

@RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

@jimp I know that, i can't get the appropriate driver installed for my com port so i can putty into the pfsense box. I need help! The support guys haven't offered me any more help beyond sending me the image link.

The USB-to-serial driver should install and work on any current Windows desktop client. Do you not have a Windows PC or laptop you could use?

Never tried to use the USB-to-serial driver on Windows Server, especially something a bit older such as 2008.

RedDelPaPa

@bmeeks Hi Bill. I gave up and tried it on a windows 7 pc and that worked. I've managed to reflash pfsense onto my SG-3100 and restore my config. Looks to still have the same problem. 1 cpu core stuck perpetually at 100%, and has not reinstalled the packages as it said it was doing at boot up.

KOM

@RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

I don't know how to restore without restoring the package data

Open up your config.xml backup in a text editor. Look for the section titled <installedpackages> and delete that entire section then save the file under a different name. Restore from that new file.

bmeeks

@RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

@bmeeks Hi Bill. I gave up and tried it on a windows 7 pc and that worked. I've managed to reflash pfsense onto my SG-3100 and restore my config. Looks to still have the same problem. 1 cpu core stuck perpetually at 100%, and has not reinstalled the packages as it said it was doing at boot up.

One thing that can cause this is for the box to not have Internet access during the package installation stage. It will try forever to contact the pkg repository. Are you sure the box has a good Internet connection and that DNS is working?

Can you log in to the web GUI and then go to DIAGNOSTICS > DNS LOOKUP and try to look up a common web site by name such as google.com or cnn.com. See if you get back valid IP addresses.

@KOM has given you a method to manually edit your config.xml file to have the firewall skip attempting to auto-reinstall packages. You can try that as well, but make that change on a copy of your backup file and not to the original!

RedDelPaPa

@bmeeks Yes I've got internet access. DNS test is working fine. I can get on all my normal sites. And I'm responding to you.

So it seems that even a fresh, factory install cannot/will not install suricata. It just sits hung at 100% cpu.

What in the world is happening here? I'm ready to stomp on this thing.

bmeeks

@RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

@bmeeks Yes I've got internet access. DNS test is working fine. I can get on all my normal sites. And I'm responding to you.

So it seems that even a fresh, factory install cannot/will not install suricata. It just sits hung at 100% cpu.

What in the world is happening here? I'm ready to stomp on this thing.

Look in the system log and see where it is stalling with the installation. I would check if pfBlocker is perhaps blocking an IP address that Suricata wants to access. That has happened before since some pfBlocker lists target sections of AWS, and the Snort rules (if you are using them in Suricata) are hosted on AWS infrastructure.

The pkg utility will install the binary and GUI package code and then call a post-install PHP script within the Suricata package. That script detects your previous installation's configuration in config.xml and starts restoring it. One step in that process is downloading the configured rules.