pkg-static update still using 100% cpu! Unacceptable!



  • I've had problems with this before, but tonight I messed something up on my SG-3100 and had to do a full config restore from a backup. Now, pfsense has been trying to reinstall the packages I use for several hours maxed out at 100% CPU! So apparently, I can't get my packages reinstalled without burning this thing up. Swell.

    Doing some searching on the internet quickly reveals that I'm 1 of many with the exact same problem.

    I'm just about ready to throw this thing in the trash and go buy a firewall solution that actually works.

    Sorry, but I'm having a very hard time keeping my cool and not cussing you guys up one side and down the other. This is totally unacceptable.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    I'm just about ready to throw this thing in the trash and go buy a firewall solution that actually works.

    Send it to me. I'll even pay the freight.

    Instead of venting in a year-old thread, why not try to troubleshoot the problem?

    Did you try rebooting the unit and then restoring again?

    Did you notice if there are any messages that might help debug this, such as which package is being installed?

    Is there anything in the System log about the issue?

    How many packages do you have installed? Worst case, could you restore without package data and then install them manually?



  • @KOM I've tried everything I know how to do.

    I've rebooted, power cycled, restored at least 3 times. Nothing has worked.

    Couldn't find anything on the internet that matches my exact problem.

    There are only 2 critical packages that I need installed. PF blocker and Suricata. Suricata absolutely refuses to install. It just sits there stuck with the CPU maxed out.

    I only have 5 packages installed. The other 3 are ntopng, mail report, and open vpn client config export. They all finally reinstalled.

    I don't know how to restore without restoring the package data, and I don't see any log files anywhere that indicate why this thing is broken.

    I'm sorry. I liked this pfsense box at first, but it has really soured my opinion of it as of last night. I've been up the whole night fighting with it. There is no reason in the world a piece of production hardware/software should endlessly max out a cpu just because it can't check for updates correctly.



  • I just hit the "Install suricata" button :

    63b70a85-1299-44ec-983d-835dfa6e90e8-image.png

    It installed :

    >>> Installing pfSense-pkg-suricata... 
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    The following 14 package(s) will be affected (of 0 checked):
    
    New packages to be INSTALLED:
    	pfSense-pkg-suricata: 4.1.5 [pfSense]
    	suricata: 4.1.5 [pfSense]
    	libyaml: 0.1.6_2 [pfSense]
    	nss: 3.39 [pfSense]
    	nspr: 4.20 [pfSense]
    	libpcap: 1.8.1 [pfSense]
    	libnet: 1.1.6_5,1 [pfSense]
    	py27-yaml: 5.1 [pfSense]
    	jansson: 2.11 [pfSense]
    	hyperscan: 4.6.0 [pfSense]
    	hiredis: 0.13.3 [pfSense]
    	barnyard2: 1.13_1 [pfSense]
    	broccoli: 1.97,1 [pfSense]
    	GeoIP: 1.6.12 [pfSense]
    
    Number of packages to be installed: 14
    
    The process will require 38 MiB more space.
    7 MiB to be downloaded.
    [1/13] Fetching pfSense-pkg-suricata-4.1.5.txz: .......... done
    [2/13] Fetching suricata-4.1.5.txz: .......... done
    [3/13] Fetching libyaml-0.1.6_2.txz: ......... done
    [4/13] Fetching nss-3.39.txz: .......... done
    [5/13] Fetching nspr-4.20.txz: .......... done
    [6/13] Fetching libpcap-1.8.1.txz: .......... done
    [7/13] Fetching libnet-1.1.6_5,1.txz: .......... done
    [8/13] Fetching py27-yaml-5.1.txz: .......... done
    [9/13] Fetching jansson-2.11.txz: ..... done
    [10/13] Fetching hyperscan-4.6.0.txz: .......... done
    [11/13] Fetching hiredis-0.13.3.txz: .......... done
    [12/13] Fetching barnyard2-1.13_1.txz: .......... done
    [13/13] Fetching broccoli-1.97,1.txz: .......... done
    Checking integrity... done (0 conflicting)
    [1/14] Installing nspr-4.20...
    [1/14] Extracting nspr-4.20: .......... done
    [2/14] Installing GeoIP-1.6.12...
    [2/14] Extracting GeoIP-1.6.12: .......... done
    [3/14] Installing libyaml-0.1.6_2...
    [3/14] Extracting libyaml-0.1.6_2: ......... done
    [4/14] Installing nss-3.39...
    [4/14] Extracting nss-3.39: .......... done
    [5/14] Installing libpcap-1.8.1...
    [5/14] Extracting libpcap-1.8.1: .......... done
    [6/14] Installing libnet-1.1.6_5,1...
    [6/14] Extracting libnet-1.1.6_5,1: .......... done
    [7/14] Installing py27-yaml-5.1...
    [7/14] Extracting py27-yaml-5.1: .......... done
    [8/14] Installing jansson-2.11...
    [8/14] Extracting jansson-2.11: .......... done
    [9/14] Installing hyperscan-4.6.0...
    [9/14] Extracting hyperscan-4.6.0: .......... done
    [10/14] Installing hiredis-0.13.3...
    [10/14] Extracting hiredis-0.13.3: .......... done
    [11/14] Installing broccoli-1.97,1...
    [11/14] Extracting broccoli-1.97,1: .......... done
    [12/14] Installing suricata-4.1.5...
    [12/14] Extracting suricata-4.1.5: .......... done
    [13/14] Installing barnyard2-1.13_1...
    [13/14] Extracting barnyard2-1.13_1: ...... done
    [14/14] Installing pfSense-pkg-suricata-4.1.5...
    [14/14] Extracting pfSense-pkg-suricata-4.1.5: .......... done
    Saving updated package information...
    done.
    Loading package configuration... done.
    Configuring package components...
    Loading package instructions...
    Custom commands...
    Executing custom_php_install_command()...   
      Setting up initial configuration.
      Setting package version in configuration file.
    done.
    Executing custom_php_resync_config_command()...done.
    Menu items... done.
    Services... done.
    Writing configuration... done.
    Message from GeoIP-1.6.12:
    
    GeoIP does not ship with the actual data files. You must download
    them yourself! To obtain the free database, run:
    # /usr/local/bin/geoipupdate.sh
    Message from suricata-4.1.5:
    
    ===========================================================================
    
    If you want to run Suricata in IDS mode, add to /etc/rc.conf:
    
    	suricata_enable="YES"
    	suricata_interface="<if>"
    
    NOTE: Declaring suricata_interface is MANDATORY for Suricata in IDS Mode.
    
    However, if you want to run Suricata in Inline IPS Mode in divert(4) mode,
    add to /etc/rc.conf:
    
    	suricata_enable="YES"
    	suricata_divertport="8000"
    
    NOTE:
    	Suricata won't start in IDS mode without an interface configured.
    	Therefore if you omit suricata_interface from rc.conf, FreeBSD's
    	rc.d/suricata will automatically try to start Suricata in IPS Mode
    	(on divert port 8000, by default).
    
    Alternatively, if you want to run Suricata in Inline IPS Mode in high-speed
    netmap(4) mode, add to /etc/rc.conf:
    
    	suricata_enable="YES"
    	suricata_netmap="YES"
    
    NOTE:
    	Suricata requires additional interface settings in the configuration
    	file to run in netmap(4) mode.
    
    RULES: Suricata IDS/IPS Engine comes without rules by default. You should
    add rules by yourself and set an updating strategy. To do so, please visit:
    
     http://www.openinfosecfoundation.org/documentation/rules.html
     http://www.openinfosecfoundation.org/documentation/emerging-threats.html
    
    You may want to try BPF in zerocopy mode to test performance improvements:
    
    	sysctl -w net.bpf.zerocopy_enable=1
    
    Don't forget to add net.bpf.zerocopy_enable=1 to /etc/sysctl.conf
    
    ===========================================================================
    Message from barnyard2-1.13_1:
    
    Read the notes in the barnyard2.conf file for how to configure
    /usr/local/etc/barnyard2.conf after installation.  For addtional information
    see the Securixlive FAQ at http://www.securixlive.com/barnyard2/faq.php.
    
    In order to enable barnyard2 to start on boot, you must edit /etc/rc.conf
    with the appropriate flags, etc.  See the FreeBSD Handbook for syntax:
    http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-rcng.html
    
    For the various options available, type % barnyard2 -h after install or read
    the options in the startup script - in /usr/local/etc/rc.d.
    
    Barnyard2 can process unified2 files from snort or suricata.  It can also
    interact with snortsam firewall rules as well as the sguil-sensor. Those
    ports must be installed separately if you wish to use them.
    
    ************************************************************************
    >>> Cleaning up cache... done.
    Success
    

    Please note : I did not set up the package - neither do I have set up info in my config (that can reak / make / make work / do something else ....). I'm not an Suricata expert, don't know where it used for - and so I'm not going to use it.

    Right now, the package server at 'pfSense' end (Netgate) is up, the Internet works, my system is ok, so I could install it.

    What I do know : If a package does not install :

    • The DNS are not default any more - may be broken (happens a lot).
    • The package maintenance database is 'broken' => many posts on this forum explain how to repair that. Start going to the console, and do an 'update' from there.
    • More ... see forum.

    Also : very important : I'm using

    2.4.4-RELEASE-p3 (amd64)
    built on Wed May 15 18:53:44 EDT 2019
    FreeBSD 11.2-RELEASE-p10
    

  • Rebel Alliance Developer Netgate

    Have you tried doing a fresh firmware install/recovery?

    If the package databases are corrupt, a simple config restore or factory reset won't help that.

    If you need the recovery image, it's freely available by contacting support at https://go.netgate.com

    Once you have a fresh installation, then restore your configuration.



  • @jimp I haven't tried that yet. I just did a factory reset and pfblocker is still installed. What the hell? This thing is screwed up beyond belief and I have no idea how it got like this. How do I get that image to do a REAL factory restore?


  • Rebel Alliance Developer Netgate

    Sounds like all the symptoms point to either the package database being corrupt, or something in the filesystem being corrupted. A reinstall is the next best thing to try.

    If you go to https://go.netgate.com and open a request, the support crew will send you a link to the image.



  • @jimp I opened a ticket. Hope they're quick about it.

    I thought a factory reset would restore it to a fresh pfsense install. Apparently not. This thing is totally screwed. Upon factory reset, it walks me through the initial setup wizard. I input all my info and hit save. It doesn't save it. The thing reboots and everything I entered is gone.



  • @jimp I'm totally lost! I downloaded the image, but how the heck am I supposed to get it loaded on the box? I have a little USB cable in the box. What am I supposed to do with that? Do I just write the image to a usb stick and reboot it or what?


  • Rebel Alliance Developer Netgate

    You'll use a USB stick. Did they not give you a pointer to the instructions? They are on the documentation site:

    https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html



  • @jimp yes they sent the link, but all i have is the usb cable. I don't have a serial cable



  • @jimp Now apparently i need a driver for my com port in order to use putty? This is a disaster.



  • @jimp I can't get the com port driver installed on my windows server 2008 box. Tried to open a putty session and get only a blank screen. I need help!


  • Rebel Alliance Developer Netgate

    The USB cable is (effectively) a serial console cable for these devices. Directions for the driver are all on the doc site if you need them. If you get hung up, send a message back to the support crew, they'll help you through the process.



  • @jimp I know that, i can't get the appropriate driver installed for my com port so i can putty into the pfsense box. I need help! The support guys haven't offered me any more help beyond sending me the image link.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @jimp I know that, i can't get the appropriate driver installed for my com port so i can putty into the pfsense box. I need help! The support guys haven't offered me any more help beyond sending me the image link.

    The USB-to-serial driver should install and work on any current Windows desktop client. Do you not have a Windows PC or laptop you could use?

    Never tried to use the USB-to-serial driver on Windows Server, especially something a bit older such as 2008.



  • @bmeeks Hi Bill. I gave up and tried it on a windows 7 pc and that worked. I've managed to reflash pfsense onto my SG-3100 and restore my config. Looks to still have the same problem. 1 cpu core stuck perpetually at 100%, and has not reinstalled the packages as it said it was doing at boot up.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    I don't know how to restore without restoring the package data

    Open up your config.xml backup in a text editor. Look for the section titled <installedpackages> and delete that entire section then save the file under a different name. Restore from that new file.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @bmeeks Hi Bill. I gave up and tried it on a windows 7 pc and that worked. I've managed to reflash pfsense onto my SG-3100 and restore my config. Looks to still have the same problem. 1 cpu core stuck perpetually at 100%, and has not reinstalled the packages as it said it was doing at boot up.

    One thing that can cause this is for the box to not have Internet access during the package installation stage. It will try forever to contact the pkg repository. Are you sure the box has a good Internet connection and that DNS is working?

    Can you log in to the web GUI and then go to DIAGNOSTICS > DNS LOOKUP and try to look up a common web site by name such as google.com or cnn.com. See if you get back valid IP addresses.

    @KOM has given you a method to manually edit your config.xml file to have the firewall skip attempting to auto-reinstall packages. You can try that as well, but make that change on a copy of your backup file and not to the original!



  • @bmeeks Yes I've got internet access. DNS test is working fine. I can get on all my normal sites. And I'm responding to you.

    So it seems that even a fresh, factory install cannot/will not install suricata. It just sits hung at 100% cpu.

    What in the world is happening here? I'm ready to stomp on this thing.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @bmeeks Yes I've got internet access. DNS test is working fine. I can get on all my normal sites. And I'm responding to you.

    So it seems that even a fresh, factory install cannot/will not install suricata. It just sits hung at 100% cpu.

    What in the world is happening here? I'm ready to stomp on this thing.

    Look in the system log and see where it is stalling with the installation. I would check if pfBlocker is perhaps blocking an IP address that Suricata wants to access. That has happened before since some pfBlocker lists target sections of AWS, and the Snort rules (if you are using them in Suricata) are hosted on AWS infrastructure.

    The pkg utility will install the binary and GUI package code and then call a post-install PHP script within the Suricata package. That script detects your previous installation's configuration in config.xml and starts restoring it. One step in that process is downloading the configured rules.



  • @bmeeks pfblocker is not yet installed. And again, even a fresh factory image cannot install suricata. I will check the logs.



  • @bmeeks I've tried everything multiple times. I don't know what else to do. This box has turned itself into a doorstop. I give up.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @bmeeks I've tried everything multiple times. I don't know what else to do. This box has turned itself into a doorstop. I give up.

    You can easily remove the Suricata package configuration section. Just make a copy of the config.xml file and then open the copy in a text editor. Find the section that says <installedpackages> and remove all the suricata from that section. You will find several XML elements with Suricata info. There will be a <menu></menu> entry, a <service></service> entry, and then finally a <suricata></suricata> entry. Remove all of those tags and Suricata-related info enclosed by them. Save the newly modified file on the firewall and try rebooting again.



  • @bmeeks I've tried all that, Bill. It still won't work. I've tried installing the packages I need on a fresh image just after I entered all of my IP, DNS, and WAN data to get the Internet working. I would think that if it doesn't work then, it surely isn't gonna work at any other step either.



  • @bmeeks I'm afraid pfsense is just not a production ready software suite. It's just not. It's glitchy, and full of bugs. And I've just ran face first into two big ones. #1 The fact that the package updater pegs the cpu at 100% if it doesn't get the responses it expects. And #2 The package update service is wholly unreliable, if it even works at all.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @bmeeks I've tried all that, Bill. It still won't work. I've tried installing the packages I need on a fresh image just after I entered all of my IP, DNS, and WAN data to get the Internet working. I would think that if it doesn't work then, it surely isn't gonna work at any other step either.

    Your replies to me and @KOM have been a little confusing. I thought you said the package reinstall was hanging during the initial reboot after first installing an image. Is that the case? Or does the box boot up fine and then you are attempting to install the packages onto a clean image (one where you did NOT import an existing config.xml)?

    If the latter, then your machine has a gremlin in it for sure. If you are restoring a config that had your list of installed packages in it, then try to restore a config with all the packages removed from the config.xml file.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @bmeeks I've tried all that, Bill. It still won't work. I've tried installing the packages I need on a fresh image just after I entered all of my IP, DNS, and WAN data to get the Internet working. I would think that if it doesn't work then, it surely isn't gonna work at any other step either.

    We're trying to help you. There are thousands and thousands of successful pfSense installs around the world. The vast majority of them in production situations. This problem appears isolated to your setup.

    If pfSense truly had the issues you describe as a common situation, this board would be overrun with complaints and posts. There are none (or very, very few considering the number of pfSense installs around the world). Ranting and raving won't fix your problem. Maybe you need to stop for today, take a long rest, and try again tomorrow?



  • Dude, you admit in the first post that you messed something thing up and you have to resist 'cussing us up one side and down the other' You realize that you are asking for help from other users, right? This is not official support. You sound like a whiny twist- you can't get the usb driver to work on your out of support, over ten year old server, etc, etc. What's totally unacceptable is your attitude toward the people offering you help. Calm down and grow up.



  • @bmeeks It hangs during the initial reboot, and it hangs everywhere else too. It simply will not install suricata even after a clean re-image just after I enter the data needed to get the internet working. I can't even get it to install openvpn client export now. Just hangs and pegs the CPU. And nothing of any value is getting written into the system log either.

    I know you're trying to help me and I greatly appreciate it. It just seems like I/we have ran out of ammo. I can't think of anything left to try. I've been working on this straight since midnight last night.

    Yes Bill. CLEAN BRAND NEW RE-IMAGE. With nothing done except going through the wizard to get the internet working. Still will not work.



  • @dotdash I changed a setting that I couldn't figure out how to get it back how it was. I had backup's made assuming that it would be a simple restore and 10 minutes I'd be back to where I was before I started. I was so very wrong. So yes, I screwed up in assuming that a backup could actually be properly restored.



  • I've never seen that behavior, but I don't have experience with the arm version. I would re-image so I knew I was starting from scratch. Then I would NOT restore the config, but manually get connected to the internet, then verify I could ping from the box. Then I would try loading a package and see if it worked or returned errors. Knowing the error message might help to track down what's happening.



  • @RedDelPaPa said in pkg-static update still using 100% cpu! Unacceptable!:

    @bmeeks It hangs during the initial reboot, and it hangs everywhere else too. It simply will not install suricata even after a clean re-image just after I enter the data needed to get the internet working. I can't even get it to install openvpn client export now. Just hangs and pegs the CPU. And nothing of any value is getting written into the system log either.

    I know you're trying to help me and I greatly appreciate it. It just seems like I/we have ran out of ammo. I can't think of anything left to try. I've been working on this straight since midnight last night.

    Yes Bill. CLEAN BRAND NEW RE-IMAGE. With nothing done except going through the wizard to get the internet working. Still will not work.

    I would check to be sure that the /var/db/pkg directory is empty and then try the image restore again. It really looks like your pkg database files are trashed. I don't know if simply reinstalling a factory image will actually clear that directory out and start over or not.

    To be sure, here is what I would do. Get to a shell prompt on the firewall and run this command --

    rm -rf /var/db/pkg
    

    After running this command,pkg will definitely be hosed up. Refer to this pfSense documentation page and perform the steps there to recreate a new pkg database structure: https://docs.netgate.com/pfsense/en/latest/packages/fixing-a-broken-pkg-database.html.

    You might also want to force a filesystem check at boot. Do that by following the instructions here: https://docs.netgate.com/pfsense/en/latest/hardware/forcing-a-filesystem-check.html.


  • LAYER 8 Netgate

    Sounds to me like you might be seeing failing storage. You might try adding an M.2 and seeing if your issues are resolved.

    https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/m-2-sata-installation.html

    @RedDelPaPa So yes, I screwed up in assuming that a backup could actually be properly restored.

    It can be. Perhaps not for you in your specific set of circumstances based on all of the facts I see, like a possibly failing storage.

    If you have successfully recovered using the recovery image and are still experiencing problems, installing to new storage is probably your best path forward.



  • @dotdash I have tried exactly what you stated. I wanted to know if my config was really the problem. So I tried loading the packages I need right after booting from a fresh re-image and walking through the wizard and getting my internet connection working. It too was a no go.



  • @bmeeks Ok, here is what I have in there now after getting 3 of my 5 desired packages installed:
    2019-10-25 20_37_33-pfsense.wattscc.net - Diagnostics_ Edit File.png

    I will trying your next suggestions shortly.



  • @Derelict Interesting. What is used in an SG-3100 for the native storage? Is it flash memory or a regular magnetic hard drive?

    If it were failing storage, wouldn't I likely see garbled log files and such?



  • @dotdash I got this after between 1 to 4 hours waiting for openvpn client export to install:
    2019-10-25 20_10_20-pfsense.wattscc.net - System_ Package Manager_ Package Installer.png

    I can copy that link into my browser and it goes right to it immediately. So pfsense is also broken here if it doesn't perform any retry's.



  • @Derelict Can I install to a USB stick plugged into an SG-3100 as a test for bad storage?



  • Ok guys. BRAND NEW RE-IMAGE. I run through the wizard to get my lan/wan info entered so I can connect to the internet. CPU usage normal. Everything appears normal. No installed packages.

    I then go to package manager / installed packages and it hangs for about 5 minutes and then fails with this:
    c15dba42-9a4a-4932-a2f4-8692f7ae4f51-image.png

    Meanwhile it pegs the cpu at 100% and still pegged:
    fec29b08-8978-49ca-bf21-0fdbdd2ded8b-image.png


Log in to reply