Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Update to 2.1 Nano, 'file system full'.

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    31 Posts 12 Posters 15.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      phil.davis
      last edited by 

      Warning: array_combine(): Both parameters should have an equal number of elements in /usr/local/www/includes/functions.inc.php on line 123 
Warning: array_combine(): Both parameters should have an equal number of elements in /usr/local/www/includes/functions.inc.php on line 125 
Warning: array_sum() expects parameter 1 to be array, boolean given in /usr/local/www/includes/functions.inc.php on line 127 
Warning: array_sum() expects parameter 1 to be array, boolean given in /usr/local/www/includes/functions.inc.php on line 128 
Warning: Division by zero in /usr/local/www/includes/functions.inc.php on line 220 Warning: Division by zero in /usr/local/www/includes/functions.inc.php on line 175
 Warning: Division by zero in /usr/local/www/includes/functions.inc.php on line 155 0%

      Those can all result from system calls out of php code that have failed, presumably due to something like a lack of memory to fork the system processes needed to run the commands to gather the data.
      @firegood: how much memory on your system? is this still happening? does the system log have any "killed" or other messages that look like lack of memory problems?

      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

      1 Reply Last reply Reply Quote 0
      • F
        firegood
        last edited by

        Yes its certainly still going on. Through the webGUI none of the "system log" tabs show any information, they are all empty. SSH closes the connection after entering a valid username and password, WinSCP will not connect either. I am not able to reboot it through the GUI either. It comes up that it is stopping the package services but never actually reboots. I am going to have to console in and see if i can retrieve the logs through the shell. Any idea what logs i should look at in particular? I only know enough about the command line to squeak by.

        I believe it has 512MB of ram (certainly no less) and it is not a wrap or alix board, but rather a mini itx mother board with a P4 processor running on the CF card via an ATA adapter.

        I have tried:
        Hard power off/on
        Attempting to manually flash the latest upgrade package through the GUI - Get message that the file is corrupt.
        Config restore - didnt help any.

        1 Reply Last reply Reply Quote 0
        • K
          kejianshi
          last edited by

          I haven't had these problems, but what I wonder is how close to full were these drives before the upgrades?

          1 Reply Last reply Reply Quote 0
          • D
            doktornotor Banned
            last edited by

            Sounds like about time to reimage and restore the config… Still don't see any info on what packages you installed there.

            1 Reply Last reply Reply Quote 0
            • F
              firegood
              last edited by

              I was going to pickup a new CF tomorrow and re-flash it.

              Packages:
              Avahi
              Cron
              Iperf
              Open VPN Client Export Utility
              pfflowd

              1 Reply Last reply Reply Quote 0
              • D
                doktornotor Banned
                last edited by

                Hmmm… the avahi thing is pretty huge when starting. Also tons of dependencies there. Try to nuke and see if it helps.

                1 Reply Last reply Reply Quote 0
                • P
                  phil.davis
                  last edited by

                  It sounds like /var is getting filled by something. A few simple commands:

                  [2.1-RELEASE][root@myrouter]/home/phil.davis(2): df
Filesystem        512-blocks   Used   Avail Capacity  Mounted on
/dev/ufs/pfsense0    1859358 510953 1199657    30%    /
devfs                      2      2       0   100%    /dev
/dev/ufs/cf           101055   7293   85678     8%    /cf
/dev/md0               78812   1424   71084     2%    /tmp
/dev/md1              118492  40848   68168    37%    /var
devfs                      2      2       0   100%    /var/dhcpd/dev

                  See which has Avail down to 0.

                  [2.1-RELEASE][root@myrouter]/home/phil.davis(3): cd /var/log
[2.1-RELEASE][root@myrouter]/var/log(4): ls
dhcpd.log      gateways.log   lastlog        ntpd.log       portalauth.log relayd.log     system.log     wireless.log
dmesg.boot     ipsec.log      lighttpd.log   openvpn.log    ppp.log        resolver.log   userlog
filter.log     l2tps.log      ntp            poes.log       pptps.log      routing.log    vpn.log

                  There should be a bunch of log files in /var/log

                  [2.1-RELEASE][root@myrouter]/var/log(15): clog system.log | grep ROUTING
Sep 15 12:20:57 myrouter php: rc.bootup: ROUTING: setting default route to 10.49.94.250

                  Most of the logs are circular logs. Use "clog" to display the content, and then "grep" for interesting strings…

                  As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                  If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                  1 Reply Last reply Reply Quote 0
                  • D
                    doktornotor Banned
                    last edited by

                    Pretty much convinced it's the Avahi thing messing this up. Cf. http://forum.pfsense.org/index.php?topic=61289.0

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      It wasn't until my box didn't come back up some time after hitting the upgrade button that I realised I'd not tried it on any box with <1GB of RAM. My box didn't/doesn't have any large packages running. The package that failed to install was lcdproc-dev which uses almost no resources under normal conditions.
                      Perhaps 60MB just isn't enough for /var? Have the ram disk sizes changed between 2.0.3 and 2.1?
                      Anyway it seems like it was the actual package re-installation process that was causing problems, once I was past that I've had no further problems.
                      For reference, after initial reboot when the box had stopped doing anything I couldn't reboot even from the console menu. I have to drop to the command line and issue 'reboot now'. Even then it errored a bit but did reboot.

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • D
                        doktornotor Banned
                        last edited by

                        @stephenw10:

                        Perhaps 60MB just isn't enough for /var? Have the ram disk sizes changed between 2.0.3 and 2.1?

                        This is what I have managed to load on a poor Alix testbox, with default 60MB /var, snapshots updated very frequently without any such issues.

                        (Note: the gwled/blinkled stuff is only used when needed, since it's a CPU hog on this poor box). So… definitely not generic issue there.

                        As for avahi, this one should most definitely be unavailable on nanobsd. And looking at the footprint and resulting attack surface, I frankly don't think anyone wants to run such thing on their firewall, at all.

                        P.S. Trying to repeat the above setup is NOT suggested for production boxes, LOL...  :P

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          You haven't even loaded Squid or Snort, not even trying!  :P

                          Yes, I agree though, it's not a matter of how many packages you have installed but of actually installing them. Downloading and extracting the package and it's dependencies (now all in the pbi?) uses a lot of space. Somehow during the upgrade there is less space available and the file system ran out in my case. Since the size of /var and /tmp are fixed (unless you've changed them) I can't see how the RAM size has much effect.  :-\

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • K
                            kejianshi
                            last edited by

                            Thats just not smart!

                            1 Reply Last reply Reply Quote 0
                            • M
                              mrsunfire
                              last edited by

                              I've tried again with no packages installed and got the same error. Now im flashing 2.1 by hand :(

                              Netgate 6100 MAX

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Which error specifically are you referring to?
                                Even if none of us have an answer at the moment it helps to at least document the problems you're having and what you've tried to correct it.

                                Steve

                                1 Reply Last reply Reply Quote 0
                                • P
                                  pubston
                                  last edited by

                                  I am having a similar issue upgrading from 2.03 to 2.1 on an Alix 2D3 w/4 GB card and nano image:

                                  Both procedures below end up with the following errors:

                                  
Generating RRD graphs...done.
Updating configuration...............Generating RRD graphs...done.
...Update RRD database wan-traffic.rrd.
Update RRD database wan-packets.rrd.
Update RRD database lan-traffic.rrd.
Update RRD database lan-packets.rrd.
Update RRD database ipsec-traffic.rrd.
Update RRD database ipsec-packets.rrd.
Generating RRD graphs...done.
pid 786 (rrdtool), uid 0 inumber 2050 on /var: filesystem full

/var: write failed, filesystem is full
....done.

                                  At reboot I get:

                                  
Enter an option: 
pfSense is now shutting down ...

/var: write failed, filesystem is full

/var: write failed, filesystem is full
Sep 16 18:26:51 lighttpd[33063]: (server.c.1546) server stopped by UID = 0 PID = 1

                                  Procedures (How I got the errors:)

                                  1. Auto upgrade using webgui (2.03 -> 2.1)

                                  2a. Backup 2.03 config
                                  2b. Wipe SD card; flash 2.1
                                  2c. Restore 2.03 config

                                  So it seems that I have no good upgrade path from 2.03 to 2.1 unless I'm missing something. Also, following a 2.1 upgrade, I get following at subsequent boots:

                                  
Generating RRD graphs...ERROR: No  tag found
ERROR: Incompatible file version, detected version . This is not supported by the version 0003 restore tool.

ERROR: Incompatible file version, detected version . This is not supported by the version 0003 restore tool.

ERROR: Incompatible file version, detected version . This is not supported by the version 0003 restore tool.

ERROR: Incompatible file version, detected version . This is not supported by the version 0003 restore tool.

done.

                                  Many thanks in any help with this.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    I assume you mean CF card not SD?

                                    I had those RRD incompatible version errors. They didn't seem to cause much of a problem. I think lost some traffic data from one WAN. Only had it one one boot though.

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Hmm, may have spoken too soon:

                                      
Sep 18 00:00:05	kernel: pid 60237 (rrdtool), uid 0 inumber 4091 on /var: filesystem full
Sep 18 00:00:04	kernel: pid 59098 (rrdtool), uid 0 inumber 4084 on /var: filesystem full

                                      Time to increase /var perhaps.

                                      Steve

                                      1 Reply Last reply Reply Quote 0
                                      • K
                                        kejianshi
                                        last edited by

                                        Ohhhhhhhh how the mighty have fallen…  ;D

                                        Give it 10 minutes of tinkering and you will be all aces again  ;)

                                        1 Reply Last reply Reply Quote 0
                                        • NetViciousN
                                          NetVicious
                                          last edited by

                                          I got too system full but my hard disk had 29G free  ???

                                          ..//\/ e t . \/ i c i o u s ..

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Hmm, odd. You hadn't chosen to use a ramdisk for /var?

                                            Steve

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.