SG-1100, outages, no DHCP, 10 days log missing
-
Hi all,
On one of my remote locations, the netgate 1100 is giving trouble. I’ve had a dozen outages last month. At first the unit came back without user intervention and fully functional.But this month it’s been different. I had 2 outages last Thursday, duration 10-20 minutes. After the outage the unit came online, but crippled: I could ping, but there was no DHCP and no VPN. And this same thing happened once before two weeks ago.
After power cycle today all functionality came back to normal. However I am missing system logs from may 8th through 18th. I do get logs from today since the power cycle.
What could be wrong? Defect eMMC maybe? I could use some help re how to start troubleshooting.
Thank you!
-
Is it not logging at all currently?
Are you using ram disks?
It's possible it could be a bad eMMC. Try creating a test file in /root and then rebooting. Make sure the file is still present.
-
Thank you for responding @stephenw10, much appreciated.
@stephenw10 said in SG-1100, outages, no DHCP, 10 days log missing:
Is it not logging at all currently?
It is logging fine ATM. When I look at the different log files, including pfBlocker, logging ceased on May 8th and resumed today after the reboot.
Are you using ram disks?
No, but I'm determined to start using them once I got this sorted out.
It's possible it could be a bad eMMC. Try creating a test file in /root and then rebooting. Make sure the file is still present.
I'm not on site. Is there a way to create a test file remotely? I can OpenVPN in. Might SSH work? No experience SSH'ing into pfSense, only UniFi.
Thanks!
-
I'm not on site. Is there a way to create a test file remotely?
Hang on. When I go to diagnostics -> Command Prompt I can upload and download files.
So through this page:
-
I uploaded "bsConfig.txt". I got confirmation it was uploaded to "/tmp/bsConfig.txt"
-
Then I downloaded "/tmp/bsConfig.txt" and it went to my Downloads folder.
-
After reboot I tried downloading this file again but nothing happened.
-
And guess what -- after the reboot all of today's logs are gone. So the logs now show the same logs until May 8th, followed by the logs as from reboot.
Is this confirmation that the eMMC is bad?
-
-
Things in /tmp would get removed at reboot anyway so that's not a good test. Try moving a file from there to /root. Or just creating a file in /root from Diag > Edit File.
-
@stephenw10
Okay /tmp/ is flushed at boot. Figures. My bad. Followed your instructions and created a file called test.tmp in /root:
After reboot I see this:
The test.tmp file is no longer there.
-
Hmm, doesn't look good.
Did you run the eMMC utils test?
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc -
@stephenw10
Well I believe I need to be on console access for that and I'm still not on site. I do think I tried that once though, but the two essential figures:eMMC Life Time Estimation A eMMC Life Time Estimation Bwere not in the output so I figured my unit is too old to check.
Serial: NTG1933xxxxxx
May be 2019? I got mine used so no idea.I suppose it's time to write off this unit and move forward with another one, which has NTG21xxxxxxxx serial #. That unit has good eMMC Life Time figures, 0x01 and 0x02.
-
@stephenw10 said in SG-1100, outages, no DHCP, 10 days log missing:
Are you using ram disks?
Now that I'm going to deploy these, where might I find size recommendations for the SG-1100? It only has 1GB RAM and at this point it's using:
Dashboard says 48%
From Diagnostics / System Activity:
Mem: 189M Active, 92M Inact, 41M Laundry, 255M Wired, 355M FreeThanks.
-
It depends if you have any packages running and what they are. I usually start at double the defaults so 80 and 120MB.
-
Hi @stephenw10,
Thank you for all the support! The unit only runs Avahi, OpenVPN (remote admin) and pfBlocker with only the PRI1 list.
I just had my wife replace the old 1100 with a newer 21xx serial 1100 and I get around 54% RAM usage.
I prepared the unit with 23.09.1 and UFS and I disabled pfB logging for PRI1.
So I'll start at what you recommended 80/120 and see how that works out.
Kind regards!
-
Yup pfBlocker will be the biggest user of /var there if you enable it.
-
@stephenw10
4 days in and here are some screen shots. I had apparently already set 150M each, so I just went with these figures to see what would happen.
Apparently not much activity, now that I disabled most of the logs.eMMC storage usage seems a lot to me, but I may be wrong.
What do you think?
-
2.2GB? That's reasonable IMO. Here's a test 1100 I have:
[24.03-RELEASE][admin@1100-3.stevew.lan]/root: df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/663e8ac8b2733b8a 14G 2.1G 10G 17% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 80M 96K 80M 0% /tmp tmpfs 120M 6.9M 113M 6% /var devfs 1.0K 0B 1.0K 0% /var/dhcpd/devThat one's running from USB.
-
@stephenw10 Okay thanks, this is what I get:
Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/6647542c1c44d84c 6.7G 2.2G 4.0G 35% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 150M 2.3M 148M 2% /tmp tmpfs 150M 48M 102M 32% /varSo when the /var RAMdisk gets full, will those logs be offloaded to eMMC automatically, to make room for new logs?
-
No, it should not get full. The log management should limit the total log size.
The RAM disks do get backed up to eMMC and re-created at reboot.
-
@Cabledude said in SG-1100, outages, no DHCP, 10 days log missing:
when the /var RAMdisk gets full
It's basically like running out of disk space. We've found RAM disk usage is reasonably small, however, we disable a lot of the default logging such as the default block rules, pfBlocker DNSBL, and Suricata HTTP logging. pfBlocker usage depends heavily on the lists chosen...there is one "adult site" list that takes over 1 GB to download and unzip for instance.
OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.
There are a couple of packages that do not transfer to eMMC at boot...I think bandwidthd is one IIRC, at least last I knew.
-
Hi @SteveITS , thanks for your insights. When looking at the log files I noticed pfBlocker is a major contributor. So I started by disabling logging for all IP block lists (one PRI1 and two GeoIP) and default block rules (under Status / System Logs / System / General).
However, what about pfB DNSBL logging options? I see this in global DNSBL settings:
So I suppose I could select "Null Block (no logging)" but will it still block anything? In the "DNSBL Groups" section, all individual Groups are set to "DNSBL WebServer/VIP".
OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.
What are you trying to say with this? I understand the RAM disk is empty upon creation and will be used by the system logs to store the logs, just like any drive. Just wondering what you mean specifically or what you're referring to.
Thanks,
-
I run pfBlocker in RAM disks and don't really see an issue. Just see how it goes. Set the max lines options lower if you see the logs growing too large.
-
@Cabledude said in SG-1100, outages, no DHCP, 10 days log missing:
OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.What do you mean by this?
In "the old days" pfSense would preallocate the 80+120 MB or whatever RAM. Technically speaking (if it let you) you could allocate 8 GB to RAM disk and it would work until you actually ran out of RAM. On our clients' 2100s we usually set 512 and 1024 but the entire "memory in use" is normally about 1 GB because /tmp and /var are not large.
OTOH if you set 1 GB for /var and try to use the UT1 list it will run out of "disk space" and fail. All our lists are WAY smaller.
For DNSBL I want to say this is on by default? (could be misremembering)
"DNS Reply Logging
Enable the logging of all DNS Replies that were not blocked via DNSBL. "...and then yeah the lists can be logged too.
