SG-1100, outages, no DHCP, 10 days log missing

Cabledude · May 18, 2024, 7:55 PM

Thank you for responding @stephenw10, much appreciated.

@stephenw10 said in SG-1100, outages, no DHCP, 10 days log missing:

Is it not logging at all currently?

It is logging fine ATM. When I look at the different log files, including pfBlocker, logging ceased on May 8th and resumed today after the reboot.

Are you using ram disks?

No, but I'm determined to start using them once I got this sorted out.

It's possible it could be a bad eMMC. Try creating a test file in /root and then rebooting. Make sure the file is still present.

I'm not on site. Is there a way to create a test file remotely? I can OpenVPN in. Might SSH work? No experience SSH'ing into pfSense, only UniFi.

Thanks!

Cabledude · May 18, 2024, 8:28 PM

I'm not on site. Is there a way to create a test file remotely?

Hang on. When I go to diagnostics -> Command Prompt I can upload and download files.

So through this page:

I uploaded "bsConfig.txt". I got confirmation it was uploaded to "/tmp/bsConfig.txt"
Then I downloaded "/tmp/bsConfig.txt" and it went to my Downloads folder.
After reboot I tried downloading this file again but nothing happened.
And guess what -- after the reboot all of today's logs are gone. So the logs now show the same logs until May 8th, followed by the logs as from reboot.

Is this confirmation that the eMMC is bad?

stephenw10 · May 18, 2024, 10:27 PM

Things in /tmp would get removed at reboot anyway so that's not a good test. Try moving a file from there to /root. Or just creating a file in /root from Diag > Edit File.

Cabledude · May 18, 2024, 10:53 PM

@stephenw10
Okay /tmp/ is flushed at boot. Figures. My bad. Followed your instructions and created a file called test.tmp in /root:

After reboot I see this:

The test.tmp file is no longer there.

stephenw10 · May 18, 2024, 10:54 PM

Hmm, doesn't look good.

Did you run the eMMC utils test?
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

Cabledude · May 18, 2024, 11:11 PM

@stephenw10
Well I believe I need to be on console access for that and I'm still not on site. I do think I tried that once though, but the two essential figures:

eMMC Life Time Estimation A 
eMMC Life Time Estimation B

were not in the output so I figured my unit is too old to check.
Serial: NTG1933xxxxxx
May be 2019? I got mine used so no idea.

I suppose it's time to write off this unit and move forward with another one, which has NTG21xxxxxxxx serial #. That unit has good eMMC Life Time figures, 0x01 and 0x02.

Cabledude · May 18, 2024, 11:54 PM

@stephenw10 said in SG-1100, outages, no DHCP, 10 days log missing:

Are you using ram disks?

Now that I'm going to deploy these, where might I find size recommendations for the SG-1100? It only has 1GB RAM and at this point it's using:

Dashboard says 48%

From Diagnostics / System Activity:
Mem: 189M Active, 92M Inact, 41M Laundry, 255M Wired, 355M Free

Thanks.

stephenw10 · May 19, 2024, 12:44 AM

It depends if you have any packages running and what they are. I usually start at double the defaults so 80 and 120MB.

Cabledude · May 19, 2024, 1:00 PM

Hi @stephenw10,

Thank you for all the support! The unit only runs Avahi, OpenVPN (remote admin) and pfBlocker with only the PRI1 list.

I just had my wife replace the old 1100 with a newer 21xx serial 1100 and I get around 54% RAM usage.

I prepared the unit with 23.09.1 and UFS and I disabled pfB logging for PRI1.

So I'll start at what you recommended 80/120 and see how that works out.

Kind regards!

stephenw10 · May 19, 2024, 1:15 PM

Yup pfBlocker will be the biggest user of /var there if you enable it.

Cabledude · May 23, 2024, 2:59 PM

@stephenw10
4 days in and here are some screen shots. I had apparently already set 150M each, so I just went with these figures to see what would happen.

Apparently not much activity, now that I disabled most of the logs.

eMMC storage usage seems a lot to me, but I may be wrong.

What do you think?

stephenw10 · May 23, 2024, 3:12 PM

2.2GB? That's reasonable IMO. Here's a test 1100 I have:

[24.03-RELEASE][admin@1100-3.stevew.lan]/root: df -h
Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/663e8ac8b2733b8a     14G    2.1G     10G    17%    /
devfs                          1.0K      0B    1.0K     0%    /dev
tmpfs                           80M     96K     80M     0%    /tmp
tmpfs                          120M    6.9M    113M     6%    /var
devfs                          1.0K      0B    1.0K     0%    /var/dhcpd/dev

That one's running from USB.

Cabledude · May 23, 2024, 3:18 PM

@stephenw10 Okay thanks, this is what I get:

Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/6647542c1c44d84c    6.7G    2.2G    4.0G    35%    /
devfs                          1.0K      0B    1.0K     0%    /dev
tmpfs                          150M    2.3M    148M     2%    /tmp
tmpfs                          150M     48M    102M    32%    /var

So when the /var RAMdisk gets full, will those logs be offloaded to eMMC automatically, to make room for new logs?

stephenw10 · May 23, 2024, 3:35 PM

No, it should not get full. The log management should limit the total log size.

The RAM disks do get backed up to eMMC and re-created at reboot.

SteveITS · May 23, 2024, 4:00 PM

@Cabledude said in SG-1100, outages, no DHCP, 10 days log missing:

when the /var RAMdisk gets full

It's basically like running out of disk space. We've found RAM disk usage is reasonably small, however, we disable a lot of the default logging such as the default block rules, pfBlocker DNSBL, and Suricata HTTP logging. pfBlocker usage depends heavily on the lists chosen...there is one "adult site" list that takes over 1 GB to download and unzip for instance.

OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.

There are a couple of packages that do not transfer to eMMC at boot...I think bandwidthd is one IIRC, at least last I knew.

Cabledude · May 23, 2024, 9:04 PM

Hi @SteveITS , thanks for your insights. When looking at the log files I noticed pfBlocker is a major contributor. So I started by disabling logging for all IP block lists (one PRI1 and two GeoIP) and default block rules (under Status / System Logs / System / General).

However, what about pfB DNSBL logging options? I see this in global DNSBL settings:

So I suppose I could select "Null Block (no logging)" but will it still block anything? In the "DNSBL Groups" section, all individual Groups are set to "DNSBL WebServer/VIP".

OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.

What are you trying to say with this? I understand the RAM disk is empty upon creation and will be used by the system logs to store the logs, just like any drive. Just wondering what you mean specifically or what you're referring to.

Thanks,

stephenw10 · May 23, 2024, 9:06 PM

I run pfBlocker in RAM disks and don't really see an issue. Just see how it goes. Set the max lines options lower if you see the logs growing too large.

SteveITS · May 23, 2024, 9:08 PM

@Cabledude said in SG-1100, outages, no DHCP, 10 days log missing:

OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.

What do you mean by this?

In "the old days" pfSense would preallocate the 80+120 MB or whatever RAM. Technically speaking (if it let you) you could allocate 8 GB to RAM disk and it would work until you actually ran out of RAM. On our clients' 2100s we usually set 512 and 1024 but the entire "memory in use" is normally about 1 GB because /tmp and /var are not large.

OTOH if you set 1 GB for /var and try to use the UT1 list it will run out of "disk space" and fail. All our lists are WAY smaller.

For DNSBL I want to say this is on by default? (could be misremembering)
"DNS Reply Logging
Enable the logging of all DNS Replies that were not blocked via DNSBL. "

...and then yeah the lists can be logged too.

Cabledude · May 24, 2024, 2:11 PM

@SteveITS said in SG-1100, outages, no DHCP, 10 days log missing:

For DNSBL I want to say this is on by default? (could be misremembering)
"DNS Reply Logging
Enable the logging of all DNS Replies that were not blocked via DNSBL. "

So do you uncheck this one on your clients' devices?

...and then yeah the lists can be logged too.

Well here it gets confusing (to me at least). See my previous post and screen shot. I get three options for Logging/Blocking:

Null Block (logging)
DNSBL Webserver/VIP
Null Block (no logging)
From which I conclude that the third would result in the smallest log files, but I wonder if it will still block anything. Only the "DNSBL Webserver/VIP" option will sinkhole the bad domains.

stephenw10 · May 24, 2024, 11:20 AM

I have the logs set to the default 20k lines. I have DNSBL set to the default 'No Global mode' and I see:

[24.03-RELEASE][admin@fw1.stevew.lan]/root: ls -ls /var/log/pfblockerng
total 5524
   0 -rw-------  1 unbound unbound       0 May 24 00:00 dns_reply.log
2324 -rw-------  1 unbound unbound 2375830 May 24 12:05 dnsbl.log
   4 -rw-r--r--  1 root    wheel      1535 Feb 15  2023 dnsbl_error.log
   4 -rw-------  1 root    wheel      1562 May 24 00:00 dnsbl_parsed_error.log
   4 -rw-------  1 root    wheel      1846 May 24 00:00 error.log
 232 -rw-------  1 root    wheel    234766 May 24 04:01 extras.log
   4 -rw-r--r--  1 root    wheel       121 May 24 04:00 maxmind_ver
 628 -rw-------  1 root    wheel    641033 May 24 00:00 pfblockerng.log
2324 -rw-------  1 unbound unbound 2375830 May 24 12:05 unified.log

So ~5MB of logs.

I do only use a few small lists though.