SG-1100, outages, no DHCP, 10 days log missing
-
Things in /tmp would get removed at reboot anyway so that's not a good test. Try moving a file from there to /root. Or just creating a file in /root from Diag > Edit File.
-
@stephenw10
Okay /tmp/ is flushed at boot. Figures. My bad. Followed your instructions and created a file called test.tmp in /root:
After reboot I see this:
The test.tmp file is no longer there.
-
Hmm, doesn't look good.
Did you run the eMMC utils test?
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc -
@stephenw10
Well I believe I need to be on console access for that and I'm still not on site. I do think I tried that once though, but the two essential figures:eMMC Life Time Estimation A eMMC Life Time Estimation Bwere not in the output so I figured my unit is too old to check.
Serial: NTG1933xxxxxx
May be 2019? I got mine used so no idea.I suppose it's time to write off this unit and move forward with another one, which has NTG21xxxxxxxx serial #. That unit has good eMMC Life Time figures, 0x01 and 0x02.
-
@stephenw10 said in SG-1100, outages, no DHCP, 10 days log missing:
Are you using ram disks?
Now that I'm going to deploy these, where might I find size recommendations for the SG-1100? It only has 1GB RAM and at this point it's using:
Dashboard says 48%
From Diagnostics / System Activity:
Mem: 189M Active, 92M Inact, 41M Laundry, 255M Wired, 355M FreeThanks.
-
It depends if you have any packages running and what they are. I usually start at double the defaults so 80 and 120MB.
-
Hi @stephenw10,
Thank you for all the support! The unit only runs Avahi, OpenVPN (remote admin) and pfBlocker with only the PRI1 list.
I just had my wife replace the old 1100 with a newer 21xx serial 1100 and I get around 54% RAM usage.
I prepared the unit with 23.09.1 and UFS and I disabled pfB logging for PRI1.
So I'll start at what you recommended 80/120 and see how that works out.
Kind regards!
-
Yup pfBlocker will be the biggest user of /var there if you enable it.
-
@stephenw10
4 days in and here are some screen shots. I had apparently already set 150M each, so I just went with these figures to see what would happen.
Apparently not much activity, now that I disabled most of the logs.eMMC storage usage seems a lot to me, but I may be wrong.
What do you think?
-
2.2GB? That's reasonable IMO. Here's a test 1100 I have:
[24.03-RELEASE][admin@1100-3.stevew.lan]/root: df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/663e8ac8b2733b8a 14G 2.1G 10G 17% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 80M 96K 80M 0% /tmp tmpfs 120M 6.9M 113M 6% /var devfs 1.0K 0B 1.0K 0% /var/dhcpd/devThat one's running from USB.
-
@stephenw10 Okay thanks, this is what I get:
Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/6647542c1c44d84c 6.7G 2.2G 4.0G 35% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 150M 2.3M 148M 2% /tmp tmpfs 150M 48M 102M 32% /varSo when the /var RAMdisk gets full, will those logs be offloaded to eMMC automatically, to make room for new logs?
-
No, it should not get full. The log management should limit the total log size.
The RAM disks do get backed up to eMMC and re-created at reboot.
-
@Cabledude said in SG-1100, outages, no DHCP, 10 days log missing:
when the /var RAMdisk gets full
It's basically like running out of disk space. We've found RAM disk usage is reasonably small, however, we disable a lot of the default logging such as the default block rules, pfBlocker DNSBL, and Suricata HTTP logging. pfBlocker usage depends heavily on the lists chosen...there is one "adult site" list that takes over 1 GB to download and unzip for instance.
OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.
There are a couple of packages that do not transfer to eMMC at boot...I think bandwidthd is one IIRC, at least last I knew.
-
Hi @SteveITS , thanks for your insights. When looking at the log files I noticed pfBlocker is a major contributor. So I started by disabling logging for all IP block lists (one PRI1 and two GeoIP) and default block rules (under Status / System Logs / System / General).
However, what about pfB DNSBL logging options? I see this in global DNSBL settings:
So I suppose I could select "Null Block (no logging)" but will it still block anything? In the "DNSBL Groups" section, all individual Groups are set to "DNSBL WebServer/VIP".
OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.
What are you trying to say with this? I understand the RAM disk is empty upon creation and will be used by the system logs to store the logs, just like any drive. Just wondering what you mean specifically or what you're referring to.
Thanks,
-
I run pfBlocker in RAM disks and don't really see an issue. Just see how it goes. Set the max lines options lower if you see the logs growing too large.
-
@Cabledude said in SG-1100, outages, no DHCP, 10 days log missing:
OTOH pfSense uses tmpfs now, so the RAM isn't allocated until used.What do you mean by this?
In "the old days" pfSense would preallocate the 80+120 MB or whatever RAM. Technically speaking (if it let you) you could allocate 8 GB to RAM disk and it would work until you actually ran out of RAM. On our clients' 2100s we usually set 512 and 1024 but the entire "memory in use" is normally about 1 GB because /tmp and /var are not large.
OTOH if you set 1 GB for /var and try to use the UT1 list it will run out of "disk space" and fail. All our lists are WAY smaller.
For DNSBL I want to say this is on by default? (could be misremembering)
"DNS Reply Logging
Enable the logging of all DNS Replies that were not blocked via DNSBL. "...and then yeah the lists can be logged too.
-
@SteveITS said in SG-1100, outages, no DHCP, 10 days log missing:
For DNSBL I want to say this is on by default? (could be misremembering)
"DNS Reply Logging
Enable the logging of all DNS Replies that were not blocked via DNSBL. "So do you uncheck this one on your clients' devices?
...and then yeah the lists can be logged too.
Well here it gets confusing (to me at least). See my previous post and screen shot. I get three options for Logging/Blocking:
- Null Block (logging)
- DNSBL Webserver/VIP
- Null Block (no logging)
From which I conclude that the third would result in the smallest log files, but I wonder if it will still block anything. Only the "DNSBL Webserver/VIP" option will sinkhole the bad domains.
-
I have the logs set to the default 20k lines. I have DNSBL set to the default 'No Global mode' and I see:
[24.03-RELEASE][admin@fw1.stevew.lan]/root: ls -ls /var/log/pfblockerng total 5524 0 -rw------- 1 unbound unbound 0 May 24 00:00 dns_reply.log 2324 -rw------- 1 unbound unbound 2375830 May 24 12:05 dnsbl.log 4 -rw-r--r-- 1 root wheel 1535 Feb 15 2023 dnsbl_error.log 4 -rw------- 1 root wheel 1562 May 24 00:00 dnsbl_parsed_error.log 4 -rw------- 1 root wheel 1846 May 24 00:00 error.log 232 -rw------- 1 root wheel 234766 May 24 04:01 extras.log 4 -rw-r--r-- 1 root wheel 121 May 24 04:00 maxmind_ver 628 -rw------- 1 root wheel 641033 May 24 00:00 pfblockerng.log 2324 -rw------- 1 unbound unbound 2375830 May 24 12:05 unified.logSo ~5MB of logs.
I do only use a few small lists though.
-
@stephenw10 said in SG-1100, outages, no DHCP, 10 days log missing:
I have the logs set to the default 20k lines. I have DNSBL set to the default 'No Global mode'
Right.
the default 'No Global mode' setting basically means logging is set to the individual group setting, which is always "DNSBL Webserver/VIP" by default
This below is my remote 1100:
total 2056 0 -rw------- 1 unbound unbound 0 May 24 13:00 dns_reply.log 8 -rw------- 1 root wheel 612 May 24 13:00 dnsbl_parsed_error.log 128 -rw------- 1 unbound unbound 62255 May 24 13:00 dnsbl.log 24 -rw------- 1 root wheel 9607 May 24 13:00 error.log 16 -rw------- 1 root wheel 5819 May 24 13:00 extras.log 192 -rw------- 1 root wheel 94504 May 24 13:00 ip_block.log 8 -rw-r--r-- 1 root wheel 121 May 24 10:00 maxmind_ver 1368 -rw------- 1 root wheel 696798 May 24 13:00 pfblockerng.log 312 -rw------- 1 unbound unbound 157331 May 24 13:00 unified.logAnd here is my home 2100:
total 4610 977 -rw------- 1 unbound unbound 3264595 May 24 13:38 dns_reply.log 9 -rw------- 1 root wheel 7986 May 24 13:01 dnsbl_parsed_error.log 793 -rw------- 1 unbound unbound 2978044 May 24 13:37 dnsbl.log 17 -rw------- 1 root wheel 57040 May 24 13:01 error.log 17 -rw------- 1 root wheel 39429 May 24 13:01 extras.log 1625 -rw------- 1 root wheel 3437077 May 24 13:01 ip_block.log 9 -rw-r--r-- 1 root unbound 121 May 17 21:02 maxmind_ver 169 -rw------- 1 root wheel 607620 May 24 13:01 pfblockerng.log 1 -rw-r--r-- 1 unbound unbound 0 Oct 2 2023 py_error.log 993 -rw------- 1 unbound unbound 3277651 May 24 13:38 unified.logI have the 2100 Max version with 128GB SSD. Currently no RAMdisks configured because SSD is far less sensitive to rewrites than eMMC. Would you agree? Or would you still set RAMdisks on an SSD unit?
-
Oh my, by reading these figures I notice the "dns_reply.log" is zero, which just made me discover that this unit (my remote 1100) is still running pfB DNSBL unbound mode and not python mode. Oops. This substantially takes more memory. I just changed it to python mode.
Here is the output after the change to python mode:
total 2272 104 -rw------- 1 unbound unbound 52668 May 24 14:13 dns_reply.log 8 -rw------- 1 root wheel 860 May 24 14:00 dnsbl_parsed_error.log 128 -rw------- 1 unbound unbound 62255 May 24 14:00 dnsbl.log 24 -rw------- 1 root wheel 9607 May 24 14:00 error.log 16 -rw------- 1 root wheel 5819 May 24 14:00 extras.log 192 -rw------- 1 root wheel 94504 May 24 14:00 ip_block.log 8 -rw-r--r-- 1 root wheel 121 May 24 10:00 maxmind_ver 1376 -rw------- 1 root wheel 700880 May 24 14:00 pfblockerng.log 0 -rw-r--r-- 1 unbound unbound 0 May 24 13:50 py_error.log 416 -rw------- 1 unbound unbound 209999 May 24 14:13 unified.log
