4100/6100 Base or Max

dennypage

I would generally recommend getting the Max for safety.

That said, if you have a UPS you can mitigate the eMMC risk by using RAM disks.

Having lost an eMMC on a 5100, I became concerned about the eMMC on my (non Max) 6100, and recently made the switch to RAM disks. The two graphs below illustrate the dramatic effect of the change:

In case you are going to ask... the dip shown for July 26-27 was checking the impact of disabling default rule and pfBlockerNG logging.

The little blips you see every 24 hours are /var/log and /var/db/rrd being copied from memory to the eMMC.

SteveITS

@Cabledude said in 4100/6100 Base or Max:

pfBlocker doesn't require SSD, am I right? Based on the above posts I am sold on SSD though, if for nothing else I get peace of mind.

It does not require an SSD, no. Everything that "requires" an SSD depends on logging. If you disable logging of the default block rules, and don't use or log DNSBL, then there's not much disk writing. Others log everything or run Suricata on web servers, etc., and log a lot of stuff.

The 2100 also has a separate WAN port instead of using the switch, and does not use VLANs, which is why it can get over 500 Mbps (500 in + 500 out = 1000). So it adds +3 GB and a NIC.

mcury

421 GB ÷ 2280 hours
0,184649123 GB per hour x 24
= 4,431578947 GB per day x 30
= 132,94736841 GB per month x 12
= 1595,36842092 GB per year or 1.6 TB per year

SN520 = 100 TBW endurance
100 / 1,6 = 62.5 years

[23.05.1-RELEASE][root@pfsense.home.arpa]/root: cat /boot/loader.conf.local 
hint.mmcsd.0.disabled="1"

[23.05.1-RELEASE][root@pfsense.home.arpa]/root: iostat -x
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
nvd0           0       5      0.4     50.0     0     0     0     0    0   0

In my opinion, the best option is to use a remote syslog server, disable all pfsense logs to the disk and use a SSD.
Also, disable eMMC using loader.conf.local.

There are some drawbacks in using RAM disk, but now I can't remember exactly what the problems were, I think it was something with geoIP database download with pfblockerNG and lists not being written to the disk when RAM disk was enabled.

Once you disable RAM disk, you would have to download the geoIP database and lists again.

Cabledude

@SteveITS

Here is my current SG-1100 System activity:

last pid: 48936; load averages: 0.72, 0.52, 0.45 up 3+16:25:58 23:19:46
186 threads: 3 running, 165 sleeping, 18 waiting
CPU: 10.7% user, 0.7% nice, 2.8% system, 2.3% interrupt, 83.5% idle
Mem: 54M Active, 523M Inact, 12K Laundry, 222M Wired, 94M Buf, 154M Free

This is UFS, pfBlocker DNSBL AdsBasic, IP PRI1, no GeoIP, avahi, 8 vlans, night time so the rest of the family is asleep. I will re-enable geoIP top spammers all countries for a couple of hours and then disable pfblocker and see what that does to the graph

SteveITS

@mcury said in 4100/6100 Base or Max:

drawbacks in using RAM disk

Not sure about disabling it, though that requires a reboot like enabling it does. I am pretty sure pfB and Suricata/Snort have been updated to save the files out of the RAM disk at reboot? Not 100% sure though. A RAM disk can be an issue with large lists, e.g. the UT1 list takes over 1 GB to extract. And of course losing logs (pfSense has options to copy logs and other things to disk every "n" hours).

Overall have used a RAM disk on pretty almost all of our clients' and our routers for a couple years now. Probably would not on an 1100 with 1GB.

mcury

@SteveITS said in 4100/6100 Base or Max:

Overall have used a RAM disk on pretty almost all of our clients' and our routers for a couple years now. Probably would not on an 1100 with 1GB.

I used for a long time in a SG-3100, first thing you notice is a longer time to boot, pfblockerNG takes a long time finish.
Some logs about /var/db/ missing during boot once you disable RAM disk, and other weird things along of the already mentioned.

But, yes, RAM disk is a very good thing to do in case you don't have a SSD and don't want to rely on the eMMC.

dennypage

@mcury I'm not sure what all that data was intended to convey, but what the heck...

Your IO rates and other people's IO rates are not necessarily comparable. Too much depends upon the packages installed, the amount of firewall logging, and the general activity in the system.

FWIW, my IO rate prior to RAM disks was 14,674 GB per year. Now it is reduced to just 69 GB per year. [Btw, I believe you have a slight math error. With 421GB over 2280 hours, your projected usage should be 1614.20 GB per year.]

You are correct about an issue with GeoIP as maintained by pfBlockerNG. When you reboot, the GeoIP list will not be present until pfBlocker runs its cron entry. It's a minor inconvenience that is fixable with a boot shell script. Even if it weren't fixable, it would totally be worth it to me for the 99.5% reduction in write to the eMMC.

mcury

@dennypage said in 4100/6100 Base or Max:

I'm not sure what all that data was intended to convey, but what the heck...

Those are to help other users to be able to do their own math.
I had a SG-3100 and at that time I didn't have a SSD.
What helped me to understand and how to do the math was a post here: https://forum.netgate.com/topic/170128/emmc-write-endurance?_=1663100394507
So, it is just to help users that just like me, needed help to better understand how this works. You can just ignore it.

@dennypage said in 4100/6100 Base or Max:

With 421GB over 2280 hours, your projected usage should be 1614.20 GB per year.]

Yes, it helps a lot but as I see it, it is not the best option..

@dennypage said in 4100/6100 Base or Max:

You are correct about an issue with GeoIP as maintained by pfBlockerNG. When you reboot, the GeoIP list will not be present until pfBlocker runs its cron entry. It's a minor inconvenience that is fixable with a boot shell script. Even if it weren't fixable, it would totally be worth it to me for the 99.5% reduction in write to the eMMC.

There are other problems, not only that one..

Everything resumes to, it is my system, I'm the administrator and I'll choose what I want.
For me, SSD is better option.

dennypage

@mcury Please understand, I was not advocating against the Max. Just saying that if you have one, using ram disk can help things quite a bit.

@mcury said in 4100/6100 Base or Max:

There are other problems, not only that one..

Can you share please? I'm only aware of the one...

mcury

@dennypage said in 4100/6100 Base or Max:

Can you share please? I'm only aware of the one...

Sure, time to boot is one, if you check through the console you will be able to see that pfblockerNG is one of the packages that takes long time to load..

Second one, is that if you disable RAM disk, you will start to see some logs about /var/db/unbound cannot be mounted or something like that (I don't remember exactly).

Third one is that if you calculate something wrong, there is a chance you cannot boot anymore, can lead to data loss or other unexpected failures.

Another one, that didn't happen to me but I think it will happen with others, or already happened, is if you try to update your system, packages will be extracted and etc, what could happen if you don't have enough RAM disk for that task ? This may happen when installing packages, but again, didn't happen to me when I was using it.

dennypage

@mcury said in 4100/6100 Base or Max:

Sure, time to boot is one, if you check through the console you will be able to see that pfblockerNG is one of the packages that takes long time to load..

I don't myself see a delay associated with pfBlockerNG. I do see a delay associated with restoring /var (from the backup created at shutdown) of maybe 20 seconds, but overall this is a minor thing.

The other issues listed are associated with configuring the ram disks incorrectly, rather than recurring issues associated with each reboot.

mcury

@dennypage said in 4100/6100 Base or Max:

I don't myself see a delay associated with pfBlockerNG. I do see a delay associated with restoring /var (from the backup created at shutdown) of maybe 20 seconds, but overall this is a minor thing.

hm.. with the SG-3100, DNSBL and pfblockerNG enabled, could take up to two minutes, just with the wizard stuff..

@dennypage said in 4100/6100 Base or Max:

The other issues listed are associated with configuring the ram disks incorrectly, rather than recurring issues associated with each reboot.

Indeed.. I used RAM disk for a long time, and I can say that I was able to sell my SG-3100 with the eMMC working because of it.. 6 years using that device.
It is a life saver, but I prefer to use a SSD for that and keep my eMMC intact when the time to replace/sell happens.

Cabledude

@Cabledude said in 4100/6100 Base or Max:

This is UFS, pfBlocker DNSBL AdsBasic, IP PRI1, no GeoIP, avahi, 8 vlans, night time so the rest of the family is asleep. I will re-enable geoIP top spammers all countries for a couple of hours and then disable pfblocker and see what that does to the graph

Here we go. Around 23:45 (11:45PM) I added the GeoIP "Top Spammers" list, and played with settings for 1.5 hours, then went to bed. It did raise the CPU usage a little but after 2 AM it was largely the same as before.

Then at 6AM pfBlocker OFF. Note that CPU activity is almost zero now. Here is a one hour view:

And here is the RAM usage:
Schermafbeelding 2023-08-29 om 06.40.53.png

Schermafbeelding 2023-08-29 om 06.41.37.png

Schermafbeelding 2023-08-29 om 06.42.35.png

So after pfBlocker OFF there is a huge RAM release.

JonathanLee

@Cabledude said in 4100/6100 Base or Max:

I can recommend it only if you use the cache and want the proxy to cache running.

Could you please do some noob translate and elaborate? I don't understand any of this. But I don't use proxies.

The proxy when using SSL intercept will cache (keep on the SSD) the what passes through the firewall and eventually you will have a nice percentage of traffic that the cache will serve when requests are sent to the firewall. When the item requested is already on the firewall, the firewall will send the information without redownloading it from the Internet. Again this requires certificates installed to work. It accelerates traffic. In the past this was known as content acceleration. Again it now has extra security tools to protect from viruses and invasive containers. The MAX has a big SSD to use with the cache, the bigger the cache the more you can save. Most Internet traffic is the same logos, scripts, photos over and over on website and some changes each day on the website. The cache saves the stuff that is repeated over and over wasting energy, and network bandwidth. It holds it locally.

Dynamic Example: Windows updates Squid uses what's called "dynamic caching" this is for when a huge update is needed for multiple devices. Leading to the network doesn't need to download that same update five or six times once for every laptop on your network, it's the same update. So, the firewalls proxy just caches that and saves it, so it just send it over and over when requested. This works, I have tested it. It even works in reverse for server traffic being requested.

This works great, I use it all the time, plus its energy friendly.

It does require some extra config.

Ref:
https://wiki.squid-cache.org/SquidFaq/SquidLogs

dennypage

@Cabledude said in 4100/6100 Base or Max:

So after pfBlocker OFF there is a huge RAM release.

I don’t really see much of a drop off in you graphs. Wire and Active are the most important measures, and there isn’t that much change there. Inactive (allocated but not being used) drops a bit, but that isn’t nearly as important.

Are you using DNSBL? If so, are you using Python mode?

Cabledude

Thank you @dennypage for having a look!

@dennypage said in 4100/6100 Base or Max:

I don’t really see much of a drop off in you graphs. Wire and Active are the most important measures, and there isn’t that much change there. Inactive (allocated but not being used) drops a bit, but that isn’t nearly as important.

Free goes up from 13 to 36 after stopping pfBlocker. I figured that is a lot, but maybe I need to learn more about memory management.

Are you using DNSBL? If so, are you using Python mode?

Unbound mode

Thanks,

Gertjan

@Cabledude said in 4100/6100 Base or Max:

Unbound mode

Why ? can you motivate this choice ?

If it is performance, memory utilization, etc you're after :

NLLab, the authors of Unbound, said themselves : "use Python mode, that's why we've included it".

Cabledude

@Gertjan said in 4100/6100 Base or Max:

@Cabledude said in 4100/6100 Base or Max:
Unbound mode

Why ? can you motivate this choice ?

Ummmm…well, how shall I put this. Okay I’ll be honest and say I just left it at its default setting and didn’t notice that the python option existed. Thank you for pointing this out. I will certainly try the python setting now.

Maybe I’ve been too far conditioned to assuming autodetect mechanisms, i.e. to expect the software to select the right version depending on the environment.
However pfSense isn’t quite the “automatic” firewall, it expects us to do research and manually configure the settings most appropriate for our situation or use case.

dennypage

@Cabledude said in 4100/6100 Base or Max:

Maybe I’ve been too far conditioned to assuming autodetect mechanisms, i.e. to expect the software to select the right version depending on the environment.
However pfSense isn’t quite the “automatic” firewall, it expects us to do research and manually configure the settings most appropriate for our situation or use case.

For the most case, the defaults are good and do not require changing. FWIW, pfBlockerNG defaults to Unbound Mode because prior versions of Unbound could crash based upon pfBlockerNG's configuration. This has since been fixed.

stephenw10

If you enable pfBlocker with a large number of lists then you will see an increase in CPU usage for a given volume of traffic. pf has to filter against all the list of IPs for pfBlocker auto-rules and that can be non-trivial.