4100/6100 Base or Max

JonathanLee

@Gertjan or log with a security onion box.

Cabledude

@Gertjan said in 4100/6100 Base or Max:
Hi Gertjan thank you for your reply, interesting stats pages. Your CPU % usage looks a lot like mine and I have an SG-1100, but it does get quite warm.

Please note that the increased CPU starting around midnight is when I activated pfBlocker. It's a fairly basic config:
IP - IPv4:

IP - Geo:

DNSBL:

The reason I want to get a 4100 or 6100 is that the 1100 just doesn't seem to cut it. It gets very hot occasionally, I did have some issues such as WAN outages and DHCP missing gateway advertisements. Even the 4100 seems heavy for my network, but I feel the 4100/6100 may be more future proof than the 2100.

So, if you plan to use pfblockerng, just forget the nvme drive right away. Go for SSD direct (MAX).

That is quite a statement. Is this because you feel the eMMC is written to extensively by pfBlocker alone?

SteveITS

@Cabledude pfBlocker in DNSBL mode I believe by default is set to log all blocks. You can turn that off. Without that it doesn’t use much disk.

If you are largely using your existing configuration, use https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html to see what your current wear is on your eMMC. Then factor in new packages.

I believe both the 4100 and 6100 are passively cooled like the lower models.

The 1100 may not quite do 500/500 because it is one switch and VLANs to isolate ports.

Cabledude

@JonathanLee said in 4100/6100 Base or Max:

@Cabledude It is great the only time I see it MAX out on it is on system start up, reboots. Snort rebuilds take up some CPU and ram when that occurs too.

I am genuinely confused after letting it rest for a couple of weeks and after re-reading your posts. Could the 2100 prove to be adequate for my needs is what I am tinkering about now.

Its warm on the bottom <...> it's good good airflow.

Warm on the bottom is not preferred, too warm is a. no-go. Of course any device gets a little warm just by having it powered on for a while. The 4100/6100 heatsink is considerably heavier than the 2100's so it should stay cooler and be more long lasting.

And I am looking for some leeway to keep things cool, guaranteed. But the 4100/6100 cost around double and consume about triple the energy.

This is it running with my son playing roblox right now on the xbox one.

I also use 7 VLANs and I have about 40 static DHCP v4 mappings.

It is impressive to see this processor run with some good code like pfSense.

So yes. I'm sold on pfSense, that is for sure.

All input considering, the 2100 Max appears to be adequate, the 4100 Max will do the job quite easily and the 6100 Max would be more than I ever need. So the 4100 is probably the obvious choice...

Gertjan

@Cabledude said in 4100/6100 Base or Max:

So the 4100 is probably the obvious choice...

And get the MAX
I've one and you win some very nice feature : No more 'disk space' issues.
You can go wild with the logs.
Install snort and forget about for many days, or weeks, before your system comes down crashing on you because the 'drive is full' (with the standard 4100 this will be a matter of hours ... and you have the 'log management doesn't work' issue)
You can create a lot of ZFS "System > Boot Environments", so, before adding a package, upgrade pfSense, make a snap shot. If there is just one issue : two clicks and your back using a working router.

Also : I hope I'm telling your BS now : the 4100 is an Intel device. The 2100 an arm. So, yes, it runs hotter. So, yes, the arm - 2100 will be more greener ;)
If I have the choice, I would go for an Intel device.

I never looked in my 4100 (a bit hard to open) but I presume the bigger MAX version NVME drive is easier to replace.
And lasts longer ...as it is a bigger drive so more choice to overwrite cells before failure, because more memory.

Cabledude

Hey @JonathanLee I was debating 4100/6100 but after reading your posts the 2100 Max is in my shopping bag waiting for me to pull the trigger.
A huge pro for me is low energy cons. (5W)
But I have two big doubts still, hope you could think along.

@JonathanLee said in 4100/6100 Base or Max:

I have the 2100-MAX with the expanded SSD works like a champ. I abuse it I have Squid, [...] It's a tank, I have been testing it for some time now. It's impressive. I have an external AP running with 12 or so devices

I have around 40, not all at the same time

I can recommend it only if you use the cache and want the proxy to cache running.

Could you please do some noob translate and elaborate? I don't understand any of this. But I don't use proxies.

I wish it had 8GB ram is all.

What would you do if you'd have 8GB?

So my concerns are:
#1. SG-2100 CPU is identical to the one in my SG-1100, which I find not very powerful. GUI dash can take 6-9 secs to load.

Its warm on the bottom

Not a technician myself but I would say a more powerful CPU that keeps cool (4100) should live longer than a weak CPU that gets warm.

@Cabledude It is great the only time I see it MAX out on it is on system start up, reboots.

#2. I use pfBlocker which can whip up the CPU to 40-50% peaks. Do you use pfBlocker? I didn't see it in the list. You do use Snort which may or may not be similarly CPU hungry.

Snort rebuilds take up some CPU and ram when that occurs too.

So bottom line is I don't use very large pfBlocker lists, I probably won't use snort/suricata or ntopng. My main goal is to have a stable system which at the moment I don't have, probably mainly due to insufficient RAM.
I also went back to UFS (not ZFS) which improved stability a lot. ZFS was also too RAM hungry.

@Gertjan said in 4100/6100 Base or Max:

If 'logging' is on your list, then you should already have some network syslogger available

Not really. I do care for the basic pfBlocker logs, that is what I meant when I said SSD not eMMC.

Thanks,

Cabledude

@Gertjan said in 4100/6100 Base or Max:

Btw : I use a 4100 MAX over a close to 1 Gbit/sec symmetric connection, about 30 local devices and about 10 or so active captive portal users (hotel here) and the 4100 handles that just fine. See here for some pfSense stats.

That stats page is awesome. Is this Zabbix? Care to share how you built this?

Thanks,

SteveITS

@Cabledude As alluded to above if you get the 2100 you will likely need to replace it if you upgrade to 1G fiber. From various posts here it should top out around 700-800 Mbps without Suricata/Snort.

pfBlocker shouldn't normally use CPU outside of processing lists and feeds. It basically just puts info in to firewall aliases or DNSBL lists.

re: SSD/eMMC see this list.

@Cabledude said in 4100/6100 Base or Max:

ZFS was also too RAM hungry

That's cache.
https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html#zfs-memory-tuning

Cabledude

Thank you @SteveITS for your swift response.

@SteveITS said in 4100/6100 Base or Max:

@Cabledude As alluded to above if you get the 2100 you will likely need to replace it if you upgrade to 1G fiber.

Yes thank you for pointing that out and I am well aware of that. We have 200/40 cable atm and gigabit fiber was more future proofing than a current requirement. For what we do 500/500 would be way more than adequate.

From various posts here it should top out around 700-800 Mbps without Suricata/Snort.

That would be quite satisfying.

pfBlocker shouldn't normally use CPU outside of processing lists and feeds. It basically just puts info in to firewall aliases or DNSBL lists.

I appreciate that, but my monitoring suggests otherwise (see above post):

Please note that the increased CPU starting around midnight is when I activated pfBlocker. It's a fairly basic config

However, given you are much more experienced than myself, you are probably right and the graph points to some other process. I haven't checked the process names.

re: SSD/eMMC see this list.

Thanks, I remember seeing that page before. You may have mentioned this now to make me find out that pfBlocker doesn't require SSD, am I right? Based on the above posts I am sold on SSD though, if for nothing else I get peace of mind.

@Cabledude said in 4100/6100 Base or Max:
ZFS was also too RAM hungry

That's cache.
https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html#zfs-memory-tuning

I studied that page. I had some erratic behaviour before on my SG-1100 when it was on ZFS. I don't know what triggered that, but someone mentioned ZFS uses more RAM so I reconfigured using UFS. The unit has been much much more stable since.
However, at a complete flash and config restore, more things get reset so it may have had nothing to do with ZFS after all. Fact is I ran ZFS and pfBlocker + some other stuff so 1GB RAM is still on the small side.

The SG-2100 is basically an SG-1100 with more RAM and more switch ports (which I absolutely don't need) so the premium in terms of unit cost seems high.
However 4GB up from 1GB just might solve all the issues I had with my SG-1100.

dennypage

I would generally recommend getting the Max for safety.

That said, if you have a UPS you can mitigate the eMMC risk by using RAM disks.

Having lost an eMMC on a 5100, I became concerned about the eMMC on my (non Max) 6100, and recently made the switch to RAM disks. The two graphs below illustrate the dramatic effect of the change:

In case you are going to ask... the dip shown for July 26-27 was checking the impact of disabling default rule and pfBlockerNG logging.

The little blips you see every 24 hours are /var/log and /var/db/rrd being copied from memory to the eMMC.

SteveITS

@Cabledude said in 4100/6100 Base or Max:

pfBlocker doesn't require SSD, am I right? Based on the above posts I am sold on SSD though, if for nothing else I get peace of mind.

It does not require an SSD, no. Everything that "requires" an SSD depends on logging. If you disable logging of the default block rules, and don't use or log DNSBL, then there's not much disk writing. Others log everything or run Suricata on web servers, etc., and log a lot of stuff.

The 2100 also has a separate WAN port instead of using the switch, and does not use VLANs, which is why it can get over 500 Mbps (500 in + 500 out = 1000). So it adds +3 GB and a NIC.

mcury

421 GB ÷ 2280 hours
0,184649123 GB per hour x 24
= 4,431578947 GB per day x 30
= 132,94736841 GB per month x 12
= 1595,36842092 GB per year or 1.6 TB per year

SN520 = 100 TBW endurance
100 / 1,6 = 62.5 years

[23.05.1-RELEASE][root@pfsense.home.arpa]/root: cat /boot/loader.conf.local 
hint.mmcsd.0.disabled="1"

[23.05.1-RELEASE][root@pfsense.home.arpa]/root: iostat -x
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
nvd0           0       5      0.4     50.0     0     0     0     0    0   0

In my opinion, the best option is to use a remote syslog server, disable all pfsense logs to the disk and use a SSD.
Also, disable eMMC using loader.conf.local.

There are some drawbacks in using RAM disk, but now I can't remember exactly what the problems were, I think it was something with geoIP database download with pfblockerNG and lists not being written to the disk when RAM disk was enabled.

Once you disable RAM disk, you would have to download the geoIP database and lists again.

Cabledude

@SteveITS

Here is my current SG-1100 System activity:

last pid: 48936; load averages: 0.72, 0.52, 0.45 up 3+16:25:58 23:19:46
186 threads: 3 running, 165 sleeping, 18 waiting
CPU: 10.7% user, 0.7% nice, 2.8% system, 2.3% interrupt, 83.5% idle
Mem: 54M Active, 523M Inact, 12K Laundry, 222M Wired, 94M Buf, 154M Free

This is UFS, pfBlocker DNSBL AdsBasic, IP PRI1, no GeoIP, avahi, 8 vlans, night time so the rest of the family is asleep. I will re-enable geoIP top spammers all countries for a couple of hours and then disable pfblocker and see what that does to the graph

SteveITS

@mcury said in 4100/6100 Base or Max:

drawbacks in using RAM disk

Not sure about disabling it, though that requires a reboot like enabling it does. I am pretty sure pfB and Suricata/Snort have been updated to save the files out of the RAM disk at reboot? Not 100% sure though. A RAM disk can be an issue with large lists, e.g. the UT1 list takes over 1 GB to extract. And of course losing logs (pfSense has options to copy logs and other things to disk every "n" hours).

Overall have used a RAM disk on pretty almost all of our clients' and our routers for a couple years now. Probably would not on an 1100 with 1GB.

mcury

@SteveITS said in 4100/6100 Base or Max:

Overall have used a RAM disk on pretty almost all of our clients' and our routers for a couple years now. Probably would not on an 1100 with 1GB.

I used for a long time in a SG-3100, first thing you notice is a longer time to boot, pfblockerNG takes a long time finish.
Some logs about /var/db/ missing during boot once you disable RAM disk, and other weird things along of the already mentioned.

But, yes, RAM disk is a very good thing to do in case you don't have a SSD and don't want to rely on the eMMC.

dennypage

@mcury I'm not sure what all that data was intended to convey, but what the heck...

Your IO rates and other people's IO rates are not necessarily comparable. Too much depends upon the packages installed, the amount of firewall logging, and the general activity in the system.

FWIW, my IO rate prior to RAM disks was 14,674 GB per year. Now it is reduced to just 69 GB per year. [Btw, I believe you have a slight math error. With 421GB over 2280 hours, your projected usage should be 1614.20 GB per year.]

You are correct about an issue with GeoIP as maintained by pfBlockerNG. When you reboot, the GeoIP list will not be present until pfBlocker runs its cron entry. It's a minor inconvenience that is fixable with a boot shell script. Even if it weren't fixable, it would totally be worth it to me for the 99.5% reduction in write to the eMMC.

mcury

@dennypage said in 4100/6100 Base or Max:

I'm not sure what all that data was intended to convey, but what the heck...

Those are to help other users to be able to do their own math.
I had a SG-3100 and at that time I didn't have a SSD.
What helped me to understand and how to do the math was a post here: https://forum.netgate.com/topic/170128/emmc-write-endurance?_=1663100394507
So, it is just to help users that just like me, needed help to better understand how this works. You can just ignore it.

@dennypage said in 4100/6100 Base or Max:

With 421GB over 2280 hours, your projected usage should be 1614.20 GB per year.]

Yes, it helps a lot but as I see it, it is not the best option..

@dennypage said in 4100/6100 Base or Max:

You are correct about an issue with GeoIP as maintained by pfBlockerNG. When you reboot, the GeoIP list will not be present until pfBlocker runs its cron entry. It's a minor inconvenience that is fixable with a boot shell script. Even if it weren't fixable, it would totally be worth it to me for the 99.5% reduction in write to the eMMC.

There are other problems, not only that one..

Everything resumes to, it is my system, I'm the administrator and I'll choose what I want.
For me, SSD is better option.

dennypage

@mcury Please understand, I was not advocating against the Max. Just saying that if you have one, using ram disk can help things quite a bit.

@mcury said in 4100/6100 Base or Max:

There are other problems, not only that one..

Can you share please? I'm only aware of the one...

mcury

@dennypage said in 4100/6100 Base or Max:

Can you share please? I'm only aware of the one...

Sure, time to boot is one, if you check through the console you will be able to see that pfblockerNG is one of the packages that takes long time to load..

Second one, is that if you disable RAM disk, you will start to see some logs about /var/db/unbound cannot be mounted or something like that (I don't remember exactly).

Third one is that if you calculate something wrong, there is a chance you cannot boot anymore, can lead to data loss or other unexpected failures.

Another one, that didn't happen to me but I think it will happen with others, or already happened, is if you try to update your system, packages will be extracted and etc, what could happen if you don't have enough RAM disk for that task ? This may happen when installing packages, but again, didn't happen to me when I was using it.

dennypage

@mcury said in 4100/6100 Base or Max:

Sure, time to boot is one, if you check through the console you will be able to see that pfblockerNG is one of the packages that takes long time to load..

I don't myself see a delay associated with pfBlockerNG. I do see a delay associated with restoring /var (from the backup created at shutdown) of maybe 20 seconds, but overall this is a minor thing.

The other issues listed are associated with configuring the ram disks incorrectly, rather than recurring issues associated with each reboot.