4100/6100 Base or Max

JonathanLee · Jun 30, 2023, 3:45 AM

@Cabledude It is great the only time I see it MAX out on it is on system start up, reboots. Snort rebuilds take up some CPU and ram when that occurs too.

Its warm on the bottom I have it elevated with a older cooke tray that fits right into my wooden side table perfectly, so it's good good airflow.

This is it running with my son playing roblox right now on the xbox one.

login-to-view

I am running a Cortex-A53 and it does have some AVX-512 instruction set commands on it. I just learned about AVX-512 (Advanced Vector Extensions) while taking an intro to assembly code class recently.

ARM = Advanced RISC Machine (WE USE THIS WITH THE 2100)

RISC = Reduced Instruction Set Computer

CISC = Complex Instruction Set Computer Older NASA VAX systems use to CISC, they could do anything with assembly code it had so many instructions (VAX was huge in 1990s)

It is impressive to see this processor run with some good code like pfSense.

SteveITS · Jun 30, 2023, 4:43 AM

@Cabledude
https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#ram-disk-settings

It can be set for /tmp and/or /var. it doesn’t preallocate memory. On ufs it’s a bit safer upon power loss. One can lose logs etc upon power loss. Of course don’t run out of space, check your usage first before setting it. Very large pfBlocker lists or heavy logging can take up lots of space. It does take a reboot to change.

Either should be ok for that speed. If you were going to use Suricata or something that scans packets the 6100 would have more CPU headroom at gigabit.

Gertjan · Jun 30, 2023, 5:52 AM

@Cabledude said in 4100/6100 Base or Max:

#1 Base or Max: logging?

If 'logging' is on your list, then you should already have some network syslogger available on one of your LANs, and you've chosen to use an external log storage.
'logging' means : you want actually look at the logs, use them for post mortem analyzes, like knowing what happened when things went bad.
Checking log files is a good thing, although time consuming. You might as well make your live more conformable while doing so.
Know that pfBlocker is using it's own log files - here : /var/log/pfblockerng. These are handled by pfBlocker, and not send over to some remote syslogger. So, if you plan to use pfblockerng, just forget the nvme drive right away. Go for SSD direct (MAX).

Also, a bigger drive means : you can use, even abuse, the snapshot ZFS facilities.

Btw : I use a 4100 MAX over a close to 1 Gbit/sec symmetric connection, about 30 local devices and about 10 or so active captive portal users (hotel here) and the 4100 handles that just fine. See here for some pfSense stats.

@Cabledude said in 4100/6100 Base or Max:

Our network has 5-10 users, 4 switches, 6 AP's, 30 client devices. Currently running pfBlocker, Avahi, OpenVPN (just for GUI access, not to "work from home").

That's what I use. I've added FreeRadius for authentication, because why would I do things the easy way if much harder is available ? I also use some quality of live packages like acme, Notes, nut, ShellCMD, Cron.

JonathanLee · Jun 30, 2023, 6:16 AM

@Gertjan or log with a security onion box.

Cabledude · Jun 30, 2023, 9:23 AM

@Gertjan said in 4100/6100 Base or Max:
Hi Gertjan thank you for your reply, interesting stats pages. Your CPU % usage looks a lot like mine and I have an SG-1100, but it does get quite warm.
login-to-view

Please note that the increased CPU starting around midnight is when I activated pfBlocker. It's a fairly basic config:
IP - IPv4:
login-to-view
IP - Geo:
login-to-view
DNSBL:
login-to-view

The reason I want to get a 4100 or 6100 is that the 1100 just doesn't seem to cut it. It gets very hot occasionally, I did have some issues such as WAN outages and DHCP missing gateway advertisements. Even the 4100 seems heavy for my network, but I feel the 4100/6100 may be more future proof than the 2100.

So, if you plan to use pfblockerng, just forget the nvme drive right away. Go for SSD direct (MAX).

That is quite a statement. Is this because you feel the eMMC is written to extensively by pfBlocker alone?

SteveITS · Jun 30, 2023, 12:54 PM

@Cabledude pfBlocker in DNSBL mode I believe by default is set to log all blocks. You can turn that off. Without that it doesn’t use much disk.

If you are largely using your existing configuration, use https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html to see what your current wear is on your eMMC. Then factor in new packages.

I believe both the 4100 and 6100 are passively cooled like the lower models.

The 1100 may not quite do 500/500 because it is one switch and VLANs to isolate ports.

Cabledude · Aug 17, 2023, 1:08 PM

@JonathanLee said in 4100/6100 Base or Max:

@Cabledude It is great the only time I see it MAX out on it is on system start up, reboots. Snort rebuilds take up some CPU and ram when that occurs too.

I am genuinely confused after letting it rest for a couple of weeks and after re-reading your posts. Could the 2100 prove to be adequate for my needs is what I am tinkering about now.

Its warm on the bottom <...> it's good good airflow.

Warm on the bottom is not preferred, too warm is a. no-go. Of course any device gets a little warm just by having it powered on for a while. The 4100/6100 heatsink is considerably heavier than the 2100's so it should stay cooler and be more long lasting.

And I am looking for some leeway to keep things cool, guaranteed. But the 4100/6100 cost around double and consume about triple the energy.

This is it running with my son playing roblox right now on the xbox one.

I also use 7 VLANs and I have about 40 static DHCP v4 mappings.

It is impressive to see this processor run with some good code like pfSense.

So yes. I'm sold on pfSense, that is for sure.

All input considering, the 2100 Max appears to be adequate, the 4100 Max will do the job quite easily and the 6100 Max would be more than I ever need. So the 4100 is probably the obvious choice...

Gertjan · Aug 17, 2023, 1:08 PM

@Cabledude said in 4100/6100 Base or Max:

So the 4100 is probably the obvious choice...

And get the MAX
I've one and you win some very nice feature : No more 'disk space' issues.
You can go wild with the logs.
Install snort and forget about for many days, or weeks, before your system comes down crashing on you because the 'drive is full' (with the standard 4100 this will be a matter of hours ... and you have the 'log management doesn't work' issue)
You can create a lot of ZFS "System > Boot Environments", so, before adding a package, upgrade pfSense, make a snap shot. If there is just one issue : two clicks and your back using a working router.

Also : I hope I'm telling your BS now : the 4100 is an Intel device. The 2100 an arm. So, yes, it runs hotter. So, yes, the arm - 2100 will be more greener ;)
If I have the choice, I would go for an Intel device.

I never looked in my 4100 (a bit hard to open) but I presume the bigger MAX version NVME drive is easier to replace.
And lasts longer ...as it is a bigger drive so more choice to overwrite cells before failure, because more memory.

Cabledude · Aug 28, 2023, 8:05 PM

Hey @JonathanLee I was debating 4100/6100 but after reading your posts the 2100 Max is in my shopping bag waiting for me to pull the trigger.
A huge pro for me is low energy cons. (5W)
But I have two big doubts still, hope you could think along.

@JonathanLee said in 4100/6100 Base or Max:

I have the 2100-MAX with the expanded SSD works like a champ. I abuse it I have Squid, [...] It's a tank, I have been testing it for some time now. It's impressive. I have an external AP running with 12 or so devices

I have around 40, not all at the same time

I can recommend it only if you use the cache and want the proxy to cache running.

Could you please do some noob translate and elaborate? I don't understand any of this. But I don't use proxies.

I wish it had 8GB ram is all.

What would you do if you'd have 8GB?

So my concerns are:
#1. SG-2100 CPU is identical to the one in my SG-1100, which I find not very powerful. GUI dash can take 6-9 secs to load.

Its warm on the bottom

Not a technician myself but I would say a more powerful CPU that keeps cool (4100) should live longer than a weak CPU that gets warm.

@Cabledude It is great the only time I see it MAX out on it is on system start up, reboots.

#2. I use pfBlocker which can whip up the CPU to 40-50% peaks. Do you use pfBlocker? I didn't see it in the list. You do use Snort which may or may not be similarly CPU hungry.

Snort rebuilds take up some CPU and ram when that occurs too.

So bottom line is I don't use very large pfBlocker lists, I probably won't use snort/suricata or ntopng. My main goal is to have a stable system which at the moment I don't have, probably mainly due to insufficient RAM.
I also went back to UFS (not ZFS) which improved stability a lot. ZFS was also too RAM hungry.

@Gertjan said in 4100/6100 Base or Max:

If 'logging' is on your list, then you should already have some network syslogger available

Not really. I do care for the basic pfBlocker logs, that is what I meant when I said SSD not eMMC.

Thanks,

Cabledude · Aug 28, 2023, 8:05 PM

@Gertjan said in 4100/6100 Base or Max:

Btw : I use a 4100 MAX over a close to 1 Gbit/sec symmetric connection, about 30 local devices and about 10 or so active captive portal users (hotel here) and the 4100 handles that just fine. See here for some pfSense stats.

That stats page is awesome. Is this Zabbix? Care to share how you built this?

Thanks,

SteveITS · Aug 28, 2023, 8:12 PM

@Cabledude As alluded to above if you get the 2100 you will likely need to replace it if you upgrade to 1G fiber. From various posts here it should top out around 700-800 Mbps without Suricata/Snort.

pfBlocker shouldn't normally use CPU outside of processing lists and feeds. It basically just puts info in to firewall aliases or DNSBL lists.

re: SSD/eMMC see this list.

@Cabledude said in 4100/6100 Base or Max:

ZFS was also too RAM hungry

That's cache.
https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html#zfs-memory-tuning

Cabledude · Aug 28, 2023, 9:00 PM

Thank you @SteveITS for your swift response.

@SteveITS said in 4100/6100 Base or Max:

@Cabledude As alluded to above if you get the 2100 you will likely need to replace it if you upgrade to 1G fiber.

Yes thank you for pointing that out and I am well aware of that. We have 200/40 cable atm and gigabit fiber was more future proofing than a current requirement. For what we do 500/500 would be way more than adequate.

From various posts here it should top out around 700-800 Mbps without Suricata/Snort.

That would be quite satisfying.

pfBlocker shouldn't normally use CPU outside of processing lists and feeds. It basically just puts info in to firewall aliases or DNSBL lists.

I appreciate that, but my monitoring suggests otherwise (see above post):

login-to-view

Please note that the increased CPU starting around midnight is when I activated pfBlocker. It's a fairly basic config

However, given you are much more experienced than myself, you are probably right and the graph points to some other process. I haven't checked the process names.

re: SSD/eMMC see this list.

Thanks, I remember seeing that page before. You may have mentioned this now to make me find out that pfBlocker doesn't require SSD, am I right? Based on the above posts I am sold on SSD though, if for nothing else I get peace of mind.

@Cabledude said in 4100/6100 Base or Max:
ZFS was also too RAM hungry

That's cache.
https://docs.netgate.com/pfsense/en/latest/hardware/tune-zfs.html#zfs-memory-tuning

I studied that page. I had some erratic behaviour before on my SG-1100 when it was on ZFS. I don't know what triggered that, but someone mentioned ZFS uses more RAM so I reconfigured using UFS. The unit has been much much more stable since.
However, at a complete flash and config restore, more things get reset so it may have had nothing to do with ZFS after all. Fact is I ran ZFS and pfBlocker + some other stuff so 1GB RAM is still on the small side.

The SG-2100 is basically an SG-1100 with more RAM and more switch ports (which I absolutely don't need) so the premium in terms of unit cost seems high.
However 4GB up from 1GB just might solve all the issues I had with my SG-1100.

dennypage · Aug 28, 2023, 9:04 PM

I would generally recommend getting the Max for safety.

That said, if you have a UPS you can mitigate the eMMC risk by using RAM disks.

Having lost an eMMC on a 5100, I became concerned about the eMMC on my (non Max) 6100, and recently made the switch to RAM disks. The two graphs below illustrate the dramatic effect of the change:

login-to-view
login-to-view

In case you are going to ask... the dip shown for July 26-27 was checking the impact of disabling default rule and pfBlockerNG logging.

The little blips you see every 24 hours are /var/log and /var/db/rrd being copied from memory to the eMMC.

SteveITS · Aug 28, 2023, 9:23 PM

@Cabledude said in 4100/6100 Base or Max:

pfBlocker doesn't require SSD, am I right? Based on the above posts I am sold on SSD though, if for nothing else I get peace of mind.

It does not require an SSD, no. Everything that "requires" an SSD depends on logging. If you disable logging of the default block rules, and don't use or log DNSBL, then there's not much disk writing. Others log everything or run Suricata on web servers, etc., and log a lot of stuff.

The 2100 also has a separate WAN port instead of using the switch, and does not use VLANs, which is why it can get over 500 Mbps (500 in + 500 out = 1000). So it adds +3 GB and a NIC.

mcury · Aug 28, 2023, 9:31 PM

login-to-view

421 GB ÷ 2280 hours
0,184649123 GB per hour x 24
= 4,431578947 GB per day x 30
= 132,94736841 GB per month x 12
= 1595,36842092 GB per year or 1.6 TB per year

SN520 = 100 TBW endurance
100 / 1,6 = 62.5 years

[23.05.1-RELEASE][root@pfsense.home.arpa]/root: cat /boot/loader.conf.local 
hint.mmcsd.0.disabled="1"

[23.05.1-RELEASE][root@pfsense.home.arpa]/root: iostat -x
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
nvd0           0       5      0.4     50.0     0     0     0     0    0   0

In my opinion, the best option is to use a remote syslog server, disable all pfsense logs to the disk and use a SSD.
Also, disable eMMC using loader.conf.local.

There are some drawbacks in using RAM disk, but now I can't remember exactly what the problems were, I think it was something with geoIP database download with pfblockerNG and lists not being written to the disk when RAM disk was enabled.

Once you disable RAM disk, you would have to download the geoIP database and lists again.

Cabledude · Aug 28, 2023, 9:32 PM

@SteveITS

Here is my current SG-1100 System activity:

last pid: 48936; load averages: 0.72, 0.52, 0.45 up 3+16:25:58 23:19:46
186 threads: 3 running, 165 sleeping, 18 waiting
CPU: 10.7% user, 0.7% nice, 2.8% system, 2.3% interrupt, 83.5% idle
Mem: 54M Active, 523M Inact, 12K Laundry, 222M Wired, 94M Buf, 154M Free

This is UFS, pfBlocker DNSBL AdsBasic, IP PRI1, no GeoIP, avahi, 8 vlans, night time so the rest of the family is asleep. I will re-enable geoIP top spammers all countries for a couple of hours and then disable pfblocker and see what that does to the graph

SteveITS · Aug 28, 2023, 10:00 PM

@mcury said in 4100/6100 Base or Max:

drawbacks in using RAM disk

Not sure about disabling it, though that requires a reboot like enabling it does. I am pretty sure pfB and Suricata/Snort have been updated to save the files out of the RAM disk at reboot? Not 100% sure though. A RAM disk can be an issue with large lists, e.g. the UT1 list takes over 1 GB to extract. And of course losing logs (pfSense has options to copy logs and other things to disk every "n" hours).

Overall have used a RAM disk on pretty almost all of our clients' and our routers for a couple years now. Probably would not on an 1100 with 1GB.

mcury · Aug 28, 2023, 10:24 PM

@SteveITS said in 4100/6100 Base or Max:

Overall have used a RAM disk on pretty almost all of our clients' and our routers for a couple years now. Probably would not on an 1100 with 1GB.

I used for a long time in a SG-3100, first thing you notice is a longer time to boot, pfblockerNG takes a long time finish.
Some logs about /var/db/ missing during boot once you disable RAM disk, and other weird things along of the already mentioned.

But, yes, RAM disk is a very good thing to do in case you don't have a SSD and don't want to rely on the eMMC.

dennypage · Aug 28, 2023, 11:16 PM

@mcury I'm not sure what all that data was intended to convey, but what the heck...

Your IO rates and other people's IO rates are not necessarily comparable. Too much depends upon the packages installed, the amount of firewall logging, and the general activity in the system.

FWIW, my IO rate prior to RAM disks was 14,674 GB per year. Now it is reduced to just 69 GB per year. [Btw, I believe you have a slight math error. With 421GB over 2280 hours, your projected usage should be 1614.20 GB per year.]

You are correct about an issue with GeoIP as maintained by pfBlockerNG. When you reboot, the GeoIP list will not be present until pfBlocker runs its cron entry. It's a minor inconvenience that is fixable with a boot shell script. Even if it weren't fixable, it would totally be worth it to me for the 99.5% reduction in write to the eMMC.

mcury · Aug 28, 2023, 11:26 PM

@dennypage said in 4100/6100 Base or Max:

I'm not sure what all that data was intended to convey, but what the heck...

Those are to help other users to be able to do their own math.
I had a SG-3100 and at that time I didn't have a SSD.
What helped me to understand and how to do the math was a post here: https://forum.netgate.com/topic/170128/emmc-write-endurance?_=1663100394507
So, it is just to help users that just like me, needed help to better understand how this works. You can just ignore it.

@dennypage said in 4100/6100 Base or Max:

With 421GB over 2280 hours, your projected usage should be 1614.20 GB per year.]

Yes, it helps a lot but as I see it, it is not the best option..

@dennypage said in 4100/6100 Base or Max:

You are correct about an issue with GeoIP as maintained by pfBlockerNG. When you reboot, the GeoIP list will not be present until pfBlocker runs its cron entry. It's a minor inconvenience that is fixable with a boot shell script. Even if it weren't fixable, it would totally be worth it to me for the 99.5% reduction in write to the eMMC.

There are other problems, not only that one..

Everything resumes to, it is my system, I'm the administrator and I'll choose what I want.
For me, SSD is better option.