Swapfile on SG-1100 running 21.05?
-
On our small appliances, we have always had issues updating pfblockerng; it taking a long time and causing issues with service availability until complete. We therefore only do them every month or so, and only manually. I just completed an update of pfblockerng on one of our SG-1100 devices and it took 2+ hours to finish, during which time unbound was offline (as was the web interface). This system is still running 2.4.5-RELEASE-p1. I found another post indicating that increasing the swapfile size would help these updates, which I've now done on this system and will test as soon as I can afford another potential 2-hour downtime for DNS.
However, we have 2 additional remote systems (both SG-1100) that have already been upgraded to 21.05 but have not had the recent pfblockerng update done. I only just noticed that neither of these appear to have a swapfile configured. Is there some reason I would not want to add one?
Will adding a 2G swapfile [help|hurt|do nothing] ???
-
On an standard x86 pfSense install, where swap is added by default, if you start using swap regularly you usually see performance drastically reduced.
Basically if you start using swap you probably have something misconfigured.The SG-1100 has only 1G of RAM so it's easier to exhaust the available memory. 2hrs to update pfBlocker is like 1h50m too long! Unless you have a very slow WAN connection I would suggest you simply have too many lists loaded. I would expect it to throw some errors when it does exhaust it though.
Adding swap may help but swap itself is very slow especially if it's on eMMC. There's no easy way to add it on the SG-1100 either.
Perhaps more importantly though is that it would dramatically increase drive writes if it's regularly in use and the write-life of eMMC is limited. So I would not recommend it.
The better way forward here is to tune pfBlocker, and anything else you have loaded, to use less RAM.
Steve
-
@stephenw10 said in Swapfile on SG-1100 running 21.05?:
There's no easy way to add it on the SG-1100 either
I'm not sure I follow. "There's no easy way to add it on the SG-1100 either"...the SG-1100 comes configured with 1G swap using swapfile out of the box. Our last one running 2.4.5 still has this located at "/usr/pagefile.bin". Our two SG-1100s that have been upgraded to 21.05, do not. This is not a change I made on any of the devices. What I did do today is create a larger pagefile.bin on the one that had it, increasing from 1G to 2G, based on a post here from 10/2019.
FWIW, it is also on a 250Mbit symmetric connection. The block list itself is very highly tuned feed and is a 20MB text file. Once loaded, it runs very well and the FW usually hums along at around 30% RAM utilization and 0% SWAP...
I'm surprised and disappointed that updating it causes these issues, quite frankly. I was hoping to hear that this is one of the (many) issues 21.05 fixes...
-
That is not standard on the SG-1100. It's not how it would be done on other systems, with a swap partition.
There are instructions for creating that on the forum though. Perhaps that was done on that unit in the past?Do you actually see it exhausting the memory during the updates?
What sort of table count numbers do you see when it's finished updating?
For example:====================[ Last Updated List Summary ]============== Jul 21 21:24 Spamhaus_drop Jul 23 00:00 Google Jul 23 00:00 Facebook Jul 23 00:00 Netflix IPv4 alias tables IP count ----------------------------- 8659 IPv6 alias tables IP count ----------------------------- 0 Alias table IP Counts ----------------------------- 8659 total 7628 /var/db/aliastables/pfB_ASN_List.txt 1031 /var/db/aliastables/pfB_Spamhaus.txt pfSense Table Stats ------------------- table-entries hard limit 1000000 Table Usage Count 11595Steve
-
I'm pretty confident I did not add a pagefile to the system, but if you're certain I'll concede the point. I intend to have this FW upgraded to 21.05 long before the next pfblocker update. I am much more concerned about what might happen on this system once it has no swapfile, assuming that 21.05 removes the existing one as part of the update...Here's the output from this morning:
[ Force Reload Task - All ] UPDATE PROCESS START [ v3.0.0_3 ] [ 07/28/21 06:20:07 ] ===[ DNSBL Process ]================================================ Loading DNSBL Statistics... completed Loading DNSBL SafeSearch... disabled Loading DNSBL Whitelist... completed [ DBL_OISD_NL ] Downloading update .. 200 OK. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 916586 916586 0 0 0 916586 ---------------------------------------------------------------------- ------------------------------------------------------------------------ Assembling DNSBL database...... completed [ 07/28/21 06:22:11 ] TLD: Blocking full TLD/Sub-Domain(s)... |jwpcdn.com|jwplatform.com|jwplayer.com|jwpsrv.com|jwpltx.com|jwpsrv-vh.akamaihd.net|flowplayer.com|doubleclick.net| completed TLD analysis.xxxxxxxx completed [ 07/28/21 06:24:38 ] ** TLD Domain count exceeded. [ 100000 ] All subsequent Domains listed as-is ** TLD finalize... ---------------------------------------- Original Matches Removed Final ---------------------------------------- 916586 12393 68968 847618 ----------------------------------------- TLD finalize... completed [ 07/28/21 06:25:57 ] Saving DNSBL statistics... completed [ 07/28/21 06:26:08 ] Resolver Live Sync analysis... completed [ 07/28/21 06:26:46 ] Resolver Live Sync finalizing: Remove local-zone(s): no changes Remove local-data(s): no changes Add local-zone(s): added 861071 zones Add local-data(s): Resolver Live Sync ... FAILED! Additional mounts: No changes required. Starting Unbound Resolver... Not completed. [ 07/28/21 07:59:40 ] error: SSL handshake failed DNSBL update [ 847618 | PASSED ]... completed [ 07/28/21 07:59:46 ] ------------------------------------------------------------------------ ===[ GeoIP Process ]============================================ ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload No Changes to Aliases, Skipping pfctl Update ===[ Kill States ]================================================== Firewall state(s) validation for [ 25 ] IPv4 address(es)... No matching states found ====================================================================== ===[ FINAL Processing ]===================================== [ Original IP count ] [ 0 ] ===[ DNSBL Domain/IP Counts ] =================================== 847618 total 847610 /var/db/pfblockerng/dnsbl/DBL_OISD_NL.txt 8 /var/db/pfblockerng/dnsbl/DNSBL_TLD.txt ====================[ DNSBL Last Updated List Summary ]============== Jul 27 20:40 DBL_OISD_NL Alias table IP Counts ----------------------------- pfSense Table Stats ------------------- table-entries hard limit 1000000 Table Usage Count 1413 UPDATE PROCESS ENDED [ 07/28/21 08:00:06 ] -
@stephenw10 Also...TBH, I did not actually verify it was a RAM issue. It was purely knee-jerk assessment and the content of the earlier post (https://forum.netgate.com/topic/157735/running-out-of-memory-on-sg-1100-on-pfblockerng-updates)
-
Ah, well at least my messaging is consistent across threads.
From that log it looks like you are using almost entirely DNS-BL rules.
But it looks like you have TLD enabled on a list of 900K domains and that is chocking it. That requires significant processing power, so much that pfBlocker doesn't allow anything over 100K domains on any system no matter how powerful. It's actually only running on the first 100K list items from your list(s):** TLD Domain count exceeded. [ 100000 ] All subsequent Domains listed as-is **However all the time there seems to be after that when it's creating the Unbound conf file from the lists:
Saving DNSBL statistics... completed [ 07/28/21 06:26:08 ] Resolver Live Sync analysis... completed [ 07/28/21 06:26:46 ] Resolver Live Sync finalizing: Remove local-zone(s): no changes Remove local-data(s): no changes Add local-zone(s): added 861071 zones Add local-data(s): Resolver Live Sync ... FAILED! Additional mounts: No changes required. Starting Unbound Resolver... Not completed. [ 07/28/21 07:59:40 ] error: SSL handshake failedThat list is probably just too large for the SG-1100.
For reference the whole process takes ~15s with a list containing ~19k items on an SG-3100 here:
UPDATE PROCESS START [ 07/29/21 13:51:27 ] ===[ DNSBL Process ]================================================ [ Easylist_Default ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # Alexa Final ---------------------------------------------------------------------- 16312 15484 0 0 0 15484 ---------------------------------------------------------------------- [ Easylist_Privacy ] Reload [ 07/29/21 13:51:28 ] . completed .. Whitelist: amazonaws.com| ---------------------------------------------------------------------- Orig. Unique # Dups # White # Alexa Final ---------------------------------------------------------------------- 4402 4387 61 1 0 4325 ---------------------------------------------------------------------- [ Custom_List_custom ] Reload. ---------------------------------------------------------------------- Orig. Unique # Dups # White # Alexa Final ---------------------------------------------------------------------- 7 7 0 0 0 7 ---------------------------------------------------------------------- ------------------------------------------ Assembling database... completed Executing TLD TLD analysis. completed Finalizing TLD... completed ---------------------------------------- Original Matches Removed Final ---------------------------------------- 19816 19714 18 19798 ----------------------------------------- Validating database... completed [ 07/29/21 13:51:34 ] Reloading Unbound.... completed DNSBL update [ 19798 | PASSED ]... completed [ 07/29/21 13:51:37 ] ------------------------------------------ ===[ Continent Process ]============================================ ===[ IPv4 Process ]================================================= [ Spamhaus_drop ] Reload . completed .. [ Google ] Reload . completed .. [ Facebook ] Reload . completed .. [ Netflix ] Reload . completed .. ===[ IPv6 Process ]================================================= ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload Updating: pfB_Spamhaus no changes. Updating: pfB_ASN_List no changes. Archiving Aliastable folder Archiving selected pfBlockerNG files. ===[ FINAL Processing ]===================================== [ Original IP count ] [ 8663 ] ===[ Native List IP Counts ] =================================== 8659 total 7383 /var/db/pfblockerng/native/Google.txt 1031 /var/db/pfblockerng/native/Spamhaus_drop.txt 192 /var/db/pfblockerng/native/Facebook.txt 53 /var/db/pfblockerng/native/Netflix.txt ===[ DNSBL Domain/IP Counts ] =================================== 19798 total 15470 /var/db/pfblockerng/dnsbl/Easylist_Default.txt 4322 /var/db/pfblockerng/dnsbl/Easylist_Privacy.txt 6 /var/db/pfblockerng/dnsbl/Custom_List_custom.txt ====================[ Last Updated List Summary ]============== Jul 21 21:24 Spamhaus_drop Jul 23 00:00 Google Jul 23 00:00 Facebook Jul 23 00:00 Netflix IPv4 alias tables IP count ----------------------------- 8659 IPv6 alias tables IP count ----------------------------- 0 Alias table IP Counts ----------------------------- 8659 total 7628 /var/db/aliastables/pfB_ASN_List.txt 1031 /var/db/aliastables/pfB_Spamhaus.txt pfSense Table Stats ------------------- table-entries hard limit 1000000 Table Usage Count 11598 UPDATE PROCESS ENDED [ 07/29/21 13:51:43 ]What do the list counts on the dashboard show after that has run?
Is it actually blocking everything?Anyway I can only recommend you remove that SWAP file. Running that and actually using it is going to be adding significant write cycles to the eMMC. There is no way to replace that if it fails.
Steve
-
@stephenw10 Thanks for the feedback. I'll nuke the swap when I do the upgrade then. More importantly, I will not add one to the existing 21.05 boxes and just make sure to do the pfblockerng update in the wee hours.
FWIW, the block list is pretty solid and works very well for our use case, hits/blocks ~= 50%. I've never had to retroactively bless anything, and only two sites ever have come back as having functional issues. Upon inspection, those were so laden with adware and trackers that I said "tough cookies" and said to visit them off-network if they really needed to.
Here's a screenie.
-
@f1d094 said in Swapfile on SG-1100 running 21.05?:
were so laden with adware and trackers that I said "tough cookies"
Ha, sounds fair.
I mean, yeah, it looks like it's definitely working for you.Steve
