Regex logs

smolka_J

Not certain if others have encountered this, it seems if once somewhere around 600 lines or more of REGEX is entered, Reports on the Unified tab for REGEX blocks stop appearing in logs altogether. I verified the REGEX blocking is in fact working as I was intending it but no longer showing viewable logs of it on either my 5100 or bare metal. Wondering if there's a log or memory limit somewhere I can tweak to bring regex logs back. I started to try to label lines individually but didn't seem to help populate anything to logs. Have 806 lines currently

smolka_J

pf regex 4-24-23.txt

Gertjan

@smoke_a_j

You probably broke the rules.

Here :

the parser that reads all these lines really want a "space # sometext" as a line terminator.
"sometext" will be used as the comment shown in the logs when the line applies.
The "sometext" have to be unique for every line.

Try for yourself : remove everything after the "REGEX683" : it should work now.
Also : there is no regex input validation : the slightest reg expression error will also produce unknown results.

smolka_J

@gertjan I haven't seen regex logs though since reaching over 600 lines, never had any lines commented with a space # description on any before that. Read up on similar claims hoping it might be that simple and then had added to every line I had at the time which was only 677 lines total now each with "space # descriptions" added but still nothing. So I kept adding a few more lines testing and gave up adding description comments after that point because they didn't affect logs not populating, only changes how they are displayed IF they were captured in logs. I can finish off the # descriptions once I'm out of the hospital for the day when I have time later but rather not remove any lines unless there's specific lines to de-bloat, some excess is just from testing. At 677 lines and prior, I was getting regex to do only HALF of what I was expecting it to for certain specifics with no logs for any. Once I added the last lines through to 806, logs are still non-existent but now regex is acting as expected otherwise finally: fdsakjganythinglytics.net and lyticsfsddhahg.com and any "lytics" variation either side of it inbetween are now filtered instead of just ONLY prefixed variations like abcanalytics.com

Gertjan

@smoke_a_j

What I have :

So, 15 lines.

Compare the GUI (image above) with this file : /var/unbound/pfb_unbound.ini :

Have a look at the documentation : /var/unbound/pfb_unbound.py - look for this text regexDB
There is no real limit of the number of 'regex' entries.

edit : wonder how tiktok wind up in there

smolka_J

@gertjan I'm not getting a functional limit as it seems, straight through to line 806 is working and shows in the .ini file. I just see zero regex block alerts logged hindering the case if there's something needing to be whitelisted that it catches as some regex randomly is expected to do. Regular standard DNSBL and IP block events from feed lists all propagate alert logs fine otherwise in the Reports tab. 400 lines or less I always seemed to get ALL regex block logs. Approaching 600 lines those regex alerts themselves dwindled in numbers while still actively blocking invisibly for 75% of regex blocks. Once I reached the number of lines I have currently, only regex block logs are completely not present while my regex functionally is blocking perfectly fine, just cannot visibly see or verify what exactly preemptively beyond running nslookup commands specifically to an already known "domain names/fake domain names" to test

Gertjan

@smoke_a_j

I copy pasted your list :

and, as the GUI is to slow and tedious, I :

tail -f /var/unbound/var/log/pfblockerng/dnsbl.log

The regex functionality skyrocketed :

DNSBL-1x1,Apr 26 09:27:03,ipv6.msftconnecttest.com,192.168.1.191,-|GET /connecttest.txt HTTP/1.1|Microsoft NCSI,Python_CNAME_A,DNSBL_Regex,v6ncsi.msedge.net,regex174,+
DNSBL-HTTPS,Apr 26 09:27:09,init.push.apple.com,192.168.1.191,Unknown,Python_HTTPS,DNSBL_Regex,init.push.apple.com,regex200,+
DNSBL-HTTPS,Apr 26 09:27:09,init.push.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,init.push.apple.com,regex200,+
DNSBL-HTTPS,Apr 26 09:27:12,updates.cdn-apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,updates.cdn-apple.com,regex_687,+
DNSBL-Full,Apr 26 09:27:15,x1.c.lencr.org,192.168.1.191,-|GET / HTTP/1.1|Microsoft-CryptoAPI/10.0,Python_CNAME_A,DNSBL_Regex,crl.root-x1.letsencrypt.org.edgekey.net,regex_787,+
DNSBL-HTTPS,Apr 26 09:27:20,mesu.apple.com,192.168.1.191,Unknown,Python_CNAME_HTTPS,DNSBL_Regex,mesu-cdn.apple.com.akadns.net,regex_774,+
DNSBL-HTTPS,Apr 26 09:27:23,updates.cdn-apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,updates.cdn-apple.com,regex_687,+
DNSBL-HTTPS,Apr 26 09:27:24,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,+
DNSBL-HTTPS,Apr 26 09:27:24,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:30,mesu.apple.com,192.168.2.52,Unknown,Python_CNAME_HTTPS,DNSBL_Regex,mesu-cdn.apple.com.akadns.net,regex_774,+
DNSBL-HTTPS,Apr 26 09:27:34,skydrive.wns.windows.com,192.168.1.191,Unknown,Python_CNAME_A,DNSBL_Regex,wns.notifytrafficmanager.net,regex_787,+
DNSBL-1x1,Apr 26 09:27:34,tile-service.weather.microsoft.com,192.168.1.191,-|GET /fr-FR/livetile/preinstall?region=FR&amp;appid=C98EA5B0842DBB9405BBF071E1DA76512D21FE36&amp;FORM=Threshold HTTP/1.1|Microsoft-WNS/10.0,Python_A,DNSBL_Regex,tile-service.weather.microsoft.com,regex617,+
DNSBL-HTTPS,Apr 26 09:27:35,licensing.mp.microsoft.com,192.168.1.191,Unknown,Python_CNAME_A,DNSBL_Regex,consumer-licensing-aks2aks.md.mp.microsoft.com.akadns.net,regex_774,+
DNSBL-HTTPS,Apr 26 09:27:36,query.ess.apple.com,192.168.1.191,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,+
DNSBL-HTTPS,Apr 26 09:27:36,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,+
DNSBL-HTTPS,Apr 26 09:27:36,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:36,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:43,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:43,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:43,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:43,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:52,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:52,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:52,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-
DNSBL-HTTPS,Apr 26 09:27:52,query.ess.apple.com,192.168.2.52,Unknown,Python_HTTPS,DNSBL_Regex,query.ess.apple.com,regex201,-

This regex counter went up fast :

@smoke_a_j said in Regex logs:

only regex block logs are completely not present

and then go half way the page.

You'll find :

smolka_J

@gertjan Thank you thank you, makes those ones much easier to locate, I was always using the Unified tab for most avoiding too much scrolling before it refreshes the page. Many thank yous. Running nslookup commands from my Kubuntu Server to make sure blocks were hitting my 10.10.10.1 redirect page added to my confusion a little I think, domain lookups this way aren't showing in Unified or DNSBL Python logs but entering them in a browser does populate the viable log entries so I think all is good. If you do choose to keep running this list, you may want the last 2 game lines removed unless you have kids like mine that actively want to like games over homework every day. Thanks!

Gertjan

@smoke_a_j said in Regex logs:

have kids like

Noop.
That is, yes, I've one @home, 13 - but not mine, so less strict rules apply
Here, where I live, they already know very young that 'they have rights' (to communicate and so on) so I'm not going to firewall her out of the world.
I added another right "you have the right to dig your own hole, and then fall into it" as nearly nothing is learned from the successes and achievements, only 'errors' are useful in the educational process.

I'm mean : nothings stops here from getting a SIM card with a boatload of 4G/5G data, and then I Iose control for sure.

smolka_J

@gertjan Yeah my 9 yr old keeps digging deeper n deeper into trying to learn coding to mod his Minecraft and other games, 2 years of COVID homeschooling pushed me to redirect all on his device logins and my moms she keeps handing him when she's over, directed all including google.com straight to his school's landing page login for a while trying to get him to learn to balance his priorities, once I get the call his homeworks done when Im not home, I have a NAT rule kill switch I vpn into to unlock whats whitelisted age appropriate approved or another to kill all TV access too if thats whats needed at the moment to get his work started. I'll give him weeks at a time full access, but every time he's late to school or late on his homework repeatedly, those kill switches still prove useful every other week sometimes every other day lol. Teenage years, talking with his older half-brother, I don't think I wanna know what they try looking up. Trying to set it all up on one pfSense box was a headache trying to factor in Squidguard left too many holes open and breaking other facets but setting up a second pf box for the parental controls side really smoothed out its effectiveness and control moderation dividing that task. Once he has a better understanding of how much Spyware/trojans are embedded into certain game sites and downloads and once I can get my ma to stop opening each and every phishing email she comes across I might open boundaries some, anything with a download link or "click here" link they like to click daily, if credit cards or passwords didn't exist that'd be a different story

smolka_J

I went through and re-organized my list, strengthened the character sets on a few hundred lines to combine and eliminate others and added a few, de-duplicated what was left, so far no errors on either box. All was gathered from through the communities, slightly tweaked, and here for any whom want to use, test, or improve upon. pfSene pfBlockerng regex list 5-4-23.txt