pfBlockerNG_devel commit reverse

lohphat

I did I just clipped the result. My config for ASN lookup is attached. Try AS15169 and it returns a bunch of IPv4 and IPv6 networks. The pfBlocker-NG IPv4 lookups work, the IPv6 lookups return no networks. The API returns a lot of IPv6 networks.

BBcan177

@lohphat try to add those to that IPv6 alias but where you would enter URL feeds. Choose the ASN format. I will check the custom list tomorrow.

lohphat

@BBcan177

Nope. Adding them above as individual ASNs didn't work either:

[ AS8075_v6 ]			 Downloading update .
  Collecting ASN: AS8075... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ AS13335_v6 ]			 Downloading update .
  Collecting ASN: AS13335... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ AS15169_v6 ]			 Downloading update [ 10/2/24 23:38:10 ] .
  Collecting ASN: AS15169... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ AS16509_v6 ]			 Downloading update .
  Collecting ASN: AS16509... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ AS19551_v6 ]			 Downloading update [ 10/2/24 23:38:11 ] .
  Collecting ASN: AS19551... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ AS20940_v6 ]			 Downloading update .
  Collecting ASN: AS20940... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ AS15133_v6 ]			 Downloading update [ 10/2/24 23:38:12 ] .
  Collecting ASN: AS15133... Failed to collect ASN... Creating empty file
. completed ..
  Empty file, Adding '::127.1.7.7' to avoid download failure.

[ QUIC_ASN_List_custom_v6 ]	 Downloading update
  Collecting host IP: mask.icloud.com... completed
  Collecting host IP: mask-h2.icloud.com... completed
  Collecting ASN: AS8075... Failed to collect ASN  Collecting ASN: AS13335... Failed to collect ASN  Collecting ASN: AS15169... Failed to collect ASN  Collecting ASN: AS16509... Failed to collect ASN  Collecting ASN: AS19551... Failed to collect ASN  Collecting ASN: AS20940... Failed to collect ASN  Collecting ASN: AS15133... Failed to collect ASN... Restoring previous data
 . completed ..
[ pfB_QUIC_ASN_List_v6 QUIC_ASN_List_custom_v6 ] Custom List: No IPs found! Ensure only IP based Feeds are used! ]

FCS001FCS

@BBcan177

Updated today to pfBlockerNG-devel 3.2.0_18 and ASNs function is working fine again with the IPInfo.io service.

I set the ASN Cache to 1 hour and entered my IPInfp.io token in the IP setup tab. The old entries I had for the ASN lookups did not work at first, so I deleted them and then started from the beginning and re-entered the ASN numbers and then it all worked.

Many thanks to @BBcan177 for all his hard work on getting this up and running again.

leres

Where is the repo that has PORTREVISION 18 of pfSense-pkg-pfBlockerNG-devel? I have been using the devel branch of https://github.com/pfsense/FreeBSD-ports.git but although it contains commits from today, pfSense-pkg-pfBlockerNG-devel only shows PORTREVISION 15 (Sept 23).

jrey

@FCS001FCS said in pfBlockerNG_devel commit reverse:

The old entries I had for the ASN lookups did not work at first, so I deleted them and then started from the beginning and re-entered the ASN numbers and then it all worked.

you really shouldn't have to delete all the ASN numbers and reenter them -- although I could see a possibility where the new ASN data has not been downloaded,(it is one file now, not individual files) and a cron job starts to update before that download is available and therefore they fail - that case should be in the pfblockerng.log I would think.
through all the testing over the past several weeks, I've not had to reenter any ASN.

you might also want to check this as well in case anything there that may impact you

https://forum.netgate.com/topic/190361/pfblockerng-devel-3-2-0_18?_=1727956822873

FCS001FCS

@jrey said in pfBlockerNG_devel commit reverse:

you really shouldn't have to delete all the ASN numbers and reenter them

Yes, I assumed so also but when I updated and forced a "Reload" after the update the ASNs did not populate the related .txt file. So, I just deleted the old ASN entries and then rebuilt them and did a "Reload" again and all worked. I did not want to spend too much time trying to determine the exact issue, so that was quickest for me.

I saw the thread you referenced earlier today, but I do not remember if I had any special characters in the entries, but it could have been the issue. The new entries do not have any special characters.

jrey

@FCS001FCS said in pfBlockerNG_devel commit reverse:

I just deleted the old ASN entries and then rebuilt them and did a "Reload" again and all worked

Yes,
It is important for people to realize that with the old system, each ASN was downloaded as an individual file
with the new system all the ASN data is in one file, and that one file only downloads once per day after initial load. Now when the routine updates asks for an ASN - the data is pulled from the one local master file, not the internet.
So all I am suggesting is that when you ran the first reload the data may not have been available in the master file yet. Thus the extraction part fails and it appears you get nothing.

@FCS001FCS said in pfBlockerNG_devel commit reverse:

but I do not remember if I had any special characters in the entries

On this point - the old data source didn't have any special characters, so the only way you run into the problem is if you had an ASN that under the old system would not have had any special characters but now under the new one does and ran an update (those are harder, but not impossible to find) and / or you now tried to add new one with these characters. Then those ASN's would be a problem.

The issue here is that it couldn't happen on the old data, and with the new data the underlying config save functions do not properly handle international strings and therefore those ASNs do not get saved.

It has only impacted my testing - there is no way I can use this to production (although the method provided of "Add the ASN to the custom list", yes works, it messes up my analytics over in Graylog because anything you list in custom gets attributed to custom not the ASN - so you can not longer track which specific ASN caused the event. Others who don't care about this can certainly use the custom list and get by the problem. for me it is "No Go" on production.

BBcan177

@lohphat

See this to fix ASN for IPv6

https://www.reddit.com/r/pfBlockerNG/s/Kv6252BTcK

lohphat

@BBcan177 said in pfBlockerNG_devel commit reverse:

kerNG/s/Kv6252BTcK

That did it!

I'm NOT crazy.

Today.

Popolou

Has everything now settled down so as to proceed with an update to _18 with a functional ASN lookup?

incith

Getting errors trying to remove countries from top spammers list (using _18)

Fatal error: Uncaught TypeError: array_path_enabled(): Argument #1 ($arr) must be of type array, int given, called in /etc/inc/config.lib.inc on line 1250 and defined in /etc/inc/util.inc:3662 Stack trace: #0 /etc/inc/config.lib.inc(1250): array_path_enabled(-1, 'notifications/s...', 'disable') #1 /etc/inc/notices.inc(379): config_path_enabled('notifications/s...', 'disable') #2 /etc/inc/notices.inc(662): notify_via_smtp('pfSense is rest...') #3 /etc/inc/notices.inc(151): notify_all_remote('pfSense is rest...') #4 /etc/inc/config.lib.inc(239): file_notice('config.xml', 'pfSense is rest...', 'pfSenseConfigur...', '') #5 /etc/inc/config.lib.inc(695): restore_backup('/cf/conf/backup...') #6 /usr/local/www/pfblockerng/pfblockerng_Top_Spammers.php(291): write_config('[pfBlockerNG] s...') #7 {main} thrown in /etc/inc/util.inc on line 3662 PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 3662, Message: Uncaught TypeError: array_path_enabled(): Argument #1 ($arr) must be of type array, int given, called in /etc/inc/config.lib.inc on line 1250 and defined in /etc/inc/util.inc:3662 Stack trace: #0 /etc/inc/config.lib.inc(1250): array_path_enabled(-1, 'notifications/s...', 'disable') #1 /etc/inc/notices.inc(379): config_path_enabled('notifications/s...', 'disable') #2 /etc/inc/notices.inc(662): notify_via_smtp('pfSense is rest...') #3 /etc/inc/notices.inc(151): notify_all_remote('pfSense is rest...') #4 /etc/inc/config.lib.inc(239): file_notice('config.xml', 'pfSense is rest...', 'pfSenseConfigur...', '') #5 /etc/inc/config.lib.inc(695): restore_backup('/cf/conf/backup...') #6 /usr/local/www/pfblockerng/pfblockerng_Top_Spammers.php(291): write_config('[pfBlockerNG] s...') #7 {main} thrown Fatal error: Uncaught TypeError: array_path_enabled(): Argument #1 ($arr) must be of type array, int given, called in /etc/inc/config.lib.inc on line 1250 and defined in /etc/inc/util.inc:3662 Stack trace: #0 /etc/inc/config.lib.inc(1250): array_path_enabled(-1, 'notifications/s...', 'disable') #1 /etc/inc/notices.inc(379): config_path_enabled('notifications/s...', 'disable') #2 /etc/inc/notices.inc(662): notify_via_smtp('PHP ERROR: Type...') #3 /etc/inc/notices.inc(151): notify_all_remote('PHP ERROR: Type...') #4 /etc/inc/config.lib.inc(1154): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors') #5 [internal function]: pfSense_clear_globals() #6 {main} thrown in /etc/inc/util.inc on line 3662

jrey

@incith

and what version of pfSense are you running _18 on ?

incith

@jrey said in pfBlockerNG_devel commit reverse:

@incith

and what version of pfSense are you running _18 on ?

Sorry about that!

Version	2.7.2-RELEASE (amd64)
built on Mon Mar 4 14:53:00 EST 2024
FreeBSD 14.0-CURRENT

incith

Just noticed there's better logs --

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec  6 20:45:47 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/F

Crash report details:

PHP Errors:
[07-Oct-2024 10:03:46 EST5EDT] PHP Fatal error:  Uncaught TypeError: array_path_enabled(): Argument #1 ($arr) must be of type array, int given, called in /etc/inc/config.lib.inc on line 1250 and defined in /etc/inc/util.inc:3662
Stack trace:
#0 /etc/inc/config.lib.inc(1250): array_path_enabled(-1, 'notifications/s...', 'disable')
#1 /etc/inc/notices.inc(379): config_path_enabled('notifications/s...', 'disable')
#2 /etc/inc/notices.inc(662): notify_via_smtp('pfSense is rest...')
#3 /etc/inc/notices.inc(151): notify_all_remote('pfSense is rest...')
#4 /etc/inc/config.lib.inc(239): file_notice('config.xml', 'pfSense is rest...', 'pfSenseConfigur...', '')
#5 /etc/inc/config.lib.inc(695): restore_backup('/cf/conf/backup...')
#6 /usr/local/www/pfblockerng/pfblockerng_Top_Spammers.php(291): write_config('[pfBlockerNG] s...')
#7 {main}
  thrown in /etc/inc/util.inc on line 3662
[07-Oct-2024 10:03:46 EST5EDT] PHP Fatal error:  Uncaught TypeError: array_path_enabled(): Argument #1 ($arr) must be of type array, int given, called in /etc/inc/config.lib.inc on line 1250 and defined in /etc/inc/util.inc:3662
Stack trace:
#0 /etc/inc/config.lib.inc(1250): array_path_enabled(-1, 'notifications/s...', 'disable')
#1 /etc/inc/notices.inc(379): config_path_enabled('notifications/s...', 'disable')
#2 /etc/inc/notices.inc(662): notify_via_smtp('PHP ERROR: Type...')
#3 /etc/inc/notices.inc(151): notify_all_remote('PHP ERROR: Type...')
#4 /etc/inc/config.lib.inc(1154): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#5 [internal function]: pfSense_clear_globals()
#6 {main}
  thrown in /etc/inc/util.inc on line 3662



No FreeBSD crash data found.

pfSenseConfigurator
pfSense is restoring the configuration /cf/conf/backup/config-1728309802.xml @ 2024-10-07 10:03:46
PHP errors
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 3662, Message: Uncaught TypeError: array_path_enabled(): Argument #1 ($arr) must be of type array, int given, called in /etc/inc/config.lib.inc on line 1250 and defined in /etc/inc/util.inc:3662
Stack trace:
#0 /etc/inc/config.lib.inc(1250): array_path_enabled(-1, 'notifications/s...', 'disable')
#1 /etc/inc/notices.inc(379): config_path_enabled('notifications/s...', 'disable')
#2 /etc/inc/notices.inc(662): notify_via_smtp('pfSense is rest...')
#3 /etc/inc/notices.inc(151): notify_all_remote('pfSense is rest...')
#4 /etc/inc/config.lib.inc(239): file_notice('config.xml', 'pfSense is rest...', 'pfSenseConfigur...', '')
#5 /etc/inc/config.lib.inc(695): restore_backup('/cf/conf/backup...')
#6 /usr/local/www/pfblockerng/pfblockerng_Top_Spammers.php(291): write_config('[pfBlockerNG] s...')
#7 {main}
thrown @ 2024-10-07 10:03:47
 
S.M.A.R.T. Status  
Drive	Ident	S.M.A.R.T. Status
nvme0		PASSED
UPS Status 
Summary status:	On line

jrey

@incith

and was the item you were trying to "remove countries from top spammers list" an ASN?

can you show the screen and the item you are trying to delete ?

incith

@jrey pfblocker > IP > geoip

jrey

@incith

okay, I'm not going to be able to troubleshoot that I don't use MaxMind for GeoIP data

but there were a lot of changes under the hood with _18 related to how configuration items are handled. There are still a few issues with ASN names where for example strings contains International characters (accented) in the fields that will cause the config save to fail and auto restore said config.

So what I'm suggesting is that if anything in TopSpammers list you are trying to remove has international characters that "might" be a problem based on the issue we see on the ASN side where those strings cause similar issues.
Why I asked for the screen capture of what you are specifically trying to remove, is because I have no idea what is on that screen. I'd be looking however for international (accented) characters just to see if that "could be" the issue here.

Specifically for MaxMind in this case suggest we tag @BBcan177 for more insight.

incith

@jrey

Maybe this? Not the one I'm trying to remove (it's already unchecked that is), but maybe it's parsing through it.

jrey

@incith

that would are my guess, but sorry no way for me to troubleshoot this case specifically- it just "looks" like a similar issue to me.

if you want to see a similar error (where the config gets restored) on the ASN side
create or go to a list you already have with ASN data items (assuming you are using ASNs somewhere and have ipinfo.io up and running)

simply try to add (any ASN that contains international characters in the name) and save the list. the data won't be saved and it will "restore" the config.

the new ipinfo data ASN names contain these, the old bgpview ASN data did not.

here are a couple samples (with the new ASN data) that will cause it to fail.. (of course there are many more)

AS29571
AS273867