URL Table not working correctly

johns

How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.

1/ Stop feeding invalid crap to aliases.
2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller.

That's exactly the issue…when I look at the files, it shows legit info, but that's not what's being loaded into the aliases, thus the reason I posted here. I'm going to remove the /32's and see if that doesn't help.

doktornotor

I think you are just very confused? You have no /32 anywhere. You have /24 there.

johns

@doktornotor:

I think you are just very confused? You have no /32 anywhere. You have /24 there.

Sorry, perhaps I'm not being clear, but I do have /32's listed, just not a lot of them:


[root@]# grep '\/32' manualblocklist.txt
70.49.52.23/32
69.46.128.29/32
66.51.128.42/32
109.169.22.220/32
162.250.189.125/32
198.23.140.98/32
104.206.96.58/32

And to clarify…I've tested and the /24 are imported and auto changed to .0/24, but the /32's are not imported, so I'm wondering if they're causing issues.

johns

I take it back, they are being imported….screenshot one is my text file and screen shot two is from diag -> tables. So I'm back to square one. Any other ideas?

![8-28-2015 1-28-35 pm.png](/public/imported_attachments/1/8-28-2015 1-28-35 pm.png)
![8-28-2015 1-27-57 pm.png](/public/imported_attachments/1/8-28-2015 1-27-57 pm.png)

johns

I tried using pfbng to no avail…file imports correctly, but lists shows as blank and status shows no blocks:

 UPDATE PROCESS START [ 08/28/15 13:31:55 ]

[ pfB_Europe_v4 ]	 exists, Reloading File 
[ pfB_Top_v4 ]		 exists, Reloading File 
[ cbl ]			 Downloading New File ----------------------------------------------------------
Original   Masterfile Outfile     [ Post Duplication count ]   
----------------------------------------------------------
225549     213102     213102      [ Passed ]                   
----------------------------------------------------------

===[  Aliastables / Rules  ]================================

Firewall Rule Changes Found, Applying Filter Reload

===[ FINAL Processing ]=====================================

   [ Original count   ]  [ 242994 ]

   [ Processed Count  ]  [ 230547 ]

===[ Deny List IP Counts ]===========================

  230547 total
  213102 /var/db/pfblockerng/deny/cbl.txt

BBcan177

According to that screenshot, it collected 213102 IPs… The others are probably duplicates...

I think your going to have strange results until you convert the file which has /24 to its proper format. ("0" in the last octet"... Don't worry about the /32 it will work with or without the /32.

Here is a small script that will convert the /24's. (Change the /tmp/filename)


#!/bin/sh
data=$(grep '/24' /tmp/filename)

for ip in $data; do
        i="$(echo $ip | cut -d '.' -f 1-3)"
        echo $i".0/24"
done
exit

To collect all other IPs in the file and exclude the /24's
grep -v '/24' filename

If you want to sort that file…
sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n filename

johns

@BBcan177:

According to that screenshot, it collected 213102 IPs… The others are probably duplicates...

I think your going to have strange results until you convert the file which has /24 to its proper format. ("0" in the last octet"... Don't worry about the /32 it will work with or without the /32.

Here is a small script that will convert the /24's. (Change the /tmp/filename)
#!/bin/sh
data=$(grep '/24' /tmp/filename)

for ip in $data; do
        i="$(echo $ip | cut -d '.' -f 1-3)"
        echo $i".0/24"
done
exit
To collect all other IPs in the file and exclude the /24's
grep -v '/24' filename

If you want to sort that file…
sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n filename

Thanks, I'll try it.

johns

I converted all the IP's from x.x.x.x/24 to x.x.x.0/24, and imported it via pfbng, it imported successfully (just as it did previously), but in the pfbng dashboard widget the status is not displaying (shows the new line, but no stats), and when I click on the alias in the rule, 0 IP's are displayed (i.e. for other aliases when you put the mouse over the alias it shows x IP's and then lists the IP's).

Any other ideas?

BBcan177

Send a screenshot of the Widget and of the Rule for this Alias.

doktornotor

As a side note regarding this, I'm also noting that the is_subnet*() functions in /etc/inc/util.inc are very useless, they don't verify a valid subnet at all.

MikeJJ

In this case, the limit seems to be linked to dirty a data type restriction, because out of these problems is easy,Casesam but there is no problem to check my previous post codes can only be regarded as a conjecture at this stage, but I suspect there may be a reason for exchange limits.Samsung Galaxy S6 edge cover

BBcan177

@johns:

I converted all the IP's from x.x.x.x/24 to x.x.x.0/24, and imported it via pfbng, it imported successfully (just as it did previously), but in the pfbng dashboard widget the status is not displaying (shows the new line, but no stats), and when I click on the alias in the rule, 0 IP's are displayed (i.e. for other aliases when you put the mouse over the alias it shows x IP's and then lists the IP's).

Any other ideas?

If you see the widget showing blank status and the mouseover of the Alias is blank, that means that there is some incorrect IPs in the list…

pfBNG v1.09 uses a Regex function to match IPs including CIDRs. The regex is not 100% and there are some corner cases where it can allow an incorrect CIDR value... The next release v2.0 uses all string functions to process and validate each IP which will avoid these corner cases...

There has to be some malformed IPs of some sort in this file of yours. Try to split the file into smaller segments, then load each into pfBNG to see which segment is of issue. Then narrow it down again until you find the issue. You can also post the whole file in pastebin/dropbox etc and PM me a link to the file and I can help figure it out for you.

johns

So…thanks for all the help, I got it working! I cleaned up all the records, replacing x.x.x.x/24 with x.x.x.0/24 and replacing /32 with just the IP. After cleaning up, I was able to get it to work both as a URL Table and in pfBNG. In fact, in pfBNG, it found 120k dups, so half of the IP's listed were dups...so my list is now down to 165k'ish.

One anomaly I noticed with pfBNG (I only mention it here instead of in a new post, since it was suggested/referenced here)...is that changing the state to OFF had no affect. I would change it to off, then save, then force update (Update Custom List was set to Update Custom List), and it would still block and show up in the dashboard widget. While it was still set to off, I went into Firewall -> Rules and deleted the rules associated with the pfBNG alias, and when I forced update, the rules recreated themselves. The only way I found to disabled the rules and blocking, was to delete them from pfBNG -> IPv4.

Other than that, it's working perfectly now! Thanks again for all the suggestions, help and code snippets!

BBcan177

@johns:

replacing /32 with just the IP

/32 shouldn't cause an issue with/without

is that changing the state to OFF had no affect. I would change it to off, then save, then force update (Update Custom List was set to Update Custom List)

Are you using the Custom List box for these IPs? or are you saving the IPs to a file in the pfSense box and adding it to the URL / local file entry?

The "state" is only for the URL / Local file entry. The Custom list needs to be empty or it will create a Firewall rule automatically.

johns

Are you using the Custom List box for these IPs? or are you saving the IPs to a file in the pfSense box and adding it to the URL / local file entry?

The "state" is only for the URL / Local file entry. The Custom list needs to be empty or it will create a Firewall rule automatically.

I'm using a URL (not local) and the file is not empty. So there's no way to "disable" without deleting the entry?