URL Table not working correctly
-
Shouldn't the last octet by "0" when using a /24 ?
I don't think those IPs will load into a packet fence table..
I've used them successfully in small alias tables and URL Tables, and from testing, appear to work correctly.
-
I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.
I am the Dev of pfBNG … So I can confirm that it can use localfiles.. :)
In the IPv4/6 Tab, enter the localfile path/filename in the URL field....
Cool! And I see I can set the update frequency! Will this handle the 250k+ records? If so, is there a limit, if not was is the limit?
-
The first post that Dok posted was about some issues with the pf Tables… I don't personally have a single table over 200,000IPs, but I do have over 200,000 IPs in total table size.
I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..
I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc... Just start with a new 3rd file to keep the size down.
grep -c '/24' filename will show how many lines are /24.
As a test : (Change the pfB_PRI1 to any existing pf Table)
pfctl -t pfB_PRI1 -T add 20.203.159.14/24
0/1 addresses added.But if I add the IP with a "0" in the last octet
pfctl -t pfB_PRI1 -T show | grep "20.203.159."
20.203.159.0
20.203.159.0/24 -
Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.
-
I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..
I'm starting to think the same thing….however....I have a much smaller list, it's setup the exact same way...the difference is the size...this one has about 300 ip's, that's it...and it auto set the last octect to 0....
[root@]# head manualblocklist.txt 178.120.172.209/24 186.82.25.216/24 77.44.161.22/24 181.118.75.200/24 188.209.49.117/24 119.94.47.83/24 81.92.120.13/24 118.98.115.16/24 180.191.104.244/24 81.213.208.9/24
![8-28-2015 1-05-55 pm.png](/public/imported_attachments/1/8-28-2015 1-05-55 pm.png)
![8-28-2015 1-05-55 pm.png_thumb](/public/imported_attachments/1/8-28-2015 1-05-55 pm.png_thumb) -
I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc… Just start with a new 3rd file to keep the size down.
I was thinking the same thing, thus my reason for asking what limits there were. Thanks.
-
How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.
1/ Stop feeding invalid crap to aliases.
2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller. Like this: -
Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.
Ah, this may be the issue….using my smaller url table I just checked, and the /24 are imported but not the single IP's listed as /32. So perhaps removing these from my lists will solve the issue? I'll try it and repost.
As a side note, I tried using pfbng to import and use the list and it semi working, the file was imported and did find dups, but it's not blocking anything (nothing shown in pfbng status widget) and when I try to view the table (via diag -> tables, nothing shows)...so I assume it may be the same issue.
-
How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.
1/ Stop feeding invalid crap to aliases.
2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller.That's exactly the issue…when I look at the files, it shows legit info, but that's not what's being loaded into the aliases, thus the reason I posted here. I'm going to remove the /32's and see if that doesn't help.
-
I think you are just very confused? You have no /32 anywhere. You have /24 there.
-
I think you are just very confused? You have no /32 anywhere. You have /24 there.
Sorry, perhaps I'm not being clear, but I do have /32's listed, just not a lot of them:
[root@]# grep '\/32' manualblocklist.txt 70.49.52.23/32 69.46.128.29/32 66.51.128.42/32 109.169.22.220/32 162.250.189.125/32 198.23.140.98/32 104.206.96.58/32
And to clarify…I've tested and the /24 are imported and auto changed to .0/24, but the /32's are not imported, so I'm wondering if they're causing issues.
-
I take it back, they are being imported….screenshot one is my text file and screen shot two is from diag -> tables. So I'm back to square one. Any other ideas?
![8-28-2015 1-28-35 pm.png](/public/imported_attachments/1/8-28-2015 1-28-35 pm.png)
![8-28-2015 1-27-57 pm.png](/public/imported_attachments/1/8-28-2015 1-27-57 pm.png) -
I tried using pfbng to no avail…file imports correctly, but lists shows as blank and status shows no blocks:
UPDATE PROCESS START [ 08/28/15 13:31:55 ] [ pfB_Europe_v4 ] exists, Reloading File [ pfB_Top_v4 ] exists, Reloading File [ cbl ] Downloading New File ---------------------------------------------------------- Original Masterfile Outfile [ Post Duplication count ] ---------------------------------------------------------- 225549 213102 213102 [ Passed ] ---------------------------------------------------------- ===[ Aliastables / Rules ]================================ Firewall Rule Changes Found, Applying Filter Reload ===[ FINAL Processing ]===================================== [ Original count ] [ 242994 ] [ Processed Count ] [ 230547 ] ===[ Deny List IP Counts ]=========================== 230547 total 213102 /var/db/pfblockerng/deny/cbl.txt
-
According to that screenshot, it collected 213102 IPs… The others are probably duplicates...
I think your going to have strange results until you convert the file which has /24 to its proper format. ("0" in the last octet"... Don't worry about the /32 it will work with or without the /32.
Here is a small script that will convert the /24's. (Change the /tmp/filename)
#!/bin/sh data=$(grep '/24' /tmp/filename) for ip in $data; do i="$(echo $ip | cut -d '.' -f 1-3)" echo $i".0/24" done exit
To collect all other IPs in the file and exclude the /24's
grep -v '/24' filenameIf you want to sort that file…
sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n filename -
According to that screenshot, it collected 213102 IPs… The others are probably duplicates...
I think your going to have strange results until you convert the file which has /24 to its proper format. ("0" in the last octet"... Don't worry about the /32 it will work with or without the /32.
Here is a small script that will convert the /24's. (Change the /tmp/filename)
#!/bin/sh data=$(grep '/24' /tmp/filename) for ip in $data; do i="$(echo $ip | cut -d '.' -f 1-3)" echo $i".0/24" done exit
To collect all other IPs in the file and exclude the /24's
grep -v '/24' filenameIf you want to sort that file…
sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n filenameThanks, I'll try it.
-
I converted all the IP's from x.x.x.x/24 to x.x.x.0/24, and imported it via pfbng, it imported successfully (just as it did previously), but in the pfbng dashboard widget the status is not displaying (shows the new line, but no stats), and when I click on the alias in the rule, 0 IP's are displayed (i.e. for other aliases when you put the mouse over the alias it shows x IP's and then lists the IP's).
Any other ideas?
-
Send a screenshot of the Widget and of the Rule for this Alias.
-
As a side note regarding this, I'm also noting that the is_subnet*() functions in /etc/inc/util.inc are very useless, they don't verify a valid subnet at all.
-
In this case, the limit seems to be linked to dirty a data type restriction, because out of these problems is easy,Casesam but there is no problem to check my previous post codes can only be regarded as a conjecture at this stage, but I suspect there may be a reason for exchange limits.Samsung Galaxy S6 edge cover
-
I converted all the IP's from x.x.x.x/24 to x.x.x.0/24, and imported it via pfbng, it imported successfully (just as it did previously), but in the pfbng dashboard widget the status is not displaying (shows the new line, but no stats), and when I click on the alias in the rule, 0 IP's are displayed (i.e. for other aliases when you put the mouse over the alias it shows x IP's and then lists the IP's).
Any other ideas?
If you see the widget showing blank status and the mouseover of the Alias is blank, that means that there is some incorrect IPs in the list…
pfBNG v1.09 uses a Regex function to match IPs including CIDRs. The regex is not 100% and there are some corner cases where it can allow an incorrect CIDR value... The next release v2.0 uses all string functions to process and validate each IP which will avoid these corner cases...
There has to be some malformed IPs of some sort in this file of yours. Try to split the file into smaller segments, then load each into pfBNG to see which segment is of issue. Then narrow it down again until you find the issue. You can also post the whole file in pastebin/dropbox etc and PM me a link to the file and I can help figure it out for you.