URL Table not working correctly
-
Any help/feedback on my actual issues/original questions?
See this: https://forum.pfsense.org/index.php?topic=98698.msg549855#msg549855
-
That link takes me to your comment that I've pasted below…how does that help? You want me to try Diagnostics -Tables to see if it loads?
BTW, @OP:
You can see what's in the tables when you go to Diagnostics - Tables and select the one you need from the dropdown. (The javascript fancy popup is not usable anyway for 200K+ or what entries... cannot search in that at all.)
-
No. I want you to compare what you see there with what you SHOULD see there, i.e. whether or not it matches the file downloaded. The popup is something that just cannot be worked with on 250K IPs, I'd figure it's very obvious?
-
No. I want you to compare what you see there with what you SHOULD see there, i.e. whether or not it matches the file downloaded. The popup is something that just cannot be worked with on 250K IPs, I'd figure it's very obvious?
It shows the same thing as in the first screen shot, IP's that are not in my text file. I've included a screen shot of it.
![8-28-2015 12-15-21 pm.png](/public/imported_attachments/1/8-28-2015 12-15-21 pm.png)
![8-28-2015 12-15-21 pm.png_thumb](/public/imported_attachments/1/8-28-2015 12-15-21 pm.png_thumb) -
Afraid that unless you make your blocklist available here, this won't get anywhere.
-
Why don't you try to use pfBlockerNG to load these text files… It will also have the option of de-duplicating the Lists (if there are any dups...)
A small note... from the 2nd screen shot... You don't need to :
cat filename | wc -lYou can just use:
wc -l filenameThe following is actually faster if your counting ms :)
grep -c ^ filename -
Afraid that unless you make your blocklist available here, this won't get anywhere.
I'm not sure the whole list is needed, but I've searched for the IP's showing up in the table in my file, but to no avail.
[root@]# grep "1\.0\.209\.0" blocklist.txt [root@]# grep "1\.0\.155\.0" blocklist.txt [root@]# grep "1\.0\.167\.0" blocklist.txt [root@]# head blocklist.txt 120.203.159.14/24 118.244.254.17/24 117.26.227.207/24 27.153.210.22/24 183.232.55.193/24 211.119.86.147/24 175.44.29.77/24 125.77.142.168/24 122.96.59.106/24 190.216.229.68/24
Here's to show grep is working… (selected an IP from the head command above):
[root@]# grep "27\.153\.210\.22" blocklist.txt 27.153.210.22/24
-
Why don't you try to use pfBlockerNG to load these text files… It will also have the option of de-duplicating the Lists (if there are any dups...)
A small note... from the 2nd screen shot... You don't need to :
cat filename | wc -lYou can just use:
wc -l filenameThe following is actually faster if your counting ms :)
grep -c ^ filenameI'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.
And thanks for the wc tip, I didn't know that!
-
Shouldn't the last octet be a "0" when using a /24 ?
I don't think those IPs will load into a packet fence table..
-
I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.
I am the Dev of pfBNG … So I can confirm that it can use localfiles.. :)
In the IPv4/6 Tab, enter the localfile path/filename in the URL field....
-
Shouldn't the last octet by "0" when using a /24 ?
I don't think those IPs will load into a packet fence table..
I've used them successfully in small alias tables and URL Tables, and from testing, appear to work correctly.
-
I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.
I am the Dev of pfBNG … So I can confirm that it can use localfiles.. :)
In the IPv4/6 Tab, enter the localfile path/filename in the URL field....
Cool! And I see I can set the update frequency! Will this handle the 250k+ records? If so, is there a limit, if not was is the limit?
-
The first post that Dok posted was about some issues with the pf Tables… I don't personally have a single table over 200,000IPs, but I do have over 200,000 IPs in total table size.
I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..
I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc... Just start with a new 3rd file to keep the size down.
grep -c '/24' filename will show how many lines are /24.
As a test : (Change the pfB_PRI1 to any existing pf Table)
pfctl -t pfB_PRI1 -T add 20.203.159.14/24
0/1 addresses added.But if I add the IP with a "0" in the last octet
pfctl -t pfB_PRI1 -T show | grep "20.203.159."
20.203.159.0
20.203.159.0/24 -
Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.
-
I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..
I'm starting to think the same thing….however....I have a much smaller list, it's setup the exact same way...the difference is the size...this one has about 300 ip's, that's it...and it auto set the last octect to 0....
[root@]# head manualblocklist.txt 178.120.172.209/24 186.82.25.216/24 77.44.161.22/24 181.118.75.200/24 188.209.49.117/24 119.94.47.83/24 81.92.120.13/24 118.98.115.16/24 180.191.104.244/24 81.213.208.9/24
![8-28-2015 1-05-55 pm.png](/public/imported_attachments/1/8-28-2015 1-05-55 pm.png)
![8-28-2015 1-05-55 pm.png_thumb](/public/imported_attachments/1/8-28-2015 1-05-55 pm.png_thumb) -
I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc… Just start with a new 3rd file to keep the size down.
I was thinking the same thing, thus my reason for asking what limits there were. Thanks.
-
How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.
1/ Stop feeding invalid crap to aliases.
2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller. Like this: -
Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.
Ah, this may be the issue….using my smaller url table I just checked, and the /24 are imported but not the single IP's listed as /32. So perhaps removing these from my lists will solve the issue? I'll try it and repost.
As a side note, I tried using pfbng to import and use the list and it semi working, the file was imported and did find dups, but it's not blocking anything (nothing shown in pfbng status widget) and when I try to view the table (via diag -> tables, nothing shows)...so I assume it may be the same issue.
-
How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.
1/ Stop feeding invalid crap to aliases.
2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller.That's exactly the issue…when I look at the files, it shows legit info, but that's not what's being loaded into the aliases, thus the reason I posted here. I'm going to remove the /32's and see if that doesn't help.
-
I think you are just very confused? You have no /32 anywhere. You have /24 there.