URL Table not working correctly

doktornotor

@firewalluser:

So you know everyone now, that must make you the NSA. ;D

Example of relevant hint: https://lists.freebsd.org/pipermail/freebsd-pf/2011-May/006139.html

(Except that this one and others mentioned on that thread do NOT work.)

Example of totally irrelevant "hint":

I didn't read the code
Swap is slow
Get an SSD
signed long is 2^xyz

:( >:( >:(

P.S. That box has 8GiB of RAM and is NOT swapping. This is NOT related to running out of physical RAM!!!

:( >:( >:(

doktornotor

BTW, @OP:

You can see what's in the tables when you go to Diagnostics - Tables and select the one you need from the dropdown. (The javascript fancy popup is not usable anyway for 200K+ or what entries… cannot search in that at all.)

firewalluser

@firewalluser:

Discs are notoriously slow, someone may have put a hard/soft limit in the code to stop too much data being loaded into memory which could then force the OS to use swap more.

I saw that thread and one of the thoughts that crossed my mind was the coder has come up with something/algo to limit how much can get loaded into memory to maintain performance. Bear in mind alot of old code was written before the HW developments like loads of RAM & SSD drives we have today, so designs & code compensated or restricted for poor HW performance.

Until the code is examined and the reasons for the behaviour is established as it might even be by design, everything is speculation on everyone's part here and in the freebsd thread, but they would be the areas I would look into and for some of the reasons why.

doktornotor

KOM

Heh, "right now"…

johns

I'm not having any issues with swap space, however I've also not enabled any firewall rules to use the new URL Table…I had when I first implemented this, but only for a few minutes while testing how things were working after enabling. I didn't see any HW or OS issues....just the two anomalies I've mentioned, so rules were enabled for 5-10 mins, without performance issues.

Any help/feedback on my actual issues/original questions?

So for some background, the reason I started using this was because I was originally using a Network aliases, but when I starting adding more IP's, I received an error stating something about reaching a limit of 1000 and to change something in php.ini, I was about to do that, but that's when I learned about the URL Tables. However in my original IP aliases list I only have about 320 IP's (not 1000). Since I was making some changes I decided to use my large block list I've been gathering, thus the reason for the 200k entries....but originally it was only 320 or so on a Network Alias.

So if I can't have 250k entries, how do I solve this, is there a way to have large numbers of entries without breaking anything (but again, the OS and HW performance were fine for the 5-10 mins)? Why is it showing numbers that I don't even have in my URL table?

Thanks,

doktornotor

@johns:

Any help/feedback on my actual issues/original questions?

See this: https://forum.pfsense.org/index.php?topic=98698.msg549855#msg549855

johns

That link takes me to your comment that I've pasted below…how does that help? You want me to try Diagnostics -Tables to see if it loads?

BTW, @OP:

You can see what's in the tables when you go to Diagnostics - Tables and select the one you need from the dropdown. (The javascript fancy popup is not usable anyway for 200K+ or what entries... cannot search in that at all.)

doktornotor

No. I want you to compare what you see there with what you SHOULD see there, i.e. whether or not it matches the file downloaded. The popup is something that just cannot be worked with on 250K IPs, I'd figure it's very obvious?

johns

@doktornotor:

No. I want you to compare what you see there with what you SHOULD see there, i.e. whether or not it matches the file downloaded. The popup is something that just cannot be worked with on 250K IPs, I'd figure it's very obvious?

It shows the same thing as in the first screen shot, IP's that are not in my text file. I've included a screen shot of it.

![8-28-2015 12-15-21 pm.png](/public/imported_attachments/1/8-28-2015 12-15-21 pm.png)
![8-28-2015 12-15-21 pm.png_thumb](/public/imported_attachments/1/8-28-2015 12-15-21 pm.png_thumb)

doktornotor

Afraid that unless you make your blocklist available here, this won't get anywhere.

BBcan177

Why don't you try to use pfBlockerNG to load these text files… It will also have the option of de-duplicating the Lists (if there are any dups...)

A small note... from the 2nd screen shot... You don't need to :
cat filename | wc -l

You can just use:
wc -l filename

The following is actually faster if your counting ms :)
grep -c ^ filename

johns

@doktornotor:

Afraid that unless you make your blocklist available here, this won't get anywhere.

I'm not sure the whole list is needed, but I've searched for the IP's showing up in the table in my file, but to no avail.


[root@]# grep "1\.0\.209\.0" blocklist.txt
[root@]# grep "1\.0\.155\.0" blocklist.txt
[root@]# grep "1\.0\.167\.0" blocklist.txt
[root@]# head blocklist.txt
120.203.159.14/24
118.244.254.17/24
117.26.227.207/24
27.153.210.22/24
183.232.55.193/24
211.119.86.147/24
175.44.29.77/24
125.77.142.168/24
122.96.59.106/24
190.216.229.68/24

Here's to show grep is working… (selected an IP from the head command above):

[root@]# grep "27\.153\.210\.22" blocklist.txt
27.153.210.22/24

johns

@BBcan177:

Why don't you try to use pfBlockerNG to load these text files… It will also have the option of de-duplicating the Lists (if there are any dups...)

A small note... from the 2nd screen shot... You don't need to :
cat filename | wc -l

You can just use:
wc -l filename

The following is actually faster if your counting ms :)
grep -c ^ filename

I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.

And thanks for the wc tip, I didn't know that!

BBcan177

Shouldn't the last octet be a "0" when using a /24 ?

I don't think those IPs will load into a packet fence table..

BBcan177

@johns:

I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.

I am the Dev of pfBNG … So I can confirm that it can use localfiles.. :)

In the IPv4/6 Tab, enter the localfile path/filename in the URL field....

johns

@BBcan177:

Shouldn't the last octet by "0" when using a /24 ?

I don't think those IPs will load into a packet fence table..

I've used them successfully in small alias tables and URL Tables, and from testing, appear to work correctly.

johns

@BBcan177:

@johns:

I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.

I am the Dev of pfBNG … So I can confirm that it can use localfiles.. :)

In the IPv4/6 Tab, enter the localfile path/filename in the URL field....

Cool! And I see I can set the update frequency! Will this handle the 250k+ records? If so, is there a limit, if not was is the limit?

BBcan177

The first post that Dok posted was about some issues with the pf Tables… I don't personally have a single table over 200,000IPs, but I do have over 200,000 IPs in total table size.

I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..

I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc... Just start with a new 3rd file to keep the size down.

grep -c '/24' filename will show how many lines are /24.

As a test : (Change the pfB_PRI1 to any existing pf Table)

pfctl -t pfB_PRI1 -T add 20.203.159.14/24
0/1 addresses added.

But if I add the IP with a "0" in the last octet

pfctl -t pfB_PRI1 -T show | grep "20.203.159."
20.203.159.0
20.203.159.0/24

doktornotor

Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.