Modify aliases from ssh shell

[robi]

Hi guys, a quick question:

I need to modify an alias containing several IP addresses from command line via SSH. Is that possible?

Reason is, there's a service port-forwarded to the public internet, but I want to block access to it from some various, dynamic IP addresses. I can get the actual list of IP addresses which need to be blocked from the service running on the internal host, and I'd like to add them automatically to pfSense's alias list via a ssh session. Actually, what I would do is replace the IPs in the alias list with a new one every time, say, once a day.

[BBcan177]

Hi robi,

You can do this with pfBlocker.

Create a new list and point the list to a local file on the box and define the frequency of update. Set the list as an "alias"

In rules you can set how this pfBlocker alias will function with block/reject etc…

Then SCP the file to the box as required and pfBlocker will reload that file as you specified.

[rjcrowder]

You can do it with a standard alias if it is one that reads a list from a URL… I can't remember where the file is kept locally, but it copies the list form the URL to a local file under the pfctl directories somewhere, then you tell it to reload the rules using pfctl.

[BBcan177]

I see in the Alias Edit screen when you define an "Alias" as a "URL" that is has an update frequency.

It would be nice if that Freq setting had settings below "1 day" so it could be done without any additional pfctl commands to reload the list and/or add/delete to the list.

[rjcrowder]

It creates a cron job to reload (again - can't remember what it is called)… but you can install the cron package and then schedule the reload job to occur more frequently.

[BBcan177]

@rjcrowder:

It creates a cron job to reload (again - can't remember what it is called)… but you can install the cron package and then schedule the reload job to occur more frequently.

The only bad thing with having the reloading happen too frequently is that at that particular windows of time, you have potentially no blocklists in pf until it completes the reload.

I was thinking that once the aliases are defined, wouldn't the following work to update ips in the existing aliases.

pfctl -t list -T delete 1.2.3.4
pfctl -t list -T add 1.2.3.4

[rjcrowder]

@BBcan177:

@rjcrowder:

It creates a cron job to reload (again - can't remember what it is called)… but you can install the cron package and then schedule the reload job to occur more frequently.

The only bad thing with having the reloading happen too frequently is that at that particular windows of time, you have potentially no blocklists in pf until it completes the reload.

I was thinking that once the aliases are defined, wouldn't the following work to update ips in the existing aliases.

pfctl -t list -T delete 1.2.3.4
pfctl -t list -T add 1.2.3.4

Don't you have the problem regardless if you delete from the list?

I believe the issues are…

1.) for the "non-URL" lists, the list of aliased IP's is kept in the config.xml and the pfctl table file. You can directly update the pfctl table by using pfctl command. However, you would also need to update the config.xml or the table would be overwritten at some point.
2.) for the "URL" lists, the list of aliased IP's is kept in the file referenced by the URL and in the pfctl table file. You can update the URL file and the pfctl table file and tell it to reload with no risk that it will be overwritten by something from the config.xml

I think either approach would work, but in my opinion it is easier to update the files (approach #2) than it is to update the config.xml.

[BBcan177]

@rjcrowder:

Don't you have the problem regardless if you delete from the list?

2.) for the "URL" lists, the list of aliased IP's is kept in the file referenced by the URL and in the pfctl table file. You can update the URL file and the pfctl table file and tell it to reload with no risk that it will be overwritten by something from the config.xml

If you goto Diagnostics:Table Editor, you can remove any IP from the URL Tables. I don't think that you would have to reload?

I just tried this now, and it deleted an IP that was there in the snort2c file, I assume that this would not require a reload?

pfctl -t snort2c -T delete 23.95.25.71
1/1 addresses deleted.

Is there any good documentation on pfctl?

[robi]

Oooh thanks guys. A lot of good ideas.
How often does the URL list update by default?

[rjcrowder]

@BBcan177:

I don't think that you would have to reload?

Gotta say I don't know which things would cause a complete delete/reload and would (to your point) create a "point in time" exposure. I have something that does it using the URL function and then I run the command to tell pfctl to reload the entire list… your probably right... that is probably the riskiest way to do it. But it's easy.

@BBcan177:

Is there any good documentation on pfctl?

It's part of FreeBSD. I'm sure there are some good tutorials, but I haven't seen anything that describes how this works…

[BBcan177]

@rjcrowder:

Gotta say I don't know which things would cause a complete delete/reload and would (to your point) create a "point in time" exposure. I have something that does it using the URL function and then I run the command to tell pfctl to reload the entire list… your probably right... that is probably the riskiest way to do it. But it's easy.

Thanks. I run a script that downloads all of my Blocklists and I perform other functions on the lists. I than point pfBlocker to the local files to update the tables etc..
If I can do this updating more efficiently and not open as many holes I would love to see any advice on that.

It's part of FreeBSD. I'm sure there are some good tutorials, but I haven't seen anything that describes how this works…

If I find some, I will be sure to post the links!  :)

[BBcan177]

I think I found a solution to my original question after reading the following:

http://www.freebsd.org/doc/handbook/firewalls-pf.html

29.3.3.4. Using Tables

Some types of data are relevant to filtering and redirection at a given time, but their definition is too long to be included in the ruleset file. PF supports the use of tables, which are defined lists that can be manipulated without needing to reload the entire ruleset, and which can provide fast lookups. Table names are always enclosed within < >, like this:

table <clients>{ 192.168.2.0/24, !192.168.2.5 }
In this example, the 192.168.2.0/24 network is part of the table, except for the address 192.168.2.5, which is excluded using the ! operator. It is also possible to load tables from files where each item is on a separate line, as seen in this example /etc/clients:

192.168.2.0/24
!192.168.2.5

–--------------------

So I believe that I could create a Blocklist with IPs as usual and create a "Block" Rule.

Example
23.95.25.71
224.3.67.3

I then create a new Blocklist and use a "Match Rule" with the following entries as example

!23.95.25.71
23.95.25.0.24
( Not sure if the ! IPs need to be at the top of the list? )

This could allow the blocking of the specifc IPs and only match the Inverse IPs in that Range.</clients>

[BBcan177]

I tried to add an entry in an "Alias" list with a "!" and it failed with the following command:

I wonder if this a pfSense issue and not a PF issue (FreeBSD)? Maybe the input script needs to be updated to allow "!" characters?

The following input errors were detected:
!75.119.222.16 is not a valid network alias.

[rjcrowder]

@BBcan177:

I tried to add an entry in an "Alias" list with a "!" and it failed with the following command:

I wonder if this a pfSense issue and not a PF issue (FreeBSD)? Maybe the input script needs to be updated to allow "!" characters?

The following input errors were detected:
!75.119.222.16 is not a valid network alias.

Obviously it is the screen code that is checking it and determining that it is not "valid"… You could try manually adding it into the config.xml and then rebooting the box - to see if it properly creates the table. If it works, you could fix the screen code...

[BBcan177]

@rjcrowder:

Obviously it is the screen code that is checking it and determining that it is not "valid"… You could try manually adding it into the config.xml and then rebooting the box - to see if it properly creates the table. If it works, you could fix the screen code...

I have gone that road once before! Trying to copy a large table of IPs and it completely crashed and I had to restore the config file locally.

I believe that these two files are involved, but I'm not proficient in php to try those changes.

/usr/local/www/guiconfig.inc
/usr/local/www/firewall_aliases_edit.php

Question is what will happen if you allow the "!" for other parts of the code that might use these "htmlspecialchars"

[rjcrowder]

You could try it with a real simple one…

Create an alias in the gui first. Then edit the config.xml and find the one you just created. Add a "!" in front of the IP address and save it... then reboot the box. After reboot, check that it still has the "!" in the config.xml and look at what it created for the pfctl table.

If it actually works, I could probably help with the change. I'm not a php expert, but 've got a pretty customized UI that I've created by modifying the base screens.

[BBcan177]

@rjcrowder:

You could try it with a real simple one…

Create an alias in the gui first. Then edit the config.xml and find the one you just created. Add a "!" in front of the IP address and save it... then reboot the box. After reboot, check that it still has the "!" in the config.xml and look at what it created for the pfctl table.

If it actually works, I could probably help with the change. I'm not a php expert, but 've got a pretty customized UI that I've created by modifying the base screens.

I gave it a shot one of my non-production box and it booted back without issue and the "!" is listed infront of the IP. I guess a proper test would be to use real addresses to see if the Match rule would work with this extra character?

See attached Pic

EDIT:

Can pfctl edit an "Alias"? Maybe I can do this from the Commandline?

I guess the proper way would be for the GUI to have an icon for "!" negation? Instead of allowing the user to enter characters?


[rjcrowder]

Is it in the pfctl table that way? I think it is under /var/db/aliastables/table_name.txt… Is it actually working?

And yes... just adding an option to put the negation would be a way to handle it...

[BBcan177]

@rjcrowder:

Is it in the pfctl table that way? I think it is under /var/db/aliastables/table_name.txt… Is it actually working?

And yes... just adding an option to put the negation would be a way to handle it...

I added that to an "Alias" so I assume thats only in the config file. I will try to add this to an existing Table or create a new table tomorrow.

Do you think its feasible to patch the code for this?

[BBcan177]

So this is actually quite easy to do and for my case won't require any hacking of the code  ;)

First step is to create a folder in  [  [b]/usr/local/www ]  example  [ [b] /usr/local/www/aliastables ]

In the [ [b]aliastables ] folder, create a file [ [b]iplist ] with the list of address(es) for the Alias Table
(Make sure the file is never empty as pf doesn't like an empty file (use 1.1.1.1 as a filler if required)

In Firewall:Aliases, add a new "Alias", Select "URL Table" and enter the URL as:

[  [b]https://[pfsense's ip]:[webgui port]/aliastables/iplist  ]    (Thanks to jflsakfja for that part)

The Freq. setting doesn't really matter as you can issue a shell command to update this table. (See next step)
Give the Alias a Name which will be used in the Firewall:Rules Definitions.

In Firewall:Rules:Floating Rules - Add a new rule as required using the Alias you just created above. (Enable Logging)

So for my setup, I want to Block certain IPs, but I want to "Monitor" the other IPs in the /24 Range.

So in the [  [b]/usr/local/www/aliastables/iplist ] file, I can do the following (as example)

Note: to exlude addresses use the exclamation point "!"

66.240.192.0/24
**!**66.240.192.138

( I would also have  [ [b]66.240.192.138 ] in a Block list file using another Blocking Alias )

To get the Table to reload you can use the following commands:

This will show what is in the current Alias Table (iplist)

[  [b]pfctl -t iplist -T show  ]

This will replace the Alias Table (iplist) with the contents of the file (iplist)

[  [b]pfctl -t iplist -T replace -f /usr/local/www/aliastables/iplist  ]

You can use these commands below to add or delete from the table, but it won't save to the local file.

[  [b]pfctl -t iplist -T add 1.1.1.2  ]
    [  [b]pfctl -t iplist -T delete 1.1.1.2  ]

I tested it out and it works well. The pfctl (replace) command is actually quick as it only "deletes" and "adds" IPs instead of a dump and load. Which hopefully makes if faster
and leaves a smaller window exposed.

So for me, this means that I am that much closer to eliminating pfBlocker and using my own Custom Script to Download my Blocklists, Remove Duplication. It can also download a lot of other sources as pfblocker is limited to ".txt" and ".gz" only. It also does the following:

• Look for How many Malicious IPs are in a /24 range in all of the downloaded Blocklists. If you use pfBlocker now and you want to see how many repeat offenders you have, you could run the following command:

The "5" is the number of repeat offenders to find. You can change as required. This will output a range format.

[  cat /var/db/aliastables/* | cut -d '.' -f 1-3 | awk '{a[$0]++}END{for(i in a){if(a _> 5){print i}}}' | sed 's/$/./' | sed 's/^/^/' | sed 's/./\./g'  ]

Take one of the lines from above, and do the following:

[  [b]grep "^x.x.x." /var/db/aliastables/* ]

This will show how many repeated IPs are found in a /24 range.

If you want to see how many Duplicates you have in your pfBlocker lists, run the following command:

[  [b]cat /var/db/aliastables/* | sort | uniq -d  ]

Once the repeated offenders Ranges are found I can do a few things:

• Based upon the IPs Country code (Maxmind Geoip Lookup), block certain country range completely.
• Whitelisted Countries can be put into a " Monitor " list as above to further Monitor their activity in your network.
• I also do a final check which is lookup any ranges where there are over 50 repeated IPs in a /24 range and block those ranges completely.

The Script has its own Scheduler so I just have pfSense CRON load the script once an hour, and the script does the rest.

I will be adding to the script this new "Monitor" method and also the reloading of the Newly downloaded Blocklists as they occur.

EDIT:

After the pfctl replace command is issued, I think it needs a filter reload? Need to test further. I notice that If I open a Rule, I can click on the "Alias" in the source column, and it shows the newly added/removed IPs. But occasionally I need to re-open the rule and then it will show the updated IPs?

[  /etc/rc.filter_configure ]_