Squid & QuidGuard - Update webinterface after manual modifications

bezourox · Nov 3, 2010, 10:30 AM

Hi,
For beginning, sorry for my poor english. ;D

I want to synchronize many pfsense which are on the same configuration :
pfsense (1.2.3) + squid(2.7.9_1) + squidguard(1.3-2) + lightsquid(1.7.1)

I configured 2 local users on squid, a basic user 'internet' who access only to my whitelist, and a vip user 'vip' who have no restrictions.
On my first pfsense, I added domains (like google.com or yahoo.com) in SquidGuard's destinations, I restart the service and all is okey.

Now, I would like to export my whitelist in all others pfsense.

How to do this ?

I tested a solution but it's not perfect :

1. I copy my whitelist's domains on a second pfsense (with a rsync of /var/db/squidGuard/whitelist/domains)
2. On my second pfsense, I update the domain.db (with /usr/local/bin/squidGuard -C domains)
3. I restart the service on this second pfsense (with /usr/local/sbin/squid.sh restart)

On , my second pfsense, the active configuration is okey, but web interface isn't update and that's why, if I do an other modification on web interface and I restart the service, my domains.db is regenerated with domains present in the GUI.(modifications done manually are lost)

I search a solution to update domains in the section "Destinations" of the web interface.

I suppose that there is a function whitch read domains, URL and expressions in a temporary file and not directly in /var/db/squidGuard/whitelist/domains …
Need some help to resolve this problem.
Thanks

dvserg · Nov 3, 2010, 10:48 AM

File /conf/config.xml
Section <squidguarddest>…</squidguarddest>, Copy From & Replace To

bezourox · Nov 3, 2010, 1:19 PM

Thanks for your quick answer.
I will dev a shell script to do this.

dvserg · Nov 3, 2010, 3:35 PM

@bezourox:

Thanks for your quick answer.
I will dev a shell script to do this.

Now, I would like to export my whitelist in all others pfsense.

I mean this is for copy 'Destinations' only.

bezourox · Nov 4, 2010, 11:48 AM

Hi,

Yes, Destinations only.
Now, config.xml is updated successfuly, but when I refresh my web browser (F5), news URL are not present… ??? >:( >:( ???

I proceeded like this :

1. Add news domains in /var/db/squidGuard/whitelist/domains (one domain by line)
2. Update domain.db (with /usr/local/bin/squidGuard -C domains)
3. Execute my script (whitch update /cf/conf/config.xml)
4. Restart squid (with /usr/etc/rc.d/squid.sh restart)

bezourox · Nov 8, 2010, 10:06 AM

UP.
Anyone help me ?

dvserg · Nov 8, 2010, 10:36 AM

@bezourox:

Hi,

Yes, Destinations only.
Now, config.xml is updated successfuly, but when I refresh my web browser (F5), news URL are not present… ??? >:( >:( ???

I proceeded like this :

1. Add news domains in /var/db/squidGuard/whitelist/domains (one domain by line)
2. Update domain.db (with /usr/local/bin/squidGuard -C domains)
3. Execute my script (whitch update /cf/conf/config.xml)
4. Restart squid (with /usr/etc/rc.d/squid.sh restart)

This is not right. Explain what you want to achieve?
Exists blacklist (downloaded as is) and exists user-defined destinations (defined via WebGUI).

bezourox · Nov 8, 2010, 10:53 AM

I want to synchronize the whitelist of squid from my main pfsense to other pfsense.
To do this, the goal is :
1. Update whitelist on my main pfsense
2. Every day, launch a cronjob witch update all others pfsense ' whitelists

This is impossible in graphic mode, that's why I want to do with rsync
My proxy as no blacklist definied…
Just 2 access : A VIP user (with no restriction), and a restricted user who access only to the whitelist

dvserg · Nov 8, 2010, 11:16 AM

@bezourox:

I want to synchronize the whitelist of squid from my main pfsense to other pfsense.
To do this, the goal is :
1. Update whitelist on my main pfsense
2. Every day, launch a cronjob witch update all others pfsense ' whitelists

This is impossible in graphic mode, that's why I want to do with rsync
My proxy as no blacklist definied…
Just 2 access : A VIP user (with no restriction), and a restricted user who access only to the whitelist

1. You must create a Destination element with the same name in both pfSense's; Define ACL's and rules; Apply config.
2.
Way A.

Copy You /var/db/squidGuard/whitelist/domains from first pfSense to second pfsense
Update domain.db (with /usr/local/bin/squidGuard -C domains)
Restart squid (with /usr/etc/rc.d/squid.sh restart)
But this way up-to the first reboot or squidGuard Web-gui change.

Way B

copy Destination (update /cf/conf/config.xml) from first pfSense to second pfSense
Execute php script with

require_once ('/usr/local/pkg/squidguard.inc');
sg_reconfigure();

This way the best.

bezourox · Nov 8, 2010, 12:47 PM

For you, second solution is the best, but for me it's first.
I would like to have a bidirectional synchronization… but I tkink it will be impossible.
I have one person by site who update URL list of her pfsense.
When one of these persons update her URL whitelist, 24 hours later, all whitelist are updated.

Site B ---->
Site C ----> rsync whitelist ----> Main site
Site D ----> |
etc... |
^ |
|_______________________________|
rsync new whitelist

Sorry for this horrible schema ;)

bezourox · Nov 9, 2010, 4:15 PM

This my script :

update_whitelist.sh

while read ligne
do
newchaine=$newchaine$ligne' '

done < /var/db/squidGuard/listeblanche/domains
echo New Chain : $newchaine

sed "s/(<domains>)[^<]*/\1$newchaine/" /cf/conf/config.xml > /cf/conf/result
mv -f /cf/conf/result /cf/conf/config.xml

/usr/local/bin/squidGuard -C domains
/usr/local/etc/rc.d/squid.sh restart</domains>

1. In /var/db/squidGuard/listeblanche/domains I add new URL (one per line) on my main pfsense
2. Every day, I synchronize my file "domains" with cronjob (From my main pfsense to others pfsense)
3. Every day, after this synchronization, I launch the same script on others pfsense with a cronjob.

It's okey on one pfsense. I have to test with others.

One problem stays.
I hope it will be possible to improve this process to do a bidirectional synchronization.