I am now creating new DansGuardian and Squid3 binaries.

aaronouthier

Brief synopsis:

DansGuardian was working for me, but RegEx didn't seem to work properly. I have already recompiled with RegEx support. It took a bit of work to get libpcre working, but I now have the binary functioning I have not tested it very thoroughly - only tested that the binary runs.

Squid3-dev did not work properly, even though it seemed to run. Binary executable would exit with a missing shared library.

Nitty-gritty: Both packages were being "configure"d to use the default PREFIX path. This didn't work correctly. I am working on remedying this.

Stay tuned. Check my previous thread for info, or if you wish to help test my version.

jimp

Make the build tags right in the xml and we can build it. We won't host binaries we didn't build ourselves, for security (and reproduceability) reasons. If you can get it building right using the tags in the XML to make the PBI, we can compile them here. If it takes special handling, document every part of it and we'll see what can be done.

aaronouthier

@jimp:

Make the build tags right in the xml and we can build it. We won't host binaries we didn't build ourselves, for security (and reproduceability) reasons. If you can get it building right using the tags in the XML to make the PBI, we can compile them here. If it takes special handling, document every part of it and we'll see what can be done.

Right!

I could almost follow that conversation. I've done html and even xml coding in the past - albeit several years ago. I am completely in the dark, however, about building a pbi file. Most of my experience with this kind of stuff revolves around linux tinkering and maintenance. FreeBSD isn't so hard coming from that background, but it's not as familiar to me as linux, or Windows for that matter…

If someone could point me in the right direction as to learning the basics of a pbi file/package. I would imagine it is actually some other file type that is just renamed, such as a compressed tar file, or similar - is this correct? Is there a sample pbi file or a template somewhere? I should be able to figure everything out, but I could use a push in the right direction...

Thanks!

Cino

I could be wrong but I think this is what he is referring too

https://github.com/pfsense/pfsense-packages

pkg_config.10.xml
pkg_config.8.xml
pkg_config.8.xml.amd64

from pkg_config.8


	 <package><name>squid3</name>
<internal_name>squid</internal_name>
 <descr>It combines squid as a proxy server with it's capabilities of acting as a HTTP / HTTPS reverse proxy.

It includes an Exchange-Web-Access (OWA) Assistant.]]></descr>
<pkginfolink>https://forum.pfsense.org/index.php/topic,48347.0.html</pkginfolink>
<website>http://www.squid-cache.org/</website>
<category>Network</category>
<version>3.1.20 pkg 2.1.1</version>
<status>beta</status>
<required_version>2.0</required_version>
<maintainer>marcellocoutinho@gmail.com fernando@netfilter.com.br seth.mos@dds.nl mfuchs77@googlemail.com jimp@pfsense.org</maintainer>
<depends_on_package_base_url>https://files.pfsense.org/packages/8/All/</depends_on_package_base_url>
<depends_on_package>squid-3.1.20.tbz</depends_on_package>
<depends_on_package>libwww-5.4.0_4.tbz</depends_on_package>
 <build_pbi><ports_before>www/libwww</ports_before>
<port>www/squid31</port>
<ports_after>www/squid_radius_auth</ports_after></build_pbi> 
<build_options>c-icap_UNSET_FORCE=IPV6;squid_UNSET_FORCE=AUTH_SMB AUTH_SQL DNS_HELPER FS_COSS ESI SNMP ECAP STACKTRACES STRICT_HTTP TP_IPF TP_IPFW VIA_DB DEBUG DOCS EXAMPLES;squid_SET=ARP_ACL AUTH_KERB AUTH_LDAP AUTH_NIS AUTH_SASL CACHE_DIGESTS DELAY_POOLS FOLLOW_XFF TP_PF MSSL_CRTD WCCP WCCPV2 FS_AUFS HTCP ICAP ICMP IDENT IPV6 KQUEUE LARGEFILE SSL SSL_CRTD</build_options>

<config_file>https://packages.pfsense.org/packages/config/squid3/31/squid.xml</config_file>
<configurationfile>squid.xml</configurationfile>
<depends_on_package_pbi>squid-3.1.22_1-i386.pbi</depends_on_package_pbi></package> 
 <package><name>squid3-dev</name>
<internal_name>squid</internal_name>
 <descr>It combines squid as a proxy server with it's capabilities of acting as a HTTP / HTTPS reverse proxy.

It includes an Exchange-Web-Access (OWA) Assistant, ssl filtering and antivirus integration via i-cap]]></descr>
<pkginfolink>https://forum.pfsense.org/index.php/topic,48347.0.html</pkginfolink>
<website>http://www.squid-cache.org/</website>
<category>Network</category>
<version>3.3.10 pkg 2.2.6</version>
<status>beta</status>
<required_version>2.0</required_version>
<maintainer>marcellocoutinho@gmail.com fernando@netfilter.com.br seth.mos@dds.nl mfuchs77@googlemail.com jimp@pfsense.org</maintainer>
<depends_on_package_base_url>https://files.pfsense.org/packages/8/All/</depends_on_package_base_url>
<depends_on_package>squid-3.3.5.tbz</depends_on_package>
<depends_on_package>libltdl-2.4.2.tbz</depends_on_package>
<depends_on_package>libwww-5.4.0_4.tbz</depends_on_package>
<depends_on_package>squidclamav-6.10_1.tbz</depends_on_package>
<depends_on_package>clamav-0.97.8.tbz</depends_on_package>
<depends_on_package>cyrus-sasl-2.1.26_2.tbz</depends_on_package>
<depends_on_package>ca_root_nss-3.14.1.tbz</depends_on_package>
 <build_pbi><ports_before>www/libwww security/cyrus-sasl2</ports_before>
<port>www/squid33</port>
<ports_after>www/squid_radius_auth security/clamav www/squidclamav security/ca_root_nss www/c-icap-modules</ports_after></build_pbi> 
<build_options>c-icap_UNSET_FORCE=IPV6;squid_UNSET_FORCE=AUTH_SMB AUTH_SQL DNS_HELPER FS_COSS ESI SNMP ECAP STACKTRACES STRICT_HTTP TP_IPF TP_IPFW VIA_DB DEBUG DOCS EXAMPLES AUTH_SASL;squid_SET=ARP_ACL AUTH_KERB AUTH_LDAP AUTH_NIS CACHE_DIGESTS DELAY_POOLS FOLLOW_XFF TP_PF MSSL_CRTD WCCP WCCPV2 FS_AUFS HTCP ICAP ICMP IDENT IPV6 KQUEUE LARGEFILE SSL SSL_CRTD</build_options>
<config_file>https://packages.pfsense.org/packages/config/squid3/33/squid.xml</config_file>
<configurationfile>squid.xml</configurationfile>
<depends_on_package_pbi>squid-3.3.10-i386.pbi</depends_on_package_pbi></package>

aaronouthier

Thanks Cino,

I'll have a look. Still trying to recompile squid. I had some setbacks. I'll keep everyone posted.

rjcrowder

@aaronouthier:

Brief synopsis:

DansGuardian was working for me, but RegEx didn't seem to work properly. I have already recompiled with RegEx support. It took a bit of work to get libpcre working, but I now have the binary functioning I have not tested it very thoroughly - only tested that the binary runs.

Squid3-dev did not work properly, even though it seemed to run. Binary executable would exit with a missing shared library.

Nitty-gritty: Both packages were being "configure"d to use the default PREFIX path. This didn't work correctly. I am working on remedying this.

Stay tuned. Check my previous thread for info, or if you wish to help test my version.

Just FYI… I can't remember which version, but only one of the versions had a problem with PCRE... I've been overlaying the .exe with a version off of Marcello's site (out of town - could get you the version on Thursday) and it works fine.

rjcrowder

Also… if you are going to the trouble of recompiling, it would be nice to just switch to the e2guardian application. I'd be willing to work on the updates to the package if you are interested in helping with it... I think it is mostly directory name changes. I got started on it at one point, but never finished...

aaronouthier

Interesting. I'm not sure which version I had tried when I had squid working. However, I do recall that, although DansGuardian didn't report any problems, I was unable to get regex replacements working. I tried to use Google SafeSearch enforcement as a test. I couldn't ever seem to force safesearch on. I assumed this was a problem with that particular function of DG. Everything else seemed to work. As I was having many troubles getting squid proxy to work, so DG has no upstream proxy to which to connect. Hence, the reason for all of this trouble.

Also, I have already recompiled DansGuardian (DG) binaries, and have copied them to my pfSense test system, overwriting the original stuff.

Overall status update: Almost finished, I hope! I have compiled Squid 3.4 from source using the ports feature of FreeBSD 10.1. Almost have it running. It looks like I still have some symlinks to create.

Legion

If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

aaronouthier

Good grief!!

All of this time!

The reason that I couldn't connect to squid service on my box: I disabled IPv6 and blocked all v6 traffic on my network. Although I can't imagine why this would affect squid, well, it apparently did. I was beginning to wonder if there wasn't some obscure bug that prevented it from working on just my hardware type or some such thing.

The whole reason I set out to recompile this stuff, is because it didn't work for me…

Sigh. Still, I'm helping (hopefully) the pfSense community by getting squid updated. I don't think there will be too many complaints about that. I'm still in awe. It was so simple!

It's late, I'll see about publishing my work in the morning. It is now 20 minutes before 1 AM. Good night. everybody.

Cino

@Legion:

If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

a little off topic, but what do you mean? squidguard itself hasn't been updated in a couple of years, its 1.5 beta on their website…

rjcrowder

@aaronouthier:

Interesting. I'm not sure which version I had tried when I had squid working. However, I do recall that, although DansGuardian didn't report any problems, I was unable to get regex replacements working. I tried to use Google SafeSearch enforcement as a test. I couldn't ever seem to force safesearch on. I assumed this was a problem with that particular function of DG.

The PCRE stuff is an option on the compile. I think Marcello had at least one version out there that did not have it turned on… Unfortunately, I also used it for a while and couldn't get the safesearch to work (think I listed it in one of the DG threads). I found another version that works fine. You'll need to make sure you turn it on when you compile.

Would be very interested in getting e2guardian working if I could get a little help on it... I think he fixed some of the abend issues that were occuring.

aaronouthier

Ok, rather than doing PMs I'm going to post the tarballs and instructions here. It is against my better judgment, but in the interest of my time, etc…

Instructions (I hope I remember everything):

Copy the gzip'd tarballs onto your pfsense box or VM, etc. This is for amd64 distro only - for now.

cd /usr/pbi/squid-amd64/
```- you need to have squid-dev package installed and not previously modified or touched.

Make a note of the symlinks that go from squid-amd64 to squid-amd64/local . For example there is a symlink from /usr/pbi/squid-amd64/www that points to /usr/pbi/squid-amd64/local/www . Again, make a note of the names of these folders, you will need this later.

Carefully, very carefully, remove the aforementioned symlinks. Do an mv from those folders in the squid-amd64/local folder to the squid-amd64 folder - basically, you're moving the symlink destinations to the parent folder. example:```
cd /usr/pbi/squid-amd64/local/ ; mv www ../

Do that for each previous symlink. Only for the ones that were previously symlinked.

cd /usr/pbi/

Extract the attached gzip'd tarball for squid with```
tar xzvf /[path where you copied my attachment]/squid-amd64.tar.gz

cd /usr/pbi/squid-adm64/sbin

ldd squid


Make a note of any missing shared libraries and where it expects to find them. Copy (or possibly Symlink?) the mentioned libraries as required. you will find the libraries in /usr/pbi/squid-amd64/lib/

Fix the symlinks in /usr/local/bin and /usr/local/sbin to point to the correct locations. Some things have moved between MarcelloC's version and mine, specifically, some things have moved from squid-amd64/bin to sbin.

Try running squid now, but with the "-d 9" command-line switch for debugging info. Note any errors or problems. If you can't figure out how to get it to run, post a reply back here - I may have forgotten to mention something.

I couldn't seem to make dansguardian work this time around, but everyone's welcome to try it out themselves. It should be the same basic procedure. I have large file support turned on, so if you want a 10 GB cache, go for it!

Update: It won't let me attach the files - both are too big. I'll put them on my dropbox. Give me a few minutes.

aaronouthier

https://www.dropbox.com/sh/e6ys6w6oj82633x/AABPezSq-MYVWKy0SGZAdCCha?dl=0

This is for testing only. After feedback and critique, I'll try to make a package as suggested by jimp.

TTFN.

aaronouthier

Update: After checking things again, there are a few missing libraries. Look in the link above, and also grab the squid-libs.tar.gz

Extract the contents to /usr/lib/

If anyone is using a windows box to try to get these files onto a pfsense box, try http file server or "HFS". Its is free, and very convenient. You can then "fetch" the files from your windows machine to your pfSense system.

Legion

@Cino:

@Legion:

If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

a little off topic, but what do you mean? squidguard itself hasn't been updated in a couple of years, its 1.5 beta on their website…

Yeah, I would hope that there would be no problem but I thought that maybe squidguard (and other packages) would have to be updated to their FreeBSD 10.x versions so while he was working with new Squid 3.4 he could test that the associated packages work as well. By the sounds of it dansguardian is the focus but I am using squidguard at the moment rather than DG since I found it more suitable for my needs.

aaronouthier

I had previously tried SquidGuard. Although it seemed to work, I need to be able to auto-update the blacklist, among other things. Even though DansGuardian isn't quite stable, it seemed to have more potential than SquidGuard, once finished.

I'd be happy to check out e2guardian, but one thing at a time.

Lastly, this is mostly directed at jimp, but anyone else interested is welcome to use as desired.

I have build instructions for building from source.

You will need to grab the 'ports' option/project/whatever for FreeBSD 10.1. This can be done either at the point of install, or post-installation, with the commands:

portsnap fetch
portsnap extract

Then, cd into the /usr/ports folder. cd further into the pkg-mgmt folder, and do amake install. If you already have the pkg binary installed, you will instead need to run```
make reinstall


Then, cd to /usr/ports/www/squid . Run```
make PREFIX=/usr/pbi/squid-[b][i]archetecture[/i][/b]/ install
```.

You will be presented with some dialog boxes the first time around, I used the following settings:

> ARP_ACL=on: ARP/MAC/EUI based authentification
>     AUTH_KERB=on: Install Kerberos authentication helpers
>     AUTH_LDAP=on: Install LDAP authentication helpers
>     AUTH_NIS=off: Install NIS/YP authentication helpers
>     AUTH_SASL=off: Install SASL authentication helpers
>     AUTH_SMB=off: Install SMB auth. helpers (req. Samba)
>     AUTH_SQL=off: Install SQL based auth (uses MySQL)
>     CACHE_DIGESTS=on: Use cache digests
>     DEBUG=off: Build with extended debugging support
>     DELAY_POOLS=on: Delay pools (bandwidth limiting)
>     DNS_HELPER=on: Use external dnsserver processes for DNS
>     DOCS=off: Build and/or install documentation
>     ECAP=off: Loadable content adaptation modules
>     ESI=on: ESI support
>     EXAMPLES=off: Build and/or install examples
>     FOLLOW_XFF=on: Support for the X-Following-For header
>     FS_AUFS=on: AUFS (threaded-io) support
>     FS_DISKD=on: DISKD storage engine controlled by separate service
>     FS_ROCK=off: ROCK (unstable)
>     HTCP=on: HTCP support
>     ICAP=on: the ICAP client
>     ICMP=on: ICMP pinging and network measurement
>     IDENT=off: Ident lookups (RFC 931)
>     IPV6=on: IPv6 protocol support
>     KQUEUE=on: Kqueue(2) support
>     LARGEFILE=on: Support large (>2GB) cache and log files
>     LAX_HTTP=off: Do not enforce strict HTTP compliance
>     SNMP=off: SNMP support
>     SSL=on: SSL gatewaying support
>     SSL_CRTD=on: Use ssl_crtd to handle SSL cert requests
>     STACKTRACES=on: Enable automatic backtraces on fatal errors
>     TP_IPF=off: Transparent proxying with IPFilter
>     TP_IPFW=off: Transparent proxying with IPFW
>     TP_PF=on: Transparent proxying with PF
>     VIA_DB=off: Forward/Via database
>     WCCP=on: Web Cache Coordination Protocol
>     WCCPV2=on: Web Cache Coordination Protocol v2

It will also ask you about options for the various libraries needed by squid, and will compile them along the way. I didn't make notes of my answers for these, but they shouldn't be too hard to figure out. Usually, just use the defaults. Turn off documentation and debugging options, etc., but mostly the defaults. Depending on your hardware, this may take 20 minutes or so to finish compiling.

jimp

If you look in pkg_config.10.xml you'll notice for squid3 there are some build options made from variables similar to the ones you see there. If you (or someone else) can turn the list of yours into the build options style and submit a pull request, we can recompile it with the new options to see if it works.

aaronouthier

JimP,

I can try that, but as I was using the FreeBSD "ports" repo for testing, there is no ./configure file until after running make. In this scenario, "make install" is the only command needed, and that creates the "configure" program, adds the build options, configures, compiles, and links the program and all requisite libraries. This makes it easy and convenient to build, but not so easy to pass configure options to somebody else. I guess I'll try to build from official Squid source, but keep in mind, that that source doesn't have all of the FreeBSD 10 specific patches and compatibility testing which is included in the Ports repo.

Stay tuned.

jimp

The build options in the xml are the build options from freebsd ports, not configure. You can also see them in /var/db/ports/<portname>/options, where portname is xxx_yyy, xxx = the ports category, yyy being the port name, such as www_squid</portname>