I am now creating new DansGuardian and Squid3 binaries.



  • Brief synopsis:

    DansGuardian was working for me, but RegEx didn't seem to work properly. I have already recompiled with RegEx support. It took a bit of work to get libpcre working, but I now have the binary functioning I have not tested it very thoroughly - only tested that the binary runs.

    Squid3-dev did not work properly, even though it seemed to run. Binary executable would exit with a missing shared library.

    Nitty-gritty: Both packages were being "configure"d to use the default PREFIX path. This didn't work correctly. I am working on remedying this.

    Stay tuned. Check my previous thread for info, or if you wish to help test my version.


  • Rebel Alliance Developer Netgate

    Make the build tags right in the xml and we can build it. We won't host binaries we didn't build ourselves, for security (and reproduceability) reasons. If you can get it building right using the tags in the XML to make the PBI, we can compile them here. If it takes special handling, document every part of it and we'll see what can be done.



  • @jimp:

    Make the build tags right in the xml and we can build it. We won't host binaries we didn't build ourselves, for security (and reproduceability) reasons. If you can get it building right using the tags in the XML to make the PBI, we can compile them here. If it takes special handling, document every part of it and we'll see what can be done.

    Right!

    I could almost follow that conversation. I've done html and even xml coding in the past - albeit several years ago. I am completely in the dark, however, about building a pbi file. Most of my experience with this kind of stuff revolves around linux tinkering and maintenance. FreeBSD isn't so hard coming from that background, but it's not as familiar to me as linux, or Windows for that matter…

    If someone could point me in the right direction as to learning the basics of a pbi file/package. I would imagine it is actually some other file type that is just renamed, such as a compressed tar file, or similar - is this correct? Is there a sample pbi file or a template somewhere? I should be able to figure everything out, but I could use a push in the right direction...

    Thanks!



  • I could be wrong but I think this is what he is referring too

    https://github.com/pfsense/pfsense-packages

    pkg_config.10.xml
    pkg_config.8.xml
    pkg_config.8.xml.amd64

    from pkg_config.8

    
    	 <package><name>squid3</name>
    <internal_name>squid</internal_name>
     <descr>It combines squid as a proxy server with it's capabilities of acting as a HTTP / HTTPS reverse proxy.
    
    It includes an Exchange-Web-Access (OWA) Assistant.]]></descr>
    <pkginfolink>https://forum.pfsense.org/index.php/topic,48347.0.html</pkginfolink>
    <website>http://www.squid-cache.org/</website>
    <category>Network</category>
    <version>3.1.20 pkg 2.1.1</version>
    <status>beta</status>
    <required_version>2.0</required_version>
    <maintainer>marcellocoutinho@gmail.com fernando@netfilter.com.br seth.mos@dds.nl mfuchs77@googlemail.com jimp@pfsense.org</maintainer>
    <depends_on_package_base_url>https://files.pfsense.org/packages/8/All/</depends_on_package_base_url>
    <depends_on_package>squid-3.1.20.tbz</depends_on_package>
    <depends_on_package>libwww-5.4.0_4.tbz</depends_on_package>
     <build_pbi><ports_before>www/libwww</ports_before>
    <port>www/squid31</port>
    <ports_after>www/squid_radius_auth</ports_after></build_pbi> 
    <build_options>c-icap_UNSET_FORCE=IPV6;squid_UNSET_FORCE=AUTH_SMB AUTH_SQL DNS_HELPER FS_COSS ESI SNMP ECAP STACKTRACES STRICT_HTTP TP_IPF TP_IPFW VIA_DB DEBUG DOCS EXAMPLES;squid_SET=ARP_ACL AUTH_KERB AUTH_LDAP AUTH_NIS AUTH_SASL CACHE_DIGESTS DELAY_POOLS FOLLOW_XFF TP_PF MSSL_CRTD WCCP WCCPV2 FS_AUFS HTCP ICAP ICMP IDENT IPV6 KQUEUE LARGEFILE SSL SSL_CRTD</build_options>
    
    <config_file>https://packages.pfsense.org/packages/config/squid3/31/squid.xml</config_file>
    <configurationfile>squid.xml</configurationfile>
    <depends_on_package_pbi>squid-3.1.22_1-i386.pbi</depends_on_package_pbi></package> 
     <package><name>squid3-dev</name>
    <internal_name>squid</internal_name>
     <descr>It combines squid as a proxy server with it's capabilities of acting as a HTTP / HTTPS reverse proxy.
    
    It includes an Exchange-Web-Access (OWA) Assistant, ssl filtering and antivirus integration via i-cap]]></descr>
    <pkginfolink>https://forum.pfsense.org/index.php/topic,48347.0.html</pkginfolink>
    <website>http://www.squid-cache.org/</website>
    <category>Network</category>
    <version>3.3.10 pkg 2.2.6</version>
    <status>beta</status>
    <required_version>2.0</required_version>
    <maintainer>marcellocoutinho@gmail.com fernando@netfilter.com.br seth.mos@dds.nl mfuchs77@googlemail.com jimp@pfsense.org</maintainer>
    <depends_on_package_base_url>https://files.pfsense.org/packages/8/All/</depends_on_package_base_url>
    <depends_on_package>squid-3.3.5.tbz</depends_on_package>
    <depends_on_package>libltdl-2.4.2.tbz</depends_on_package>
    <depends_on_package>libwww-5.4.0_4.tbz</depends_on_package>
    <depends_on_package>squidclamav-6.10_1.tbz</depends_on_package>
    <depends_on_package>clamav-0.97.8.tbz</depends_on_package>
    <depends_on_package>cyrus-sasl-2.1.26_2.tbz</depends_on_package>
    <depends_on_package>ca_root_nss-3.14.1.tbz</depends_on_package>
     <build_pbi><ports_before>www/libwww security/cyrus-sasl2</ports_before>
    <port>www/squid33</port>
    <ports_after>www/squid_radius_auth security/clamav www/squidclamav security/ca_root_nss www/c-icap-modules</ports_after></build_pbi> 
    <build_options>c-icap_UNSET_FORCE=IPV6;squid_UNSET_FORCE=AUTH_SMB AUTH_SQL DNS_HELPER FS_COSS ESI SNMP ECAP STACKTRACES STRICT_HTTP TP_IPF TP_IPFW VIA_DB DEBUG DOCS EXAMPLES AUTH_SASL;squid_SET=ARP_ACL AUTH_KERB AUTH_LDAP AUTH_NIS CACHE_DIGESTS DELAY_POOLS FOLLOW_XFF TP_PF MSSL_CRTD WCCP WCCPV2 FS_AUFS HTCP ICAP ICMP IDENT IPV6 KQUEUE LARGEFILE SSL SSL_CRTD</build_options>
    <config_file>https://packages.pfsense.org/packages/config/squid3/33/squid.xml</config_file>
    <configurationfile>squid.xml</configurationfile>
    <depends_on_package_pbi>squid-3.3.10-i386.pbi</depends_on_package_pbi></package> 
    
    


  • Thanks Cino,

    I'll have a look. Still trying to recompile squid. I had some setbacks. I'll keep everyone posted.



  • @aaronouthier:

    Brief synopsis:

    DansGuardian was working for me, but RegEx didn't seem to work properly. I have already recompiled with RegEx support. It took a bit of work to get libpcre working, but I now have the binary functioning I have not tested it very thoroughly - only tested that the binary runs.

    Squid3-dev did not work properly, even though it seemed to run. Binary executable would exit with a missing shared library.

    Nitty-gritty: Both packages were being "configure"d to use the default PREFIX path. This didn't work correctly. I am working on remedying this.

    Stay tuned. Check my previous thread for info, or if you wish to help test my version.

    Just FYI… I can't remember which version, but only one of the versions had a problem with PCRE... I've  been overlaying the .exe with a version off of Marcello's site (out of town - could get you the version on Thursday) and it works fine.



  • Also… if you are going to the trouble of recompiling, it would be nice to just switch to the e2guardian application. I'd be willing to work on the updates to the package if you are interested in helping with it... I think it is mostly directory name changes. I got started on it at one point, but never finished...



  • Interesting. I'm not sure which version I had tried when I had squid working. However, I do recall that, although DansGuardian didn't report any problems, I was unable to get regex replacements working. I tried to use Google SafeSearch enforcement as a test. I couldn't ever seem to force safesearch on. I assumed this was a problem with that particular function of DG. Everything else seemed to work. As I was having many troubles getting squid proxy to work, so DG has no upstream proxy to which to connect. Hence, the reason for all of this trouble.

    Also, I have already recompiled DansGuardian (DG) binaries, and have copied them to my pfSense test system, overwriting the original stuff.

    Overall status update: Almost finished, I hope! I have compiled Squid 3.4 from source using the ports feature of FreeBSD 10.1. Almost have it running. It looks like I still have some symlinks to create.



  • If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.



  • Good grief!!

    All of this time!

    The reason that I couldn't connect to squid service on my box: I disabled IPv6 and blocked all v6 traffic on my network. Although I can't imagine why this would affect squid, well, it apparently did. I was beginning to wonder if there wasn't some obscure bug that prevented it from working on just my hardware type or some such thing.

    The whole reason I set out to recompile this stuff, is because it didn't work for me…

    Sigh. Still, I'm helping (hopefully) the pfSense community by getting squid updated. I don't think there will be too many complaints about that. I'm still in awe. It was so simple!

    It's late, I'll see about publishing my work in the morning. It is now 20 minutes before 1 AM. Good night. everybody.



  • @Legion:

    If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

    a little off topic, but what do you mean? squidguard itself hasn't been updated in a couple of years, its 1.5 beta on their website…



  • @aaronouthier:

    Interesting. I'm not sure which version I had tried when I had squid working. However, I do recall that, although DansGuardian didn't report any problems, I was unable to get regex replacements working. I tried to use Google SafeSearch enforcement as a test. I couldn't ever seem to force safesearch on. I assumed this was a problem with that particular function of DG.

    The PCRE stuff is an option on the compile. I think Marcello had at least one version out there that did not have it turned on… Unfortunately, I also used it for a while and couldn't get the safesearch to work (think I listed it in one of the DG threads). I found another version that works fine.  You'll need to make sure you turn it on when you compile.

    Would be very interested in getting e2guardian working if I could get a little help on it... I think he fixed some of the abend issues that were occuring.



  • Ok, rather than doing PMs I'm going to post the tarballs and instructions here. It is against my better judgment, but in the interest of my time, etc…

    Instructions (I hope I remember everything):

    Copy the gzip'd tarballs onto your pfsense box or VM, etc. This is for amd64 distro only - for now.

    cd /usr/pbi/squid-amd64/
    ```- you need to have squid-dev package installed and not previously modified or touched.
    
    Make a note of the symlinks that go from squid-amd64 to squid-amd64/local . For example there is a symlink from /usr/pbi/squid-amd64/www that points to /usr/pbi/squid-amd64/local/www . Again, make a note of the names of these folders, you will need this later.
    
    Carefully, very carefully, remove the aforementioned symlinks. Do an mv from those folders in the squid-amd64/local folder to the squid-amd64 folder - basically, you're moving the symlink destinations to the parent folder. example:```
    cd /usr/pbi/squid-amd64/local/ ; mv www ../
    

    Do that for each previous symlink. Only for the ones that were previously symlinked.

    cd /usr/pbi/
    

    Extract the attached gzip'd tarball for squid with```
    tar xzvf /[path where you copied my attachment]/squid-amd64.tar.gz

    
    

    cd /usr/pbi/squid-adm64/sbin

    
    

    ldd squid

    
    Make a note of any missing shared libraries and where it expects to find them. Copy (or possibly Symlink?) the mentioned libraries as required. you will find the libraries in /usr/pbi/squid-amd64/lib/
    
    Fix the symlinks in /usr/local/bin and /usr/local/sbin to point to the correct locations. Some things have moved between MarcelloC's version and mine, specifically, some things have moved from squid-amd64/bin to sbin.
    
    Try running squid now, but with the "-d 9" command-line switch for debugging info. Note any errors or problems. If you can't figure out how to get it to run, post a reply back here - I may have forgotten to mention something.
    
    I couldn't seem to make dansguardian work this time around, but everyone's welcome to try it out themselves. It should be the same basic procedure. I have large file support turned on, so if you want a 10 GB cache, go for it!
    
    Update: It won't let me attach the files - both are too big. I'll put them on my dropbox. Give me a few minutes.


  • https://www.dropbox.com/sh/e6ys6w6oj82633x/AABPezSq-MYVWKy0SGZAdCCha?dl=0

    This is for testing only. After feedback and critique, I'll try to make a package as suggested by jimp.

    TTFN.



  • Update: After checking things again, there are a few missing libraries. Look in the link above, and also grab the squid-libs.tar.gz

    Extract the contents to /usr/lib/

    If anyone is using a windows box to try to get these files onto a pfsense box, try http file server or "HFS". Its is free, and very convenient. You can then "fetch" the files from your windows machine to your pfSense system.



  • @Cino:

    @Legion:

    If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

    a little off topic, but what do you mean? squidguard itself hasn't been updated in a couple of years, its 1.5 beta on their website…

    Yeah, I would hope that there would be no problem but I thought that maybe squidguard (and other packages) would have to be updated to their FreeBSD 10.x versions so while he was working with new Squid 3.4 he could test that the associated packages work as well. By the sounds of it dansguardian is the focus but I am using squidguard at the moment rather than DG since I found it more suitable for my needs.



  • I had previously tried SquidGuard. Although it seemed to work, I need to be able to auto-update the blacklist, among other things. Even though DansGuardian isn't quite stable, it seemed to have more potential than SquidGuard, once finished.

    I'd be happy to check out e2guardian, but one thing at a time.

    Lastly, this is mostly directed at jimp, but anyone else interested is welcome to use as desired.

    I have build instructions for building from source.

    You will need to grab the 'ports' option/project/whatever for FreeBSD 10.1. This can be done either at the point of install, or post-installation, with the commands:

    portsnap fetch
    portsnap extract
    

    Then, cd into the /usr/ports folder. cd further into the pkg-mgmt folder, and do amake install. If you already have the pkg binary installed, you will instead need to run```
    make reinstall

    
    Then, cd to /usr/ports/www/squid . Run```
    make PREFIX=/usr/pbi/squid-[b][i]archetecture[/i][/b]/ install
    ```.
    
    You will be presented with some dialog boxes the first time around, I used the following settings:
    
    > ARP_ACL=on: ARP/MAC/EUI based authentification
    >     AUTH_KERB=on: Install Kerberos authentication helpers
    >     AUTH_LDAP=on: Install LDAP authentication helpers
    >     AUTH_NIS=off: Install NIS/YP authentication helpers
    >     AUTH_SASL=off: Install SASL authentication helpers
    >     AUTH_SMB=off: Install SMB auth. helpers (req. Samba)
    >     AUTH_SQL=off: Install SQL based auth (uses MySQL)
    >     CACHE_DIGESTS=on: Use cache digests
    >     DEBUG=off: Build with extended debugging support
    >     DELAY_POOLS=on: Delay pools (bandwidth limiting)
    >     DNS_HELPER=on: Use external dnsserver processes for DNS
    >     DOCS=off: Build and/or install documentation
    >     ECAP=off: Loadable content adaptation modules
    >     ESI=on: ESI support
    >     EXAMPLES=off: Build and/or install examples
    >     FOLLOW_XFF=on: Support for the X-Following-For header
    >     FS_AUFS=on: AUFS (threaded-io) support
    >     FS_DISKD=on: DISKD storage engine controlled by separate service
    >     FS_ROCK=off: ROCK (unstable)
    >     HTCP=on: HTCP support
    >     ICAP=on: the ICAP client
    >     ICMP=on: ICMP pinging and network measurement
    >     IDENT=off: Ident lookups (RFC 931)
    >     IPV6=on: IPv6 protocol support
    >     KQUEUE=on: Kqueue(2) support
    >     LARGEFILE=on: Support large (>2GB) cache and log files
    >     LAX_HTTP=off: Do not enforce strict HTTP compliance
    >     SNMP=off: SNMP support
    >     SSL=on: SSL gatewaying support
    >     SSL_CRTD=on: Use ssl_crtd to handle SSL cert requests
    >     STACKTRACES=on: Enable automatic backtraces on fatal errors
    >     TP_IPF=off: Transparent proxying with IPFilter
    >     TP_IPFW=off: Transparent proxying with IPFW
    >     TP_PF=on: Transparent proxying with PF
    >     VIA_DB=off: Forward/Via database
    >     WCCP=on: Web Cache Coordination Protocol
    >     WCCPV2=on: Web Cache Coordination Protocol v2
    
    It will also ask you about options for the various libraries needed by squid, and will compile them along the way. I didn't make notes of my answers for these, but they shouldn't be too hard to figure out. Usually, just use the defaults. Turn off documentation and debugging options, etc., but mostly the defaults. Depending on your hardware, this may take 20 minutes or so to finish compiling.

  • Rebel Alliance Developer Netgate

    If you look in pkg_config.10.xml you'll notice for squid3 there are some build options made from variables similar to the ones you see there. If you (or someone else) can turn the list of yours into the build options style and submit a pull request, we can recompile it with the new options to see if it works.



  • JimP,

    I can try that, but as I was using the FreeBSD "ports" repo for testing, there is no ./configure file until after running make. In this scenario, "make install" is the only command needed, and that creates the "configure" program, adds the build options, configures, compiles, and links the program and all requisite libraries. This makes it easy and convenient to build, but not so easy to pass configure options to somebody else. I guess I'll try to build from official Squid source, but keep in mind, that that source doesn't have all of the FreeBSD 10 specific patches and compatibility testing which is included in the Ports repo.

    Stay tuned.


  • Rebel Alliance Developer Netgate

    The build options in the xml are the build options from freebsd ports, not configure. You can also see them in /var/db/ports/<portname>/options, where portname is xxx_yyy, xxx = the ports category, yyy being the port name, such as www_squid</portname>



  • Interesting. Good to know! I'll check that in the morning.



  • Sorry for the silence this weekend. My computer was having video issues on Friday. On Saturday, I made some new VMs, based on FreeBSD 10.2 RC1, instead of Beta 3 as before. Build process is erroring-out on glib20, while running "make" in "/usr/ports/ports-mgmt/". I am unable to compile and install the updated "pkg" binary, which is required by the rest of the ports repo. This is occurring in both amd64 and i386 versions. I'll update "ports" tomorrow and try again, to see if this is fixed.

    Thank you for your continued patience.


  • Rebel Alliance Developer Netgate

    pkg can be a little odd to upgrade sometimes, but it usually prints instructions about it when it fails.

    try this:

    pkg delete -f pkg; cd /usr/ports/ports-mgmt/pkg; env UPGRADEPKG=1 make clean install clean
    


  • @Jimp: No, I don't think that's it. I am having no end of headaches with ports on RC1. Things that worked without issues. Squid, Avahi, pkg, and more are constantly getting stuck while building dependencies. It is not happening during install phase. Also, the same errors are occurring in the same places on both i386 and amd64 builds. Everything from "aclocal 1.13 missing", to missing dependencies during configure, because they are supposed to be compiled ahead of time, but aren't. None of this was happening with beta 3. The fix has always been to look at the logs and figure out where the process is breaking down and cd to that location and "make clean && make install". Compiling squid alone took more than 3 hours, and I had to manually build about 20 different dependencies. Every time I'd fix one thing, something else further down the line would fail.

    I then spend just under 4 hours on Avahi before calling it a night. Going to try to finish up in a few minutes.

    All of this was with a fresh, clean install of freebsd 10.1 - not an upgrade or install over-the-top of the existing.


  • Rebel Alliance Developer Netgate

    Strange, I haven't used an RC yet myself, my workstation is on a late beta, but there were a ton of updates to ports over the last couple weeks.

    When all else fails, pkg delete -fa, and start over with a fresh ports tree (portsnap fetch extract)



  • I've been trying to resolve them manually. That said, I just did a portsnap fetch and portsnap update, followed closely behind by a few _make distclean_s. Portsnap fetch grabbed over 200 patches to my existing tree. Crossing my fingers…



  • Yeah, going to start with a new ports install. Running rm -rf /usr/ports now. Just ran pkg delete -fa a moment ago. Thanks for that, by the way. I didn't previously know I could force remove all pkgs! I'm assuming that's what is meant by -fa (force all).



  • I'm back on the case.

    FreeBSD was updated to RC2 on Friday. I had some things going over the weekend. It's now about midnight Monday morning. GTG.



  • Aaron - Thanks for working on this.  Any update on progress.  I am looking to install and would appreciate any guidance you can provide.
    -Chanaka



  • Actually, I've stopped working on this, as the original maintainer of squid and DansGuardian, namely user MarcelloC, managed to find the time to update them about a month ago. I assume you're having trouble? If so, you're in the right place… (Pfsense forums).



  • @aaronouthier:

    Actually, I've stopped working on this, as the original maintainer of squid and DansGuardian, namely user MarcelloC, managed to find the time to update them about a month ago. I assume you're having trouble? If so, you're in the right place… (Pfsense forums).

    That's awesome news… so now the normal Squid3 and DG packages should work under 2.2?



  • Do you know where he updated them too?

    I checked github for php/inc file changes: squid3 hasn't been updated in 2 months, Dansguardian 5 months.

    binary changes:
    Checked http://files.pfsense.org/packages/8/All/
    dansguardian-2.12.0.3_2-i386.pbi                  23-Jun-2014 13:57            19952423
    squid-3.3.10-i386.pbi                              26-Nov-2013 20:06            17598644

    Checked http://files.pfsense.org/packages/10/All/
    dansguardian-2.12.0.3_2-i386.pbi                  27-Jun-2014 03:42            16177170
    squid-3.3.11-i386.pbi                              22-Apr-2014 12:12            17568448
    squid-3.3.11_1-i386.pbi                            17-Jul-2014 22:26            17702572

    amd64 pbi have the same dates



  • Ok. I did some more checking, and, now I'm not sure who updated it, or when. I just know that around October 15 or so, I reinstalled my box, and everything worked, whereas a fresh install previously didn't work right without some modifications.

    Also, note that I am using squid3-dev, not regular squid3, and I am running it on the 2.2 beta, not the 2.1.x stable.



  • I have a fresh install of 2.2-BETA (amd64) built on Mon Oct 27 15:31:41 CDT 2014 FreeBSD 10.1-RC3
    If I install squid3-dev beta 3.3.11_1 pkg 2.2.7 platform: 2.2 - I've never managed to start it.

    On the previous install, I tried installing libraries it complained were missing ect. to see if I could get it up,
    but eventually I gave up, and reinstalled from scratch.

    Would you mind sharing which versions you're running?

    Thanks in advanced.



  • More or less the same situation here, I'm running 2.2 beta snapshot and tried to install Squid 3.3.11_1 pkg 2.2.7, it wouldn't start.
    I used the workaround described elsewhere on this forum, and now it runs.
    Downside is that the "workaround" (console commandos) have to be entered again after each update.

    So it's either "do not update" or "workaround".

    For my purposes, Squid proxy (and if possible with ad blocking) is really a must-have. 
    I'm not a programmer, but can test packages if needed.
    Please keep up the effort.

    Cheers.



  • Hmm. Not sure what happened. I just did a fresh reinstall myself. Squid now segfaults upon launch with core dump. This is with the official versions of everything. Nothing was custom-compiled or copied from another box. Amd64 build. I don't know what to say.



  • Exactly.
    It did that on my box also, but it turned out to be the cache filesystem.
    If you set it to "aufs", Squid will complain.
    Leaving it at "ufs" (default) and it runs.

    I have found a way to block ads with the help of a regex list added to Squid, and that works fine.
    So for now, all is dandy. No updates though.

    Cheers.



  • Well, I am having some frustrating issues. I have gotten squid to compile just fine, but when I do amake install, it hangs with a series of```
    lstat: file not found



  • Hmm I'm not familiar with the process, wish I could help in some way.
    Take your time.

    Cheers.



  • Today, a new Squid package was made available.
    As I was feeling adventurous, decided to hit the pkg button and….
    Installed just fine, all configs retained. No errors.

    I haven't tried to update the beta snapshot yet, better wait until it gets final.

    Cheers.