I am now creating new DansGuardian and Squid3 binaries.

aaronouthier

Good grief!!

All of this time!

The reason that I couldn't connect to squid service on my box: I disabled IPv6 and blocked all v6 traffic on my network. Although I can't imagine why this would affect squid, well, it apparently did. I was beginning to wonder if there wasn't some obscure bug that prevented it from working on just my hardware type or some such thing.

The whole reason I set out to recompile this stuff, is because it didn't work for me…

Sigh. Still, I'm helping (hopefully) the pfSense community by getting squid updated. I don't think there will be too many complaints about that. I'm still in awe. It was so simple!

It's late, I'll see about publishing my work in the morning. It is now 20 minutes before 1 AM. Good night. everybody.

Cino

@Legion:

If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

a little off topic, but what do you mean? squidguard itself hasn't been updated in a couple of years, its 1.5 beta on their website…

rjcrowder

@aaronouthier:

Interesting. I'm not sure which version I had tried when I had squid working. However, I do recall that, although DansGuardian didn't report any problems, I was unable to get regex replacements working. I tried to use Google SafeSearch enforcement as a test. I couldn't ever seem to force safesearch on. I assumed this was a problem with that particular function of DG.

The PCRE stuff is an option on the compile. I think Marcello had at least one version out there that did not have it turned on… Unfortunately, I also used it for a while and couldn't get the safesearch to work (think I listed it in one of the DG threads). I found another version that works fine. You'll need to make sure you turn it on when you compile.

Would be very interested in getting e2guardian working if I could get a little help on it... I think he fixed some of the abend issues that were occuring.

aaronouthier

Ok, rather than doing PMs I'm going to post the tarballs and instructions here. It is against my better judgment, but in the interest of my time, etc…

Instructions (I hope I remember everything):

Copy the gzip'd tarballs onto your pfsense box or VM, etc. This is for amd64 distro only - for now.

cd /usr/pbi/squid-amd64/
```- you need to have squid-dev package installed and not previously modified or touched.

Make a note of the symlinks that go from squid-amd64 to squid-amd64/local . For example there is a symlink from /usr/pbi/squid-amd64/www that points to /usr/pbi/squid-amd64/local/www . Again, make a note of the names of these folders, you will need this later.

Carefully, very carefully, remove the aforementioned symlinks. Do an mv from those folders in the squid-amd64/local folder to the squid-amd64 folder - basically, you're moving the symlink destinations to the parent folder. example:```
cd /usr/pbi/squid-amd64/local/ ; mv www ../

Do that for each previous symlink. Only for the ones that were previously symlinked.

cd /usr/pbi/

Extract the attached gzip'd tarball for squid with```
tar xzvf /[path where you copied my attachment]/squid-amd64.tar.gz

cd /usr/pbi/squid-adm64/sbin

ldd squid


Make a note of any missing shared libraries and where it expects to find them. Copy (or possibly Symlink?) the mentioned libraries as required. you will find the libraries in /usr/pbi/squid-amd64/lib/

Fix the symlinks in /usr/local/bin and /usr/local/sbin to point to the correct locations. Some things have moved between MarcelloC's version and mine, specifically, some things have moved from squid-amd64/bin to sbin.

Try running squid now, but with the "-d 9" command-line switch for debugging info. Note any errors or problems. If you can't figure out how to get it to run, post a reply back here - I may have forgotten to mention something.

I couldn't seem to make dansguardian work this time around, but everyone's welcome to try it out themselves. It should be the same basic procedure. I have large file support turned on, so if you want a 10 GB cache, go for it!

Update: It won't let me attach the files - both are too big. I'll put them on my dropbox. Give me a few minutes.

aaronouthier

https://www.dropbox.com/sh/e6ys6w6oj82633x/AABPezSq-MYVWKy0SGZAdCCha?dl=0

This is for testing only. After feedback and critique, I'll try to make a package as suggested by jimp.

TTFN.

aaronouthier

Update: After checking things again, there are a few missing libraries. Look in the link above, and also grab the squid-libs.tar.gz

Extract the contents to /usr/lib/

If anyone is using a windows box to try to get these files onto a pfsense box, try http file server or "HFS". Its is free, and very convenient. You can then "fetch" the files from your windows machine to your pfSense system.

Legion

@Cino:

@Legion:

If you're inclined, it would be cool if you made sure squidguard was updated and worked with your squid-3.4 as well.

a little off topic, but what do you mean? squidguard itself hasn't been updated in a couple of years, its 1.5 beta on their website…

Yeah, I would hope that there would be no problem but I thought that maybe squidguard (and other packages) would have to be updated to their FreeBSD 10.x versions so while he was working with new Squid 3.4 he could test that the associated packages work as well. By the sounds of it dansguardian is the focus but I am using squidguard at the moment rather than DG since I found it more suitable for my needs.

aaronouthier

I had previously tried SquidGuard. Although it seemed to work, I need to be able to auto-update the blacklist, among other things. Even though DansGuardian isn't quite stable, it seemed to have more potential than SquidGuard, once finished.

I'd be happy to check out e2guardian, but one thing at a time.

Lastly, this is mostly directed at jimp, but anyone else interested is welcome to use as desired.

I have build instructions for building from source.

You will need to grab the 'ports' option/project/whatever for FreeBSD 10.1. This can be done either at the point of install, or post-installation, with the commands:

portsnap fetch
portsnap extract

Then, cd into the /usr/ports folder. cd further into the pkg-mgmt folder, and do amake install. If you already have the pkg binary installed, you will instead need to run```
make reinstall


Then, cd to /usr/ports/www/squid . Run```
make PREFIX=/usr/pbi/squid-[b][i]archetecture[/i][/b]/ install
```.

You will be presented with some dialog boxes the first time around, I used the following settings:

> ARP_ACL=on: ARP/MAC/EUI based authentification
>     AUTH_KERB=on: Install Kerberos authentication helpers
>     AUTH_LDAP=on: Install LDAP authentication helpers
>     AUTH_NIS=off: Install NIS/YP authentication helpers
>     AUTH_SASL=off: Install SASL authentication helpers
>     AUTH_SMB=off: Install SMB auth. helpers (req. Samba)
>     AUTH_SQL=off: Install SQL based auth (uses MySQL)
>     CACHE_DIGESTS=on: Use cache digests
>     DEBUG=off: Build with extended debugging support
>     DELAY_POOLS=on: Delay pools (bandwidth limiting)
>     DNS_HELPER=on: Use external dnsserver processes for DNS
>     DOCS=off: Build and/or install documentation
>     ECAP=off: Loadable content adaptation modules
>     ESI=on: ESI support
>     EXAMPLES=off: Build and/or install examples
>     FOLLOW_XFF=on: Support for the X-Following-For header
>     FS_AUFS=on: AUFS (threaded-io) support
>     FS_DISKD=on: DISKD storage engine controlled by separate service
>     FS_ROCK=off: ROCK (unstable)
>     HTCP=on: HTCP support
>     ICAP=on: the ICAP client
>     ICMP=on: ICMP pinging and network measurement
>     IDENT=off: Ident lookups (RFC 931)
>     IPV6=on: IPv6 protocol support
>     KQUEUE=on: Kqueue(2) support
>     LARGEFILE=on: Support large (>2GB) cache and log files
>     LAX_HTTP=off: Do not enforce strict HTTP compliance
>     SNMP=off: SNMP support
>     SSL=on: SSL gatewaying support
>     SSL_CRTD=on: Use ssl_crtd to handle SSL cert requests
>     STACKTRACES=on: Enable automatic backtraces on fatal errors
>     TP_IPF=off: Transparent proxying with IPFilter
>     TP_IPFW=off: Transparent proxying with IPFW
>     TP_PF=on: Transparent proxying with PF
>     VIA_DB=off: Forward/Via database
>     WCCP=on: Web Cache Coordination Protocol
>     WCCPV2=on: Web Cache Coordination Protocol v2

It will also ask you about options for the various libraries needed by squid, and will compile them along the way. I didn't make notes of my answers for these, but they shouldn't be too hard to figure out. Usually, just use the defaults. Turn off documentation and debugging options, etc., but mostly the defaults. Depending on your hardware, this may take 20 minutes or so to finish compiling.

jimp

If you look in pkg_config.10.xml you'll notice for squid3 there are some build options made from variables similar to the ones you see there. If you (or someone else) can turn the list of yours into the build options style and submit a pull request, we can recompile it with the new options to see if it works.

aaronouthier

JimP,

I can try that, but as I was using the FreeBSD "ports" repo for testing, there is no ./configure file until after running make. In this scenario, "make install" is the only command needed, and that creates the "configure" program, adds the build options, configures, compiles, and links the program and all requisite libraries. This makes it easy and convenient to build, but not so easy to pass configure options to somebody else. I guess I'll try to build from official Squid source, but keep in mind, that that source doesn't have all of the FreeBSD 10 specific patches and compatibility testing which is included in the Ports repo.

Stay tuned.

jimp

The build options in the xml are the build options from freebsd ports, not configure. You can also see them in /var/db/ports/<portname>/options, where portname is xxx_yyy, xxx = the ports category, yyy being the port name, such as www_squid</portname>

aaronouthier

Interesting. Good to know! I'll check that in the morning.

aaronouthier

Sorry for the silence this weekend. My computer was having video issues on Friday. On Saturday, I made some new VMs, based on FreeBSD 10.2 RC1, instead of Beta 3 as before. Build process is erroring-out on glib20, while running "make" in "/usr/ports/ports-mgmt/". I am unable to compile and install the updated "pkg" binary, which is required by the rest of the ports repo. This is occurring in both amd64 and i386 versions. I'll update "ports" tomorrow and try again, to see if this is fixed.

Thank you for your continued patience.

jimp

pkg can be a little odd to upgrade sometimes, but it usually prints instructions about it when it fails.

try this:

pkg delete -f pkg; cd /usr/ports/ports-mgmt/pkg; env UPGRADEPKG=1 make clean install clean

aaronouthier

@Jimp: No, I don't think that's it. I am having no end of headaches with ports on RC1. Things that worked without issues. Squid, Avahi, pkg, and more are constantly getting stuck while building dependencies. It is not happening during install phase. Also, the same errors are occurring in the same places on both i386 and amd64 builds. Everything from "aclocal 1.13 missing", to missing dependencies during configure, because they are supposed to be compiled ahead of time, but aren't. None of this was happening with beta 3. The fix has always been to look at the logs and figure out where the process is breaking down and cd to that location and "make clean && make install". Compiling squid alone took more than 3 hours, and I had to manually build about 20 different dependencies. Every time I'd fix one thing, something else further down the line would fail.

I then spend just under 4 hours on Avahi before calling it a night. Going to try to finish up in a few minutes.

All of this was with a fresh, clean install of freebsd 10.1 - not an upgrade or install over-the-top of the existing.

jimp

Strange, I haven't used an RC yet myself, my workstation is on a late beta, but there were a ton of updates to ports over the last couple weeks.

When all else fails, pkg delete -fa, and start over with a fresh ports tree (portsnap fetch extract)

aaronouthier

I've been trying to resolve them manually. That said, I just did a portsnap fetch and portsnap update, followed closely behind by a few _make distclean_s. Portsnap fetch grabbed over 200 patches to my existing tree. Crossing my fingers…

aaronouthier

Yeah, going to start with a new ports install. Running rm -rf /usr/ports now. Just ran pkg delete -fa a moment ago. Thanks for that, by the way. I didn't previously know I could force remove all pkgs! I'm assuming that's what is meant by -fa (force all).

aaronouthier

I'm back on the case.

FreeBSD was updated to RC2 on Friday. I had some things going over the weekend. It's now about midnight Monday morning. GTG.

dooldeniya

Aaron - Thanks for working on this. Any update on progress. I am looking to install and would appreciate any guidance you can provide.
-Chanaka