Need help understanding logging options, esp with proxy related packages



  • Hello.

    EDIT: forgot to mention, using v2.0.2 on 2ghz machine dedicated as router, with dsl connection and only two intel nics (wan and lan).

    So I am a pc enthusiast, which is why I like messing with pfsense for my home network. I don't know much linux related stuff, but can manage if given instructions.

    I have two questions here. One is more complex and one is easier. I will start with the complex question.

    I have three kids who range from 8 to 15. They each have thier own machine. I also have various other machines in the house, mostly my machines, as well as the periodic guest machines. I use static dhcp mapping. I use squid and squid guard. I developed 3 ACLs in SG that start with a few specific IPs with few restrictions, to a CIDR of 27 for my or guests machines with only some blacklists and finally a CIDR of 24 for everything else, which is whitelist only. This works well. I use Norton DNS servers and have iptable rules in place to deny any port 53 other than Norton ones and to deny ports 80 and 443 so users must use the proxy port. I have exclusions in place of course, but the overall picture is that my kids and guests must use the proxy and norton dns servers, which lets me have some measure of control over the content that they get.

    The problem I have is this is a somewhat labor intensive method, maintaining the whitelists. I don't mind it, but it is not really easy to see what is going on. Normally I would install a firewall on the client, like outpost, and when the kids are trying to do something that fails, I would see what is happing - what URL is trying to be accessed, etc.

    In PFSense I am wondering, what would the best method be to log, only for a given time period like say 5 minutes, the connection attempts. So that I can see, where is that game trying to connect, what ports and protocols is it using. I realize that a client firewall would show processes, and pfsense will not, only src and dest addresses and such, but surely there must be a way.

    Now, the second question sort of fits in with the first question, maybe.

    I have used lightsquid, and like the ability to periodically see just what has been allowed. I also used Sarg, although it is fickle to get to work. That looks like it has the ability to show me what was denied, presuming with squid guard.

    So, how do the logs at my disposal of either pfsense itself or squid and squid guard work? What I mean is, if I want to capture only a weeks worth of overall info to look at, how do the rotate and replace options work exactly? And what if I want to do more of a diagnostic report when something is not connecting. Are there options available that will rotate to a new log file when desired, or do I have to delete or rename existing log files?

    In short, I can enable logging, that is no problem. But I don't find any implicit information on how the rototate or replace/overwrite logs will affect things, much less what logs or packages I might use to track down what is being denied by squid/squidguard.

    On a side note, I do understand that each iptable entry allows me to log it and I can tell when an iptable entry is causing the problem. But I think it is always squid guard that is blocking, which is fine because I am whitelisting, but need to understand how all the logging options work to understand how to keep from having gigabytes of log files. I don't need that, only a week time period maximum.

    I hope this makes sense. I don't yet know all the terminology to easily explain what I am after.

    Thank you to anyone who can help.



  • To add to my plea for help ;)

    By viewing the squidguard log (block log) from the GUI I can see when my "kids_acl" is blocking something. Then I can add an entry to the whitelist and apply.

    By viewing the squid log (access log) I can see where anyone using the proxy went. Most of the entries though are links on pages. For example going to minecraft website will put many URLs in the squid log that weren't actually entered by users.

    I have lightsquid working and SARG. Both show similar results, which is a display of what addresses were requested and what IP did the requesting, along with different stats. I am unable to get SARG to show me when an ACL blocks a site for squidguard.

    So, let me shift my approach and find if anyone has any ideas.

    If there is no way for lightsquid or SARG to determine what URLs are the originators (that is, what was typed into the address bar of a browser) and what are children of the originator (all those other addresses a site requests to load itself) then I don't need those tools except on a periodic and specific basis when I want to troubleshoot or just want an overview. I'll likely never know what addresses were intended by any machine without looking the list over carefully to find sites I know would have been the originator.

    So if those squid access logs are not giving me a view of what the machines intended to do, its up to squidguard to tell me at least what was blocked.

    Other than SARG, is there a tool available which will parse the squidguard log and display the blocked connections? SARG has that option, but no matter how I configure it, I don't see any blocked or denied entries at all, even though they exist in the actual log.

    If there is no package or tool that does what I want, is there a way I can create a shared resource on the pfsense box that allows my machine (windows 7) to access (download) the squidguard block log? I'm pretty experienced in windows networking, but not unix bases stuff. I assume a shell script run by cron could copy the block.log somewhere, but can I share it? Wouldn't I need samba?

    I ask because I could easily write a script that could parse the block.log file myself. I only want to see what is causing problems. For example, I could leave the logging off of both squid and squidguard, and only enable squidguard logging when one of my kids encounters a problem. Right now I can use file manager package to download the file to my pc, but I would like to automate it.

    Anyone have any thoughts on this?

    Or, would it be more advantageous for me to learn some new script language that works on the pfsense box that I can build my own output to do this very specific feature? Not opposed to doing that, but would need some idea of what I would need to research to do it. Such as some php or perl or unix shell commands, etc.



  • Is there a config file for the squidguard (proxy filter) page that I can modify how many blocked entries are displayed in the GUI? It limits it to 50 by default. The file squidguard_configurator.inc has a define statement that sets max log lines at 500. Is this meaning max gui entries or max entries to the block.log file? I cannot seem to find the file that has this value in it.

    Anyone know the location of the file, if it exists and this is possible?