Shipping Proxy access.log and cache.log to ELK stack over syslog

LeeArchinal · Sep 23, 2020, 1:18 PM

Good day everyone,

I am trying to get my squid access.log and cache.log to my ELK stack. I already have my system logs shipping over port 514 to my stack and I can see the logs. I am now trying to find where to configure my squid proxy to ship the logs over the same port. I accessed the pfsense through Putty, opened a shell and inspected the /squid.conf file and it stated "Do not edit manually". So my next idea is that some configuration needs to be added into the advanced configuration options.

Please help!

stephenw10 · Sep 23, 2020, 2:40 PM

Squid can send it's access logs directly. For example put this line in the Custom Options (Before Auth) field:

access_log udp://172.21.16.12:514

I have never tried sending the cache log but it's entirely possible. Check the Squid man pages.

Steve

LeeArchinal · Sep 23, 2020, 2:54 PM

@stephenw10 said in Shipping Proxy access.log and cache.log to ELK stack over syslog:

access_log udp://172.21.16.12:514

Thank you very much. I have added my statement to the custom options (Before Auth) and I restarted the service. I also verified that those statements made it to the squid.conf. Is a full reboot necessary?

stephenw10 · Sep 23, 2020, 3:02 PM

No it should not be. Squid should restart when you make that change and pull in the new config.

Steve

LeeArchinal · Sep 24, 2020, 4:22 PM

@stephenw10
Do you know how I can verify that I am receiving logs on my remote server? It is an Ubuntu 18.04 with filebeat running on it. I am still a novice at linux and cannot find the correct command.

stephenw10 · Sep 24, 2020, 4:30 PM

Not really, I have never used that. There's no way to tell from pfSense since it's UDP.

I would expect the log entries to be labelled something pretty obvious though. At least the time stamps should march so anything you see in the real-time log in pfSense should appear at that same stamp in the syslog server.

Steve

LeeArchinal · Sep 24, 2020, 5:02 PM

@stephenw10
Can I change to tcp?

stephenw10 · Sep 24, 2020, 8:21 PM

Yes, I believe Squid will do it if the syslog server can accept it.
It would have to do both though since the main pfSense syslogging is UDP only. Or you'd have to go via something else like syslog-ng as a relay.

Steve

LeeArchinal · Sep 24, 2020, 9:07 PM

@stephenw10
From the custom options can I tell the logs where I want them to be on my remote system? For example:
access_log udp://x.x.x.x:514 /var/log/syslog/squid.*

LeeArchinal · Sep 24, 2020, 9:17 PM

Disregard. So this statement did not work:
access_log udp://x.x.x.x:514

But this one did:
access_log syslog:local5.info squid

Not sure why but I am now getting them in my ELK Stack. Next question I have may not be yours to answer but I need to parse them.

stephenw10 · Sep 24, 2020, 9:23 PM

Hmm, interesting. I have always used the udp module there. Tested receiving in syslog-ng.

http://www.squid-cache.org/Versions/v4/cfgman/access_log.html

Is it just sending to the local system log with that and then being sent to the syslog server from there maybe?

Yeah, can't really help with parsing Squid logs ELK but it's probably quite common so I would expect guides/code to be available.

Steve

LeeArchinal · Sep 24, 2020, 9:39 PM

I have filebeat listening on port 514 not syslog-ng. That could be the big difference.

stephenw10 · Sep 24, 2020, 11:44 PM

Yes. Filebeat is not directly a syslog server as far as I can see. You have to configure it with the syslog input module: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-syslog.html
And possibly some other config there. As I say I've never used it.

Steve