Shipping Proxy access.log and cache.log to ELK stack over syslog

stephenw10 · Sep 23, 2020, 3:02 PM

No it should not be. Squid should restart when you make that change and pull in the new config.

Steve

LeeArchinal · Sep 24, 2020, 4:22 PM

@stephenw10
Do you know how I can verify that I am receiving logs on my remote server? It is an Ubuntu 18.04 with filebeat running on it. I am still a novice at linux and cannot find the correct command.

stephenw10 · Sep 24, 2020, 4:30 PM

Not really, I have never used that. There's no way to tell from pfSense since it's UDP.

I would expect the log entries to be labelled something pretty obvious though. At least the time stamps should march so anything you see in the real-time log in pfSense should appear at that same stamp in the syslog server.

Steve

LeeArchinal · Sep 24, 2020, 5:02 PM

@stephenw10
Can I change to tcp?

stephenw10 · Sep 24, 2020, 8:21 PM

Yes, I believe Squid will do it if the syslog server can accept it.
It would have to do both though since the main pfSense syslogging is UDP only. Or you'd have to go via something else like syslog-ng as a relay.

Steve

LeeArchinal · Sep 24, 2020, 9:07 PM

@stephenw10
From the custom options can I tell the logs where I want them to be on my remote system? For example:
access_log udp://x.x.x.x:514 /var/log/syslog/squid.*

LeeArchinal · Sep 24, 2020, 9:17 PM

Disregard. So this statement did not work:
access_log udp://x.x.x.x:514

But this one did:
access_log syslog:local5.info squid

Not sure why but I am now getting them in my ELK Stack. Next question I have may not be yours to answer but I need to parse them.

stephenw10 · Sep 24, 2020, 9:23 PM

Hmm, interesting. I have always used the udp module there. Tested receiving in syslog-ng.

http://www.squid-cache.org/Versions/v4/cfgman/access_log.html

Is it just sending to the local system log with that and then being sent to the syslog server from there maybe?

Yeah, can't really help with parsing Squid logs ELK but it's probably quite common so I would expect guides/code to be available.

Steve

LeeArchinal · Sep 24, 2020, 9:39 PM

I have filebeat listening on port 514 not syslog-ng. That could be the big difference.

stephenw10 · Sep 24, 2020, 11:44 PM

Yes. Filebeat is not directly a syslog server as far as I can see. You have to configure it with the syslog input module: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-syslog.html
And possibly some other config there. As I say I've never used it.

Steve