HOWTO: Remote Realtime pfSense Log Visualization with glTail (Revised!)

jimp

I've written a glTail parser that will display firewall logs from pfSense in glTail's realtime log visualization on a workstation. If you haven't used glTail before, I also suggest checking it out for additional monitoring such as web and mail server logs.

glTail breaks the log output down into "blocks" such as Action (pass/block), Protocol (tcp/udp/icmp/etc), Host/Interface (fxp0, vr0, em0, vlan0, etc), source host, source port, destination host, destination port, and full source/destination pair. More active entries bubble up to the top of their respective blocks, and are displayed with a "requests per minute" counter. In some ways it is sort of like top, showing the most active entries in the firewall logs. Here is a screenshot of what it looks like from a slightly older revision of the parser. (It might also look a little funny since I blacked out some IP Addresses).

It can take a bit of CPU time and some graphics card power to run, but it is worth it. It also looks pretty enough to show PHBs/Management that the firewall is "doing stuff", especially if you see a lot of blocked packets in your logs.

I had to use a slightly older version of glTail than what the author has available in his public git repository. The current version uses a library that is not currently working on Windows. It should work fine on Ubuntu. I have the source for a version of glTail that should work fine on either Windows or Ubuntu (and probably elsewhere) available on my site Download glTail source.

What follows is a set of general instructions that should get glTail up and running, and talking to your pfSense system. It's not exactly a simple process, and some things will vary from system to system, so it might take some effort beyond what I have outlined below.

Major Edit: Since I wrote this, I have found a MUCH easier way to get this running on Windows. It will likely also work with Ubuntu, but I cannot test it for a few days.

Method 1, The Easy Way: (Windows only at the moment)
Step 1 - Ensure ssh is turned on in pfSense, and accessible from your workstation.

Step 2 - Download the glTail binary executable and unzip it somewhere convenient on your workstation.

Step 3 - Edit the config file (gl_tail.yaml) as needed
- Change the host and password to suit your needs. Leave the username as root. The password is the same as admin in the webgui, but this will allow glTail to run a command and not get caught in the shell menu.
- If you are connecting to a syslog server instead of directly, comment out the first host definition and uncomment the second, and change the settings as needed.
- You might need to change the dimensions, but it does not work properly at less than about 1000px wide.
- There is also an option at the bottom to turn off/on reverse IP lookups if you want to turn them off

Step 4 - Double click the glTail.exe executable

Step 5 - Enjoy! (And read the Caveats section below)

Method 2, Using the Source and installing required dependencies: (Windows/Linux)
Step 1 - Ensure ssh is turned on in pfSense, and accessible from your workstation.

Step 2 - Download the glTail source and unzip it somewhere convenient on your workstation.
- In case you missed it above, you can get the glTail files I worked with here: Download glTail source.
- Much of the following setup is covered in the README/README.txt file included in the Zip.
- If you choose to use a different version of glTail obtained elsewhere, copy the pfsense.rb file from the lib/gl_tail/parsers directory in the zip to your local installation of glTail. If you are using the latest version of glTail from the public git repository, the pfSense parser is now included in the distribution. You will likely also need the supplied example config.yaml to work from/refer to.

Step 3 - Install Ruby and rubygems on your workstation.
- Windows:
Download and run the Ruby One-Click Installer
http://rubyforge.org/frs/download.php/29263/ruby186-26.exe
(Or most recent final from http://rubyforge.org/frs/?group_id=167&release_id=28426 )
- Ubuntu:
sudo apt-get install rubygems rake ruby1.8-dev libgl1-mesa-dev libglu1-mesa-dev libglut3-dev build-essential
- This will also install some libraries required to build the native ruby extensions needed for glTail

Step 4 - Install the prerequisite gems on your workstation:
- gem install -y file-tail -r
- gem install -y ruby-opengl -r
- Select 0.60.0 (i386-win32 for Windows, native for Ubuntu)
- gem install -y net-ssh -v 1.1.4

Step 5 - Edit the config file (config.yaml) as needed
- Change the host and password to suit your needs. Leave the username as root. The password is the same as admin in the webgui, but this will allow glTail to run a command and not get caught in the shell menu.
- If you are connecting to a syslog server instead of directly, comment out the first host definition and uncomment the second, and change the settings as needed.
- You might need to change the dimensions, but it does not work properly at less than about 1000px wide.
- There is also an option at the bottom to turn off/on reverse IP lookups if you want to turn them off

Step 6 - Run glTail from a command line on your workstation
Windows:
cd <path_to_gltail>ruby bin\gl_tail config.yaml
Ubuntu:
cd <path_to_gltail>ruby bin/gl_tail config.yaml

- If any libraries are missing, it should print out an error that tells you which ones and will even tell you how to install them.

Step 7 - Enjoy!

Caveats:
#0 If you have trouble running Ruby on Windows, try adding "C:\Ruby\bin" to your path.

#1 When connecting directly, there is no way to only view the end of the log. The clog program to view circular logs will dump the entire log to the parser, then will tail it showing new messages. Therefore, the parser I wrote has to run a simple time check and only view entries from the last 5 minutes, or the "future".

The log output from pfSense does NOT include a time zone, so it is assumed that the workstaion is set to the same time as the pfSense box. If this is not the case, you may need to edit pfsense.rb and correct the time zone. You may also need to ensure that your workstation time is correct, as well as the time on the pfSense box.

Symptoms of an incorrectly set time zone would be that either no data ever shows up, or that too much shows up all at once when glTail is started.

#2 In case it wasn't obvious from the above, this monitoring/output is done remotely via ssh from a workstation. This does not happen on the server or in the webgui.

#3 If you have a lot of IPs pointed at your pfSense box, and someone does a port scan, it might get a tad sluggish as it draws the deluge of new entries. This is entirely dependent on your workstation, and may not be noticeable if you have a fast CPU/GPU and proper OpenGL-aware drivers. This may also happen during the start of a run, if there are a lot of log entries in the last 5 minutes or so.

#4 Every once in a while there might be a log entry that isn't properly interpreted by the parser. If I can get examples of log entries that are formatted incorrectly, I can refine the parser to either handle or ignore such entries.

Let me know if this works for you!

I have submitted the parser to the author of glTail so it might be included in future versions.

Thanks go to Erlend Simonsen for writing glTail!

Edit: Added screenshot to the post.

pfsense_gltail.jpg_thumb </path_to_gltail></path_to_gltail>

jimp

The pfSense parser is now in glTail's git repository, but as a reminder the current version of glTail is a work in progress and requires a library that does not yet work properly on Windows. I have tested it on Ubuntu and it worked fine there, so it may also work on other *nix-based systems.

If you go this route, you'll still probably want to refer to my example configuration file. I've attached it to this post. It should be config.yaml, but as this forum only allows txt, I've renamed it slightly.

Also, oddly enough, I could not get it working on FreeBSD because for whatever reason I am unable to get a modern-enough version of ruby-opengl to compile and run. I'm still working on that one.

config.yaml.txt

jimp

I revised the howto to include an "easy" method that can be used on Windows. I managed to get an altered version of glTail to compile/collect with RubyScript2Exe. Now, for those running Windows workstations, it should be as easy as download, unzip, edit config, and run.

simoncpu

whoa… very cool! :)

AudiAddict

Looks awesome, nice work! The only main downside is that I have to place our webgui password in a config file.. in cleartext.

If somebody happends to get a hold of that file, they also have access to our firewall.

Not a very safe solution if you ask me, any idea's on how I could do this differently? Running a extra syslog server just for this is a bit too much.

jimp

It is not very well documented, but it can be setup to use SSH keys.

Have you tried leaving the password blank? I can't remember if that caused it to prompt for one or not.

I'll see if I can dig up some more configuration examples

jimp

Commenting out the password field does cause it to prompt, but it echos the typed characters, which is also not very good from a security standpoint.

I'll see what can be done about that.

AudiAddict

It also seems to be very CPU hungry, it's giving me 75-90% cpu load on a P4 3GHZ 2GB Work PC.

jimp

It is very resource-intensive. According to the glTail author, he has done quite a bit of work on optimization but that has not made it into a version that works on windows yet. If you are running Ubuntu or another Linux platform, you might have better luck with the current version from the author's git repo.