Graylog server on a raspberry pi

mcury

The settings here are for small networks only!

How to install Graylog server in a Raspberry Pi 4b, Ubuntu Server 20.04.5 LTS:
Use at least a 4GB model, 8GB recommended.

I didn't test this tutorial in a VM, other device and OS.

Update your device:

sudo apt update
sudo apt upgrade

Install dependencies:

sudo apt install apt-transport-https openjdk-17-jre-headless uuid-runtime pwgen

mongodb 4.4 install:

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt update
sudo apt install mongodb-org
sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl restart mongod.service

ElasticSearch install:

wget -q https://artifacts.elastic.co/GPG-KEY-elasticsearch -O myKey
sudo apt-key add myKey
echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt update 
sudo apt install elasticsearch-oss

ElasticSearch configuration: Copy and paste the following:

sudo tee -a /etc/elasticsearch/elasticsearch.yml > /dev/null <<EOT
cluster.name: graylog
action.auto_create_index: false
EOT

ElasticSearch enable service:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl restart elasticsearch.service
sudo systemctl --type=service --state=active | grep elasticsearch

Graylog install: (version 5.0 is alpha, 4.3 is the latest stable):

wget https://packages.graylog2.org/repo/packages/graylog-4.3-repository_latest.deb
sudo dpkg -i graylog-4.3-repository_latest.deb
sudo apt update
sudo apt install graylog-server

Check /etc/graylog/server/server.conf for password_secret and root_password_sha2 and http_bind_address, read https://docs.graylog.org/docs/ubuntu for more details.

Enable graylog

sudo systemctl daemon-reload
sudo systemctl enable graylog-server.service
sudo systemctl start graylog-server.service
sudo systemctl --type=service --state=active | grep graylog

At this time, Graylog will be running and you can access it through the browser http://ip:9000

user: admin
password: admin

Configure pfsense to send logs to Graylog:

Go to : Status>System Logs>Settings and configure similar to this:

Configure Graylog to receive the logs

Create input:

Go to System/Inputs / Inputs:
Drop down menu, select Syslog UDP and launch the new input.

Then click save.

Create Index:
Go to System/Inputs / Index: (This is keep your data for 3 days only and it will rotate daily).

Create stream:

Go to Streams and click, create stream:

Now, you need to get gl2_source_input value, this will allow the data to land in the correct stream in Graylog, go to System / Inputs, click in show messages, copy the field in the search:

Go to Streams, Manage rules and paste the info gathered in the previous step, similar to this:

Check the Extractor link at the bottom of this tutorial to correctly parse the data inside Graylog.
There are other extractors, feel free to change in case you want, I like this one.

Now, you can configured personalized searches and the dashboard to your liking.

Note: You can configure other devices to send logs to Graylog.

Don't forget to change your admin password.
Use SSL/TLS (optional but highly recommended)

Example of a dashboard:

Credits to Lawrence System, who first showed me Graylog.

Links:
Graylog: https://docs.graylog.org/docs/ubuntu (Links are not up to date, so use the links above)
Graylog SSL/HTTPS https://docs.graylog.org/docs/https
Extractor URL: https://github.com/loganmarchione/Graylog_Extractors_pfSense
Mongodb: https://www.mongodb.com/docs/v4.4/tutorial/install-mongodb-on-ubuntu/
Lawrence System: https://www.youtube.com/watch?v=rtfj6W5X0YA

johnpoz

@mcury looks pretty straight forward walkthru - but you might want to either share your dashboard, or walk thru how you created yours as an example so users could get something up and running quick that shows them interesting info.

I have played with graylog in the past, and it has a pretty steep learning curve..

mcury

@johnpoz said in Graylog server on a raspberry pi:

looks pretty straight forward walkthru

:)

Dashboard:

Top inbound IP:
Note: mvneta2 and mvneta0 are my WANs interfaces; 192.168.255.245 is removed from the graphic:

Top outbound IP: Excluding LAN 192.168.255.2* and 8.8.* from the graphic.

TOP inbound TCP:S WAN: I like this one to see which port the Internet is hitting mostly:

Reputable blocklists: This is the tracking ID of the firewall rule created by pfblockerNG:

pfsense logs: Removing firewall logs from it, to get a clean view of the logs (system, dhcp and etc):

Synology logs: So few logs that you don't need a specific filter for it:

keyser

@mcury Interesting post and thanks for sharing. Personally I have been a little weary of installing a major syslog server/log analytics tool because of the learning curve and emmense amount of “framework” configuration you need to make - in order for the product to actually be usefull.

But perhaps i’ll give a spin one of these days….

mcury

@keyser said in Graylog server on a raspberry pi:

@mcury Interesting post and thanks for sharing. Personally I have been a little weary of installing a major syslog server/log analytics tool because of the learning curve and emmense amount of “framework” configuration you need to make - in order for the product to actually be usefull.

But perhaps i’ll give a spin one of these days….

It is a nice tool. In case you have any doubts about this tutorial, or any other questions, just ask and I'll try to help..

betrimble

Has the mongo install broke? Tried to run this and all I get is the following:

betrimble@graylog:~$ wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt update
sudo apt install mongodb-org
sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl restart mongod.service
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:4 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu jammy-security InRelease
Hit:6 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 Release
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: https://repo.mongodb.org/apt/ubuntu/dists/focal/mongodb-org/4.4/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
mongodb-org-mongos : Depends: libssl1.1 (>= 1.1.0) but it is not installable
mongodb-org-server : Depends: libssl1.1 (>= 1.1.0) but it is not installable
mongodb-org-shell : Depends: libssl1.1 (>= 1.1.0) but it is not installable
E: Unable to correct problems, you have held broken packages.
Failed to enable unit: Unit file mongod.service does not exist.
Failed to restart mongod.service: Unit mongod.service not found.

mcury

@betrimble said in Graylog server on a raspberry pi:

mongodb-org-mongos : Depends: libssl1.1 (>= 1.1.0) but it is not installable
mongodb-org-server : Depends: libssl1.1 (>= 1.1.0) but it is not installable
mongodb-org-shell : Depends: libssl1.1 (>= 1.1.0) but it is not installable

You are running Ubuntu 22.04.1 LTS and not 20.04.5 ?
If that is the case, try the step in the last post in this thread:

https://github.com/dotnet/sdk/issues/25441 or downgrade the OS to 20.05.5

michmoor

@mcury Great job on the write up. Ive been using graylog for a bit over a year and the hardest part was creating the dashboards. The documentation wasnt great so def a lot of trial and error. I think you did a great job showing the steps on how to do this.
Truthfully, the hardest part of graylog so far [community edition] is finding extracters. For example, im on the hunt for Suricata extractor. I can get the eve json sent to graylog but its still a mess. My goal is to tie that in with GeoIP to get a pretty dashboard maybe. Been to tied up with other projects to continue researching.

mcury

@michmoor They just released Graylog 5.0. It works with Ubuntu 22.04

I can't test it here because it depends on mongodb 5.0, which is not compatible with raspberry.
MongoDB 5.0 requires ARM v8.2-A or later and the Raspberry Pi 4 uses an ARM Cortex-A72 3 which is ARM v8-A.

I suppose you can build mongodb by yourself using the correct flags, but I didn't try it.

If you are running on a x86 system you can run Graylog 5.0.
All you need to change is the following:

mongodb 5.0 install: (note here that I removed arm64, test only in x86 system).

wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list
sudo apt update
sudo apt install mongodb-org

Graylog install

wget https://packages.graylog2.org/repo/packages/graylog-5.0-repository_latest.deb
sudo dpkg -i graylog-5.0-repository_latest.deb
sudo apt-get update && sudo apt-get install graylog-server

Everything else in the first post remains the same.

michmoor

@mcury Yep i got 5.0 installed today actually. Just reading through the changelogs.
Ive been curious about one thing tho...
The 'count' in your charts. Should we assume thats how many sessions were created on the firewall, i.e. how many times a packet hit that rule?
I got my kids playing Roblox and.....

mcury

@michmoor said in Graylog server on a raspberry pi:

The 'count' in your charts. Should we assume thats how many sessions were created on the firewall, i.e. how many times a packet hit that rule?

Based on what I've observed so far, this would be the same thing you would see in System logs > Firewall in Pfsense logs.

Since its a game, it is probably using UDP, right? I never played Roblox.. So I can't tell.

You can click the play button inside this chart to take a look at each of those entries to check.