Need Log Analysis

tsmalmbe

If you want to tune and tune and tune and spend time on tuning, get Elkstack, if you want to read your logs and start analyzing them immediately, use Splunk. The license is free for 500MB logs per day, for me this is two firewalls and a lot of other stuff.

Stop playing, start analyzing :)

muswellhillbilly

Another link to a quick guide to setting up ELK with PFS.

https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

I used this myself when setting up my own log server with my PFS setup.

An updated version is here:

https://elijahpaul.co.uk/updated-monitoring-pfsense-logs-using-elk-elasticsearch-logstash-kibana-part-1/

AR15USR

@johnpoz:

…I do with we could get someone to maintain a guide or vm that logs everything with nice dashboards already made, etc. I keep meaning to make my setup available but it still needs a lot of tweaking to do everything extra pretty ;)

Yes please! ;D

mudmanc4

For those who do not use vmware

Grabbed the .ova converted it to qemu2 , flawless. Nicely done.

decompress:

tar xvf PFELK01-160408.ova

Convert:

qemu-img convert -f vmdk -O qcow2 PFELK01-160408-disk1.vmdk PFELK01.qcow2

Have a peek at the .ovf noticing the existing virtual CPU's and memory
Which happen to be 4 CPU's and 8GB ram

Import = done

Change existing password from 'pf' to {your password}

passwd

Access at {IP:5601}

Easypeasy

johnpoz

U might need change timezone too that was in utc while my pfsense in central

And u prob dont need 4 cpu and 8 gig running on only 2 and 2gb

hamed_forum

@muswellhillbilly:

Another link to a quick guide to setting up ELK with PFS.

https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

I used this myself when setting up my own log server with my PFS setup.

An updated version is here:

https://elijahpaul.co.uk/updated-monitoring-pfsense-logs-using-elk-elasticsearch-logstash-kibana-part-1/

i do it many time but dont work

hamed_forum

@johnpoz:

it does work, have it running..

You just need to set it up correctly ;)

You can find a working vm here..
https://www.reddit.com/r/PFSENSE/comments/4dymci/i_made_a_simple_bare_bones_simple_elk_vm_for/

This will do firewall logs, there are many guides on the net, many of them missing pieces or not very clear. But this vm works out of the box for firewall logs. If you want other logs you might need to make some adjustments

I do with we could get someone to maintain a guide or vm that logs everything with nice dashboards already made, etc. I keep meaning to make my setup available but it still needs a lot of tweaking to do everything extra pretty ;)

in pf sense only tike firewall logs or everthing?
tanks you do great work

AR15USR

I got that converted to the .vmdk, but not sure how to translate that into a running VM. VirtualBox says it can't recognize that type of disk. Jonpoz or mudmanc4 can you expand a bit on your instructions for getting this working please? Thanks..

Im using VirtualBox 5.0.2 running a Ubuntu 16 VM, on MacOS 10.11.5

johnpoz

pretty sure virtualbox just lets you import a ova.

AR15USR

@johnpoz:

pretty sure virtualbox just lets you import a ova.

Failed to open the disk image file /Users/xxxxxxxxx/Downloads/PFELK01-160408.ova.

Could not get the storage format of the medium '/Users/xxxxxxxxx/Downloads/PFELK01-160408.ova' (VERR_NOT_SUPPORTED).

Result Code: VBOX_E_IPRT_ERROR (0x80BB0005)
Component: MediumWrap
Interface: IMedium {4afe423b-43e0-e9d0-82e8-ceb307940dda}
Callee: IVirtualBox {0169423f-46b4-cde9-91af-1e9d5b6cd945}
Callee RC: VBOX_E_OBJECT_NOT_FOUND (0x80BB0001)

mudmanc4

@AR15USR:

<snip>Im using VirtualBox 5.0.2 running a Ubuntu 16 VM, on MacOS 10.11.5</snip>

Virtual disk space issue?
So your running Virtualbox on OS X, where Ubuntu is a guest, then trying to import a 1.6GB image into Ubuntu? Correct me?

AR15USR

@mudmanc4:

@AR15USR:

<snip>Im using VirtualBox 5.0.2 running a Ubuntu 16 VM, on MacOS 10.11.5</snip>

Virtual disk space issue?
So your running Virtualbox on OS X, where Ubuntu is a guest, then trying to import a 1.6GB image into Ubuntu? Correct me?

Not sure exactly what you mean by "import a 1.6GB image into Ubuntu". I downloaded the .ova in my Ubuntu guest so I could convert to the .vmdk. From there I copied the .vmdk to OS X and tried to create a new VM guest based on Linux with 2 processors, 16gb ram, and the .vmdk as the virtual hard disk. When selecting the .vmdk as the virtual hard disk is when I get the error.

mudmanc4

@AR15USR:

Not sure exactly what you mean by "import a 1.6GB image into Ubuntu". I downloaded the .ova in my Ubuntu guest so I could convert to the .vmdk. From there I copied the .vmdk to OS X and tried to create a new VM guest based on Linux with 2 processors, 16gb ram, and the .vmdk as the virtual hard disk. When selecting the .vmdk as the virtual hard disk is when I get the error.

Gotcha now.
You could not import the .ova itself?

AR15USR

@mudmanc4:

@AR15USR:

Not sure exactly what you mean by "import a 1.6GB image into Ubuntu". I downloaded the .ova in my Ubuntu guest so I could convert to the .vmdk. From there I copied the .vmdk to OS X and tried to create a new VM guest based on Linux with 2 processors, 16gb ram, and the .vmdk as the virtual hard disk. When selecting the .vmdk as the virtual hard disk is when I get the error.

Gotcha now.
You could not import the .ova itself?

No errors out as well. What Virtual Machine software are you using?

mudmanc4

@AR15USR:

No errors out as well. What Virtual Machine software are you using?

https://virt-manager.org/

Maybe it's corrupt in some way. I would just grab a new copy.

AR15USR

@mudmanc4:

@AR15USR:

No errors out as well. What Virtual Machine software are you using?

https://virt-manager.org/

Maybe it's corrupt in some way. I would just grab a new copy.

What version of Linux is the ova/vdmk based on? Maybe I was using the wrong type..

mudmanc4

@AR15USR:

What version of Linux is the ova/vdmk based on? Maybe I was using the wrong type..

I believe it is Ubuntu 14

vmw:osType="ubuntu64Guest

AR15USR

Hmm, since Splunk has a native OS X version I think I will start exploring that.

johnpoz

why are you going through all the extraction nonsense - pretty sure virtualbox should just import that ova as an appliance. When I get home from this road trip I will give it a go.

AR15USR

@johnpoz:

why are you going through all the extraction nonsense - pretty sure virtualbox should just import that ova as an appliance. When I get home from this road trip I will give it a go.

Yes I see that now. I'll try that when I get home. I was trying to use it as a virtual hard disk, way over thinking it..