What uses storage space for pfsense?
-
@denitrosubmena said in What uses storage space for pfsense?:
I am really looking for a network monitoring solution where i can see detail slike the live flow data and interface details to see bandwidth usage on interfaces for years of history
so opensource softwares can you recommend for that? besides ntopng
If having a log of all sessions made from all clients - including the amount of data moved is the primary goal, then you should definitively consider the pfFlow export in pfSense Plus. It has zero performance impact, and the GREATEST "addon" is that it can be activated on a per-firewall-rule basis. essentially only logging flows using the firewall rules you deem necessary.
I have no real expertise in the best netflow logging destination tool (Greylog, nProbe/NtopNG, Splunk and what not). There are many, and quite a few open source.
But to me the lack of DPI insights into the traffic (DNS names, SNI Certificate Info and Application fingerprinting) makes netflow logging less interesting. You will need some proper forensics skills and other logs/info to pair that flow info to in order to learn anything from it.
It is PERFECT for statistics though....Remember pfSense itself does permanent historical logging and summarisation of bandwidth usage.
-
@keyser said in What uses storage space for pfsense?:
Remember pfSense itself does permanent historical logging and summarisation of bandwidth usage.
where is this at? i can see historical data for years?
@keyser said in What uses storage space for pfsense?:
I have no real expertise in the best netflow logging destination tool (Greylog, nProbe/NtopNG, Splunk and what not). There are many, and quite a few open source.
But to me the lack of DPI insights into the traffic (DNS names, SNI Certificate Info and Application fingerprinting) makes netflow logging less interesting. You will need some proper forensics skills and other logs/info to pair that flow info to in order to learn anything from it.
It is PERFECT for statistics though....what i want for a start is what i see currently in the ntopng currently setup on the pfsense all i want now is historical data
so i can view the data for months and years. that is what am after and setting up the infra to do that is easy for me and i dont mind that. i just want to make sure am choosing the right tool for what i want.
and if this is getting more complicated than i thought then maybe i can reconsider sticking with fortigate/forticloud then and not have to worry about all these
-
@denitrosubmena said in What uses storage space for pfsense?:
So you pretty much recommend ntopng for network monitoring then? and that will give me all i need as far as network monitoring? and ability to view network bandwidth usage for a years of history? what are the other alternatives to that so i can dig in further
honestly i currently use fortigate and one of my issue is i wanted to view traffic metrics for long period and also view what is going on and i just couldnt till i get the forticloud which means handing access to fortigate to my firewall which i am just against to be honest.
I am just not for this trend of just handling all your access to some company in the name of they can provide cloud software and services for you. No thanks.
That is what brought me to pfsense and with ntopng onto of pfsense i thought that was all i needed not knowing this is also not so simple
I hear you :-)
There is no one perfect solution for pfSense - That requires Fortigate or Palo Alto services and loads of money.
Personally I'm using a licensed NtopNG of-host (on switch mirrorport) as that is the near perfect solution in my opinion. But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
So you will need to combine a couple of solutions in my opinion to keep it free:
1: Run NtopNG on pfSense itself to get near realtime insights into traffic and application flow.
2: Export Netflow data from pfSense (either with pfFlow in pfSense+ or the free SoftflowD in community) and get a good netflow logging and analytics system on another host. Fx. Greylog
That can be configured to do all the historical summaries and statistics you need - AND - you can see specific session flows back in time (though without session details such and DNS names, APP info and Certificate info). -
@denitrosubmena said in What uses storage space for pfsense?:
where is this at? i can see historical data for years?
STATUS -> MONITORING and use the wrench in the top-bar to look at traffic on a given interface for a give history.
But it's summarisation based, so it looses details rather quickly. If you want a specific bandwidth usage during a specific hour 3 weeks ago you can't. That will require additional monitoring. I fx. use Zabbix to monitor my pfSense, and in zabbix I can ask it to keep the bandwidth detail levels to my specific needs (minute based summaries for 90 days, 5 minutes for 365 days)
-
@keyser said in What uses storage space for pfsense?:
Personally I'm using a licensed NtopNG of-host (on switch mirrorport) as that is the near perfect solution in my opinion. But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
What do you mean by this
NtopNG of-host (on switch mirrorport)?But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
So how much historical data does it offer then? will look more into these things but wanted to get a base starting point from this chat
and thanks a lot for all the info, really appreciate them all so i can help make the best decision for my usecase
I dont mind paying but will prefer to pay in one place as opposed to paying in multiple places. that is the issue with softwares and services nowadays as they split all these features so they can make more money and just lose many potential users because no one wants to be paying in multiple places. We are all not big companies with profits to be paying multiple places. I mean i get that companies have to make money to stay alive but question i have for them is how much profit do they need ot they think they can just continue to increase the profit into infinity and keep adding more new products with new pricing. anyways let me end that rant.
i dont mind paying but will pay in one place and as long as it is not arm and leg.
between it is graylog, not greylog :)
-
@denitrosubmena said in What uses storage space for pfsense?:
What do you mean by this
NtopNG of-host (on switch mirrorport)?I have i switch on my network where i ask it to MIRROR all traffic that goes to and from the switchport connecting to the LAN interface on my pfSense. That traffic is mirrored to another switchport where it Have a NVMe SSD configured Raspberry Pi 5 with a Licensed NtopNG Enterprise Embedded edition running (and the free Clickhouse database system).
This gives me FULL details of everything happening from all clients towards the Internet. Since its licensed NtopNG, it logs all sessions to clickhouse, so I can see everything that happened - in DETAIL - up to 90 days back as that is my retention setting.. In those 90 days I have everything.
I suppose you could configure it to do years of retention if you have diskspace and performance enough :-)The Pi 5 handles full 1Gbit without packet loss, but it's at its limits doing that when every detail logging is configured and there and more than 100 clients (Lots of sessions).
But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
So how much historical data does it offer then? will look more into these things but wanted to get a base starting point from this chat
Hard to explain - but it likely offers what you need as long as its within the configured retention period. Its not unlimited - you need to set a period.
However - NtopNG is not meant as a historical bandwidth info tool, so you might want to consider using something else for that. -
between have you heard of https://www.observium.org?
i remembered a hosting provider i used in the past used that to monitor bandwidth usageand even that one too is not free for the important stuffs
so i just want one tool that combines many things in one and a great tool as a NOC tool to view and monitor traffic and view historical logs, will pay as long price is reasonable and pricing model allows to add more router/firewall devices meaning pricing supports multiple devices not per device
-
@denitrosubmena I don't know that software.
From my investigations NtopNG is the cheapest tool that delivers "almost everything". But at scale even that becomes expensive (from a private consumer perspective).
-
@keyser said in What uses storage space for pfsense?:
I don't know that software.
you should check it out, may be a good find, or not
yeah will have a look at the ntopng more and try to understand what the nprobe and clickhouse setup thing is about and what i get more than the free ntopng i have on pfsense
-
@denitrosubmena Observium seems more like a combined monitoring system and logging destination than a network analytics system. I do my monitoring (including bandwidths on interfaces) in Zabbix.
nProbe is the datacapture part of NtopNG. So you can have a central NtopNG and have nProbes running in many places and send telemetry back to NtopNG.
nProbe can also collect Netflow from Netflow exporters (like pfFlow and SoftflowD) and enrich it before sending it to a NtopNG for display, analytics and if licensed - Rentention.
-
@keyser said in What uses storage space for pfsense?:
nProbe is the datacapture part of NtopNG. So you can have a central NtopNG and have nProbes running in many places and send telemetry back to NtopNG.
nProbe can also collect Netflow from Netflow exporters (like pfFlow and SoftflowD) and enrich it before sending it to a NtopNG for display, analytics and if licensed - Rentention.
i just need this for pfsense since it is gateway to and from internet for my setup
so the solution i need will be just for pfsense
so does that mean ntopng will have that all done and i dont need to worry about nprobe? in multiple places? -
@denitrosubmena said in What uses storage space for pfsense?:
@keyser said in What uses storage space for pfsense?:
nProbe is the datacapture part of NtopNG. So you can have a central NtopNG and have nProbes running in many places and send telemetry back to NtopNG.
nProbe can also collect Netflow from Netflow exporters (like pfFlow and SoftflowD) and enrich it before sending it to a NtopNG for display, analytics and if licensed - Rentention.
i just need this for pfsense since it is gateway to and from internet for my setup
so the solution i need will be just for pfsense
so does that mean ntopng will have that all done and i dont need to worry about nprobe? in multiple places?If all the traffic you worry about goes through this one pfSense, then yes, you just need a NtopNG recieving copies of all packets - either by running on pfsense itself, or on a switch mirrorport.
No need for nProbe - it is only needed if you had more WAN links in other locations to visualize in the same NtopNG - or if you want NtopNG to visualize Netflow data as it cannot ingest netflow directly (needs to be converted and enriched by nProbe).
The reason I keep saying bandwidth might need another tool is:
NtopNG creates a full bandwitdh history for all involved and active elements (hosts, interfaces and such). So for as long as your retention is setup, you can see bandwitdth statistics for your interface, and for all active hosts. The problem is that once a host goes inactive, NtopNG removes the client from memory (all its data is still on disk) and you can no longer recall any statistics on that specific host.
To see its historical data (have NtopNG read it from disk) the host needs to be connected to the networks again and become active.
So NtopNG has the data you are asking for (historical bandwidth details), but you cannot see it unless the client is active. It makes no sense - i Know, but thats how it is. -
so what happens lets say i have 4 instances of pfsense in 4 different locations, is it better to setup ntopng for each one or have one ntopng to monitor all 4?
all pfsense will have 1 x WAN and 1 x LAN interfaces
as you recommended monitor only the LAN interface so totla 4 LAN interfaces to monitoreven if i want to pay for ntopng will the ntopng pro provide what i need to retain data on my own server for as long as i want?
i do have grafana prometheus/victoriametrics and loki/victorilogs so whatever i can ship to that am ok with
i prefer not have another logging with graylog
am sure there will be community dashboards support for network monitoring if i search -
@denitrosubmena said in What uses storage space for pfsense?:
so what happens lets say i have 4 instances of pfsense in 4 different locations, is it better to setup ntopng for each one or have one ntopng to monitor all 4?
In the free NtopNG version there is only a local install on each pfsense as an option.
If you start licencing NtopNG - which is also needed to store session history, It's a matter of preference. If you want you can have one central NtopNG instance showing the interface of all 4 LAN links on the different boxes. This will either require a manual install of nProbe on each pfSense (not recommended), or a mirrorport on each location to nProbe machines there.
Alternatively you can install a full NtopNG on each location using a mirrorport.
even if i want to pay for ntopng will the ntopng pro provide what i need to retain data on my own server for as long as i want?
NtopNG Pro does offer logging of flows. But it is rather feature limited in some scenarios. That is why I prefered an enterprise edition.
https://www.ntop.org/products/traffic-analysis/ntop/i do have grafana prometheus/victoriametrics and loki/victorilogs so whatever i can ship to that am ok with
The built in pfFlow netflow exporter or SoftflowD can export to all capable Netflow recievers.
i prefer not have another logging with graylog
am sure there will be community dashboards support for network monitoring if i searchLike I said, I use Zabbix for that. an EXCELLENT free monitoring system that really can do everything.
-
@keyser said in What uses storage space for pfsense?:
Like I said, I use Zabbix for that. an EXCELLENT free monitoring system that really can do everything.
i actually have zabbix still running i was using to monitor mikrotik router before i moved to fortigate
i dont like zabbix dashboard, except if they updated their dashboard UI to something modern, that thing is like 1990s widgets. i mean it is not the worst looking but they need to make it more modern like ntopng and other monitoring tool
so honestly i will pass on zabbix, and just look for one tool that will provide whatever zabbix will and other features am after for a few, wont mind paying. afterall forticloud is not free and one also need to have support for the fortigate so fortigate is out of question
so gotta make this pfsense thing work with ntopng.
ntopng enterprise M license is 499.95 euro per year, that is crazy for someone just trying to setup for personal to semi small app operationbut i get the pricing strategy because most people will use free and they want to charge people that some how need more than free to help pay for the free users :(
-
@denitrosubmena Yeah, with your interface speeds its going to be expensive. For me the Embedded NtopNG Enterprise M for 149€ is enough as a Raspberry Pi 5 i just fine for my 1 Gbit needs.
-
@denitrosubmena Also - doing packet capture and analytics at 40 and 100Gbe speeds (like your interfaces), is going to require some potent hardware.
Not to mention you will have to license pfring ZC also. Otherwise there is no chance of NtopNG keeping up with those traffic speeds -
@keyser said in What uses storage space for pfsense?:
@denitrosubmena Also - doing packet capture and analytics at 40 and 100Gbe speeds (like your interfaces), is going to require some potent hardware.
Not to mention you will have to license pfring ZC also. Otherwise there is no chance of NtopNG keeping up with those traffic speedsyeah am not going to be pushing anything close to 10Gbps talk less 100Gbps
everyone, including just loves the idea of having 100G just for the hell of it :)
pushing 10G will be all i need for a while
nothing will be connected to the 100G just yet, probably wont even add the card to the pfsense yet. WAN and LAN will be at 10G
-
Logs. If you use a proxy package cache items also. Also boot environments if you have multiple of them each is a huge file.
-
@JonathanLee said in What uses storage space for pfsense?:
boot environments
what are those?
@JonathanLee said in What uses storage space for pfsense?:
Logs. If you use a proxy package cache items also
nopes, will not be using any proxy cache on pfsense
this will be in datacenter not homelab
firewall and router for servers
