What uses storage space for pfsense?
-
I am trying to build a pfsense router/firewall with 2 x 100G sfp+ ports and wanted to know what can be the services that will consume storage space that i need to be aware of.
I will be shipping the logs via syslog to logging infra.
I do want to be able to view ntop graphs forever though so i will like to be able to see the ngot graphs for as long as possible, lets say last 5 years within the ntopng web ui. So i think that is what am mostly concerned aboutSo i wanted to know what packages or usecase of pfsense consumes lots of space that i need to be worried about?
I have a test pfsense am using to test things out that is barely using anything. Only packages i have installed are below and you can see current storage usage
I will not be using pfblockerNG, the below packages are the ones am mostly interested in. I will not be doin gIPS/IDS either as i will put that on the firewall in front of pfsense to do all of that including doing WAF for me, so i wont be using pfsense for that.
hardware will be as follow
Dell R620
4 x 10G sfp+ onboard
2 x 100G qsfp28 NIC
1 x 400G Intel DC S3710 SSDSo as you can see i am setting this up with 400G for the pfsense OS and data
-
@denitrosubmena There really only are 4 “big” contributors to diskspace usage:
1: pfSense’s own firewall logs - if configured to NOT to rotate logs, and allow HUGE logfiles (This is not the default settings)
2: pfBlockerNG
3: Suricata/Snort IDS/IPS
4: NtopNG - This one is a bit theoretical as the package NtopNG does not allow history logging. So only summaries for devices are logged. It will require thousands and thousands of clients for this to actually theretically become an +20Gb spaceuser.There is a difference between space use and SSD wear. Number 1 and 4 can wear a SSD fairly fast if theres a lot of clients and logs generated. But a 400G SSD can tolerate a lot of writing so it should last you at least a couple of years in a very heavy use environment.
-
@keyser said in WHat uses storage space for pfsense?:
1: pfSense’s own firewall logs - if configured to NOT to rotate logs, and allow HUGE logfiles (This is not the default settings)
2: pfBlockerNG
3: Suricata/Snort IDS/IPS
4: NtopNG - This one is a bit theoretical as the package NtopNG does not allow history logging. So only summaries for devices are logged. It will require thousands and thousands of clients for this to actually theretically become an +20Gb spaceuser.yeah i am only interested in 1 and 4, wont be installing pfblockerng or suricata/snort
#1.
if that is the case maybe i should install pfsense base OS on a different OS drive and then the logging on a different drive
how can i achieve that? what will be the ZFS setup to do that and pfsense settings for that setup to work properly?
that way the OS drive does not get written to and then the data drive is the one that gets written to
so i can have this hardware setup
Dell R620 4 x 10G sfp+ onboard 2 x 100G qsfp28 NIC 1 x 200G Intel DC S3510 SSD for pfsense OS 1 x 1.2TB Intel DC S3710 SSD for pfsense data#2.
regarding ntopng not allowing history logging. what do you mean by that? when you say logging, are you referring to the interface details history of traffic? and the other graphs?
I really need those and that is what is most important to me, just as much as the firewall logs too. I want to retain all of those graphs so i can look back years for them. That is actually main reason for me posting this thread so maybe i need to get a bigger hard drive to data thus question #1 above
-
@denitrosubmena pfSense is not really made for adding second disks. There is no way to that during the install. Neither can you do it afterwards by reconfiguring the system to a second logdisk in the UI.
It can only be done manually in the CLI, and most/all of that will be lost when upgrading your pfsense version.So I would strongly encourage you not to add a second disk and just go with one system volume SSD. If you are really worried about wear, add two SSD's, Get pfSense+ and setup a ZFS mirror so you can handle loosing one.
On the matter of NtopNG: The NtopNG package that is available in the package manager is only the commuty edition. It cannot be licensed to a more featurerich version. Community edition does not allow logging of flows (history of sessions), so it will only show you near realtime flows. You cannot go back and see who did what 30 min or 30 days ago.
But it does support logging of interface and client statistics - this is what might use a bit of space.
Interface and Client statistics are summaries of bandwidth usage and to some extent application usage. But no details - just summaries.There is one "workaround". NtopNG themselves supports installing the full NtopNG version on pfSense directly. I you do that you can license it and so on. But to store historical flows you also need the Clickhouse Database system, and you need to host that elsewhere than on pfSense. Also - this does not survive pfSense upgrades in which case you need to reinstall NtopNG from scratch again.
If you really care about NtopNG data and history setup a switch mirrorport and setup NtopNG/Clickhouse on a dedicated Traffic logging system.
Alternatively run community NtopNG for realtime data on pfSense, and set pfSense up to do flow logging of traffic to a Netflow logging/analytical system on a Raspberry pi/small computer.Do not rely on NtopNG (community or licensed) for long term statistics. Upgrades might loose your data, and NtopNG itself is not made for storing that data for more than 30 - 90 days. And since clients are not persistent in NtopNG, you will loose statistics for clients that are not online for longer periods of time. Nor can you easily browse statistics for clients that have not been online for a while.
-
@keyser said in WHat uses storage space for pfsense?:
So I would strongly encourage you not to add a second disk and just go with one system volume SSD. If you are really worried about wear, add two SSD's, Get pfSense+ and setup a ZFS mirror so you can handle loosing one.
only pfsense+ allow to do ZFS mirror of pfsense OS drive?
@keyser said in WHat uses storage space for pfsense?:
If you really care about NtopNG data and history setup a switch mirrorport and setup NtopNG/Clickhouse on a dedicated Traffic logging system.
Alternatively run community NtopNG for realtime data on pfSense, and set pfSense up to do flow logging of traffic to a Netflow logging/analytical system on a Raspberry pi/small computer.So essentially best to setup ntopng outside of pfsense. ok cool. What if i want to ship to metrics/logs to my monitoring infra like prometheus/loki? that wont work?
I am really looking for a network monitoring solution where i can see detail slike the live flow data and interface details to see bandwidth usage on interfaces for years of history
so opensource softwares can you recommend for that? besides ntopng
-
@denitrosubmena A little word of warning considering the other thread about NtopNG interfaces you started.
NtopNG is not designed/made for monitoring the WAN interface, but as I understand it you insist on doing it anyways. Each to his own, but one of the pitfalls of that is that all hosts on the intire internet is considered local hosts, so NtopNG will start a Redis RDD statistics file on each and every host you ever talk to. And each session made will add statistics to both your inside clients statistics file and the WAN side hosts statistics file (double data counting)If you configure logging/statistics in NtopNG to be as complete as possible, that will then happen for all internet hosts as well. Since you are looking at 10 and 100Gbit interfaces, I assume you will be moving a lot of traffic. That will likely kill NTopNG instantly as it will not have enough Disk IO available to log the thousands and thousands of datapoints/second.
Also: It will probably kill both your SSD space and wear fairly quickly. Because then it can easily become A LOT MORE than 20+ Gb of statistics. -
@denitrosubmena said in What uses storage space for pfsense?:
only pfsense+ allow to do ZFS mirror of pfsense OS drive?
Actually - i'm not sure. I know boot environments (ZFS Snapshots) are a plus feature. But ZFS is available in the community Edition, so it may also offer Raid Mirror setup during install. I haven't tried it myself.
So essentially best to setup ntopng outside of pfsense. ok cool. What if i want to ship to metrics/logs to my monitoring infra like prometheus/loki? that wont work?
I am really looking for a network monitoring solution where i can see detail slike the live flow data and interface details to see bandwidth usage on interfaces for years of history
so opensource softwares can you recommend for that? besides ntopng
In my opinion NtopNG is a stellar tool because is does DPI analytics of the traffic (looks and DNS requests, Certificate SNI info and application fingerprinting). Replacing all those features is impossible unless you bring out the big wallet.
But if basic session Flow Info and statistics is enough, pfSense plus offers that right out of the box with the pf Netflow export feature. That will export all sessions in Netflow v9 or IPFIX format to any netflow logging and analytical system you have (Greylog, splunk, nProbe and so on). Loki/grafana and Prometheus/grafana can also be used, but will require A LOT of work from your side "decoding the flows" to proper usefull information in Grafana.
You can also do it with the pfSense community edition and the softflowd package in the package manager. But that does not offer the same performance, and it can be tricky to configure NOT to split long sessions into smaller part-sessions. -
@keyser said in What uses storage space for pfsense?:
A little word of warning considering the other thread about NtopNG interfaces you started.
NtopNG is not designed/made for monitoring the WAN interface, but as I understand it you insist on doing it anyways.Yeash wanted to test things out and see happens and after seeing the logs under Hosts tab yeah i will disable the WAN and only use the LAN
good to find out things for once self sometimes. I think the LAN port gives me what i wanted, i thought it wont and i could only get it from the WAN port
thanks for that pointing that out again
-
@keyser said in What uses storage space for pfsense?:
Actually - i'm not sure. I know boot environments (ZFS Snapshots) are a plus feature. But ZFS is available in the community Edition, so it may also offer Raid Mirror setup during install. I haven't tried it myself.
i think this can be done on community edition except if pfsense is getting out of open source business completely
@keyser said in What uses storage space for pfsense?:
In my opinion NtopNG is a stellar tool because is does DPI analytics of the traffic (looks and DNS requests, Certificate SNI info and application fingerprinting). Replacing all those features is impossible unless you bring out the big wallet.
So you pretty much recommend ntopng for network monitoring then? and that will give me all i need as far as network monitoring? and ability to view network bandwidth usage for a years of history? what are the other alternatives to that so i can dig in further
honestly i currently use fortigate and one of my issue is i wanted to view traffic metrics for long period and also view what is going on and i just couldnt till i get the forticloud which means handing access to fortigate to my firewall which i am just against to be honest.
I am just not for this trend of just handling all your access to some company in the name of they can provide cloud software and services for you. No thanks.
That is what brought me to pfsense and with ntopng onto of pfsense i thought that was all i needed not knowing this is also not so simple
-
@denitrosubmena said in What uses storage space for pfsense?:
I am really looking for a network monitoring solution where i can see detail slike the live flow data and interface details to see bandwidth usage on interfaces for years of history
so opensource softwares can you recommend for that? besides ntopng
If having a log of all sessions made from all clients - including the amount of data moved is the primary goal, then you should definitively consider the pfFlow export in pfSense Plus. It has zero performance impact, and the GREATEST "addon" is that it can be activated on a per-firewall-rule basis. essentially only logging flows using the firewall rules you deem necessary.
I have no real expertise in the best netflow logging destination tool (Greylog, nProbe/NtopNG, Splunk and what not). There are many, and quite a few open source.
But to me the lack of DPI insights into the traffic (DNS names, SNI Certificate Info and Application fingerprinting) makes netflow logging less interesting. You will need some proper forensics skills and other logs/info to pair that flow info to in order to learn anything from it.
It is PERFECT for statistics though....Remember pfSense itself does permanent historical logging and summarisation of bandwidth usage.
-
@keyser said in What uses storage space for pfsense?:
Remember pfSense itself does permanent historical logging and summarisation of bandwidth usage.
where is this at? i can see historical data for years?
@keyser said in What uses storage space for pfsense?:
I have no real expertise in the best netflow logging destination tool (Greylog, nProbe/NtopNG, Splunk and what not). There are many, and quite a few open source.
But to me the lack of DPI insights into the traffic (DNS names, SNI Certificate Info and Application fingerprinting) makes netflow logging less interesting. You will need some proper forensics skills and other logs/info to pair that flow info to in order to learn anything from it.
It is PERFECT for statistics though....what i want for a start is what i see currently in the ntopng currently setup on the pfsense all i want now is historical data
so i can view the data for months and years. that is what am after and setting up the infra to do that is easy for me and i dont mind that. i just want to make sure am choosing the right tool for what i want.
and if this is getting more complicated than i thought then maybe i can reconsider sticking with fortigate/forticloud then and not have to worry about all these
-
@denitrosubmena said in What uses storage space for pfsense?:
So you pretty much recommend ntopng for network monitoring then? and that will give me all i need as far as network monitoring? and ability to view network bandwidth usage for a years of history? what are the other alternatives to that so i can dig in further
honestly i currently use fortigate and one of my issue is i wanted to view traffic metrics for long period and also view what is going on and i just couldnt till i get the forticloud which means handing access to fortigate to my firewall which i am just against to be honest.
I am just not for this trend of just handling all your access to some company in the name of they can provide cloud software and services for you. No thanks.
That is what brought me to pfsense and with ntopng onto of pfsense i thought that was all i needed not knowing this is also not so simple
I hear you :-)
There is no one perfect solution for pfSense - That requires Fortigate or Palo Alto services and loads of money.
Personally I'm using a licensed NtopNG of-host (on switch mirrorport) as that is the near perfect solution in my opinion. But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
So you will need to combine a couple of solutions in my opinion to keep it free:
1: Run NtopNG on pfSense itself to get near realtime insights into traffic and application flow.
2: Export Netflow data from pfSense (either with pfFlow in pfSense+ or the free SoftflowD in community) and get a good netflow logging and analytics system on another host. Fx. Greylog
That can be configured to do all the historical summaries and statistics you need - AND - you can see specific session flows back in time (though without session details such and DNS names, APP info and Certificate info). -
@denitrosubmena said in What uses storage space for pfsense?:
where is this at? i can see historical data for years?
STATUS -> MONITORING and use the wrench in the top-bar to look at traffic on a given interface for a give history.
But it's summarisation based, so it looses details rather quickly. If you want a specific bandwidth usage during a specific hour 3 weeks ago you can't. That will require additional monitoring. I fx. use Zabbix to monitor my pfSense, and in zabbix I can ask it to keep the bandwidth detail levels to my specific needs (minute based summaries for 90 days, 5 minutes for 365 days)
-
@keyser said in What uses storage space for pfsense?:
Personally I'm using a licensed NtopNG of-host (on switch mirrorport) as that is the near perfect solution in my opinion. But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
What do you mean by this
NtopNG of-host (on switch mirrorport)?But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
So how much historical data does it offer then? will look more into these things but wanted to get a base starting point from this chat
and thanks a lot for all the info, really appreciate them all so i can help make the best decision for my usecase
I dont mind paying but will prefer to pay in one place as opposed to paying in multiple places. that is the issue with softwares and services nowadays as they split all these features so they can make more money and just lose many potential users because no one wants to be paying in multiple places. We are all not big companies with profits to be paying multiple places. I mean i get that companies have to make money to stay alive but question i have for them is how much profit do they need ot they think they can just continue to increase the profit into infinity and keep adding more new products with new pricing. anyways let me end that rant.
i dont mind paying but will pay in one place and as long as it is not arm and leg.
between it is graylog, not greylog :)
-
@denitrosubmena said in What uses storage space for pfsense?:
What do you mean by this
NtopNG of-host (on switch mirrorport)?I have i switch on my network where i ask it to MIRROR all traffic that goes to and from the switchport connecting to the LAN interface on my pfSense. That traffic is mirrored to another switchport where it Have a NVMe SSD configured Raspberry Pi 5 with a Licensed NtopNG Enterprise Embedded edition running (and the free Clickhouse database system).
This gives me FULL details of everything happening from all clients towards the Internet. Since its licensed NtopNG, it logs all sessions to clickhouse, so I can see everything that happened - in DETAIL - up to 90 days back as that is my retention setting.. In those 90 days I have everything.
I suppose you could configure it to do years of retention if you have diskspace and performance enough :-)The Pi 5 handles full 1Gbit without packet loss, but it's at its limits doing that when every detail logging is configured and there and more than 100 clients (Lots of sessions).
But it does not offer years of client and interface data/Statistics. The historical statistics part if NtopNGs biggest "letdown".
So how much historical data does it offer then? will look more into these things but wanted to get a base starting point from this chat
Hard to explain - but it likely offers what you need as long as its within the configured retention period. Its not unlimited - you need to set a period.
However - NtopNG is not meant as a historical bandwidth info tool, so you might want to consider using something else for that. -
between have you heard of https://www.observium.org?
i remembered a hosting provider i used in the past used that to monitor bandwidth usageand even that one too is not free for the important stuffs
so i just want one tool that combines many things in one and a great tool as a NOC tool to view and monitor traffic and view historical logs, will pay as long price is reasonable and pricing model allows to add more router/firewall devices meaning pricing supports multiple devices not per device
-
@denitrosubmena I don't know that software.
From my investigations NtopNG is the cheapest tool that delivers "almost everything". But at scale even that becomes expensive (from a private consumer perspective).
-
@keyser said in What uses storage space for pfsense?:
I don't know that software.
you should check it out, may be a good find, or not
yeah will have a look at the ntopng more and try to understand what the nprobe and clickhouse setup thing is about and what i get more than the free ntopng i have on pfsense
-
@denitrosubmena Observium seems more like a combined monitoring system and logging destination than a network analytics system. I do my monitoring (including bandwidths on interfaces) in Zabbix.
nProbe is the datacapture part of NtopNG. So you can have a central NtopNG and have nProbes running in many places and send telemetry back to NtopNG.
nProbe can also collect Netflow from Netflow exporters (like pfFlow and SoftflowD) and enrich it before sending it to a NtopNG for display, analytics and if licensed - Rentention.
-
@keyser said in What uses storage space for pfsense?:
nProbe is the datacapture part of NtopNG. So you can have a central NtopNG and have nProbes running in many places and send telemetry back to NtopNG.
nProbe can also collect Netflow from Netflow exporters (like pfFlow and SoftflowD) and enrich it before sending it to a NtopNG for display, analytics and if licensed - Rentention.
i just need this for pfsense since it is gateway to and from internet for my setup
so the solution i need will be just for pfsense
so does that mean ntopng will have that all done and i dont need to worry about nprobe? in multiple places?
