Packet Logging to a database?
-
I've searched the forum using the term "packet logging" but could only find one link which wasnt relevent hence this post.
Is it possible to log all packets going in and out of pfsense and if so can they be logged to a database of sorts maybe an SQL server so I can analyse them?
If it is are there any db's which will run on the installation? My old pc running pfsense died so I got a cheap replacement which has plenty of capacity, dual core G840, 6Gb of ram & 250Gb hard drive & dual nic, but yet to install it maybe tomorrow if I get some time assuming it will run on this newer hardware, fingers crossed.
Anyway assuming it installs ok and if there isnt a db I can run on FreeBSD/pfsense and/or there isnt a db it can send a copy of all the packets to, is there any other way I can get all the packets out, maybe a syslog broadcast of sorts?
I'm essentially a windows programmer with some networking librarys so I can read and write my own tcp/ip/udp packets if need be.
Anyway sorry for the waffle, got to say I love pfSense, its quite capable for a freebie firewall much easier to use than ISA 2000 and ISA2003 imo, although havent installed and checked it out against MS latest offering becuase they have announced they are getting out of this line of business.
-
You don't want to log packets to a database. If you want to analyse them then you want to store them in pcap format, since that's what all the tools use. There are a mountain of suitable tools, from the venerable tcpdump, through daemonlogger to the highly capable OpenFPC. Only tcpdump comes with pfSense.
What is it you're wanting to actually do?
-
@Cry:
You don't want to log packets to a database. If you want to analyse them then you want to store them in pcap format, since that's what all the tools use. There are a mountain of suitable tools, from the venerable tcpdump, through daemonlogger to the highly capable OpenFPC. Only tcpdump comes with pfSense.
What is it you're wanting to actually do?
Ok thanks for the terminology update re pcap. I literally want to log everything coming in and going out so that I can build up a db of all the data my "visitors" are sending to me (IP Addresses, Date Time, the 16kb packet content data, user agents, source ports etc etc) and what I'm sending out as I was quite surprised at what Windows sends out as well, but I also want to see what patterns might also arise.
In time I'll build rules to ignore what I want to ignore but I do want to trap the unusual stuff coming into my custom built webservers and I just dont know what I might find until I start looking really.
Its a curiosity thing more than anything else plus I can at least satisfy myself the webservers I've built are reasonably robust.
I'll check out those mentioned above and see if I can do anything with them, I know from running ethereal and wireshark it can slow the windows machine down a bit so if anything existed already for pfsense I'd rather work with that becuase its a different platform for starters and its always good to mix things up instead of relying on any one thing.
-
I literally want to log everything coming in and going out
You have the disk bandwidth for that?
If you are prepared to accept a somewhat more modest capability you might want to look at logging flows. This will effectively record source and destination IP address, source and destination ports, data transfer counts, time stamp for every connection. There is a variety of analysis and reporting tools.
-
Netflow is generally the only highly scalable, long term storage option there. You could save full packet data via tcpdump but you'll chew up a whole lot of disk space quickly unless it's a really quiet network.
-
Do you realise that you'll never look through even a fraction of that traffic? It isn't practical to eyeball the traffic of even a home network. If you want to look for patterns then Netflow is a better starting point, coupled with some of the many Netflow graphing tools out there. If you're concerned about what's going in and out of your network then running the likes of Snort with the Emerging Threat ruleset will generally be more useful. There are also various tools for logging user agents and tools for logging traffic to web servers.
Running a packet capture on any system will slow it down, both with the CPU load and the disk I/O load. The higher your network traffic the higher the load.
-
I literally want to log everything coming in and going out
You have the disk bandwidth for that?
If you are prepared to accept a somewhat more modest capability you might want to look at logging flows. This will effectively record source and destination IP address, source and destination ports, data transfer counts, time stamp for every connection. There is a variety of analysis and reporting tools.
Thanks, I'll check out logging flow on google (Mule) has popped up as one suggestion.
@cmb:
Netflow is generally the only highly scalable, long term storage option there. You could save full packet data via tcpdump but you'll chew up a whole lot of disk space quickly unless it's a really quiet network.
It is a quiet network ATM just a few pc's and I'm only a one man band so finances are limited, but in time I can set the system to ignore what is genuine.
@Cry:
Do you realise that you'll never look through even a fraction of that traffic? It isn't practical to eyeball the traffic of even a home network. If you want to look for patterns then Netflow is a better starting point, coupled with some of the many Netflow graphing tools out there. If you're concerned about what's going in and out of your network then running the likes of Snort with the Emerging Threat ruleset will generally be more useful. There are also various tools for logging user agents and tools for logging traffic to web servers.
Running a packet capture on any system will slow it down, both with the CPU load and the disk I/O load. The higher your network traffic the higher the load.
I know from my time in Ethereal & Wireshark alot of it is genuine but by setting up a system to log this data and then have it remove what is genuine or inconsequential traffic will spare the disk to some degree and my time analysing these logs as well. I've already got snort and its ETR in use but its still being triggered. Anyway thanks for your help.
-
I'll check out logging flow on google (Mule) has popped up as one suggestion.
Some posts in http://forum.pfsense.org/index.php/topic,32256.msg238508.html give some more details on netflows, a book reference and some examples of the sort of reports that can be generated by the flow-tools package.
-
I'll check out logging flow on google (Mule) has popped up as one suggestion.
Some posts in http://forum.pfsense.org/index.php/topic,32256.msg238508.html give some more details on netflows, a book reference and some examples of the sort of reports that can be generated by the flow-tools package.
Thanks checking them out now and added my two cents as well. ;)