Network monitoring on an existing network
-
I have a pfsense firewall running on what is a relatively large home network. This has been running for many years with little to no probems. The network monitoring tools on the firewall have been very helpful over the years for troubleshooting issues with the network. Recently I started doing contract work for a local business who does not have anyone handling their IT. One of the senior employees has been handling the network for quite awhile, but after a recent expansion he has quickly realized that he is out of his element and can no longer solve the endless issues at their new location.
Currently the business has two locations. One is a studio with a Linksys RV082 Multi-wan router, a few PCs, one Windows 2003 Server (solely for file share), and a basic AP. This network has had no issues related to the network connection. The PCs being Windows XP and 2000 are the biggest issue there. But that will be remedied once they move those offices. This office is fed by an 10mb/1mb DSL connection.
The second location is the business's main office. This has all of their sales staff, a digital division (Web design etc..), IP PBX, and a Linux server I recently setup to move their file sharing off of the antique Windows server. This office has the same Linksys RV082 Multi-wan route the other location being fed by an Time Warner 20mb/2mb cable connection. Both of these routers are connected through an Ipsec VPN. Currently this location has been suffering from painfully slow connections, to intermitent connection loss. I am trying to establish where the problem lies, internal or external.
Their existing routers do not have the tools I need to monitor the network traffic. So what I would like to do is place a pfsense firewall in between the Linksys router and the main switch at the main office so I can utilize the tools in pfsense and maintain a log. What would be the best way to go about this? So far I have tried using the pfsense box with two lan connections bridged, one LAN to the Linksys and the second LAN to the switch. This works for passing traffic, DHCP, etc.., but there is no traffic monitored in RRD Graphs, BandwidthD, or Ntop. So I tried using the WAN conneted to the Linksys router and one LAN connected to the switch. Then disabled packet filtering. With this setup, no traffic or DHCP is passed to the internal network through pfsense. I also lost access to the WebGUI through the LAN connection.
I have been scratching my head over something that I imagine is probaby simple so some input on this issue would be great. If you think I should try to troubleshoot this a different way, I am open to ideas one way or the other. Also, the customer's switch does not support port mirroring so to properly monitor the network traffice I will have to utilize something in-between the router and switch.
-
Something like Solarwinds NPM if you have a budget, or Cacti / Nagios / Zabbix if you don't. There's also OpenNMS but, IMHO, it has a pretty steep learning curve still.
What you want is a network management platform that queries all your devices and interfaces using SNMP and keeps a historical database for graphs, watches for anomalies/errors/traps for notifications via email, SMS, TAP, etc. You also want a syslog server so the log entries you need to see aren't scrolled away in 15 minutes or whatever.
I'm so much happier after eliminating the IPsec Cisco RV042s I had in my world and replacing them with 2.1.5 Alix/APU and OpenVPN.
-
There's also a cool Mac App I use called PeakHour that allows you to set up a quick SNMP monitor of a few interfaces and displays them live. I think it was free. Not sure if there's something available for your platform.
-
I'll check those out, currently budget falls on me since they will not spend money until there is a definite answer to the problem. So at this point free would be ideal, but if the cost isn't too outrageous I would consider investing in tools that I would use again.
I do not have a MAC handy to run that particular SNMP program. Right now I run Visual Syslog on a Windows machine. It is a free tool that works fairly well, but my switch and one of my Access Points does not support anything other than SNMP. That is of course my personal equipment. Their setup is far less advanced. The way the new network at this business was thrown together is a small nightmare. The connection starts at the RV082 router, which feeds a couple PCs. Then it runs to one end of the building to a Netgear 8-port switch to supply the Linux file server and four more PCs. That switch then feeds another Netgear 8-port switch which then feeds their Xerox copier and a 7 year old "Business Class" Dell Access Point (running WEP security no less). I can't walk from room to room without shaking my head in disgust. I am certain of what the problem is, but without some proof to show that most of the problems lie with the internal network setup I can't get them to spend money on fixing the mess.
In a few months they will be consolidating the two locations into the main office. I already have them convinced to remove the Linksys router when that happens and put in a new pfsense applianc. In the mean time I would like to show them the major issues with the existing setup and bring in the new equipment to have the proper network in place before more equipment is in place and I have more to work around.
-
IF they have done their own cabling, dont underestimate one or more ethernet cables being terminated poorly which will cause any number of packets retries and subsequent delays & dropouts in the network.
One of my customers is a cable manufacturer and I was for ever telling them to re terminate their cables because they would phone up complaining my software was not working, back then (2000) it was an isam database which means all workstations have to communicate through a transaction file before CRUDs could take place, but with SQL servers that problem is masked now.
Likewise oplocks on windows can also cause delays with some file reads and writes and you certainly have some bottlenecks with the way the network is setup. Can they not invest in a decent switch and run cables properly instead of daisy chaining switches?
Or failing that do you have some equipment and cables you can loan them to remove the bottlenecks to show them?
Either way good luck, I've had customers like that and I choose not deal with them now if I can help it.
-
They ran over half the cable themselves, the rest was pre-existing. The nice part about that is that I ran the pre-existing cable, this was for the previous owner of the building. The cable ran by this business has it pulled through half-a$$ed, some of it is over fluorescent lights (not shielded cable), half-crimped between drop ceiling tiles and the track. Everything has been done the quickest and cheapest way so far.
I do have an extra basic switch I could put in my home network to pull my 28-port managed switch out for testing purposes there. The problem with that is, they need to have a lot of their new cables re-run to accommodate a single switch. The reason they have daisy chained switches is to get out of the longer runs. The more I think about this, the more I just have to sit down and convince them that even proper testing can't be done without some changes first.
-
Well as long as the cable isnt longer than 100metres you should be alright, although stick with Cat6 and check its shielded otherwise the EMF from flo tubes and other electrical equipment can interfere with the data like you mention. If you are doing the cable runs yourself, try to get a drum/reel of cat6e, although technically that spec is not recognised, its the best you can get and tbh the copper price has been falling a bit recently so should be getting cheaper as well if they turnover their stock alot otherwise you could be paying a high price for old stock and be wary of chinese fakes.
Theres a number of cable manufacturers on alibaba passing off other cable manufacturers images & specs as their own or claiming to be able to supply a cable manufacturers product when the cable manufacturer only sells to end users or other recognised stockists.manufacturers. If you can go direct to a major supplier/manufacturer the prices are alot lot cheaper than what most people pay for at retail for a drum/reel, markups are huge in that industry, and you'll probably want the sheath to be Low Smoke Zero Halogen to comply with fire regs.
There are some free apps around which you can run on workstations to make sure the data transmitted is perfect, ie no retries although rodents and electrical interference generally cause the biggest problems if its not poorly terminated.
On the point of cost, it pays to be assertive (most CEO's are psychopaths apparently and can sense weakness easily) so maybe do a deal where you agree a price for the work & cable upfront, switch to be decided at a later date and if all are happy, do the work, like a sort of sale or return deal, that way once you have shown them, they dont go off to someone else for the cheapest price.
One way I explain how networks run to customers is to imagine a packet is a car, and the network is like a road network, then point out the bottle necks where they have switches plugged into another switch and how networks can grind to a halt when they are really busy like when everyone logs on in the morning, just like rush hours can be in cities. Switches that support LAGG can improve throughput just like dual and quad nics can when placed in a server. Through in the difference in speed with devices, ie some home routers run slow arm chips and although stable, not always the fastest compared to a decent switch which can also help improve matters and it all starts to add up.
It will also pay to workout what machines are being used to store/serve huge amounts of data they might use and see if they can be positioned on the network better, or maybe see if you can offload some of the shares/work to other servers/machines less busy, like dept servers.
If they dont mind, although its not ideal, you might be able to have the longer cable runs lying around on the floor for a day as a temporary solution to save you having to scrabble around in the dusty roof spaces with your switch. This way provided its not something like a bad port on a switch or bad nic in a machine, you save yourself some time showing them its their bad cabling that is almost certainly at fault.
Either way good luck, I know whats it like. ;)
-
If there are errors or down/up events on the switch ports, look at the cabling, jacks, etc. If not, you're wasting your time
How far are the "long" runs?
-
The longest cable run is 120 feet, so well under the cable limits.
-
If you have the budge ($20k), SolarWinds NPM, hands down.
If not… SpiceWorks just released a stand-alone Network Monitoring application. It's a new product, customization options are limited, and it only supports WMI and SNMP at the moment. But, it's free, it works, and it has a very simple slider to control how much noise it'll generate. (Alerts)
...ct