Network Upgrade and Rack cleanup

johnpoz

So just got back from work trip.. Pretty happy with how it turned out, and wanted to share somewhere it might be understood how hard of a job it was, not just unplugging and plugging wires back in ;)

My daughter-in-law said - oh so your a professional organizer, when my son showed her the before after pictures ;)

We replaced 3 HP 5412's with 3 stacks of Cisco 9200's 1 4 stack, and 3 5 stacks. The HPs had been running for 18 years, at least that is the oldest of the bunch. How the cables got to that state I have no idea, other than years of years of cable additions and changes.. While not the worst I have ever seen, it prob has to be some of the worse I have ever had to personally work on fixing up. I have redone access closets before - but those we took down the whole closet and had the wiring redone by contract.. With this we could not take down production so had to move 1 cable at a time. Was 3 of us doing it.. The 1st one we pretty much did in a day, long day ;)

This 2nd one took us the rest of the trip to finish..

We took out the old cable managers and slid the new switches in between the panels, and then started tracing 1 wire at a time and making sure the new port was on the correct vlan and then moving the connection as fast as possible so that didn't reset the connection. We didn't get any complaints of anything going down that was not expected - there was some stuff that needed to be moved was going to cause a brief outage, we had to do those after hours. We put in UPSes and PDUs and moved to stuff that was staying to the UPS so that was a blip.. Patching in the ports that were no up and runs that were not patched before helped so that not everything had to be done fast.

There was some stuff we couldn't touch, so couldn't replace the patch with new slims, so the cable management isn't as nice as it could of been.

But overall think it came out pretty nice.

I will be retired (7 years to go - hoping maybe 5 but will see) before these should ever have to be touched again.. But we are wondering how long it will last looking like this - but since we patched all runs to ports, the only reason cables should have to be touched is if they add more runs to these racks - and if so we can just add a switch to the stack, etc. And there are some free ports that could be leveraged if they just add a few more runs.

patient0

@johnpoz but now it's boring, no more half an hour following a cable to see where it goes! :).

Great work, no incidents caused?

johnpoz

@patient0 not that aware of - have to look into something with the card readers in one of the buildings this morning - but we didn't do anything in that building so its prob not related.

We did make a few mistakes in tracing wires, but we caught them very quickly when checking devices were on the right vlan, etc..

And sure there were a few brief outages when couldn't get the cable plugged in fast enough so the interface on other end reset. If you can move the cable in like 2-3 seconds normally the other end won't reset and you might loose like 1 ping..

And sure the phones that are poe had to reset, and AP when we moved those - but did the AP after hours, or on the weekend and we only did a couple of APs at a time - so clients should of just moved over to another AP while the ones we moved rebooted on loss of poe.

But we didn't get any complaints that xyz was down, etc. But will see over the next couple of days ;)

Phizix

@johnpoz
OMG it looks like the panel had barfed spaghetti! :o

johnpoz

@Phizix yeah it was insane - its for sure not the worse have ever seen but yeah pretty bad.. There were cables that crossed from one rack to other - there were cables wrapped around bunches of other cables..

Whole trip was 5 days.. With other than the last day being 12+ hours each.. First day we spent some time unboxes and configuring all the switches. Takes some time to unbox 19 switches and configure them in stacks, and then load base config.. Then take them to the location - its campus really with multiple buildings, and you know where we had room to configure the switches was a few buildings away. At least we didn't have to sit on the floor with laptop consoled in to configure them all ;)

We had 1 ups go missing, stuff had been delivered months ago - and the ups was received, but nowhere to be found. So the switches in that first rack, the 2nd psu in each switch is still on raw power until we get replacement ups.

We spent the last day final touches on cabling - pulling the old 5412s out of the racks and configuring the sup modules for use as spares for the other 5412s left on the network - those are slated for replacement sometime later this year or early next. But this dual rack was by far the worse, the other racks are more like the 1st rack.

dennypage

@johnpoz Nice work John.

AndyRH

Excellent work!
Our DC stays clean for 2 reasons, only 10 people can get in and the floor manager is not to be trifled with, I got fussed at more than once. The closets vary by location, some of the site manages are picky, others are not.

Good luck keeping it clean.

slu

@johnpoz

I love this kind of work (hardware and wiring), you can see in the evening what you have done...

johnpoz

@slu well it took a few days to see what we had done ;)

Here are some pictures after day 1, 2 and then 3.. The fourth day was final clean up and removal of the old 5412s

Those boxes there with the old cables we filled up a few of those ;) It went faster as we worked through the cables and clearing them out - we had to go 1 cable at a time to minimize any disruption to production..

It was a real rats nest at the bottom.. Tracing each cable took a while, especially when first started.. After that first day it didn't even look like we had accomplished anything..

And we couldn't even get all the switches into the racks at first without clearing cables so their was enough slack to slide in the next switch in the stack, etc.

For a while they we were wondering if we wouldn't finish before we were schedule to fly home. But after day 2 we could actually see progress.

dennypage

@johnpoz Pic 1 looks like someone came across a "too good to pass up" discount on 5 and 7 meter cables.

stephenw10

Nice!