This is the 235th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization, mobile, networking, wireless, cloud and SaaS, or MSPs for the series PM Eric to get started.
I’ve done my fair share of contract work, bouncing from project to project while still maintaining a regular IT job — whatever “regular” is. The thing I see repeatedly is hastily thrown together network solutions (also know as “just get it working, and we’ll do it the right way later.”) This is a story about an office where they threw cables together and decided they would work out the details later.
The company had a large satellite office with about 100 users. That office had complained for a few months about network issues. You know, the normal things like, “everything is running slow,” “I randomly lose connection,” and “it takes forever for stuff to load.” We would look into the problems and most of the time we would find nothing wrong and kick it down to the tier two/remote desktop guys to handle.
This continued until one day we received a call from the remote tech assigned to that office stating they had a really bad electrical storm and one of the switches would not power up. We asked him if he had removed the failed Cisco switch and put in the “emergency” Cisco switch. Keep in mind that this was a large satellite, so we could justify keeping a few extra pieces of equipment on hand. The tech replied, “No, I haven’t removed it because it won’t slide out of the back of the rack, and I have too many cables in the way on the front side.”
Oh no, red flag! We asked for a picture of the rack, and it looked like the top of the rack threw up CAT5 all down the front of the racked equipment. A small discussion with the IT director and two of us were on the road to assess what needed to be done and “fix it” before Monday.
We arrived Friday evening and quickly discovered that the local IT had done what was convenient instead of proper. You know the times when you need an 18-inch patch cable but all you have handy is a 7-foot cable? You can only jam so much of that excess cable into the cable management device until it breaks. Then what do you? In this case they simply tucked the excess cable in between the switches and let it hang down the backside of the equipment.
We ended up finding ports that were damaged from what we assumed with the weight of the cable ball pulling down. We found custom-made cables where the wires had been pulled loose — again probably from the weight of the cable ball — and equipment that had been suffocated but somehow was still functioning even though it was hot to the touch. So, we made a call to the director and explained that we would need to re-cable almost three full racks to make sure the problems were fixed.
We then spent the rest of the night and most of the early morning documenting what was in place to allow us to strip out all of the cable balls. I don’t know what time I finished working that morning as I simply sat down at someone’s desk and fell asleep.
Saturday morning, we sent the local IT guy out to get 2,000 feet of CAT5 cable while we went to get something to eat. When we got back to the office we did the easy part — disconnecting the cable balls. We ended up throwing away almost all of the cable we removed since no one had the time to untangle them. We then proceeded to make the needed cables and literally rewire the three racks.
By Saturday evening, we decided we needed to get some rest since still had a lot of work ahead of us.
Sunday was spent testing each and every wall jack in the office and each and every Cisco switch port. We replaced two switches and marked a handful of other ports as bad. By the time we left Sunday evening, the office had a clean bill of health
The point in my story is we should always think the solution through first, then document the solution, and then apply the solution. Had the local IT support thought through the possible issues with sloppy cabling, I believe they would have managed it better.
Have you dealt with (or been responsible for) a cabling catastrophe? Share your tales, tips or questions in the comments below!