Here is an overview of the issue I am faced with. I am looking for additional troubleshooting tips and/or ideas from people who might know more about the hardware than me:
We have a new client with mostly Cisco networking equipment. Cisco ASA 5505 firewall and multiple switches. There is a satellite office across the street from the main office, and it is attached with a dedicated T1 line. The satellite office does not have its own internet connection, so everything goes through the T1 back to the network closet in the main building. There are identical Cisco boxes with Cox Communications labels in each of the offices (with only 1 cat5 coming out of each, and going into a wall jack into the Cox equipment), and I am assuming this is the hardware that is connecting the sites.
The main building's subnet is 10.0.0.1 and the satellite is 10.1.0.1.
Starting almost a month ago (before we signed on, of course), at certain points in the day, the network performance drops to abyssmal levels. Most notably after 415pm, but not only in those times. My first clue was that at 4pm, pings across the T1 to the server were averaging 5ms, but at around 4:40pm pings start averaging 60-90ms and will occasionally drop. Pings from the server back to any device in the satellite office gives the same result. Pings within the main building are <1ms.
Today, I noticed a strong correlation between internet usage and ping latency. While continuously pinging a PC from the server, I ran a speed test on the pinged PC and noticed a spike while the test was running. I pinged a completely different device while running the speed test again on the original PC, and I still see slowdowns while the test is running, even though I'm pinging another IP.
So that tells me that it has something to do with the bandwidth over the T1. We recently added a wireless AP to that building (which they didn't have before) and I'm wondering if the additional connections from phones and iPads could be messing things up.
Any ideas? My coworkers and I are looking for any smoking guns that could obviously cause any slowdowns during those times, and it is almost looking like it could be any significant (or otherwise) download that does it.