This is the 186th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization, mobile, networking, wireless, DNS, or MSPs for the series PM Eric to get started.
Shortly after I started my career with my current employer, I was tasked with researching a solution to one of our networks biggest shortcomings: downtime.
Downtime for us was a major issue. Each day we would have periods of time when services and applications were unavailable at various offices — not because our servers were causing problems, but because our ISP’s connections could be considered spotty. A few minutes here, a couple hours there — these common occurrences put our small three-person department on edge each and every day just waiting for the inevitable flood of calls.
Our “home” office is based in a fairly rural area, which means our connectivity options are limited. Our offices are scattered over three counties and up to around 60 miles apart. At the time I was tasked with this problem, our offices were connected with T1 lines and ISDN backups using Cisco routers that required a manual switchover. Every time a line went down for an extended period of time, someone would have to travel to the office and reconfigure the router to use the ISDN line, then go back to switch it back over when the problem was fixed.
This caused severe headaches for our customers and major frustration for our employees who rely on up-to-the-second data from our central servers. In the financial industry, a delay in receiving or sending data can cause huge problems.
The network I maintain is a widely distributed series of offices that provide financial services. We have offices that have as many as 60 or so active users, down to offices with as few as two to three. Can you image trying to run even three to five users on an ISDN line when your primary connection is unavailable for hours? Trust me, it’s a nightmare you’d rather not face just because someone dug a hole in the wrong place!
Suffice it to say, offices were closed sometimes because we were simply unable to provide customers and employees with the services and information they needed, when they needed them.
The problem posed to me was: How do you make a secure yet highly available and fault-tolerant network connection between 12+ offices with limited funds — and even more limited services available?
We found many options available to us, but the cost of those solutions was beyond what our small company could comfortably afford. Thankfully, I stumbled upon our “Holy Grail” for the problem: Network bonding.
Network bonding is a method for using two or more network connections that appear internally as a single larger connection. There are plenty of commercial devices out there that can perform this task, but in my case it needed to be accomplished with little to no budget.
The key factor in my “Holy Grail” solution was my discovery of a FOSS Linux build called Zeroshell. Zeroshell is a minimal Linux build that creates a router platform capable of bonding, VPN, load balancing, automatic failover, and much more. I found my platform, and to implement it I needed some hardware.
Like most companies, when we replace old PCs with new ones, we keep the old stock until the next round of replacements, so we had some reusable hardware. Next, we needed a plethora of network cards. Each of these PCs needed to have at least three NICs installed: One for the internal interface and one for each network connection we were going to bond together.
Foregoing expensive T1s and relying instead on business- or residential-class cable, DSL, and FIOS services with static IPs was the next logical choice. We needed each office to use two or more different ISPs. If one ISP had problems, the other ISP(s) would typically still be up and running. Beneficially this also meant using different connection media as cable and DSL use different physical methods for transmitting data, thus further reducing the chance that if one service was struck down that the others would be affected as well.
Using this combination of multiple ISPs with various types of media, ZeroshelI Linux, and old desktops with some extra network cards not only resolved 99 percent of our downtime problems, but it also provided us with a sustained overall bandwidth boost at every branch (and at a lower monthly cost than our old dedicated T1 lines). All this was accomplished while keeping a zero-to-minimal overhead cost for hardware by recycling old machines that we already had in stock.
No project ever goes perfectly, of course. It did take some time to get everything just right. A few of the recycled PCs had to be replaced (due to bad caps on motherboards or because they had just become unreliable) with newer or more powerful ones. When we started they were all Dell GX150s with P3s and 256mb. We had to upgrade a few of the larger offices, which thankfully were in more urban areas, to more robust Ethernet connections, but we maintained their smaller cable and DSL lines for failover. All in all it’s been one of the most successful technology rollouts I’ve been a part of, and it’s been very well worth it.
Charles Carmichael is a favored pseudonym for my use on Spiceworks where names are not important, only experiences. Often used as an alias in confidential operations, it is the subject of much humor between SpiceRex and myself. The alias is also often used in assignments where I am disguised as an IT hero.