When a weather-related or other emergency strikes, telecom disaster recovery ranks right up there with power and sanitation in terms of importance. Telecompetitor was reminded of this today when we dropped in on an AT&T telecom disaster recovery drill in Chicago – one of numerous such drills that the company conducts around the country every year.
“It’s like our own little village,” commented Kelly T. Morrison, AT&T senior technical specialist, in reference to 15 semi-truck trailers situated in a parking lot at Soldier Field. Normally home to pre-Bears game tailgate parties, the parking lot for one week will host 50 AT&T employees daily, who will set up and program all the equipment they would need in the event that, for example, a central office were to go out of service or the company were to lose a portion of its cellular network.
The Village
Some of the semi-trailers contain telecom routers, cross-connects and other typical CO equipment, while others provide power, satellite connectivity and other support functions. When necessary, AT&T also brings food for everyone and provides recreational vehicles that employees can sleep in.
The idea is for the AT&T telecom disaster recovery team to be self-sufficient for at least a week or two, Morrison explained. “We don’t want to be a stress point for local responders,” said Morrison.
The company generates its own power. Cabling connects trailers housing generators with trailers containing telecom equipment, creating what Morrison likens to a “giant multi-strip outlet.” Fiber optic cabling also interconnects some of the trailers.
There is also a command center, where AT&T employees during a disaster would coordinate activities with police, fire and other first responders.
The Telecom Disaster Recovery Team
Morrison is one of 30 AT&T employees devoted full time to telecom disaster recovery.
An additional 120 AT&T employees have traditional operational jobs but devote part of their time to telecom disaster recovery. These employees would be dispatched to a disaster site as needed. At least twice a year each of them also participates in an exercise such as the one in Chicago this week.
Their participation not only helps keep them ready to respond when needed; their daily experience with the latest telecom equipment also benefits staffers who don’t do that kind of work so continuously.
“Their skills help train us,” observed Morrison.
Network Connectivity
Telecompetitor arrived at AT&T’s telecom disaster recovery test site near the beginning of the week-long exercise, just as the company was raising a satellite dish to support cellular communications. The dish is used to provide backhaul between individual cellsites on light trucks (COLTs) and AT&T’s earth station. A smaller dish sits on the top of a mobile command center to support WiFi and wired connectivity for the disaster recovery operations.
In the event that AT&T were to lose connectivity to a central office supporting landline services, a fiber connection would have to be spliced into an existing connection to AT&T’s core network.
Several of the trailers are designed to essentially replicate the functions of a central office. AT&T keeps a central database showing the type of equipment in each of its central offices, along with detailed information about how the equipment is configured. Technicians can pull this information from the central database using a T-1 connection and can use the information to configure the equipment at the disaster recovery site. Alternatively the information can be transferred via a thumb drive.
As AT&T’s network transitions more and more toward IP communications, the task of replicating existing landline services becomes easier because many legacy services can be carried in IP format, Morrison noted.
Normally AT&T doesn’t actually move traffic from a central office to the emergency facilities as part of a disaster recovery exercise. But the company will find an unused dark fiber connection that it can use to connect to the CO as part of the drill, Morrison said.
The longest time Morrison recalls one of AT&T’s disaster recovery villages existing was two month, which occurred after the attacks on the World Trade Center in 2011. AT&T isn’t the incumbent local carrier in New York but the company “helped out with cellular coverage,” Morrison said.
Some of the toughest working conditions occur after a major flood, Morrison noted. During one such operation, recovery workers found fish a mile and a half from the water, he recalled.
A portion of AT&T’s disaster recovery team is trained on hazardous materials for working in the kinds of septic conditions that the company can encounter after a flood or certain other types of disasters.
