A Comcast voice outage yesterday illustrates the risks of network automation and virtualization and the move to the cloud.

Reports of service outages first surfaced yesterday morning, impacting business users in multiple markets nationwide, according to online news website The Verge. The report cites a statement from Comcast issued around 3:30 p.m. ET noting that the company experienced a service interruption with Comcast Business Voice and VoiceEdge Select services and that service had been restored for most customers. Comcast apologized for the outage.

Comcast Voice Outage
Comcast Business Voice Edge, launched in 2012, is a cloud-based voice and unified communications offering that eliminates the need for certain premises-based equipment and provides convenience features such as voicemail-to-email. But while the cloud-based approach enables these capabilities, it also magnifies the impact when something goes wrong. With traditional technology, outages generally were confined to relatively small geographic areas.

comcast voice outage

Comcast Voice Outage Heat Map (Source: Down Detector)

The situation is reminiscent of the 911 outage that affected seven states in 2014. A software issue with a cloud service used by 81 public safety answering points (PSAPs) caused all those PSAPs to become inoperative.

The FCC investigated the 911 outage and came up with six recommendations aimed at preventing future outages of that nature. While some of that advice pertains specifically to 911 service, some recommendations could make sense for a company like Comcast offering VoIP, UC or other cloud-based services.

For example, the FCC recommended that all parties involved in end-to-end call completion should practice situational awareness to better prepare for potential outages. The commission also recommended the development of a comprehensive set of best practices.

Ironically, the FCC noted that increased use of cloud-based 911 services were accompanied by a rise in “sunny day” 911 outages resulting for “purely technical reasons,” illustrating the dark side of technology advances.

It would be fruitless and even foolhardy to seek to prevent cloud-based services, virtualization and automation or to prolong the use of less efficient decades-old technology, however, as those technology advances can bring many benefits. Instead, outages such as the 2014 PSAP outages or Comcast’s voice outage yesterday remind us that the industry shouldn’t leap into the new technologies without carefully considering the downside and how it might be prevented.