Authored by Bill Smith, AT&T’s President of Network Operations
By all accounts, Superstorm Sandy was a massively destructive event that resulted in more than $70 billion in damage and caused devastating losses for many in the impacted area. More than eight million people lost power, which was a record for a storm-induced power outage. AT&T has a $600 million, 300-vehicle network disaster recovery organization that has a 30-year history of preparing for the full spectrum of “all threats” and that conducts full-scale field exercises across the country and overseas several times each year. We utilized these critical resources to prepare and respond to Sandy in the most effective way possible.
No matter the disaster at hand, in every case, the key to an efficient response is planning and preparation. This includes prepositioning assets and leveraging the flexibility to address different events – a blizzard is a very different challenge from a hurricane, an earthquake or terrorist attack. We try to anticipate, plan and prepare for the entire spectrum of possibilities, and we regularly train and re-certify our employees and partners that support this process. A key focus at AT&T is our ongoing post-event process to identify lessons learned from our experience and from others during each event.
As Sandy headed our way, AT&T began monitoring the projected path, intensity, and impacts with our on-staff meteorologist – that’s right, we have a dedicated Weatherpro. With our presence in Puerto Rico, we have experience with Atlantic hurricanes before they make landfall on the mainland. Resources prepositioned throughout most of the Mid-Atlantic and eventually the entire Northeast region included Emergency Communications Vehicles, Cells-on-Wheels (COWs), Satellite Cells-on-Light-Trucks (COLTs), over 3,000 generators, and a convoy of fuel tanker trucks to keep our network equipment going. We set up more than 100 staging areas across the projected impact zone, out of harm’s way, but close enough to the projected storm landfall to allow for rapid deployment. In addition to equipment, AT&T prepositioned additional personnel and provided emergency credentialing to ensure we could move throughout the area to rapidly restore communications. We had multiple command and control centers and even set up international communications channels for our overseas customers with assets in the affected areas.
We monitored the situation 24×7 at our Global Network Operations Centers (GNOC) to prioritize response efforts. We activated Local Response Centers and our Northeast Emergency Operations Center, which included 75 certified expert restoration managers. Our advanced staging of equipment allowed us to rapidly deploy emergency communications equipment to areas that were most impacted. We surged the restoration workforce by preparing 600 crews before the storm, which enabled us to dispatch at 5 a.m. when it was safe to enter the impacted areas. We declared a state of emergency with our union employees and partnered with them to implement mandatory 12-hour work days and six-day work weeks where necessary. We cancelled vacations, recalled personnel, and dispatched loaned employees from more than 20 states outside the storm-impact area. AT&T deployed more than 2,500 first responders “around-the-clock” until our network was back to normal. And we brought in fuel from out-of-state to ensure that our teams could get to places they were needed.
For cell sites to function they have to be connected to the core network through backhaul transport. For Superstorm Sandy, the loss of backhaul transport was the single biggest issue we faced. Therefore our restoration efforts focused on rapidly deploying generators to areas that were without power but that still had the transport to carry customer communications. We dispatched satellite COLTs to areas without wireless service, and rolled COWs to damaged or destroyed cell sites that had working terrestrial backhaul. In some cases where we lost terrestrial backhaul, we were able to establish temporary microwave connections. We also set up satellite data communications vehicles to support public safety operations with a number of fire departments, FEMA, and search and rescue operations.
As the breadth of the storm destruction emerged, we voluntarily instituted a sharing agreement with T-Mobile (which uses compatible GSM technology) to enable their customers and ours to roam onto whichever network was available in their location to maximize the ability of our subscribers to place calls or obtain information. We set up charging stations at AT&T stores and public aid locations throughout the affected area, providing relief from the elements as well as recharging, Wi-Fi and wireless communications.
This was a monumental effort by AT&T to ensure that our customers had the best communications network possible and to try to minimize the communications problems for all people in the affected areas. It was a tremendous effort by our employees (many whose own families and homes were impacted by the storm), and partners who traveled far and worked tirelessly to restore service.
These events were catastrophic for the people impacted by this disaster. I wish there was an “easy” button which we could hit to restore normalcy in a matter of minutes. Instead, we have to use each of these experiences to teach us how to make improvements in our processes so we are even better prepared the next time that catastrophe hits. We never know exactly where or when that next time will be, but we do know that we have assembled the best team of dedicated employees, expertise, and quality resources to tailor an optimal restoration response no matter what type of disaster strikes.