PREVIOUS
City of Oakland Strengthens Police, Fire and EMS Dispatch with Motorola Solutions
The AT&T wireless network experienced a widespread outage in the early morning hours of February 22, 2024. This outage impacted service to AT&T’s commercial customers and affected some FirstNet subscribers. AT&T took immediate action to prioritize the restoration of public safety’s communications on FirstNet.
While the prioritization of FirstNet was important, the network did not perform up to public safety’s standards that day. Both the First Responder Network Authority (FirstNet Authority) and AT&T have taken action to help prevent FirstNet from experiencing an outage like this in the future.
The FirstNet Authority is responsible for overseeing AT&T’s performance under the Nationwide Public Safety Broadband Network contract. Since February 22, we have coordinated with our network contractor, AT&T, to verify the cause of the outage, review the processes that led to it, and better understand the impact to public safety users of the network.
Through the work of AT&T and our After-Action Task Force, we now have a more comprehensive picture of the outage and events surrounding it. I directed the Task Force to review the incident from every angle, and they met the challenge.
The Task Force cast a wide net, working extensively with AT&T to review the incident in depth, and consulting with FirstNet users and other partners, including the Public Safety Advisory Committee. The team reviewed AT&T’s after-action assessment of the root-cause analysis of the outage, as well as third-party crowdsourced data to further understand the impacts and extent of the outage.
The team confirmed AT&T’s initial finding that the outage resulted from an incorrect process used during a network expansion activity. This process inadvertently triggered defense mechanisms, which ultimately prevented users from connecting to the FirstNet network between 3:45 a.m. EST and 6:00 a.m. EST, at which point public safety’s network was restored.
When I stood up the Task Force, I challenged them to not only assess the root cause of the incident, but also to determine opportunities to improve how we manage these types of incidents.
The Task Force examined areas, such as post-incident communications, information sharing with public safety, and network policies and procedures, that are critical to supporting the response to incidents of this magnitude. The Task Force then produced the following five key recommended outcomes, which aim to improve future response to network incidents:
Maintenance and Notification – This outcome focuses on coordination between the FirstNet Authority and AT&T on maintenance and network update protocols that impact, touch, or leverage systems that support the FirstNet network. This would include routine maintenance training and/or exercise opportunities, as well as improvements to network outage alerts and notifications to FirstNet subscribers.
Public-Private Partnership Communications – This outcome aims to foster more comprehensive information sharing, coordination, and communications between AT&T and the FirstNet Authority for network impacting events.
Operational Response Planning – This outcome looks to facilitate more complete All Hazards Emergency Operations planning between the FirstNet Authority and AT&T, so both entities can better prepare for, respond to, and communicate effectively during planned and no-notice network impacting events.
Stakeholder Communications – This outcome aims to share verified information on network impacting events as quickly as possible with FirstNet users and public safety stakeholders.
Continuity of Operations (COOP)/Continuity of Government Plan – This outcome ensures the FirstNet Authority’s continuity of operations plan, or COOP, is executed in coordination with AT&T to ensure the FirstNet Authority provides public safety stakeholders with accurate, timely situational information about FirstNet’s operating status.
I thank the Task Force for its dedication. While this network outage has tested us, it will ultimately make us stronger. With the Task Force’s work complete, the FirstNet Authority has been actively working with AT&T to execute improvements. In many cases, the FirstNet Authority and AT&T have already implemented measures to support these recommended outcomes.
Network outages can happen. By collaborating with AT&T and acting on the Task Force’s findings, we will be able to know and understand the impact of network disruptions sooner. These efforts will help us better address network disruptions and surge to share information with public safety.
As the federal agency overseeing FirstNet, we take our responsibility seriously and remain committed to delivering the resilient, reliable broadband network that Congress envisioned and public safety asked for following the tragedy of September 11, 2001.
