Data Centers and Wildfires

As will be discussed in our upcoming note about managing data center with the severe weather caused by climate change, we continue to highlight the need for data center managers to not only review existing emergency plans but also anticipate previously unforeseen challenges. The need to understand the new ‘Normal’

Which brings us to wildfires… For any organization, the health and safety of their workers is the first priority. As we’ve learned through the pandemic, tiered responses and good communication are key to managing unexpected risks. Wildfires, however, present a range of specific challenges to mission-critical facilities. Below we’ll discuss a few key concerns.

The fire itself

To ensure clean, inexpensive electricity, many data centers are sited near hydroelectric power sources. That generally means nearby mountains and forests — areas increasingly at risk of wildfire in an age of climate change.

Road closures and the evacuation of towns in the path of the fire could pose immediate risks to staffing levels: workers may not be able to get to the site or may need to deal with their own emergency situations. Regularly review your contingency plans, and ensure personnel are advised of updates. Stay abreast of construction, road repairs or other impediments that might impact alternate routes. Develop and test notification plans in advance.

There are other risks as well. Wildfires could burn transmission lines, interrupting the flow of power to the facility. Post-fire, burn scar erosion can cause landslides that take out fiber cable — sometimes in multiple areas and in so doing, thwart redundancies. With climate change, the risks extend beyond the imminent, and preparation is key.

Wind

There doesn’t need to be a fire for high winds to affect data centers. After the disastrous 2018 Camp Fire in California that killed 86 people, PG&E (Pacific Gas and Electric Company) announced a policy of pre-emptively interrupting service to at-risk areas when forecasts indicate conditions may increase the risk of wildfire (e.g., high winds in an already hot, dry environment). These planned outages may continue for a decade, as the utility company upgrades and hardens its systems to deal with the risks more extreme conditions bring.

On-site power generation is one option data center operators might consider as a hedge against uncertain utility power supplies. However, a smooth transition to on-site power is never a given, no matter the amount of preparation. As a result, utility power outages — planned or unplanned — increase the risk of failure for data center operators.

Facility managers need to consider not only their risk from the direct line of fire but also from blown embers. Wind-blown embers can spark dry grasses at fence lines, collect near fuel tanks and even ignite employee vehicles in the parking lot (e.g., a canvas pickup bed cover). Think expansively about the possible risks and mitigation strategies.

Smoke and ash

Facilities many miles away from an active wildfire can be affected by it, potentially for weeks at a time.

All data centers are required to have some form of fresh air supply. This is necessary to replace the carbon dioxide we exhale with oxygen. That means managing the flow of ash and other particulates into the facility via the ventilation system. Beyond that, ash can be transported into the facility on workers’ clothes, maintenance tool kits, equipment and more – even slight movements can cause it to become airborne. All data centers near a wildfire are at some risk from smoke and ash infiltration and should take action to mitigate contamination.

Engine generators — which may well be pressed into service — pose particular concerns. Most have filters on the combustion air intake (and those will likely need to be replaced more often than usual), but there is rarely any filtration on the intake for the engine cooling. Consult with manufacturers to learn more about appropriate remedial measures that do not impact performance or affect warranties.

The fuel storage tanks and the fuel itself also require attention. Almost all diesel tanks are atmospheric types, which means they have vents. These vents allow air into the tanks when fuel is withdrawn and allow a path for air to escape when fuel is added. With the anticipated increased generator operation, the vents will be pulling in more air — which means ash can get into the tanks. Over time the ash will settle to the bottom of the tank but in the short term, it can be picked up by the fuel pumps and clog fuel filters. Longer term, the impact on fuel quality should be assessed.

The cooling system of almost all facilities will be affected in some way. Condenser units for air conditioners, air-cooled chillers, or cooling towers are all susceptible to ash in the air, as it will either contaminate the water in a cooling tower or clog condenser coils. This will result in less cooling capacity, and backup units may be similarly affected. Direct air-cooled data centers need to monitor air quality and adjust their filter maintenance schedules as conditions dictate.

Data center managers experiencing the impact of wildfires should document conditions and actions taken in response, then share lessons learned with others. As in the aftermath of Superstorm Sandy, knowledge sharing through organizations like the Uptime Institute Network can be invaluable in learning to cope with the new reality of a climate-changed world.