Data center operators will face more grid disturbances

The energy crisis of 2022, resulting from Russia’s invasion of Ukraine, caused serious problems for data center operators in Europe. Energy prices leapt up and are likely to stay high. This has resulted in ongoing concerns that utilities in some European countries, which have a mismatch of supply and demand, will have to shed loads.

Even before the current crisis, long-term trends in the energy sector pointed towards less reliable electrical grids, not only in Europe, but also in North America. Data center operators — which rarely have protected status due to their generator use — are having to adapt to a new reality: electrical grids in the future may be less reliable than they are now.

Frequency and voltage disturbances may occur more frequently, even if full outages do not, with consequent risk to equipment. Data center operators may need to test their generators more often and work closely with their utilities to ensure they are collaborating to combat grid instability. In some situations, colocation providers may have to work with their customers to ensure service level agreements (SLAs) are met, and even shift some workloads ahead of power events.

There are four key factors that affect the reliability of electrical grids:

Greater adoption of intermittent renewable energy sources.
Aging of electricity transmission systems.
More frequent (and more widespread) extreme weather events.
Geopolitical instability that threatens oil and gas supplies.

Individually, these factors are unlikely to deplete grid reserve margins, which measure the difference between expected maximum available supply and expected peak demand. When combined, however, this confluence of factors can create an environment that complicates grid operation and increases the likelihood of outages.

Two out of six regional grid operators in the US have reported that their reserve margins are currently under target. One of them — the Midcontinent Independent System Operator — is expected to have a 1,300 megawatt (MW) capacity shortfall in the summer of 2023. In Europe, Germany is turning to foreign power suppliers to meet its reserve margins; even before the war in Ukraine, the country was lacking 1,400 MW of capacity. There has been a marked increase in power event outage warnings — even if they are not happening yet.

Dispatchable versus intermittent

An often-cited reason for grid instability is that dispatchable (or firm) power generation — for example, from natural gas, coal and nuclear sources — is being replaced by intermittent renewable power generated by solar and wind energy. This hinders the ability of grid operators to match supply and demand of energy and makes them more vulnerable to variations in weather. Utilities find themselves caught between competing mandates: maintain strict levels of reserve power, decommission the most reliable fossil fuel power plants, and maintain profitability.

Historically, electrical grid operators have depended on firm power generation to buttress the power system. Due to the market preference for renewable energy, many power plants that provide grid stability now face severe economic and regulatory challenges. These include lower-than-expected capacity factors, which compare power plants’ output with how much they could produce at peak capacity; higher costs of fuel and upgrades; stricter emissions requirements; and uncertainty regarding license renewals.

The effects are seen in the levelized cost of electricity (LCOE), measured in dollars per megawatt-hour (MWh). The LCOE is the average net present value of electricity generation over the lifetime of a generation asset. It factors in debt payments, fuel costs and maintenance costs, and requires assumptions regarding fuel costs and interest rates. According to financial services company Lazard’s annual US analysis (2023 Levelized cost of energy+), utility scale solar and onshore wind LCOE (without subsidies) is $24-$96 per MWh and $24-$75 per MWh respectively. Combined cycle natural gas plants, for comparison, have an LCOE of $39-$101 per MWh, while natural gas peaking plants have an LCOE of $115-$221 per MWh.

The economic viability of generation assets, as represented by LCOE, is only a part of a very complex analysis to determine the reserve margin needed to maintain grid stability. Wind and solar generation varies significantly by the hour and the season, making it essential that the grid has sufficient dispatchable assets (see 24×7 carbon-free energy (part one): expectations and realities).

Grid-scale energy storage systems, such as electrochemical battery arrays or pumped storage hydropower, are a possible solution to the replacement of fossil fuel-based firm power. Aside from a few examples, these types of storage do not have the capacity to support the grid when renewable energy output is low. It is not merely a question of more installations, but affordability: grid-scale energy storage is not currently economical.

Grids are showing their age

Much of the infrastructure supporting electricity transmission in the US and Europe was built in the 1970s and 1980s. These grids were designed to transmit power from large, high-capacity generation facilities to consumers. They were not designed to manage the multi-directional power flow of widely distributed wind and solar generation assets, electric vehicle chargers which can consume or supply power, and other vagaries of a fully electrified economy. The number of end users served by electrical grids has since increased dramatically, with the world’s population doubling — and power networks are now struggling to keep up without expensive capacity upgrades and new transmission lines.

In the US, the Department of Energy found that the average age of large power transformers, which handle 90% of the country’s electricity flow, is more than 40 years. Three-quarters of the country’s 707,000 miles (1,138,000 km) of transmission lines — which have a lifespan of 40 to 50 years — are more than 25 years old. The number of transmission outages in the US has more than doubled in the past six years compared with the previous six years, according to North American Electric Reliability Corporation. This is partially due to an increase in severe weather events, which put more stress on the power grid (more on this below).

Renewable energy in some grid regions is being wasted because there is not enough transmission capacity to transport it to other regions that have low generation and high demand. In the US, for example, the lack of high-voltage transmission lines prevents energy delivery from expansive wind farms in Iowa and Oklahoma to data centers in the southeast. Germany is facing similar problems as it struggles to connect wind capacity in the north near the Baltic Sea to the large industrial consumers in the south.

Energy experts estimate that modernizing the US power grid will require an investment of $1 trillion to $2 trillion over the next two decades. Current business and regulatory processes cannot attract and support this level of funding; new funding structures and business processes will be required to transform the grid for a decarbonized future.

Weather is changing

The Earth is becoming warmer, which means long-term changes to broad temperature and weather patterns. These changes are increasing the risk of extreme weather events, such as heat waves, floods, droughts and fires. According to a recent study published in the Proceedings of the National Academy of Sciences, individual regions within the continental US were more than twice as likely to see a “once in a hundred years” extreme temperature event in 2019 than they were in 1979.

Extreme weather events put the grid system at risk. High temperatures reduce the capacity of high-voltage transmission, high winds knock power lines off the towers, and stormy conditions can result in the loss of generation assets. In the past three years, Gulf Coast hurricanes, West Coast wildfires, Midwest heat waves and a Texas deep freeze have all caused local power systems to fail. The key issue with extreme weather is that it can disrupt grid generation, transmission, and fuel supplies simultaneously.

The Texas freeze of February 2021 serves as a cautionary tale of interdependent systems. As the power outages began, some natural gas compressors became inoperable because they were powered by electricity without backup — leading to further blackouts and interrupting the vital gas supply needed to maintain or restart generation stations.

Geopolitical tensions create fuel supply risks

Fuel supply shocks resulting from the war in Ukraine rapidly increased oil and natural gas prices in Europe during the winter of 2022. The global energy supply chain is a complex system of generators, distributors and retailers that prioritizes economic efficiency and low cost, often at the expense of resiliency, which can create the conditions for cascading failures. Many countries in the EU warned that fuel shortages would cause rolling blackouts if the weather became too cold.

In the UK, the government advised that a reasonable worst-case scenario would see rolling power cuts across the country. Large industrial customers, including data center operators, would have been affected, losing power in 3-hour blocks without any warning. In France, planning for the worst-case scenario assumed that up to 60% of the country’s population would be affected by scheduled power cuts. Even Germany, a country with one of the most reliable grids in the world, made plans for short-term blackouts.

Fuel supply shocks are reflected in high energy prices and create short- and long-term risks for the electrical grid. Prior to the conflict in Ukraine, dispatchable fossil fuel peaker plants, which only operate during periods of high demand, struggled to maintain price competitiveness with renewable energy producers. This trend was exacerbated by high fuel costs and renewable energy subsidies.

Any political upheaval in the major oil-producing regions, such as the gulf countries or North Africa, would affect energy prices and energy supply. Development of existing shale gas deposits could offer some short-term energy security benefits in markets, such as the US, the UK and Germany, but political pressures are currently preventing such projects from getting off the ground.

Being prepared for utility power problems

Loss of power is still the most common cause of data center outages. Uptime Institute’s 2022 global survey of data center managers attributes 44% of outages to power issues — greater than the second, third and fourth most common outage causes combined.

The frequency of power-related outages is partly due to power system complexity and a dependence on moving parts: a loss of grid power often reveals weaknesses elsewhere, including equipment maintenance regimens and staff training.

The most common power-related outages result from faults with uninterruptible power supply (UPS) systems, transfer switches and generators. Compared with other data center systems, such as cooling, power is more prone to fail completely, rather than operating at partial capacity.

Disconnecting from the power grid and running on emergency backup systems for extended periods (hours and days) is not a common practice in most geographies. Diesel generators remain the de facto standard for data center backup power and these systems have inherent limitations.

Any issues with utility power create risks across the data center electrical system. Major concerns include:

Load transfer risks between the grid and the generators. It is recommended that data center operators fully disconnect from the electrical grid once a year to test procedures, yet many choose not to do so out of operational concerns in a production environment. This means that lurking failures in transfer switches and paralleling switchgears may go undetected and any operational mistakes remain undiscovered.
Fuel reserves and refueling. The volume of on-site fuel storage must meet cost and available space constraints; requires system maintenance; and spill and leak management. Longer grid outages can exceed on-site fuel capacity, making operators dependent on outside vendors for fuel resupply. These vendors are, in turn, dependent on the diesel supply chain, which may be disrupted during a wide-area grid outage because some diesel terminals may lack backup power to pump the fuel. Fuel delivery procedures may come under time pressure and may not be fully observed with the potential to create accidents, such as contamination and spills.
Increased likelihood of engine failures. More frequent warm-up and cool-down cycles, as well as higher than expected runtime hours, accelerate generator wear. As many as 27% of data center operators have experienced a generator-related power outage in the past three years, according to the Uptime Institute Annual outages analysis 2023. Ongoing supply chain bottlenecks may mean that rental generators are in short supply while replacement parts or new engines may take months to arrive. This may force the data center to operate at a derated capacity, lower redundancy, or both.
Pollutant emissions. Many jurisdictions limit generator operating hours to cap emissions of nitrogen oxides, sulfur oxides, and other particulates. For example, in the US, most diesel generators are limited to 100 hours of full load per year and non-compliance can result in fines.
Battery system wear and failures. Frequent deep discharge and recharge cycles wear battery cells out faster, particularly lead-acid batteries. Lithium-ion chemistries are not immune either: discharges create thermal stress for Lithium-ion cells when currents are high. Temperature can also spike towards the end-of-cell capacity, which increases the chance of a thermal event as a result of inherent manufacturing imperfections. Often, the loss of critical load caused by a failure in an uninterruptible power supply system is due to batteries not being monitored closely enough by experienced technicians.

It will get worse before it gets better

The likely worst-case scenario facing US and European data center operators in 2023 and beyond will consist primarily of load-shedding requests, brownouts and 2- to 4-hour, controlled rolling blackouts as opposed to widespread, long-lasting grid outages.

Ultimately, how much margin electrical grids will have in reserve is not possible to predict with any accuracy beyond a few weeks. Extreme weather events, in particular, are exceedingly difficult to model. Other unexpected events — such as power station maintenance, transmission failures and geopolitical developments that affect energy supply — might contribute to a deterioration of grid reliability.

Operators can take precautions to prepare for rolling blackouts, including developing a closer relationship with their energy provider. These steps are well understood, but evidence shows they are not always followed.

Considering historically low fuel reserves and long lead times for replacement components, all measures are best undertaken in advance. Data center operators should reach out to suppliers to confirm their capability to deliver fuel and their outlook for future supply — and determine whether the supplier is equipped to sell fuel in the event of a grid outage. Based on the response, operators should assess the need for additional on-site fuel storage or consider contracting additional vendors for their resupplies.

Data center operators should also test their backup power systems regularly. The so-called pull-the-plug test, which checks the entire backup system, is recommended annually. A building load test should be performed monthly, which involves less risk than pull-the-plug testing and checks most of the backup power system. This test may not run long enough to test certain components, such as fuel pumps and level switches, and these should be tested manually. Additionally, data center operators should test the condition of their stored fuel, and filter or treat it as necessary.

The challenges to grid reliability vary by location, but in general Uptime Intelligence sees evidence that the risk of rolling blackouts at times of peak demand is escalating. Operators can mitigate risk to their data centers by testing their backup power system and arranging for additional fuel suppliers as needed. They should also revisit their emergency procedures — even those with a very low chance of occurrence.

Jacqueline Davis, Research Analyst

Lenny Simon, Research Associate

Daniel Bizo, Research Director

Max Smolaks, Research Analyst