Across the world, data center owners and managers are striving to buy more renewable energy and reduce their dependence on fossil fuels. The global internet giants and the largest colocation companies have led the way with huge green energy purchases.
But the impact of renewable energy on the grid operator’s economics, and on the critical issue of grid reliability, is not well understood by many observers and energy buyers. In some cases, the purchase and use of renewable energy can expose the buyer to financial risks and can threaten to destabilize the economics of the power supply chain. For this reason, Uptime Institute is advising operators to conduct a thorough analysis of the availability and economics of renewable energy as part of their energy buying strategy.
The problem stems from the volatile balance of supply and demand. The intermittent nature of wind and solar generation moves the electricity markets in unexpected ways, driving down wholesale prices as renewable generation capacity increases. These price movements can stress the business model of reliable generation, which is needed to ensure continuous operation of the grid.
An analysis of market conditions in grid regions with high penetration of renewably generated power offers important guideposts for data center operators evaluating renewable energy purchases and the reliability of their electricity supplies.
Generally, average wholesale power prices decrease as generation capacity — particularly intermittent renewable energy generation capacity — increases. Consider the example of ERCOT, the Electric Reliability Council of Texas. ERCOT is the Texas grid operator recently in the news as it struggled with extreme weather. In that region, wind generation capacity more than doubled from 2013 to 2020. Against this increase in capacity, the average spot market price of electricity dropped from approximately $25/MWh (megawatt-hour) in 2016 to $15/MWh in 2020. (Solar generation exhibits the same negative correlation between installed capacity and wholesale prices.)
This reduction in wholesale price in turn reduces the revenue stream of power purchase agreements (PPAs) — an instrument used by many data center operators. In a PPA, the purchaser takes on the financial risk of power generation by guaranteeing to pay the generator a fixed price for the electricity of the generated MWh over the contract term. If the spot market revenue for the generated power is less than the PPA revenue (based on the agreed contract price), the purchaser will then pay the difference and take a loss.
This does happen: Currently, PPAs signed in the Texas ERCOT region in 2016 are generating losses for their purchasers. Figure 1 illustrates how higher average available wind generation capacity (bars) reduced the daily average spot market prices at two settlement hubs (lines) during January 2021.
Figure 1: Changes in ERCOT electricity spot market prices with change in wind output
The loss of revenue resulting from falling wholesale prices also challenges the business models of nonrenewable fossil fuel and nuclear generators, making it difficult for them to meet operation, maintenance and financing costs while achieving profitability. Unless the grid has sufficient renewable energy storage and transmission, this will threaten the reliable supply of power.
The impact of renewable energy on the reliability and economics of the grid
The impact of renewable energy on the reliability and economics of the grid does not stop there. Three further issues are curtailment, congestion and reliability.
Curtailment. As intermittent renewable capacities increase, higher levels of output can exceed grid demand. To maintain grid stability, power must be sold to other grid regions or some portion of the generation supply must be taken offline. In the California grid district CAISO (the California Independent System Operator), solar generation curtailments have grown from 400 GWh (gigawatt-hours) in 2017 to over 1500 GWh in 2020. This results in lost revenue for the curtailed facilities. Oftentimes, the need to ensure that reliable generators are available to the grid requires the curtailment of the intermittent renewable generators.
Congestion. Congestion occurs as transmission capacity approaches or exceeds full utilization. To stay within the capacity constraints, generators need to be rebalanced to assure supply and demand will be matched in all local regions of the grid. This condition often results in increased curtailment of renewable generation. Transmission costs may also be increased on transmission lines that are near capacity to send an economic signal to balance capacity and demand.
Reliability. As intermittent renewable capacity increases it reduces the percentage of time fossil fuel and nuclear generation sources are dispatched to the grid, reducing their revenues. While the delivery of renewably generated energy has increased, it still varies significantly from hour to hour. In ERCOT in 2019, there were 63 hours where wind satisfied more than 50% of demand and 602 hours where it supplied less than 5%. Figure 2 provides an example of wind generation variability over two days in January 2019 in the ERCOT area. On January 8, the available wind generation capacity averaged roughly 8 gigawatts (GWs), satisfying approximately 25% of the demand. On January 14, the output was significantly reduced at 2 GWs, satisfying approximately 10% of demand.
Figure 2: Day-to-day and hour-to-hour variation in wind generation
To deal with this volatility, the system operator needs to have a capacity plan, using economic signals or other strategies to ensure there is sufficient capacity to meet demand under all possible mixes of renewable and conventional generation.
As the proportion of renewably generated electricity in their supply grid grows, data center operators will need to carefully evaluate their renewable energy procurement decisions, the overall generation capacity available under different scenarios, and the robustness of the transmission system capacity. There is no guarantee that a grid that has proved reliable in the past will prove equally reliable in the years ahead.
While purchases of renewable energy are important, data center operators must also advocate for increased transmission capacity; improved, automated grid management; and robust, actionable capacity and reliability planning to ensure a reliable electricity supply with increased renewable content at their facility meter.
https://journal.uptimeinstitute.com/wp-content/uploads/2021/04/renew-2.jpg7842660Jay Dietrich, Research Director of Sustainability, Uptime Institute, jdietrich@uptimeinstitute.comhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngJay Dietrich, Research Director of Sustainability, Uptime Institute, jdietrich@uptimeinstitute.com2021-04-26 06:10:062021-04-12 12:31:53Renewable energy and data centers: Buyer, be aware
Uptime Institute Members say one of their most vexing security concerns is the insider threat — authorized staff, vendors or visitors acting with malicious intent.
In extreme examples, trusted individuals could power down servers and other equipment, damage network equipment, cut fiber paths, or steal data from servers or wipe the associated storage. Unfortunately, data centers cannot simply screen for trusted individuals with bad intent.
Most data center operators conduct background checks. Most have policies for different levels of access. Some may insist that all visitors have security escorts, and many have policies that prevent tailgating (physically following an authorized person through a door to gain access). Many have policies to limit the use of portable memory devices in computer rooms to only authorized work; some destroy them once the work is complete, and some insist that only specific computers assigned to specific worktables can be used.
Yet vulnerabilities exist. The use of single-source authentication of identification (ID), for example, can lead to the sharing of access cards and other unintended consequences. While some ID cards and badges have measures, such as encryption, to prevent them being copied, they can be cloned using specialist devices. In some data centers, multifactor authentication is used to significantly harden ingress and egress access.
The COVID-19 pandemic increased the risk for many data centers, at least temporarily. Some of the usual on-site staff were replaced by others, and routines were changed. When this happens, security and vetting procedures can be more successfully evaded.
However, even before the pandemic, the risk of the insider threat has been growing — and it has changed. Trusted individuals are now more likely to unwittingly act in ways that lead to malicious outcomes (or fail to respond and prevent such outcomes). This is because human psychology tactics are increasingly being used to trick authorized people into providing sensitive information. Social engineering, using deception to obtain unauthorized data or access, is now prolific and becoming increasingly sophisticated.
Tactics can include a mix of digital and physical reconnaissance. The simplest approaches are often the most effective, such as manipulating people using phone or email, and using information available to the public (for example, on the internet).
Social engineering is a concern for all businesses but particularly those with mission-critical infrastructure. A growing number of data center operators use automated security systems to detect anomalies in communications, such as email phishing campaigns on staff and visitors.
However, even routine communication can be exploited by hackers. For example, the host names derived from the headers of an email may contain information about the internet protocol (IP) address of the computer that sent the email, such as its geographic location. Further information about, say, a data center employee can be obtained using online information (social media, typically), which can then be used for social manipulation — such as posing as a trusted source (spoofing caller IDs or creating unauthorized security certificates for a web domain, for example), tricking an employee into providing sensitive information. By surveilling employees, either physically or online, hackers can also obtain useful information at places they visit, such as credit card information used at a restaurant (by exploiting a vulnerability in the restaurant’s digital system, for example). Hackers often gain trust by combining information gleaned from chasing digital trails with social engineering tactics.
Reviews of policies and procedures, including separation of duties, are recommended. There are also numerous cybersecurity software and training tools to minimize the scope for social engineering and unauthorized access. Some data center operations use automated open-source intelligence (OSInt) software to scan social media and the internet for mentions of keywords, such as their organization’s name, associated with terror-related language. Some use automated cybersecurity tools to conduct open-source reconnaissance of exposed critical equipment and digital assets.
The insider threat is impossible to fully control — but it can be mitigated against by adding layers of security.
The full report Data center security: Reassessing physical, human and digital risks is available to members of Uptime Institute. Consider a guest membership here.
In early March 2021, a hacker group publicly exposed the username and password of an administrative account of a security camera vendor. The credentials enabled them to access 150,000 commercial security systems and, potentially, set up subsequent attacks on other critical equipment. A few weeks earlier, leaked credentials for the collaboration software TeamViewer gave hackers a way into a system controlling a city water plant in Florida (US). They remotely adjusted the sodium hydroxide levels to a dangerous level (the attack was detected, and harm avoided).
These are just some of the most recent examples of exploits where critical infrastructure was disrupted by remote access to IT systems, including some high-profile attacks at power plants.
The threat of cybersecurity breaches also applies to physical data centers, and it is growing. Cloud computing, and increased automation and remote monitoring have broadened the attack surface. (See our recent report Data center security: Reassessing physical, human and digital risks.)
So, how widespread is the problem of insecure facility assets? Our research of vulnerable systems on the open internet suggests it is not uncommon.
For close to a decade, the website Shodan has been used by hackers, benevolent and malevolent, to search for targets. Instead of fetching results that are webpages, Shodan crawls the internet for devices and industrial control systems (ICSs) that are connected to the internet but exposed.
Shodan and similar search engine websites (BinaryEdge, Censys and others) provide a compendium of port-scan data (locating open ports, which are a path to attack) on the internet. Expert users identify interesting characteristics about certain systems and set out to gain as much access as they can. Automation tools make the process more efficient, speeding up and also expanding what is possible for an exploit (e.g., by defeating login safeguards).
In a recent demonstration of Shodan for the Uptime Institute, the cybersecurity firm Phobos Group showed more than 98,000 ICSs exposed globally, including data center equipment and devices. Phobos quickly discovered access to the login screens of control systems for most major data center equipment providers. In Figure 1 (as in all figures), screenshots of aggregate search results are shown with specific details hidden to ensure privacy.
The login process itself can be highly problematic. Sometimes installers or users do not change the default credentials supplied by the manufacturers, which can often be found online. During our demonstration, for example, Phobos used a default login to gain access to the control system for cooling units supplied by a widely used data center equipment vendor. If this exercise were carried out by a genuine intruder, they would be able to change setpoint temperatures and alarms.
Users’ customized login credentials can sometimes be obtained from a data breach of one service and then used by a hacker to try to log into another service, a type of cyberattack known as credential stuffing. The availability of lists of credentials has proliferated, and automated credential-stuffing tools have become more sophisticated, using bots to thwart traditional login protections. (Data breaches can happen without leaving any trace in corporate systems and can go undetected.)
As cybersecurity exploits of critical infrastructure in recent years have shown, control system interfaces may be the primary targets — but access to them is often through another system. Using the Shodan tool, the security company Phobos searched for exposed remote desktops, which can then provide access to multiple systems. This method can be particularly troubling if a control system is accessible through a remote desktop and if the user employs the same or similar passwords across systems.
There are many remote desktops exposed online. As Figure 2 shows, in a recent Shodan search, over 86,700 remote desktops were exposed in the US city of Ashburn, Virginia, alone (a city known as the world’s data center capital). This list includes a set of addresses for a global data center capacity provider (not shown).
Password reuse is one of the biggest security vulnerabilities humans introduce, but it can be minimized with training and tools, and by multifactor authentication where practicable. Installers and users should also be prevented from removing password protection controls (another vulnerability that Phobos demonstrated). There are also cybersecurity tools to continuously scan for assets exposed online and to provide attack simulations. Services used at some facilities include threat intelligence and penetration tests on IP addresses and infrastructure. Low-tech approaches such as locked workstations and clean-desk policies also help protect sensitive information.
Cybersecurity of data center control systems and other internet protocol (IP)-enabled assets is multilayered and requires a combination of ongoing strategies. The threat is real and the likelihood of physical breaches, unauthorized access to information, and the destruction of or tampering with data and services is higher than ever before.
The full report Data center security: Reassessing physical, human and digital risks is available to members of the Uptime Institute community here.
The catastrophic fire that occurred at OVHcloud’s SBG2 data center in Strasbourg, France (see last week’s blog about it) has led many operators to question their vulnerability to fires.
Fires at data centers are a constant concern, but are rare. In almost all cases of data center fire, the source is quickly located, the equipment isolated, and damage contained to a small area.
Uptime Institute’s database of abnormal incidents, which documents over 8,000 incidents shared by members since its inception in 1994, records 11 fires in data centers — less than 0.5 per year. All of these were successfully contained, causing minimal damage/disruption.
A separate Uptime Institute database of publicly recorded incidents around the world — which includes only those that receive public/media attention — also shows fires are rare, with outages often caused by fire suppression equipment.
One or two of these fires have been serious and have led to some destruction of equipment and data. However, the data centers or communications rooms involved have been small, with minimal long-term disruption.
The majority of incidents begin and end in the electrical room (although “people doing dumb things” — such as overloading power strips or working with open flames near flammable materials — is also a cause, says Uptime Institute Chief Technical Officer Chris Brown). Faults in uninterruptible power supplies can create heat and smoke and can require that the equipment be immediately isolated, but the risk rarely goes beyond this, due the lack of nearby combustive materials.
If batteries are nearby, they can catch fire and will burn until the fuel is consumed — which can take some time. Lithium-ion batteries, which are commonly perceived as a fire risk, contain internal monitoring at the cell level, which cuts the battery power if heating occurs.
In recent years (before OVHcloud Strasbourg), accidental discharge of fire suppression systems, especially high pressure clean agent gas systems, has actually caused significantly more serious disruption than fires, with some banking and financial trading data centers affected by this issue. Fires near a data center, or preventative measures taken to reduce the likelihood of forest fires, have also led to some data center disruption (not included in the numbers reported above).
Responsibility for fire regulation is covered by the local AHJ (authority having jurisdiction), and requirements are usually strict. But rules may be stricter for newer facilities, so good operational management is critical for older data centers.
Uptime Institute advises that all data centers use VESDA (very early smoke detection apparatus) systems and maintain appropriate fire barriers and separation of systems. Well-maintained water sprinkler or low pressure clean agent fire suppression systems are preferred. Risk assessments, primarily aimed at reducing the likelihood of outages, will also pick up obvious issues with these systems.
The Uptime Tier IV certification requires 1 hour fire-rated partitions between complementary critical systems. This is to help ensure a fire in one area does not immediately shut down a data center. It does assume proper fire suppression in the facility.
https://journal.uptimeinstitute.com/wp-content/uploads/2021/04/fire3.jpg6921932Andy Lawrence, Executive Director of Research, Uptime Institute, alawrence@uptimeinstitute.comhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, alawrence@uptimeinstitute.com2021-04-05 08:26:002021-05-04 10:33:00Datacenter Fire Frequency
The fire that destroyed a data center (and damaged others) at the OVHcloud facility in Strasbourg, France, on March 10-11, 2021, has raised a multitude of questions from concerned data center operators and customers around the world. Chief among these is, “What was the main cause, and could it have been prevented?”
Fires at data centers are rare but do occur — Uptime Institute Intelligence has some details of more than 10 data center fires (see our upcoming blog about the frequency of fire incidents). But most of these are quickly isolated and extinguished; it is extremely uncommon for a fire to rage out of control, especially at larger data centers, where strict fire prevention and containment protocols are usually followed. Unfortunately for OVHcloud, the fire occurred just two days after the owners announced plans for a public listing on the Paris Stock Exchange in 2022.
While this Note will address some of the known facts and provide some context, more complete and informed answers will have to wait for the full analysis by OVHcloud, the fire services and other parties. OVHcloud has access to a lot of closed-circuit television and some thermal camera images that will help in the investigation.
OVHcloud
OVHcloud is a high-profile European data center operator and one of the largest hosting companies globally. Founded in 1999 by Octave Klaba, OVHcloud is centered in France but has expanded rapidly, with facilities in several countries offering a range of hosting, colocation and cloud services. It has been championed as a European alternative to the giant US cloud operators and is a key participant in the European Union’s GAIA-X cloud project. It has partnerships with big IT services operators, such as Deutsche Telekom, Atos and Capgemini.
Among OVHcloud customers are tens of thousands of small businesses running millions of websites. But it has many major enterprise, government and commercial customers, including various departments of the French government, the UK’s Vehicle Licensing Agency, and the European Space Agency. Many have been affected by the fire.
OVHcloud is hailed as a bold innovator, offering a range of cloud services and using advanced low energy, free air cooling designs and, unusually for commercial operators, direct liquid cooling. But it has also suffered some significant outages, most notably two serious incidents in 2017. After that, then-Chief Executive Officer and chairman Octave Klaba spoke of the need for OVHcloud to be “even more paranoid than it is already.” Some critics at the time believed these outages were due to poor design and operational practices, coupled with a high emphasis on innovation. The need to compete on a cost basis with large-scale competitors —Amazon Web Services, Microsoft and others – is an ever-present factor.
The campus at Strasbourg (SBG) is based on a site acquired from ArcelorMittal, a steel and mining company. It houses four data centers, serving customers internationally. The oldest and smallest two, SBG1 and SBG4, were originally based on prefab containers. SBG2, destroyed by the fire, was a 2 MW facility capable of housing 30,000 servers. It used an innovative free air cooling system. SBG3, a newer 4 MW facility that was partially damaged, uses a newer design that may have proved more resilient.
Chronology
The fire in SBG2 started after midnight and was picked up by sensors and alarms. Black smoke prevented staff from effectively intervening. The fire spread rapidly within minutes, destroying the entire data center. Using thermal cameras, firefighters identified that two uninterruptible power supplies (UPSs) were at the heart of the blaze, one of which had been extensively worked on that morning.
All of the data centers have been out of action in the days immediately following the fire, although all but SBG2 are due to come back online shortly. SBG1 suffered significant damage to some rooms, with recovery planned to take a week or so. Many customers were advised to invoke disaster recovery plans, but OVHcloud has spare capacity in other data centers and has been working to get customers up and running.
Causes, design and operation
Only a thorough root-cause analysis will reveal exactly what happened and whether this fire was preventable. However, some design and operational issues have been highlighted among the many customers and ecosystem partners of OVHcloud:
UPS and electrical fires. Early indicators point to the failure of a UPS, causing a fire that spread quickly. At least one of the UPSs had been extensively worked on earlier in the day, suggesting a maintenance issue may have been a main contributor. Although it is not best practice, battery cabinets (when using vent-regulated lead-acid, or VRLA, batteries) are often installed next to the UPS units themselves. Although this may not have been the case at SBG2, this type of configuration can create a situation where a UPS fire heats up batteries until they start to burn and can cause fire to spread rapidly.
Cooling tower design. SBG2 was built in 2011 using a tower design that has convection-cooling based “auto-ventilation.” Cool air enters, passes through a heat exchange for the (direct liquid) cooling system, and warm air rises through the tower in the center of the building. OVHcloud has four other data centers using the same principle. OVHcloud says this is an environmentally sound, energy efficient design — but since the fire, concerns have been raised that it can act rather like a chimney. Vents that allow external air to enter would need to be immediately shut in the event of a potential fire (the nearby, newer SBG3 data center, which uses an updated design, suffered less damage).
VESDA and fire suppression. It is being reported that SBG2 had neither a VESDA (very early smoke detection apparatus) system nor a water or gas fire suppression system. Rather, staff relied on smoke detectors and fire extinguishers. It is not known if these reports are accurate. Most data centers do have early detection and fire suppression systems.
Backup and cloud services. Cloud (and many hosting) companies cite high availability figures and extremely low figures for data loss. But full storage management and recovery across multiple sites costs extra, especially for hosted services. Many customers, especially smaller ones, usually pay for basic backup only. Statements from OVHcloud since the fire suggests that some customers would have lost data. Some backups were in the same data center, or on the same campus, and not all data was replicated elsewhere.
Fire and resiliency certification
Responsibility for fire prevention — and building regulations — is mostly dealt with by local planning authorities (AHJs – authorities having jurisdiction). These vary widely across geographies.
Uptime Institute has been asked whether Tier certification would help prevent fires. Uptime’s Chief Technical Officer Chris Brown responds:
“Tiers has limited fire system requirements, and they are geared to how the systems can impact the critical MEP (mechanical, electrical and plumbing) infrastructure. This is the case because in most locations, fire detection and suppression are tightly controlled by life/safety codes. If the Tier standard were to include specific fire detection and suppression requirements, it would add little value and would run the risk of clashing with local codes.
This is always under review.
Tier IV does have a compartmentalization requirement. It requires a 1 hour fire-rated barrier between complementary systems. This is to protect complementary systems from being impacted by a single fire event. This does assume the facility is properly protected by fire suppression systems.”
A separate Uptime Data Center Risk Assessment (DCRA) would document the condition (or lack of?) a fire suppression system, any lack of a double-interlocked suppression system, and even a pre-action system using only compressed air to charge the lines.
https://journal.uptimeinstitute.com/wp-content/uploads/2021/03/firefist2a.jpg6501819Andy Lawrence, Executive Director of Research, Uptime Institute, alawrence@uptimeinstitute.comhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, alawrence@uptimeinstitute.com2021-03-29 06:01:002021-03-23 15:21:42Learning from the OVHcloud data center fire
Transitioning to renewable energy use is an important, but not easily achieved, goal. Although the past decade has seen significant improvements in IT energy efficiency, there are indications that this may not continue. Moore’s Law may be slowing, more people are coming online, and internet traffic is growing faster than ever before. As energy consumption increases, data center operators will need to transition to 100% renewable energy use 100% of the time.
Uptime Institute has recently published a report covering the key components of renewable energy sustainability strategies for data centers. The tech industry has made considerable effort to improve energy efficiency and is the largest purchaser of renewable energy. Even so, most data center sustainability strategies still focus on renewable energy certificates (RECs). RECs are now considered to be low quality products because they cannot credibly be used to back claims of 100% renewable energy use.
To avoid accusations of greenwashing, data center operators must consider investing in a portfolio of renewable energy products. RECs may play a part, but power purchase agreements (PPAs) are becoming more popular, even though there can be financial risks involved.
A stepwise approach will ease the process. There are four steps that data center operators need to take on the journey toward the use of sustainable, renewable energy.
1. Measure, report, offset
Electricity is just one component of the carbon footprint of an organization, but it is relatively easy to measure because exact consumption is regularly reported for billing purposes. Tracking how much electricity comes from renewable and nonrenewable sources allows decisions to be made about offsets and renewables matching. This breakdown can be obtained from the electricity supplier, or grid-level emissions factors can be used. For example, in the US this is published annually by state by the Energy Information Administration; other countries provide similar resources. (A more formal methodology is explained in the Greenhouse Gas Protocol Scope 2 guidance.)
Once total emissions are known (that is, total emissions from electricity ― the full organizational emissions also need to be calculated), the next step is to buy offset products to mitigate the existing impact. However, there are significant challenges with ensuring offset quality, and so offsetting is only a stopgap measure. Ideally, offsets must be reserved for emissions that cannot be reduced by other measures (e.g., by switching to 100% renewable energy).
Measurement and reporting are crucial to understanding carbon footprint and are effectively becoming a necessary function of doing business. The reporting of carbon emissions is becoming a legal requirement for larger companies in some jurisdictions (e.g., the UK). Some companies are starting to require carbon reporting for their suppliers (for example, Apple and Microsoft, because of their own goals to be carbon-neutral/negative by 2030).
Data center operators who are not already tracking carbon emissions associated with electricity purchases may have to invest significant resources to catch up should reporting become required.
2. 100% renewables matching
Ideally, all electricity used should be 100% matched by renewable energy production. So far in the data center industry, 100% renewables matching has generally been achieved through purchasing RECs, but PPAs (direct, financial, or through green supply arrangements) must now take over as the dominant approach. RECs can act as a stopgap between taking no action and using tools such as PPAs, but they should eventually be a small component in the overall data center sustainability strategy.
3. Power purchase agreements
Purchasing RECs is a reasonable first step in a sustainability strategy but is insufficient on its own. Establishing direct/physical PPAs with a nearby renewable energy generator, combined with their associated RECs, is the gold standard necessary to truly claim 100% renewable energy use. However, even this does not mean 100% renewable energy is actually being used by the data center, just that an amount of purchased renewable energy equivalent to the data center’s energy use was added to the grid. Virtual/financial PPAs are an option where the power market does not allow direct retail PPAs.
Both types of PPA involve pricing risk or can act as a hedge against wholesale price changes. For direct PPAs, the fixed price provides certainty — but if wholesale prices fall, buyers may be stuck in a long-term contract paying more than the current market price. Virtual/financial PPAs introduce further complexity and financial risk: if the wholesale price falls below the agreed-upon strike price at the time of purchase, the buyer must pay the supplier the difference, which may be significant.
Despite these risks, the use of PPAs is growing rapidly in the US, particularly in the tech and communications sectors. Operators with advanced sustainability programs have been buying PPAs for several years, either directly through financial/virtual PPAs, or by using green supply agreements through a utility. Our report covers these options in more detail.
4. 24/7 renewable energy use
Most matching (of renewable energy purchased against energy actually used) happens on an annual basis but shifts in generation (the grid mix) happen at a much lower granularity. There are strategies to smooth this out: different sources of renewable energy can be combined to create blended PPAs, such as combining wind and solar energy production with storage capacity. This is useful because different sources generate at different times (for example, wind can generate energy at night when solar energy is unavailable).
In 2019, Microsoft and Vattenfall announced a new product to provide hourly renewables matching. The pilot started at Microsoft’s Sweden headquarters and will provide hourly matching to a new Azure cloud region in 2021. No data center operator has achieved 24/7 matching for its entire global fleet, although some are almost there for individual sites (e.g., in 2019, Google achieved 96% renewable energy use in Oklahoma [US], and 61% on an hourly basis globally).
This is the objective: 24/7 renewable energy use, 100% of the time. Matching on an annual basis is not enough to reach the goal of decarbonizing the electricity grid. All demand ― 100% ― must be supplied by 100% renewable energy, 100% of the time. This will, of course, take many decades to achieve in most economies.
The full report Renewable energy for data centers: Renewable energy certificates, power purchase agreements and beyond is available to members of the Uptime Institute. Membership can be found here.
https://journal.uptimeinstitute.com/wp-content/uploads/2021/03/renew3.jpg6281783David Mytton, Research Affiliate, Uptime Institutehttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngDavid Mytton, Research Affiliate, Uptime Institute2021-03-22 06:11:002021-03-12 14:01:13How data center operators can transition to renewable energy
Renewable energy and data centers: Buyer, be aware
/in Executive, Operations/by Jay Dietrich, Research Director of Sustainability, Uptime Institute, jdietrich@uptimeinstitute.comAcross the world, data center owners and managers are striving to buy more renewable energy and reduce their dependence on fossil fuels. The global internet giants and the largest colocation companies have led the way with huge green energy purchases.
But the impact of renewable energy on the grid operator’s economics, and on the critical issue of grid reliability, is not well understood by many observers and energy buyers. In some cases, the purchase and use of renewable energy can expose the buyer to financial risks and can threaten to destabilize the economics of the power supply chain. For this reason, Uptime Institute is advising operators to conduct a thorough analysis of the availability and economics of renewable energy as part of their energy buying strategy.
The problem stems from the volatile balance of supply and demand. The intermittent nature of wind and solar generation moves the electricity markets in unexpected ways, driving down wholesale prices as renewable generation capacity increases. These price movements can stress the business model of reliable generation, which is needed to ensure continuous operation of the grid.
An analysis of market conditions in grid regions with high penetration of renewably generated power offers important guideposts for data center operators evaluating renewable energy purchases and the reliability of their electricity supplies.
Generally, average wholesale power prices decrease as generation capacity — particularly intermittent renewable energy generation capacity — increases. Consider the example of ERCOT, the Electric Reliability Council of Texas. ERCOT is the Texas grid operator recently in the news as it struggled with extreme weather. In that region, wind generation capacity more than doubled from 2013 to 2020. Against this increase in capacity, the average spot market price of electricity dropped from approximately $25/MWh (megawatt-hour) in 2016 to $15/MWh in 2020. (Solar generation exhibits the same negative correlation between installed capacity and wholesale prices.)
This reduction in wholesale price in turn reduces the revenue stream of power purchase agreements (PPAs) — an instrument used by many data center operators. In a PPA, the purchaser takes on the financial risk of power generation by guaranteeing to pay the generator a fixed price for the electricity of the generated MWh over the contract term. If the spot market revenue for the generated power is less than the PPA revenue (based on the agreed contract price), the purchaser will then pay the difference and take a loss.
This does happen: Currently, PPAs signed in the Texas ERCOT region in 2016 are generating losses for their purchasers. Figure 1 illustrates how higher average available wind generation capacity (bars) reduced the daily average spot market prices at two settlement hubs (lines) during January 2021.
The loss of revenue resulting from falling wholesale prices also challenges the business models of nonrenewable fossil fuel and nuclear generators, making it difficult for them to meet operation, maintenance and financing costs while achieving profitability. Unless the grid has sufficient renewable energy storage and transmission, this will threaten the reliable supply of power.
The impact of renewable energy on the reliability and economics of the grid
The impact of renewable energy on the reliability and economics of the grid does not stop there. Three further issues are curtailment, congestion and reliability.
Curtailment. As intermittent renewable capacities increase, higher levels of output can exceed grid demand. To maintain grid stability, power must be sold to other grid regions or some portion of the generation supply must be taken offline. In the California grid district CAISO (the California Independent System Operator), solar generation curtailments have grown from 400 GWh (gigawatt-hours) in 2017 to over 1500 GWh in 2020. This results in lost revenue for the curtailed facilities. Oftentimes, the need to ensure that reliable generators are available to the grid requires the curtailment of the intermittent renewable generators.
Congestion. Congestion occurs as transmission capacity approaches or exceeds full utilization. To stay within the capacity constraints, generators need to be rebalanced to assure supply and demand will be matched in all local regions of the grid. This condition often results in increased curtailment of renewable generation. Transmission costs may also be increased on transmission lines that are near capacity to send an economic signal to balance capacity and demand.
Reliability. As intermittent renewable capacity increases it reduces the percentage of time fossil fuel and nuclear generation sources are dispatched to the grid, reducing their revenues. While the delivery of renewably generated energy has increased, it still varies significantly from hour to hour. In ERCOT in 2019, there were 63 hours where wind satisfied more than 50% of demand and 602 hours where it supplied less than 5%. Figure 2 provides an example of wind generation variability over two days in January 2019 in the ERCOT area. On January 8, the available wind generation capacity averaged roughly 8 gigawatts (GWs), satisfying approximately 25% of the demand. On January 14, the output was significantly reduced at 2 GWs, satisfying approximately 10% of demand.
To deal with this volatility, the system operator needs to have a capacity plan, using economic signals or other strategies to ensure there is sufficient capacity to meet demand under all possible mixes of renewable and conventional generation.
As the proportion of renewably generated electricity in their supply grid grows, data center operators will need to carefully evaluate their renewable energy procurement decisions, the overall generation capacity available under different scenarios, and the robustness of the transmission system capacity. There is no guarantee that a grid that has proved reliable in the past will prove equally reliable in the years ahead.
While purchases of renewable energy are important, data center operators must also advocate for increased transmission capacity; improved, automated grid management; and robust, actionable capacity and reliability planning to ensure a reliable electricity supply with increased renewable content at their facility meter.
The insider threat: Social engineering is raising security risks
/in Executive, Operations/by Rhonda Ascierto, Vice President, Research, Uptime InstituteUptime Institute Members say one of their most vexing security concerns is the insider threat — authorized staff, vendors or visitors acting with malicious intent.
In extreme examples, trusted individuals could power down servers and other equipment, damage network equipment, cut fiber paths, or steal data from servers or wipe the associated storage. Unfortunately, data centers cannot simply screen for trusted individuals with bad intent.
Most data center operators conduct background checks. Most have policies for different levels of access. Some may insist that all visitors have security escorts, and many have policies that prevent tailgating (physically following an authorized person through a door to gain access). Many have policies to limit the use of portable memory devices in computer rooms to only authorized work; some destroy them once the work is complete, and some insist that only specific computers assigned to specific worktables can be used.
Yet vulnerabilities exist. The use of single-source authentication of identification (ID), for example, can lead to the sharing of access cards and other unintended consequences. While some ID cards and badges have measures, such as encryption, to prevent them being copied, they can be cloned using specialist devices. In some data centers, multifactor authentication is used to significantly harden ingress and egress access.
The COVID-19 pandemic increased the risk for many data centers, at least temporarily. Some of the usual on-site staff were replaced by others, and routines were changed. When this happens, security and vetting procedures can be more successfully evaded.
However, even before the pandemic, the risk of the insider threat has been growing — and it has changed. Trusted individuals are now more likely to unwittingly act in ways that lead to malicious outcomes (or fail to respond and prevent such outcomes). This is because human psychology tactics are increasingly being used to trick authorized people into providing sensitive information. Social engineering, using deception to obtain unauthorized data or access, is now prolific and becoming increasingly sophisticated.
Tactics can include a mix of digital and physical reconnaissance. The simplest approaches are often the most effective, such as manipulating people using phone or email, and using information available to the public (for example, on the internet).
Social engineering is a concern for all businesses but particularly those with mission-critical infrastructure. A growing number of data center operators use automated security systems to detect anomalies in communications, such as email phishing campaigns on staff and visitors.
However, even routine communication can be exploited by hackers. For example, the host names derived from the headers of an email may contain information about the internet protocol (IP) address of the computer that sent the email, such as its geographic location. Further information about, say, a data center employee can be obtained using online information (social media, typically), which can then be used for social manipulation — such as posing as a trusted source (spoofing caller IDs or creating unauthorized security certificates for a web domain, for example), tricking an employee into providing sensitive information. By surveilling employees, either physically or online, hackers can also obtain useful information at places they visit, such as credit card information used at a restaurant (by exploiting a vulnerability in the restaurant’s digital system, for example). Hackers often gain trust by combining information gleaned from chasing digital trails with social engineering tactics.
Reviews of policies and procedures, including separation of duties, are recommended. There are also numerous cybersecurity software and training tools to minimize the scope for social engineering and unauthorized access. Some data center operations use automated open-source intelligence (OSInt) software to scan social media and the internet for mentions of keywords, such as their organization’s name, associated with terror-related language. Some use automated cybersecurity tools to conduct open-source reconnaissance of exposed critical equipment and digital assets.
The insider threat is impossible to fully control — but it can be mitigated against by adding layers of security.
The full report Data center security: Reassessing physical, human and digital risks is available to members of Uptime Institute. Consider a guest membership here.
Data center insecurity: Online exposure threatens critical systems
/in Executive, Operations/by Rhonda Ascierto, Vice President, Research, Uptime InstituteIn early March 2021, a hacker group publicly exposed the username and password of an administrative account of a security camera vendor. The credentials enabled them to access 150,000 commercial security systems and, potentially, set up subsequent attacks on other critical equipment. A few weeks earlier, leaked credentials for the collaboration software TeamViewer gave hackers a way into a system controlling a city water plant in Florida (US). They remotely adjusted the sodium hydroxide levels to a dangerous level (the attack was detected, and harm avoided).
These are just some of the most recent examples of exploits where critical infrastructure was disrupted by remote access to IT systems, including some high-profile attacks at power plants.
The threat of cybersecurity breaches also applies to physical data centers, and it is growing. Cloud computing, and increased automation and remote monitoring have broadened the attack surface. (See our recent report Data center security: Reassessing physical, human and digital risks.)
So, how widespread is the problem of insecure facility assets? Our research of vulnerable systems on the open internet suggests it is not uncommon.
For close to a decade, the website Shodan has been used by hackers, benevolent and malevolent, to search for targets. Instead of fetching results that are webpages, Shodan crawls the internet for devices and industrial control systems (ICSs) that are connected to the internet but exposed.
Shodan and similar search engine websites (BinaryEdge, Censys and others) provide a compendium of port-scan data (locating open ports, which are a path to attack) on the internet. Expert users identify interesting characteristics about certain systems and set out to gain as much access as they can. Automation tools make the process more efficient, speeding up and also expanding what is possible for an exploit (e.g., by defeating login safeguards).
In a recent demonstration of Shodan for the Uptime Institute, the cybersecurity firm Phobos Group showed more than 98,000 ICSs exposed globally, including data center equipment and devices. Phobos quickly discovered access to the login screens of control systems for most major data center equipment providers. In Figure 1 (as in all figures), screenshots of aggregate search results are shown with specific details hidden to ensure privacy.
The login process itself can be highly problematic. Sometimes installers or users do not change the default credentials supplied by the manufacturers, which can often be found online. During our demonstration, for example, Phobos used a default login to gain access to the control system for cooling units supplied by a widely used data center equipment vendor. If this exercise were carried out by a genuine intruder, they would be able to change setpoint temperatures and alarms.
Users’ customized login credentials can sometimes be obtained from a data breach of one service and then used by a hacker to try to log into another service, a type of cyberattack known as credential stuffing. The availability of lists of credentials has proliferated, and automated credential-stuffing tools have become more sophisticated, using bots to thwart traditional login protections. (Data breaches can happen without leaving any trace in corporate systems and can go undetected.)
As cybersecurity exploits of critical infrastructure in recent years have shown, control system interfaces may be the primary targets — but access to them is often through another system. Using the Shodan tool, the security company Phobos searched for exposed remote desktops, which can then provide access to multiple systems. This method can be particularly troubling if a control system is accessible through a remote desktop and if the user employs the same or similar passwords across systems.
There are many remote desktops exposed online. As Figure 2 shows, in a recent Shodan search, over 86,700 remote desktops were exposed in the US city of Ashburn, Virginia, alone (a city known as the world’s data center capital). This list includes a set of addresses for a global data center capacity provider (not shown).
Password reuse is one of the biggest security vulnerabilities humans introduce, but it can be minimized with training and tools, and by multifactor authentication where practicable. Installers and users should also be prevented from removing password protection controls (another vulnerability that Phobos demonstrated). There are also cybersecurity tools to continuously scan for assets exposed online and to provide attack simulations. Services used at some facilities include threat intelligence and penetration tests on IP addresses and infrastructure. Low-tech approaches such as locked workstations and clean-desk policies also help protect sensitive information.
Cybersecurity of data center control systems and other internet protocol (IP)-enabled assets is multilayered and requires a combination of ongoing strategies. The threat is real and the likelihood of physical breaches, unauthorized access to information, and the destruction of or tampering with data and services is higher than ever before.
The full report Data center security: Reassessing physical, human and digital risks is available to members of the Uptime Institute community here.
Datacenter Fire Frequency
/in Design, Executive, Operations/by Andy Lawrence, Executive Director of Research, Uptime Institute, alawrence@uptimeinstitute.comThe catastrophic fire that occurred at OVHcloud’s SBG2 data center in Strasbourg, France (see last week’s blog about it) has led many operators to question their vulnerability to fires.
Fires at data centers are a constant concern, but are rare. In almost all cases of data center fire, the source is quickly located, the equipment isolated, and damage contained to a small area.
Uptime Institute’s database of abnormal incidents, which documents over 8,000 incidents shared by members since its inception in 1994, records 11 fires in data centers — less than 0.5 per year. All of these were successfully contained, causing minimal damage/disruption.
A separate Uptime Institute database of publicly recorded incidents around the world — which includes only those that receive public/media attention — also shows fires are rare, with outages often caused by fire suppression equipment.
One or two of these fires have been serious and have led to some destruction of equipment and data. However, the data centers or communications rooms involved have been small, with minimal long-term disruption.
The majority of incidents begin and end in the electrical room (although “people doing dumb things” — such as overloading power strips or working with open flames near flammable materials — is also a cause, says Uptime Institute Chief Technical Officer Chris Brown). Faults in uninterruptible power supplies can create heat and smoke and can require that the equipment be immediately isolated, but the risk rarely goes beyond this, due the lack of nearby combustive materials.
If batteries are nearby, they can catch fire and will burn until the fuel is consumed — which can take some time. Lithium-ion batteries, which are commonly perceived as a fire risk, contain internal monitoring at the cell level, which cuts the battery power if heating occurs.
In recent years (before OVHcloud Strasbourg), accidental discharge of fire suppression systems, especially high pressure clean agent gas systems, has actually caused significantly more serious disruption than fires, with some banking and financial trading data centers affected by this issue. Fires near a data center, or preventative measures taken to reduce the likelihood of forest fires, have also led to some data center disruption (not included in the numbers reported above).
Responsibility for fire regulation is covered by the local AHJ (authority having jurisdiction), and requirements are usually strict. But rules may be stricter for newer facilities, so good operational management is critical for older data centers.
Uptime Institute advises that all data centers use VESDA (very early smoke detection apparatus) systems and maintain appropriate fire barriers and separation of systems. Well-maintained water sprinkler or low pressure clean agent fire suppression systems are preferred. Risk assessments, primarily aimed at reducing the likelihood of outages, will also pick up obvious issues with these systems.
The Uptime Tier IV certification requires 1 hour fire-rated partitions between complementary critical systems. This is to help ensure a fire in one area does not immediately shut down a data center. It does assume proper fire suppression in the facility.
Learning from the OVHcloud data center fire
/in Design, Executive/by Andy Lawrence, Executive Director of Research, Uptime Institute, alawrence@uptimeinstitute.comThe fire that destroyed a data center (and damaged others) at the OVHcloud facility in Strasbourg, France, on March 10-11, 2021, has raised a multitude of questions from concerned data center operators and customers around the world. Chief among these is, “What was the main cause, and could it have been prevented?”
Fires at data centers are rare but do occur — Uptime Institute Intelligence has some details of more than 10 data center fires (see our upcoming blog about the frequency of fire incidents). But most of these are quickly isolated and extinguished; it is extremely uncommon for a fire to rage out of control, especially at larger data centers, where strict fire prevention and containment protocols are usually followed. Unfortunately for OVHcloud, the fire occurred just two days after the owners announced plans for a public listing on the Paris Stock Exchange in 2022.
While this Note will address some of the known facts and provide some context, more complete and informed answers will have to wait for the full analysis by OVHcloud, the fire services and other parties. OVHcloud has access to a lot of closed-circuit television and some thermal camera images that will help in the investigation.
OVHcloud
OVHcloud is a high-profile European data center operator and one of the largest hosting companies globally. Founded in 1999 by Octave Klaba, OVHcloud is centered in France but has expanded rapidly, with facilities in several countries offering a range of hosting, colocation and cloud services. It has been championed as a European alternative to the giant US cloud operators and is a key participant in the European Union’s GAIA-X cloud project. It has partnerships with big IT services operators, such as Deutsche Telekom, Atos and Capgemini.
Among OVHcloud customers are tens of thousands of small businesses running millions of websites. But it has many major enterprise, government and commercial customers, including various departments of the French government, the UK’s Vehicle Licensing Agency, and the European Space Agency. Many have been affected by the fire.
OVHcloud is hailed as a bold innovator, offering a range of cloud services and using advanced low energy, free air cooling designs and, unusually for commercial operators, direct liquid cooling. But it has also suffered some significant outages, most notably two serious incidents in 2017. After that, then-Chief Executive Officer and chairman Octave Klaba spoke of the need for OVHcloud to be “even more paranoid than it is already.” Some critics at the time believed these outages were due to poor design and operational practices, coupled with a high emphasis on innovation. The need to compete on a cost basis with large-scale competitors —Amazon Web Services, Microsoft and others – is an ever-present factor.
The campus at Strasbourg (SBG) is based on a site acquired from ArcelorMittal, a steel and mining company. It houses four data centers, serving customers internationally. The oldest and smallest two, SBG1 and SBG4, were originally based on prefab containers. SBG2, destroyed by the fire, was a 2 MW facility capable of housing 30,000 servers. It used an innovative free air cooling system. SBG3, a newer 4 MW facility that was partially damaged, uses a newer design that may have proved more resilient.
Chronology
The fire in SBG2 started after midnight and was picked up by sensors and alarms. Black smoke prevented staff from effectively intervening. The fire spread rapidly within minutes, destroying the entire data center. Using thermal cameras, firefighters identified that two uninterruptible power supplies (UPSs) were at the heart of the blaze, one of which had been extensively worked on that morning.
All of the data centers have been out of action in the days immediately following the fire, although all but SBG2 are due to come back online shortly. SBG1 suffered significant damage to some rooms, with recovery planned to take a week or so. Many customers were advised to invoke disaster recovery plans, but OVHcloud has spare capacity in other data centers and has been working to get customers up and running.
Causes, design and operation
Only a thorough root-cause analysis will reveal exactly what happened and whether this fire was preventable. However, some design and operational issues have been highlighted among the many customers and ecosystem partners of OVHcloud:
Fire and resiliency certification
Responsibility for fire prevention — and building regulations — is mostly dealt with by local planning authorities (AHJs – authorities having jurisdiction). These vary widely across geographies.
Uptime Institute has been asked whether Tier certification would help prevent fires. Uptime’s Chief Technical Officer Chris Brown responds:
“Tiers has limited fire system requirements, and they are geared to how the systems can impact the critical MEP (mechanical, electrical and plumbing) infrastructure. This is the case because in most locations, fire detection and suppression are tightly controlled by life/safety codes. If the Tier standard were to include specific fire detection and suppression requirements, it would add little value and would run the risk of clashing with local codes.
This is always under review.
Tier IV does have a compartmentalization requirement. It requires a 1 hour fire-rated barrier between complementary systems. This is to protect complementary systems from being impacted by a single fire event. This does assume the facility is properly protected by fire suppression systems.”
A separate Uptime Data Center Risk Assessment (DCRA) would document the condition (or lack of?) a fire suppression system, any lack of a double-interlocked suppression system, and even a pre-action system using only compressed air to charge the lines.
How data center operators can transition to renewable energy
/in Executive/by David Mytton, Research Affiliate, Uptime InstituteTransitioning to renewable energy use is an important, but not easily achieved, goal. Although the past decade has seen significant improvements in IT energy efficiency, there are indications that this may not continue. Moore’s Law may be slowing, more people are coming online, and internet traffic is growing faster than ever before. As energy consumption increases, data center operators will need to transition to 100% renewable energy use 100% of the time.
Uptime Institute has recently published a report covering the key components of renewable energy sustainability strategies for data centers. The tech industry has made considerable effort to improve energy efficiency and is the largest purchaser of renewable energy. Even so, most data center sustainability strategies still focus on renewable energy certificates (RECs). RECs are now considered to be low quality products because they cannot credibly be used to back claims of 100% renewable energy use.
To avoid accusations of greenwashing, data center operators must consider investing in a portfolio of renewable energy products. RECs may play a part, but power purchase agreements (PPAs) are becoming more popular, even though there can be financial risks involved.
A stepwise approach will ease the process. There are four steps that data center operators need to take on the journey toward the use of sustainable, renewable energy.
1. Measure, report, offset
Electricity is just one component of the carbon footprint of an organization, but it is relatively easy to measure because exact consumption is regularly reported for billing purposes. Tracking how much electricity comes from renewable and nonrenewable sources allows decisions to be made about offsets and renewables matching. This breakdown can be obtained from the electricity supplier, or grid-level emissions factors can be used. For example, in the US this is published annually by state by the Energy Information Administration; other countries provide similar resources. (A more formal methodology is explained in the Greenhouse Gas Protocol Scope 2 guidance.)
Once total emissions are known (that is, total emissions from electricity ― the full organizational emissions also need to be calculated), the next step is to buy offset products to mitigate the existing impact. However, there are significant challenges with ensuring offset quality, and so offsetting is only a stopgap measure. Ideally, offsets must be reserved for emissions that cannot be reduced by other measures (e.g., by switching to 100% renewable energy).
Measurement and reporting are crucial to understanding carbon footprint and are effectively becoming a necessary function of doing business. The reporting of carbon emissions is becoming a legal requirement for larger companies in some jurisdictions (e.g., the UK). Some companies are starting to require carbon reporting for their suppliers (for example, Apple and Microsoft, because of their own goals to be carbon-neutral/negative by 2030).
Data center operators who are not already tracking carbon emissions associated with electricity purchases may have to invest significant resources to catch up should reporting become required.
2. 100% renewables matching
Ideally, all electricity used should be 100% matched by renewable energy production. So far in the data center industry, 100% renewables matching has generally been achieved through purchasing RECs, but PPAs (direct, financial, or through green supply arrangements) must now take over as the dominant approach. RECs can act as a stopgap between taking no action and using tools such as PPAs, but they should eventually be a small component in the overall data center sustainability strategy.
3. Power purchase agreements
Purchasing RECs is a reasonable first step in a sustainability strategy but is insufficient on its own. Establishing direct/physical PPAs with a nearby renewable energy generator, combined with their associated RECs, is the gold standard necessary to truly claim 100% renewable energy use. However, even this does not mean 100% renewable energy is actually being used by the data center, just that an amount of purchased renewable energy equivalent to the data center’s energy use was added to the grid. Virtual/financial PPAs are an option where the power market does not allow direct retail PPAs.
Both types of PPA involve pricing risk or can act as a hedge against wholesale price changes. For direct PPAs, the fixed price provides certainty — but if wholesale prices fall, buyers may be stuck in a long-term contract paying more than the current market price. Virtual/financial PPAs introduce further complexity and financial risk: if the wholesale price falls below the agreed-upon strike price at the time of purchase, the buyer must pay the supplier the difference, which may be significant.
Despite these risks, the use of PPAs is growing rapidly in the US, particularly in the tech and communications sectors. Operators with advanced sustainability programs have been buying PPAs for several years, either directly through financial/virtual PPAs, or by using green supply agreements through a utility. Our report covers these options in more detail.
4. 24/7 renewable energy use
Most matching (of renewable energy purchased against energy actually used) happens on an annual basis but shifts in generation (the grid mix) happen at a much lower granularity. There are strategies to smooth this out: different sources of renewable energy can be combined to create blended PPAs, such as combining wind and solar energy production with storage capacity. This is useful because different sources generate at different times (for example, wind can generate energy at night when solar energy is unavailable).
In 2019, Microsoft and Vattenfall announced a new product to provide hourly renewables matching. The pilot started at Microsoft’s Sweden headquarters and will provide hourly matching to a new Azure cloud region in 2021. No data center operator has achieved 24/7 matching for its entire global fleet, although some are almost there for individual sites (e.g., in 2019, Google achieved 96% renewable energy use in Oklahoma [US], and 61% on an hourly basis globally).
This is the objective: 24/7 renewable energy use, 100% of the time. Matching on an annual basis is not enough to reach the goal of decarbonizing the electricity grid. All demand ― 100% ― must be supplied by 100% renewable energy, 100% of the time. This will, of course, take many decades to achieve in most economies.
The full report Renewable energy for data centers: Renewable energy certificates, power purchase agreements and beyond is available to members of the Uptime Institute. Membership can be found here.