What we can expect for mission-critical digital infrastructure in 2021?
Each autumn Uptime Institute, like many other organizations, puts together a list of some of the big trends and themes for the year ahead. This time, we have focused on five big trends that might not have been so obvious 12 months ago.
Heading into 2021, during a macroeconomic downturn, the critical digital infrastructure sector itself continues to expand and to attract enviable levels of new investment. The ongoing build-out of new data centers and networks is largely being driven by cloud, hosted, and “as-a-service” workloads, as more enterprises seek to outsource more of their IT and/or data center capacity. However, for many managers, the COVID-19 pandemic has forced a reassessment — of working practices and, in particular, of risk. The global economy’s dependence on IT is growing, and this is catching the attention of an increasing number of customers, governments and watchdogs.
The coming year (and beyond) also holds new opportunities: Edge computing, artificial intelligence (AI) and new innovations in hardware and software technologies promise greater efficiencies and agility.
Here are Uptime Institute’s five trends for 2021, summarized:
1. Accountability — the “new” imperative
Enterprises want more cloud and greater agility, but they can’t outsource responsibility — for incidents, outages, security breaches or even, in the years ahead, carbon emissions. In 2021, hybrid IT, with workloads running in both on- and off-premises data centers, will continue to dominate, but investments will increasingly be constrained and shaped by the need for more transparency, oversight and accountability. More will be spent on cloud and other services, as well as in on-premises data centers.
2. Smarter, darker data centers
Following a scramble to effectively staff data centers during a pandemic, many wary managers are beginning to see remote monitoring and automation systems in a more positive light, including those driven by AI. An adoption cycle that has been slow and cautious will accelerate. But it will take more than just investment in software and services before the technology reduces staffing requirements.
3. Edge — the next frontier
Significant new demand for edge computing, fueled by technologies such as 5G, the internet of things and AI, is likely to build slowly — but the infrastructure preparation is underway. Expect new alliances and investments across enterprise, mobile and wireline networks, and for a wide range of edge data centers, small and large. Smart and automated software-defined networks and interconnections will become as important as the physical infrastructure.
4. Sustainability: More challenging, more transparent
For years, operators could claim environmental advances based on small, incremental and relatively inexpensive steps — or by adopting new technologies that would pay for themselves anyway. But the time of easy wins and greenwashing is ending: Regulators, watchdogs, customers and others will increasingly expect operators of digital infrastructure to provide hard and detailed evidence of carbon reductions, water savings and significant power savings — all while maintaining, if not improving, resiliency.
5. A surge of innovation
Data center operators (and enterprise IT) are mostly cautious, if not late, adopters of new technologies. Few beyond hyperscale operators can claim to have gained a competitive advantage through technology. However, several new technologies are maturing at the same time, promising advances in the performance and manageability of data centers and IT. Storage-class memory, silicon photonics, ARM servers and software-defined power are ready for greater adoption.
This is a summary of the full report on new Trends for 2021. The full report is available to members of the Uptime Institute. Click to learn more.
The power density per rack (kilowatts [kW] per cabinet) is a critical number in data center design, capacity planning, and cooling and power provisioning. There have been industry warnings about a meteoric rise in IT equipment rack power density for the past decade (at least). One reason for this prediction is the proliferation of compute-intensive workloads (e.g., AI, IoT, cryptocurrencies, and augmented and virtual reality), all of which drive the need for high-density racks.
Our recent annual surveys found that racks with densities of 20 kW and higher are becoming a reality for many data centers (we asked about highest rack density) — but not to the degree forewarned. Year-over-year, most respondents said their highest density racks were in 10-19 kW range, which is not enough to merit wholesale technical changes. When rack densities are higher than 20-25 kW, direct liquid cooling and precision air cooling becomes more economical and efficient. According to what we see in the field, such high densities are not pervasive enough to have an impact on most data centers.
This does not mean that the trend should be ignored. It is clear from our latest research that average mean rack density in data centers is rising steadily, as the figure below shows. Eliminating respondents with above 30 kW as high-performance outliers, the mean average density in our 2020 survey sample was 8.4 kW/rack. This is consistent with other industry estimates and safely within the provisioned range of most facilities.
PUE over the Years
In our 2020 survey, we asked about the most common (modal average) SERVER rack density, which is perhaps a better metric than overall average density. More than two-thirds (71%) reported a modal average of below 10 kW/rack, with just 16% widely deploying 20 kW or higher rack densities (Figure 7). The most common density was 5-9 kW/rack. Overprovisioning of power/cooling is probably a more common issue than underprovisioning due to rising rack densities.
The modal average power consumption, rack density
Assuming a trend that rack units will increasingly be filled with higher-powered servers that are well utilized, we anticipate that the modal average kW/rack will increase over time. Figure 8 shows that for most organizations — roughly half of those surveyed — average density is increasing, albeit only slowly.
Rack density changing…
We expect density to keep rising. Our research shows that the use of virtualization and software containers pushes IT utilization up, in turn requiring more power and cooling. With Moore’s law slowing down, improvements in IT can require more multi-core processors and, consequently, more power consumption per operation, especially if utilization is low. Even setting aside new workloads, increases in density can be regarded a long-term trend.
But, as our 2020 survey findings demonstrate, the expectation for 20 kW racks throughout the industry has not manifested. We believe that many compute-intensive workloads — those that will significantly push up power use, rack density and heat — currently reside across a relatively small group of hyperscale cloud data centers and are consumed by organizations as a service.
Want to know more about this and other data center trends and strategies? Download a copy of our complete 2020 survey.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/11/racks1.jpg6491923Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-12-07 06:00:562020-11-30 14:14:55Rack Density is Rising
Data center capacity is rapidly expanding in outsourced, third-party IT venues, such as colocation data centers and public cloud. Whether measured in megawatts (MW) of uninterruptible power supply capacity or IT load (or by some other measure, such as square feet of white or leased space, or in units of compute or storage) overall capacity is growing rapidly.
There are many factors driving this growth, including newer cloud-based services, such as social media and streaming; mobile applications and services; new enterprise services and applications; the migration of corporate workloads into colocation sites; and the adoption of software as a service (SaaS) and public cloud platforms.
Does this mean that almost all IT workloads will end up running in a third-party data center? In a word, No, not for a long time and in some cases, never. In the 2020 Uptime Institute survey, as in years past, we asked respondents to estimate, by percentage, how much of their workload/data is processed/stored in different types of data centers today and how this might look in two years’ time.
The majority (58%) said that most of their workloads run in corporate data centers today — that is, enterprise-owned, on-premises facilities. And if you add in Micro datacenters and Server closets, that figure rises to 69%!
These findings are similar to those of our previous years’ surveys. They confirm Uptime Institute’s view that the enterprise-owned data center sector, while not necessarily the most innovative, will continue to be the foundation of enterprise IT for the next decade. In our survey, nearly two-thirds of IT workloads are expected to be running in privately owned environments (large data centers, server closets and micro data centers) by 2022, with the remainder contracted to external suppliers. Although the enterprise data center sector is falling as a percentage of the whole, the absolute amount of enterprise data center capacity is still growing.
A mix of factors typically drive enterprise demand for third-party IT venues; similarly, multiple factors drive demand for on-premises data centers. Often some combination of the factors listed below come into play when deciding the best-execution venue for workloads.
COMMON DRIVERS FOR IT VENUE SELECTION DECISIONS
Choosing Outsourced:
Cost: Outsourcing can lower costs in the short to medium term. For organizations “born” in a public cloud or colo, it typically is cost-prohibitive to move to an enterprise data center.
Cost allocation: Outsourcing shifts cost allocations from capex toward more repeatable opex models.
IT agility and flexibility: Outsourcing provides the ability to readily and quickly adapt to changing capacity needs without the burden of managing the full stack of IT and applications; IT can be used for a project’s duration only (e.g., for test and development).
Access to resources: Third-parties may provide access to a wider range of resources, including technology, interconnections, software tools, services and application environments.
Security: Third-parties can offer the most advanced, highly resourced security features.
Choosing on-premise:
Cost: Ownership delivers total cost of ownership benefits over the long term; in the shorter term, owners avoid the data transport costs of moving to an outsourced venue.
Governance: On-premises environments may be necessary for compliance with data governance and regulatory requirements.
Control: Owners can closely monitor and control factors such as latency, availability and application performance. While most outsourced venues are strong in these areas, service level agreements vary and are limited.
Risk: Ownership ensures full visibility into (and the ability to adjust) the risk profile of every workload.
Security: Ownership provides the ability to maintain control and governance (dedicated rather than shared physical infrastructure) over security features.
Want to know more about venue selection, data center staffing and skills, latest technologies in use and the cloud, risk and cost management, operational strategies and more? Check out our full 2020 Annual Survey.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/11/choice.jpg8852329Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-11-23 06:00:322020-11-20 16:18:06Common Factors for IT venue selection (Choosing In-house or Outsource)
The COVID-19 pandemic has caused concerns about data center HVAC (heating, ventilation and air conditioning) filtration. So many data center operators are adjusting filtration protocols, including upgrading to finer MERV (minimum efficiency reporting value) 13 filters, to better filter out aerosols and COVID-19 virus carriers.1 But there is no free lunch. Depending on the design of the data center’s HVAC system, this increase in filtration static pressure can slightly increase energy consumption.
Technically, with a PSC (permanent split capacitor) blower motor, extra static pressure from this denser filter would result in lower airflow, reduced system performance and less air distribution. With ECMs (electronically commutated motors) — the type of motor most commonly found in data center cooling systems — this extra static pressure can result in higher blower motor power consumption.
Recently, an Uptime Institute member reported an increase of 8% energy use following a filtration upgrade from a MERV 8 to a more dense MERV 13 filter, due to the increased fan energy required to maintain static pressure and airflow. Since their data center fan power is 7-10% of total data center energy used, any increased fan energy affects their PUE (power usage effectiveness).
But this single data point may not be representative of all environments, since the quality and quantity of the media in the filter and the configuration of the HVAC system can significantly affect static pressure and airflow. Denser MERV 13 filters won’t necessarily require more fan energy if the operator installs filters that increase surface area by having more and deeper pleats.2
There is a direct correlation between airflow CFM (cubic feet per minute) and IWC (initial resistance water column), so some HVAC adjustments might be needed (see chart below).2
Source: “Is There a Downside to High-MERV Filters? The new high-MERV filters extract an energy penalty,” by David Springer, November 02, 2009, in Home energy: The Home Performance Magazine. http://www.homeenergy.org/show/article/nav/issues/page/4/id/667
Some guidance is in order: When choosing a filter remember that airflow (static pressure increase) is just as important to consider as filtration (MERV rating). Just because a filter fits doesn’t mean it’s the best filter for the system, so its very important to check the rating of the facility’s filter and the design specifications of the cooling system.
Many data center cooling systems use an airflow rate between 330–490 CFM, so the system would experience 0.10-0.16 initial resistance (static pressure increase in inches of water column) for the MERV 11 filter example shown in the image below.3
Typical data center underfloor static pressure is 20-25 pascal or 0.08-0.1 inches of water column. Therefore, while this filter would introduce some increased static pressure, it pales in comparison to the 220 pascal static pressure required to move air through the cooling coils in the CRAC (computer room air conditioning) units or the impingement and turning of cooled air on the subfloor, which can be as high as 200 pascal.4
With careful airflow planning there should be minimal airflow and pressure impacts from more dense filters, such as MERV 13. These finer filters, though, will need to be changed more frequently, since they will trap more particulates and so will clog sooner. In addition to the higher change frequency, budget for higher per-unit costs, for filters with larger surface area media will likely be more expensive than smaller, less efficient filters.
But everything considered, given the modest energy trade-offs and the increased particulate and virus filtering they provide, MERV 13 filter use is quickly becoming the new normal.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/10/ch1aa.jpg5001350Todd Traverhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngTodd Traver2020-11-16 05:00:502021-05-04 10:20:49COVID-19, air filtration and energy use
There are few organizations that have had as big an impact on data center design as ASHRAE — and specifically, their Technical Committee (TC) 9.9. ASHRAE’s 2004 publication Thermal guidelines for Data Processing Environments (now in its fourth edition) described the optimal environmental operating conditions for the electronics in a data center, and in doing so, effectively established some of the key design goals that have been followed by almost every data center builder and operator for the past 15 years.
In October 2020, ASHRAE published what it describes as a “groundbreaking” Technical Bulletin — this time looking specifically at edge data centers. The Bulletin is a lighter, shorter document than most previous TC9.9 publications, and it does not signal any changes in the thermal operating guidelines that remain so important. Nor does it reveal any significant new findings — even though TC9.9 members do have access to a lot of data and tests conducted by equipment manufacturers.
But its brevity does not imply that it lacks importance. The ASHRAE committee members want to send a warning that unless careful processes and guidelines are followed, and good technical choices made, there will be a significantly increased risk of electronic equipment failure in many edge data centers. Given that a lot more processing will be done at the edge in the next decade or so that means there could be more failures and service disruptions in the future than in the past.
In the document, and in separate conversations with Uptime Institute (which has members on the TC9.9 committee), ASHRAE TC9.9 members identify two major issues as they relate to edge data centers.
First, the simple fact is that almost all “core” enterprise, large, and colocation data centers have a strictly climate-controlled white space. Small edge data centers, however, cannot always maintain this — especially during maintenance. Edge data center doors may be open during maintenance, exposing equipment to rapid temperature change, rain and pollution. Cooling may be turned off during maintenance, leading to rapid heat rise; routine or urgent maintenance may not be possible or timely; and edge environments are more likely to have a higher density and to be hotter, colder, wetter, dustier or more insecure than big, remote, well-managed facilities.
A second issue is that a lot of the equipment on the market has been designed for use in large data centers where failure is more easily tolerated — or even deliberately allowed, as part of an overall “abandon in place” strategy. Put bluntly, the quality of IT equipment is no longer required to be as high as it used to be, because it has been designed to be quickly swapped out. Some leading suppliers no longer do the systematic, lifecycle-based testing that was once a necessity.
What can be done?
The first place to start is to avoid poor site selection — but that, of course, is not always an option for small edge data centers, which need to be near the point of use. Ideally, the actual data center (which may be prefabricated) should be designed to allow ease of maintenance without exposure to outside conditions, to minimize damage (such as to batteries and IT equipment). A data center infrastructure management system that remotely monitors key environmental variables and IT performance will help prevent failures. It also makes sense to design with some concurrent maintainability/fault tolerance, if the design goals/budgets allow.
Other key points of advice are:
Protect the edge facility by servicing only during moderate weather or when using a mantrap or tent, to avoid potentially deleterious conditions. Even time of day can be a factor (e.g., humidity is often higher in the morning).
Monitor humidity, condensation and temperature during servicing.
Monitor the rate of temperature or humidity change when doors are opened.
Beware of local particulates/pollution, such as ocean spray, dust, industrial pollutants and heavy vehicle exhaust.
Air filtration should meet necessary standards — MERV (Minimum Efficiency Reporting Value) 11 or 13.
Beware the effect of gaseous pollutants that cause corrosion (e.g., of components containing copper or silver).
Use remote monitoring where possible to track corrosion rate, filter performance, etc.
The key takeaway from this ASHRAE Bulletin is that it is far more difficult (and possibly more expensive) to control virtually all environmental variables in edge data centers. This will almost certainly lead to more failures. Vigilance and good practices can improve early detection and reduce the likelihood of failures and mitigate their impact.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/11/Edge1.jpg9202185Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-11-09 04:54:042020-10-29 08:36:15Why ASHRAE is concerned about edge data centers
When the PUE (power usage effectiveness) metric was first discussed at a meeting of The Green Grid in Santa Clara, back in 2007, a microphone stand was placed in each aisle of the auditorium. The importance of the initiative was understood even then: experts, including the founders of Uptime Institute, formed lines to give their considered input. And if there was one point that came across, it was that the industry should not treat PUE as a comparative metric, a gold standard that every designer, operator and planning authority must chase down, for comparison, presentation and applause.
Thirteen years on, that is almost exactly what has happened. PUE, thanks to its simplicity and universal applicability, has become the critical benchmark for scoring data centers for efficiency and “greenness” (environmental responsibility). But it is often used uncritically.
This has had both positive and negative consequences. The positive side is that PUE values have been forced down across the world, as operators strive to reduce waste and operating costs. To meet their goals, suppliers have focused on improving electrical efficiency of equipment and have developed nonmechanical alternatives (such as free cooling). PUEs have fallen from an average of 2.5 in 2007 to around 1.6 today. In an Uptime Institute survey, around 95% of respondents said it is important that colocation companies have a low PUE.
So what is the negative side? Its simplicity. The very quality that led to PUE’s near universal adoption also may have led to its being over-used and misapplied — increasingly, with unintended consequences.
This observation is hardly new — indeed, this is exactly what those experts feared back in 2007 as they lined up to voice their concerns. And it is why PUE was developed into a standard, with rules on how it should be measured, and why Partial PUE variants were developed.
But the unintended consequences are becoming more serious and more concerning. The metric may be driving the wrong behaviors. For this reason, it is very possible that PUE values may begin to be de-emphasized in the near future and more operators may allow their number to rise a little. (In 2019, Uptime Institute did note a marginal rise in PUE values.) If this happens, some operators may find themselves in conflict with certain regulators and planners, for whom PUE has been an easily understood but simplistic way of enforcing environmental goals (policing this is another matter). These planning authorities, then, may also need to review their policies around driving down the PUE numbers. (California, Singapore, Amsterdam, take note.)
There are, we think, three strong and strengthening reasons why focusing on the PUE of the data center might be counter-productive:
Resiliency (N to N+1). There is an industry-wide move to increase the resiliency of data centers, with the growing dependency on IT and the impact of the pandemic and extreme weather among the drivers. At the physical site level, it is difficult to improve redundancy and resiliency without adding power-consuming equipment — which drives up the PUE. (Similarly, one operator installed the highest level of MERV [Minimum Efficiency Reporting Value] air filters to filter out viruses — but fan energy use rose by 8% and the PUE value rose accordingly.)
Water use. The drive to improve energy efficiency has encouraged operators to use economizers and, in particular, adiabatic and evaporative cooling and chillers. But this has often involved a trade-off: energy use drops, but water use rises. That compromise is becoming problematic in some regions, where climate change means water is becoming more scarce and, in some instances, a bigger local environmental issue than on-site energy use. Again, PUE may be the wrong metric to track — WUE (water usage effectiveness) is a more important one.
IT efficiency. It is widely known that while facilities have become ever more energy efficient, IT use can be extremely inefficient and wasteful, with processors and memory often drawing energy while doing relatively little work. Energy-saving efficiency investments on the IT side, however, are not rewarded with a lower PUE, but rather a higher one (as PUE is the ratio of the IT to non-IT energy figures). This has been known from the outset. With a slew of new technologies becoming available to lower the overall energy figure, it is important that the need to meet a given PUE value does not discourage investments and innovation on the IT side.
There is, to be fair, a good counter-argument to these three points: that by searching for ways to lower the PUE regardless of these constraints, data center operators can find ways to meet all the goals. Liquid cooling, for example, can be highly resilient and uses little water. The argument here, then, is not for anyone to abandon PUE, or to stop watching and measuring it, but to apply it more cautiously than ever, and more flexibly.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/11/PUE-Rusty.jpg380895Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-11-02 07:32:042020-10-26 12:18:10PUE: The golden metric is looking rusty
Five Trends for 2021: accountability, automation, edge, sustainability, innovation
/in Design, Executive, Operations/by Rhonda Ascierto, Vice President, Research, Uptime InstituteWhat we can expect for mission-critical digital infrastructure in 2021?
Each autumn Uptime Institute, like many other organizations, puts together a list of some of the big trends and themes for the year ahead. This time, we have focused on five big trends that might not have been so obvious 12 months ago.
Heading into 2021, during a macroeconomic downturn, the critical digital infrastructure sector itself continues to expand and to attract enviable levels of new investment. The ongoing build-out of new data centers and networks is largely being driven by cloud, hosted, and “as-a-service” workloads, as more enterprises seek to outsource more of their IT and/or data center capacity. However, for many managers, the COVID-19 pandemic has forced a reassessment — of working practices and, in particular, of risk. The global economy’s dependence on IT is growing, and this is catching the attention of an increasing number of customers, governments and watchdogs.
The coming year (and beyond) also holds new opportunities: Edge computing, artificial intelligence (AI) and new innovations in hardware and software technologies promise greater efficiencies and agility.
Here are Uptime Institute’s five trends for 2021, summarized:
1. Accountability — the “new” imperative
Enterprises want more cloud and greater agility, but they can’t outsource responsibility — for incidents, outages, security breaches or even, in the years ahead, carbon emissions. In 2021, hybrid IT, with workloads running in both on- and off-premises data centers, will continue to dominate, but investments will increasingly be constrained and shaped by the need for more transparency, oversight and accountability. More will be spent on cloud and other services, as well as in on-premises data centers.
2. Smarter, darker data centers
Following a scramble to effectively staff data centers during a pandemic, many wary managers are beginning to see remote monitoring and automation systems in a more positive light, including those driven by AI. An adoption cycle that has been slow and cautious will accelerate. But it will take more than just investment in software and services before the technology reduces staffing requirements.
3. Edge — the next frontier
Significant new demand for edge computing, fueled by technologies such as 5G, the internet of things and AI, is likely to build slowly — but the infrastructure preparation is underway. Expect new alliances and investments across enterprise, mobile and wireline networks, and for a wide range of edge data centers, small and large. Smart and automated software-defined networks and interconnections will become as important as the physical infrastructure.
4. Sustainability: More challenging, more transparent
For years, operators could claim environmental advances based on small, incremental and relatively inexpensive steps — or by adopting new technologies that would pay for themselves anyway. But the time of easy wins and greenwashing is ending: Regulators, watchdogs, customers and others will increasingly expect operators of digital infrastructure to provide hard and detailed evidence of carbon reductions, water savings and significant power savings — all while maintaining, if not improving, resiliency.
5. A surge of innovation
Data center operators (and enterprise IT) are mostly cautious, if not late, adopters of new technologies. Few beyond hyperscale operators can claim to have gained a competitive advantage through technology. However, several new technologies are maturing at the same time, promising advances in the performance and manageability of data centers and IT. Storage-class memory, silicon photonics, ARM servers and software-defined power are ready for greater adoption.
This is a summary of the full report on new Trends for 2021. The full report is available to members of the Uptime Institute. Click to learn more.
Rack Density is Rising
/in Design, Executive/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]Density is rising
The power density per rack (kilowatts [kW] per cabinet) is a critical number in data center design, capacity planning, and cooling and power provisioning. There have been industry warnings about a meteoric rise in IT equipment rack power density for the past decade (at least). One reason for this prediction is the proliferation of compute-intensive workloads (e.g., AI, IoT, cryptocurrencies, and augmented and virtual reality), all of which drive the need for high-density racks.
Our recent annual surveys found that racks with densities of 20 kW and higher are becoming a reality for many data centers (we asked about highest rack density) — but not to the degree forewarned. Year-over-year, most respondents said their highest density racks were in 10-19 kW range, which is not enough to merit wholesale technical changes. When rack densities are higher than 20-25 kW, direct liquid cooling and precision air cooling becomes more economical and efficient. According to what we see in the field, such high densities are not pervasive enough to have an impact on most data centers.
This does not mean that the trend should be ignored. It is clear from our latest research that average mean rack density in data centers is rising steadily, as the figure below shows. Eliminating respondents with above 30 kW as high-performance outliers, the mean average density in our 2020 survey sample was 8.4 kW/rack. This is consistent with other industry estimates and safely within the provisioned range of most facilities.
PUE over the Years
In our 2020 survey, we asked about the most common (modal average) SERVER rack density, which is perhaps a better metric than overall average density. More than two-thirds (71%) reported a modal average of below 10 kW/rack, with just 16% widely deploying 20 kW or higher rack densities (Figure 7). The most common density was 5-9 kW/rack. Overprovisioning of power/cooling is probably a more common issue than underprovisioning due to rising rack densities.
The modal average power consumption, rack density
Assuming a trend that rack units will increasingly be filled with higher-powered servers that are well utilized, we anticipate that the modal average kW/rack will increase over time. Figure 8 shows that for most organizations — roughly half of those surveyed — average density is increasing, albeit only slowly.
Rack density changing…
We expect density to keep rising. Our research shows that the use of virtualization and software containers pushes IT utilization up, in turn requiring more power and cooling. With Moore’s law slowing down, improvements in IT can require more multi-core processors and, consequently, more power consumption per operation, especially if utilization is low. Even setting aside new workloads, increases in density can be regarded a long-term trend.
But, as our 2020 survey findings demonstrate, the expectation for 20 kW racks throughout the industry has not manifested. We believe that many compute-intensive workloads — those that will significantly push up power use, rack density and heat — currently reside across a relatively small group of hyperscale cloud data centers and are consumed by organizations as a service.
Want to know more about this and other data center trends and strategies? Download a copy of our complete 2020 survey.
Common Factors for IT venue selection (Choosing In-house or Outsource)
/in Executive/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]Data center capacity is rapidly expanding in outsourced, third-party IT venues, such as colocation data centers and public cloud. Whether measured in megawatts (MW) of uninterruptible power supply capacity or IT load (or by some other measure, such as square feet of white or leased space, or in units of compute or storage) overall capacity is growing rapidly.
There are many factors driving this growth, including newer cloud-based services, such as social media and streaming; mobile applications and services; new enterprise services and applications; the migration of corporate workloads into colocation sites; and the adoption of software as a service (SaaS) and public cloud platforms.
Does this mean that almost all IT workloads will end up running in a third-party data center? In a word, No, not for a long time and in some cases, never. In the 2020 Uptime Institute survey, as in years past, we asked respondents to estimate, by percentage, how much of their workload/data is processed/stored in different types of data centers today and how this might look in two years’ time.
The majority (58%) said that most of their workloads run in corporate data centers today — that is, enterprise-owned, on-premises facilities. And if you add in Micro datacenters and Server closets, that figure rises to 69%!
These findings are similar to those of our previous years’ surveys. They confirm Uptime Institute’s view that the enterprise-owned data center sector, while not necessarily the most innovative, will continue to be the foundation of enterprise IT for the next decade. In our survey, nearly two-thirds of IT workloads are expected to be running in privately owned environments (large data centers, server closets and micro data centers) by 2022, with the remainder contracted to external suppliers. Although the enterprise data center sector is falling as a percentage of the whole, the absolute amount of enterprise data center capacity is still growing.
A mix of factors typically drive enterprise demand for third-party IT venues; similarly, multiple factors drive demand for on-premises data centers. Often some combination of the factors listed below come into play when deciding the best-execution venue for workloads.
COMMON DRIVERS FOR IT VENUE SELECTION DECISIONS
Choosing Outsourced:
Cost: Outsourcing can lower costs in the short to medium term. For organizations “born” in a public cloud or colo, it typically is cost-prohibitive to move to an enterprise data center.
Cost allocation: Outsourcing shifts cost allocations from capex toward more repeatable opex models.
IT agility and flexibility: Outsourcing provides the ability to readily and quickly adapt to changing capacity needs without the burden of managing the full stack of IT and applications; IT can be used for a project’s duration only (e.g., for test and development).
Access to resources: Third-parties may provide access to a wider range of resources, including technology, interconnections, software tools, services and application environments.
Security: Third-parties can offer the most advanced, highly resourced security features.
Choosing on-premise:
Cost: Ownership delivers total cost of ownership benefits over the long term; in the shorter term, owners avoid the data transport costs of moving to an outsourced venue.
Governance: On-premises environments may be necessary for compliance with data governance and regulatory requirements.
Control: Owners can closely monitor and control factors such as latency, availability and application performance. While most outsourced venues are strong in these areas, service level agreements vary and are limited.
Risk: Ownership ensures full visibility into (and the ability to adjust) the risk profile of every workload.
Security: Ownership provides the ability to maintain control and governance (dedicated rather than shared physical infrastructure) over security features.
Want to know more about venue selection, data center staffing and skills, latest technologies in use and the cloud, risk and cost management, operational strategies and more? Check out our full 2020 Annual Survey.
COVID-19, air filtration and energy use
/in Design, Executive/by Todd TraverThe COVID-19 pandemic has caused concerns about data center HVAC (heating, ventilation and air conditioning) filtration. So many data center operators are adjusting filtration protocols, including upgrading to finer MERV (minimum efficiency reporting value) 13 filters, to better filter out aerosols and COVID-19 virus carriers.1 But there is no free lunch. Depending on the design of the data center’s HVAC system, this increase in filtration static pressure can slightly increase energy consumption.
Technically, with a PSC (permanent split capacitor) blower motor, extra static pressure from this denser filter would result in lower airflow, reduced system performance and less air distribution. With ECMs (electronically commutated motors) — the type of motor most commonly found in data center cooling systems — this extra static pressure can result in higher blower motor power consumption.
Recently, an Uptime Institute member reported an increase of 8% energy use following a filtration upgrade from a MERV 8 to a more dense MERV 13 filter, due to the increased fan energy required to maintain static pressure and airflow. Since their data center fan power is 7-10% of total data center energy used, any increased fan energy affects their PUE (power usage effectiveness).
But this single data point may not be representative of all environments, since the quality and quantity of the media in the filter and the configuration of the HVAC system can significantly affect static pressure and airflow. Denser MERV 13 filters won’t necessarily require more fan energy if the operator installs filters that increase surface area by having more and deeper pleats.2
There is a direct correlation between airflow CFM (cubic feet per minute) and IWC (initial resistance water column), so some HVAC adjustments might be needed (see chart below).2
Some guidance is in order: When choosing a filter remember that airflow (static pressure increase) is just as important to consider as filtration (MERV rating). Just because a filter fits doesn’t mean it’s the best filter for the system, so its very important to check the rating of the facility’s filter and the design specifications of the cooling system.
Many data center cooling systems use an airflow rate between 330–490 CFM, so the system would experience 0.10-0.16 initial resistance (static pressure increase in inches of water column) for the MERV 11 filter example shown in the image below.3
Typical data center underfloor static pressure is 20-25 pascal or 0.08-0.1 inches of water column. Therefore, while this filter would introduce some increased static pressure, it pales in comparison to the 220 pascal static pressure required to move air through the cooling coils in the CRAC (computer room air conditioning) units or the impingement and turning of cooled air on the subfloor, which can be as high as 200 pascal.4
With careful airflow planning there should be minimal airflow and pressure impacts from more dense filters, such as MERV 13. These finer filters, though, will need to be changed more frequently, since they will trap more particulates and so will clog sooner. In addition to the higher change frequency, budget for higher per-unit costs, for filters with larger surface area media will likely be more expensive than smaller, less efficient filters.
But everything considered, given the modest energy trade-offs and the increased particulate and virus filtering they provide, MERV 13 filter use is quickly becoming the new normal.
Endnotes
1 https://www.ashrae.org/technical-resources/commercial#general
2 “Is There a Downside to High-MERV Filters? The new high-MERV filters extract an energy penalty,” by David Springer, November 02, 2009, in Home energy: The Home Performance Magazine. http://www.homeenergy.org/show/article/nav/issues/page/4/id/667
3 https://www.hvacrschool.com/air-filter-static-pressure-drop/
4 http://asmedigitalcollection.asme.org/heattransfer/article-pdf/132/7/073001/5794918/073001_1.pdf
Why ASHRAE is concerned about edge data centers
/in Design, Executive/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]There are few organizations that have had as big an impact on data center design as ASHRAE — and specifically, their Technical Committee (TC) 9.9. ASHRAE’s 2004 publication Thermal guidelines for Data Processing Environments (now in its fourth edition) described the optimal environmental operating conditions for the electronics in a data center, and in doing so, effectively established some of the key design goals that have been followed by almost every data center builder and operator for the past 15 years.
In October 2020, ASHRAE published what it describes as a “groundbreaking” Technical Bulletin — this time looking specifically at edge data centers. The Bulletin is a lighter, shorter document than most previous TC9.9 publications, and it does not signal any changes in the thermal operating guidelines that remain so important. Nor does it reveal any significant new findings — even though TC9.9 members do have access to a lot of data and tests conducted by equipment manufacturers.
But its brevity does not imply that it lacks importance. The ASHRAE committee members want to send a warning that unless careful processes and guidelines are followed, and good technical choices made, there will be a significantly increased risk of electronic equipment failure in many edge data centers. Given that a lot more processing will be done at the edge in the next decade or so that means there could be more failures and service disruptions in the future than in the past.
In the document, and in separate conversations with Uptime Institute (which has members on the TC9.9 committee), ASHRAE TC9.9 members identify two major issues as they relate to edge data centers.
First, the simple fact is that almost all “core” enterprise, large, and colocation data centers have a strictly climate-controlled white space. Small edge data centers, however, cannot always maintain this — especially during maintenance. Edge data center doors may be open during maintenance, exposing equipment to rapid temperature change, rain and pollution. Cooling may be turned off during maintenance, leading to rapid heat rise; routine or urgent maintenance may not be possible or timely; and edge environments are more likely to have a higher density and to be hotter, colder, wetter, dustier or more insecure than big, remote, well-managed facilities.
A second issue is that a lot of the equipment on the market has been designed for use in large data centers where failure is more easily tolerated — or even deliberately allowed, as part of an overall “abandon in place” strategy. Put bluntly, the quality of IT equipment is no longer required to be as high as it used to be, because it has been designed to be quickly swapped out. Some leading suppliers no longer do the systematic, lifecycle-based testing that was once a necessity.
What can be done?
The first place to start is to avoid poor site selection — but that, of course, is not always an option for small edge data centers, which need to be near the point of use. Ideally, the actual data center (which may be prefabricated) should be designed to allow ease of maintenance without exposure to outside conditions, to minimize damage (such as to batteries and IT equipment). A data center infrastructure management system that remotely monitors key environmental variables and IT performance will help prevent failures. It also makes sense to design with some concurrent maintainability/fault tolerance, if the design goals/budgets allow.
Other key points of advice are:
The key takeaway from this ASHRAE Bulletin is that it is far more difficult (and possibly more expensive) to control virtually all environmental variables in edge data centers. This will almost certainly lead to more failures. Vigilance and good practices can improve early detection and reduce the likelihood of failures and mitigate their impact.
The full document Edge Computing: Considerations for Reliable Operation is available on the ASHRAE website..
PUE: The golden metric is looking rusty
/in Executive, Operations/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]When the PUE (power usage effectiveness) metric was first discussed at a meeting of The Green Grid in Santa Clara, back in 2007, a microphone stand was placed in each aisle of the auditorium. The importance of the initiative was understood even then: experts, including the founders of Uptime Institute, formed lines to give their considered input. And if there was one point that came across, it was that the industry should not treat PUE as a comparative metric, a gold standard that every designer, operator and planning authority must chase down, for comparison, presentation and applause.
Thirteen years on, that is almost exactly what has happened. PUE, thanks to its simplicity and universal applicability, has become the critical benchmark for scoring data centers for efficiency and “greenness” (environmental responsibility). But it is often used uncritically.
This has had both positive and negative consequences. The positive side is that PUE values have been forced down across the world, as operators strive to reduce waste and operating costs. To meet their goals, suppliers have focused on improving electrical efficiency of equipment and have developed nonmechanical alternatives (such as free cooling). PUEs have fallen from an average of 2.5 in 2007 to around 1.6 today. In an Uptime Institute survey, around 95% of respondents said it is important that colocation companies have a low PUE.
So what is the negative side? Its simplicity. The very quality that led to PUE’s near universal adoption also may have led to its being over-used and misapplied — increasingly, with unintended consequences.
This observation is hardly new — indeed, this is exactly what those experts feared back in 2007 as they lined up to voice their concerns. And it is why PUE was developed into a standard, with rules on how it should be measured, and why Partial PUE variants were developed.
But the unintended consequences are becoming more serious and more concerning. The metric may be driving the wrong behaviors. For this reason, it is very possible that PUE values may begin to be de-emphasized in the near future and more operators may allow their number to rise a little. (In 2019, Uptime Institute did note a marginal rise in PUE values.) If this happens, some operators may find themselves in conflict with certain regulators and planners, for whom PUE has been an easily understood but simplistic way of enforcing environmental goals (policing this is another matter). These planning authorities, then, may also need to review their policies around driving down the PUE numbers. (California, Singapore, Amsterdam, take note.)
There are, we think, three strong and strengthening reasons why focusing on the PUE of the data center might be counter-productive:
There is, to be fair, a good counter-argument to these three points: that by searching for ways to lower the PUE regardless of these constraints, data center operators can find ways to meet all the goals. Liquid cooling, for example, can be highly resilient and uses little water. The argument here, then, is not for anyone to abandon PUE, or to stop watching and measuring it, but to apply it more cautiously than ever, and more flexibly.