There are few organizations that have had as big an impact on data center design as ASHRAE — and specifically, their Technical Committee (TC) 9.9. ASHRAE’s 2004 publication Thermal guidelines for Data Processing Environments (now in its fourth edition) described the optimal environmental operating conditions for the electronics in a data center, and in doing so, effectively established some of the key design goals that have been followed by almost every data center builder and operator for the past 15 years.
In October 2020, ASHRAE published what it describes as a “groundbreaking” Technical Bulletin — this time looking specifically at edge data centers. The Bulletin is a lighter, shorter document than most previous TC9.9 publications, and it does not signal any changes in the thermal operating guidelines that remain so important. Nor does it reveal any significant new findings — even though TC9.9 members do have access to a lot of data and tests conducted by equipment manufacturers.
But its brevity does not imply that it lacks importance. The ASHRAE committee members want to send a warning that unless careful processes and guidelines are followed, and good technical choices made, there will be a significantly increased risk of electronic equipment failure in many edge data centers. Given that a lot more processing will be done at the edge in the next decade or so that means there could be more failures and service disruptions in the future than in the past.
In the document, and in separate conversations with Uptime Institute (which has members on the TC9.9 committee), ASHRAE TC9.9 members identify two major issues as they relate to edge data centers.
First, the simple fact is that almost all “core” enterprise, large, and colocation data centers have a strictly climate-controlled white space. Small edge data centers, however, cannot always maintain this — especially during maintenance. Edge data center doors may be open during maintenance, exposing equipment to rapid temperature change, rain and pollution. Cooling may be turned off during maintenance, leading to rapid heat rise; routine or urgent maintenance may not be possible or timely; and edge environments are more likely to have a higher density and to be hotter, colder, wetter, dustier or more insecure than big, remote, well-managed facilities.
A second issue is that a lot of the equipment on the market has been designed for use in large data centers where failure is more easily tolerated — or even deliberately allowed, as part of an overall “abandon in place” strategy. Put bluntly, the quality of IT equipment is no longer required to be as high as it used to be, because it has been designed to be quickly swapped out. Some leading suppliers no longer do the systematic, lifecycle-based testing that was once a necessity.
What can be done?
The first place to start is to avoid poor site selection — but that, of course, is not always an option for small edge data centers, which need to be near the point of use. Ideally, the actual data center (which may be prefabricated) should be designed to allow ease of maintenance without exposure to outside conditions, to minimize damage (such as to batteries and IT equipment). A data center infrastructure management system that remotely monitors key environmental variables and IT performance will help prevent failures. It also makes sense to design with some concurrent maintainability/fault tolerance, if the design goals/budgets allow.
Other key points of advice are:
Protect the edge facility by servicing only during moderate weather or when using a mantrap or tent, to avoid potentially deleterious conditions. Even time of day can be a factor (e.g., humidity is often higher in the morning).
Monitor humidity, condensation and temperature during servicing.
Monitor the rate of temperature or humidity change when doors are opened.
Beware of local particulates/pollution, such as ocean spray, dust, industrial pollutants and heavy vehicle exhaust.
Air filtration should meet necessary standards — MERV (Minimum Efficiency Reporting Value) 11 or 13.
Beware the effect of gaseous pollutants that cause corrosion (e.g., of components containing copper or silver).
Use remote monitoring where possible to track corrosion rate, filter performance, etc.
The key takeaway from this ASHRAE Bulletin is that it is far more difficult (and possibly more expensive) to control virtually all environmental variables in edge data centers. This will almost certainly lead to more failures. Vigilance and good practices can improve early detection and reduce the likelihood of failures and mitigate their impact.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/11/Edge1.jpg9202185Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-11-09 04:54:042020-10-29 08:36:15Why ASHRAE is concerned about edge data centers
When the PUE (power usage effectiveness) metric was first discussed at a meeting of The Green Grid in Santa Clara, back in 2007, a microphone stand was placed in each aisle of the auditorium. The importance of the initiative was understood even then: experts, including the founders of Uptime Institute, formed lines to give their considered input. And if there was one point that came across, it was that the industry should not treat PUE as a comparative metric, a gold standard that every designer, operator and planning authority must chase down, for comparison, presentation and applause.
Thirteen years on, that is almost exactly what has happened. PUE, thanks to its simplicity and universal applicability, has become the critical benchmark for scoring data centers for efficiency and “greenness” (environmental responsibility). But it is often used uncritically.
This has had both positive and negative consequences. The positive side is that PUE values have been forced down across the world, as operators strive to reduce waste and operating costs. To meet their goals, suppliers have focused on improving electrical efficiency of equipment and have developed nonmechanical alternatives (such as free cooling). PUEs have fallen from an average of 2.5 in 2007 to around 1.6 today. In an Uptime Institute survey, around 95% of respondents said it is important that colocation companies have a low PUE.
So what is the negative side? Its simplicity. The very quality that led to PUE’s near universal adoption also may have led to its being over-used and misapplied — increasingly, with unintended consequences.
This observation is hardly new — indeed, this is exactly what those experts feared back in 2007 as they lined up to voice their concerns. And it is why PUE was developed into a standard, with rules on how it should be measured, and why Partial PUE variants were developed.
But the unintended consequences are becoming more serious and more concerning. The metric may be driving the wrong behaviors. For this reason, it is very possible that PUE values may begin to be de-emphasized in the near future and more operators may allow their number to rise a little. (In 2019, Uptime Institute did note a marginal rise in PUE values.) If this happens, some operators may find themselves in conflict with certain regulators and planners, for whom PUE has been an easily understood but simplistic way of enforcing environmental goals (policing this is another matter). These planning authorities, then, may also need to review their policies around driving down the PUE numbers. (California, Singapore, Amsterdam, take note.)
There are, we think, three strong and strengthening reasons why focusing on the PUE of the data center might be counter-productive:
Resiliency (N to N+1). There is an industry-wide move to increase the resiliency of data centers, with the growing dependency on IT and the impact of the pandemic and extreme weather among the drivers. At the physical site level, it is difficult to improve redundancy and resiliency without adding power-consuming equipment — which drives up the PUE. (Similarly, one operator installed the highest level of MERV [Minimum Efficiency Reporting Value] air filters to filter out viruses — but fan energy use rose by 8% and the PUE value rose accordingly.)
Water use. The drive to improve energy efficiency has encouraged operators to use economizers and, in particular, adiabatic and evaporative cooling and chillers. But this has often involved a trade-off: energy use drops, but water use rises. That compromise is becoming problematic in some regions, where climate change means water is becoming more scarce and, in some instances, a bigger local environmental issue than on-site energy use. Again, PUE may be the wrong metric to track — WUE (water usage effectiveness) is a more important one.
IT efficiency. It is widely known that while facilities have become ever more energy efficient, IT use can be extremely inefficient and wasteful, with processors and memory often drawing energy while doing relatively little work. Energy-saving efficiency investments on the IT side, however, are not rewarded with a lower PUE, but rather a higher one (as PUE is the ratio of the IT to non-IT energy figures). This has been known from the outset. With a slew of new technologies becoming available to lower the overall energy figure, it is important that the need to meet a given PUE value does not discourage investments and innovation on the IT side.
There is, to be fair, a good counter-argument to these three points: that by searching for ways to lower the PUE regardless of these constraints, data center operators can find ways to meet all the goals. Liquid cooling, for example, can be highly resilient and uses little water. The argument here, then, is not for anyone to abandon PUE, or to stop watching and measuring it, but to apply it more cautiously than ever, and more flexibly.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/11/PUE-Rusty.jpg380895Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-11-02 07:32:042020-10-26 12:18:10PUE: The golden metric is looking rusty
When Uptime Institute recently asked over 300 data center managers how the pandemic would change their operations, one answer stood out: Two-thirds expect to increase the resiliency of their core data center(s) in the years ahead. Many said they expected their costs to increase as a result.
The reasoning is clear: the pandemic — or any future one — can mean operating with fewer staff, and possibly with disrupted service and supply chains. Remote monitoring and preventive maintenance will help to reduce the likelihood of an incident, but machines will always fail. It makes sense to reduce the impact of failure by increasing system redundancy.
But even before the pandemic, there was a trend toward higher levels of redundancy. As shown in the figure below, roughly half of those participating in the Uptime Institute 2020 global survey of suppliers, designers and advisors reported their customers have increased redundancy levels in the last three to five years.
This trend may seem unsurprising to some, but it was not entirely predictable. The growth of cloud has been accompanied by the much greater use of multisite resiliency and regional availability zones. In theory, at least, these substantially reduce the impact of single-site facility outages, because traffic and workloads can be diverted elsewhere. Backed by this capability, some operators — Facebook is an example — have proceeded with lower levels of redundancy than was common in the past (thereby saving costs and energy).
The use of availability zones, however, has come with its own problems, with networking and software issues often causing service interruptions. And the loss of one data center immediately places capacity and traffic demand on others, heightening risks. For this reason, even the big cloud providers and internet application operators manage mostly concurrently maintainable facilities, and it is common for them to stipulate that colocation partners have N+2 level facilities.
With a variety of options, the overall shift to increased resiliency is still slow and quite nuanced, with designers mostly favoring either N+1 or N+2 configurations, according to site and business needs, and, often, according to the creativity of the designers. Overall, there is actually a marginal decrease in the number of data centers that are 2N, but a steady three-year shift from N+1 to N+2 — not only in power, but also in cooling (see figure below). There is also an increase in the use of active-active availability zones, as discussed in our recent report Uptime Institute global data center survey 2020.
Demand patterns and growing IT dependency partly account for these higher levels of redundancy/resiliency. The level of resiliency needed for each service or by each customer is dictated by business requirements, but this is not fixed in time. The growing criticality of many IT services highlights the importance of mitigating risk through increased resiliency. “Creeping criticality” — a situation in which infrastructure and processes have not been upgraded or updated to reflect the growing criticality of the applications or business processes they support — may require redundancy upgrades.
Uptime Institute expects operators to make more use of distributed resiliency in the future — especially as more workloads are designed using cloud or microservices architectures (workloads are more portable, and instances are more easily copied). But there is no sign that this is diminishing the need for site-level resiliency. The software running these distributed services is often opaque, complex and may be prone to errors of programming or configuration. Annual outage data shows these types of issues are proliferating. Further, any big component failures can cascade, making recovery difficult and expensive, with data and applications synchronized across multiple sites.
The trend for now is clear: more resiliency at every level is the least risky approach — even if it means some extra expense and duplication of effort.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/10/type1a.jpg3551475Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]https://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngAndy Lawrence, Executive Director of Research, Uptime Institute, [email protected]2020-10-26 06:00:342021-05-04 10:19:35Why data center operators are investing in more redundancy
In mid-September 2020, The US Bureau of Labor and Statistics published their updated 2019-2029 Employment Projection summary news release and the associated handbook which discusses various job roles and hiring and salary expectations over the next decade. It identified a number of easy to consume trends which caught my attention and perhaps worth your consideration.
In the report, they state that in 2019 we had 162.8 million workers in the USA which they project to grow by 6-8 million jobs (in total) over the next ten years, an annual growth of less than HALF of 1%, compared to the last decade’s growth of 1.3%. Not surprising (and due to the aging of America), 60% of those new jobs will be in the health and medical fields. You can browse through the interactive handbook and click on any number of filters and report types, but BE SURE to try the links in the “Browse Occupations” section on the bottom half of the first screen. Try “Most New Jobs” or “Highest Paid Jobs”. (Spoiler alerts: Highest-Paid Jobs has Psychiatrists listed at the top, and while Most-New Jobs is listed as “Home Health”, the second biggest increase in jobs is Fast Food workers!)
BLS talks about some notable related and supporting projections which can be summarized as follows:
More people over 55 years old will continue to be employed full-time (financial stability has decreased)
Fewer people below 34 years old will choose to be employed full-time (the ‘desk job’ has less appeal to many in this age group who want freelance or “gig” style work)
About 57% of women will work full-time outside the home, compared to 66% of men (and both will be down slightly from today)
The BLS offers their narrative: “The decline in labor force participation is due to the aging of the babyboom generation, a continuation of the declining trend in men’s participation, and a slight decline in women’s participation”. And throughout the report they cite the aging of the baby-boomers as a root cause.
BLS spends billions of dollars and countless resources gathering data and analyzing it and ultimately come to the same conclusion and projections that we all implicitely feel: Baby-Boomers are timing out, or as I like to say ‘Greying out’. Couple this workforce aging with the URGENT macro topics of:
COVID – all of the long-term shifts and economics caused by it
Trade Policies – shifting of raw materials/goods/transportation/macro-economics
‘GIG work Mentality’ – intentionally skipping full-time employment altogether
Online everything – The death of brick and mortar retail
and we have a crazy ride over the next 10 years to be sure. The world we knew has changed, and those changes are STRUCTURAL. We will never go back to pre-2020 life…. This is not a fad, nor a short-term manageable incident. The world has changed and the quicker each of us decides how they wish to participate in the new structure, the easier life will be for that person. And when enough people have gotten on board, life will become easier once again.
Note: While the report above is USA-centric, similiar patterns are already being seen worldwide.
https://journal.uptimeinstitute.com/wp-content/uploads/2020/09/BLS-blog.jpg8202320Mark Harrishttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngMark Harris2020-10-19 06:00:012020-10-20 09:38:45Job Projections, 2019-2029, Macro shifts, Gig work and the Baby-Boomers
Periodically we get questions about data center temperatures and energy savings. The questions usually ask about possible energy savings in commercial data centers by raising the temperature of the data hall. Some form of this question surfaces regularly over time because it just seems to be a common sense opportunity. But there is alot to consider before running out and adjusting your “thermostat”.
The most recent question took the form of “I have followed ASHRAE for some years and have increased temperature thresholds in rooms over which I have full control. If there are cost and efficiency gains by accepting higher temperatures, is there talk of marketing this — i.e., segregated/containerized hotter rooms — to colo customers who believe they can take the heat to save on monthly recurring charges? If so, I have not seen it yet.”.
This really is a great question, because it means we are thinking about energy conservation and optimization. At first glance this seems like easy avenue to pursue. In fact many of the IT gear makers are already supporting higher inlet temperatures, some as high as 95-degrees F or more. Earlier this year, our CTO Chris Brown provided the short answer, “In general, there are cost savings and energy efficiency gains to be realized by increasing supply temperatures in data halls.” and then went on to add that, ‘BUT it is not as simple as it sounds, because alot of factors need to be considered before it’s a ‘slam-dunk'”.
So lets dive into these factors in some more detail:
Increasing the inlet supply temperature often results in increased air flow requirements. Depending on the design of the facility, this may over tax the fans and significant increase their energy consumption if the system is tightly designed around a given air flow. Remember that a non-trivial amount of the energy spent for cooling is consumed by the fan motors themselves, so running fans faster WILL reduce any savings seen with higher temperatures. Using an Energystar report on power consumption of fans, they state that “a reduction of 10% in fan speed reduces that fan’s use of electricity by approximately 25%. A 20% speed reduction yields electrical savings of roughly 45%.” so for this discussion we’d be doing the reverse… Increasing the fan speed with the resulting increase in power draw.
Manufacturers usually warranty their equipment when used within their specified operational conditions, including the range of temperatures at the inlet. And while some manufacturers may have increased those ranges to allow for warmer inlet temperatures, the data hall must be cooled to the LOWEST operating condition specified across all equipment installed in that hall. In other words, your thousand Dell servers may happily support inlets of 85 or 95-degrees, but your Cisco NEXUS sitting at the end of each row may require 80F or less. And the storage appliance may specifiy 78F. Each piece of gear has it’s own allowable environment to be warrantied. And there is no practicle way to separate the various types of gear across different ‘temperature zones’ in a commercial data center even if such zones were available, which they are not.
Most enterprise grade IT gear also includes in-chassis fans, which vary in speed as the inlet temperature goes up. Again, raising the hall temperatures and the inlet temperatures will likely cause all of the in-chassis fans to run much faster, and draw significantly more current. (Which ironicly is great for the old macro PUE calculation, but horrible when considering actual energy usage for cooling). And as Chris pointed out in his response, there is a secondary concern when using in-chassis fans more heavily: “doing so can significantly raise the sizing requirements on UPS systems, because all of that new in-chassis fan power is now considered ‘IT load’ from a design standpoint.” A huge factor to be sure.
The incentive may not exist. Most colocation providers write SLAs (service level agreements) specifying inlet temperatures UP to and including the ASHRAE “Recommended” 80-degrees. (Many still contractually guarantee an inlet of 74F). Chris advises that he has not run across any of those type of client agreements which offer different rates based on willingness to operate in other environmental conditions (i.e. ASHRAE A1 through A4 for instance). This is likely because providers with existing facilities do not have their data hall designed to allow for varying the supply temperature in such granular neighborhoods. Hot/Cold aisels may exist, but the inlet temperatures are not offered on a slide scale pricing schedule. And since not all tenants in any given hall will want an elevated inlet supply air temperature, those operators and their contracts do not provide any variants.
Failure rates of IT gear have not been studied sufficiently at the A3 and above higher temperature ranges. Intel has been studing this question for years and regularly concludes that servers running up through the A2 range (95 degrees F) seem to have typical failure rates. They also state that more study is required to make the same statements around A3 and A4. And then you have the network, storage and security gear from hundrerds of manufacturers that also have the same requirements.
And keep in mind that running a data center above ASHRAE’s “Recommended” levels might limit the number of hours the data center technicians could remain on the IT floor due to local regulatory, health and safety concerns. As cited by OSHA, working conditions between 68 F and 78 F are ideal for the workplace. A number of other health and safety journals (including Healthday) go into more detail about what would be expected if technicians are required to work hotter data centers. Their summary includes higher rates of accidents and more mis-judgments as the temperature increases above 73-degress, with significant concerns for employees working in the conditions identified by ASHRAE’s A3 and A4.
So can higher temperature halls be effective? Absolutely, if they were designed to be higher. If the mechanical design accounted for the hall to be much hotter (gasp: A3 or A4 ?) and if ALL of the IT gear that was to be installed in that hall was purchased and warrantied for that range, then absolutely this works. But few if any halls are built and/or operated that way today, because doing so would severely limit the choice of gear that could be included on the raised floor.
Directionally a great discussion… it’s just the retrofit of reality discussion that gets awkward…
https://journal.uptimeinstitute.com/wp-content/uploads/2020/10/thermo3.jpg7352420Mark Harrishttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngMark Harris2020-10-12 06:00:522020-09-21 15:37:57Increasing data center temperatures to reduce energy costs
Why ASHRAE is concerned about edge data centers
/in Design, Executive/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]There are few organizations that have had as big an impact on data center design as ASHRAE — and specifically, their Technical Committee (TC) 9.9. ASHRAE’s 2004 publication Thermal guidelines for Data Processing Environments (now in its fourth edition) described the optimal environmental operating conditions for the electronics in a data center, and in doing so, effectively established some of the key design goals that have been followed by almost every data center builder and operator for the past 15 years.
In October 2020, ASHRAE published what it describes as a “groundbreaking” Technical Bulletin — this time looking specifically at edge data centers. The Bulletin is a lighter, shorter document than most previous TC9.9 publications, and it does not signal any changes in the thermal operating guidelines that remain so important. Nor does it reveal any significant new findings — even though TC9.9 members do have access to a lot of data and tests conducted by equipment manufacturers.
But its brevity does not imply that it lacks importance. The ASHRAE committee members want to send a warning that unless careful processes and guidelines are followed, and good technical choices made, there will be a significantly increased risk of electronic equipment failure in many edge data centers. Given that a lot more processing will be done at the edge in the next decade or so that means there could be more failures and service disruptions in the future than in the past.
In the document, and in separate conversations with Uptime Institute (which has members on the TC9.9 committee), ASHRAE TC9.9 members identify two major issues as they relate to edge data centers.
First, the simple fact is that almost all “core” enterprise, large, and colocation data centers have a strictly climate-controlled white space. Small edge data centers, however, cannot always maintain this — especially during maintenance. Edge data center doors may be open during maintenance, exposing equipment to rapid temperature change, rain and pollution. Cooling may be turned off during maintenance, leading to rapid heat rise; routine or urgent maintenance may not be possible or timely; and edge environments are more likely to have a higher density and to be hotter, colder, wetter, dustier or more insecure than big, remote, well-managed facilities.
A second issue is that a lot of the equipment on the market has been designed for use in large data centers where failure is more easily tolerated — or even deliberately allowed, as part of an overall “abandon in place” strategy. Put bluntly, the quality of IT equipment is no longer required to be as high as it used to be, because it has been designed to be quickly swapped out. Some leading suppliers no longer do the systematic, lifecycle-based testing that was once a necessity.
What can be done?
The first place to start is to avoid poor site selection — but that, of course, is not always an option for small edge data centers, which need to be near the point of use. Ideally, the actual data center (which may be prefabricated) should be designed to allow ease of maintenance without exposure to outside conditions, to minimize damage (such as to batteries and IT equipment). A data center infrastructure management system that remotely monitors key environmental variables and IT performance will help prevent failures. It also makes sense to design with some concurrent maintainability/fault tolerance, if the design goals/budgets allow.
Other key points of advice are:
The key takeaway from this ASHRAE Bulletin is that it is far more difficult (and possibly more expensive) to control virtually all environmental variables in edge data centers. This will almost certainly lead to more failures. Vigilance and good practices can improve early detection and reduce the likelihood of failures and mitigate their impact.
The full document Edge Computing: Considerations for Reliable Operation is available on the ASHRAE website..
PUE: The golden metric is looking rusty
/in Executive, Operations/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]When the PUE (power usage effectiveness) metric was first discussed at a meeting of The Green Grid in Santa Clara, back in 2007, a microphone stand was placed in each aisle of the auditorium. The importance of the initiative was understood even then: experts, including the founders of Uptime Institute, formed lines to give their considered input. And if there was one point that came across, it was that the industry should not treat PUE as a comparative metric, a gold standard that every designer, operator and planning authority must chase down, for comparison, presentation and applause.
Thirteen years on, that is almost exactly what has happened. PUE, thanks to its simplicity and universal applicability, has become the critical benchmark for scoring data centers for efficiency and “greenness” (environmental responsibility). But it is often used uncritically.
This has had both positive and negative consequences. The positive side is that PUE values have been forced down across the world, as operators strive to reduce waste and operating costs. To meet their goals, suppliers have focused on improving electrical efficiency of equipment and have developed nonmechanical alternatives (such as free cooling). PUEs have fallen from an average of 2.5 in 2007 to around 1.6 today. In an Uptime Institute survey, around 95% of respondents said it is important that colocation companies have a low PUE.
So what is the negative side? Its simplicity. The very quality that led to PUE’s near universal adoption also may have led to its being over-used and misapplied — increasingly, with unintended consequences.
This observation is hardly new — indeed, this is exactly what those experts feared back in 2007 as they lined up to voice their concerns. And it is why PUE was developed into a standard, with rules on how it should be measured, and why Partial PUE variants were developed.
But the unintended consequences are becoming more serious and more concerning. The metric may be driving the wrong behaviors. For this reason, it is very possible that PUE values may begin to be de-emphasized in the near future and more operators may allow their number to rise a little. (In 2019, Uptime Institute did note a marginal rise in PUE values.) If this happens, some operators may find themselves in conflict with certain regulators and planners, for whom PUE has been an easily understood but simplistic way of enforcing environmental goals (policing this is another matter). These planning authorities, then, may also need to review their policies around driving down the PUE numbers. (California, Singapore, Amsterdam, take note.)
There are, we think, three strong and strengthening reasons why focusing on the PUE of the data center might be counter-productive:
There is, to be fair, a good counter-argument to these three points: that by searching for ways to lower the PUE regardless of these constraints, data center operators can find ways to meet all the goals. Liquid cooling, for example, can be highly resilient and uses little water. The argument here, then, is not for anyone to abandon PUE, or to stop watching and measuring it, but to apply it more cautiously than ever, and more flexibly.
Why data center operators are investing in more redundancy
/in Design, Executive, Operations/by Andy Lawrence, Executive Director of Research, Uptime Institute, [email protected]When Uptime Institute recently asked over 300 data center managers how the pandemic would change their operations, one answer stood out: Two-thirds expect to increase the resiliency of their core data center(s) in the years ahead. Many said they expected their costs to increase as a result.
The reasoning is clear: the pandemic — or any future one — can mean operating with fewer staff, and possibly with disrupted service and supply chains. Remote monitoring and preventive maintenance will help to reduce the likelihood of an incident, but machines will always fail. It makes sense to reduce the impact of failure by increasing system redundancy.
But even before the pandemic, there was a trend toward higher levels of redundancy. As shown in the figure below, roughly half of those participating in the Uptime Institute 2020 global survey of suppliers, designers and advisors reported their customers have increased redundancy levels in the last three to five years.
This trend may seem unsurprising to some, but it was not entirely predictable. The growth of cloud has been accompanied by the much greater use of multisite resiliency and regional availability zones. In theory, at least, these substantially reduce the impact of single-site facility outages, because traffic and workloads can be diverted elsewhere. Backed by this capability, some operators — Facebook is an example — have proceeded with lower levels of redundancy than was common in the past (thereby saving costs and energy).
The use of availability zones, however, has come with its own problems, with networking and software issues often causing service interruptions. And the loss of one data center immediately places capacity and traffic demand on others, heightening risks. For this reason, even the big cloud providers and internet application operators manage mostly concurrently maintainable facilities, and it is common for them to stipulate that colocation partners have N+2 level facilities.
With a variety of options, the overall shift to increased resiliency is still slow and quite nuanced, with designers mostly favoring either N+1 or N+2 configurations, according to site and business needs, and, often, according to the creativity of the designers. Overall, there is actually a marginal decrease in the number of data centers that are 2N, but a steady three-year shift from N+1 to N+2 — not only in power, but also in cooling (see figure below). There is also an increase in the use of active-active availability zones, as discussed in our recent report Uptime Institute global data center survey 2020.
Demand patterns and growing IT dependency partly account for these higher levels of redundancy/resiliency. The level of resiliency needed for each service or by each customer is dictated by business requirements, but this is not fixed in time. The growing criticality of many IT services highlights the importance of mitigating risk through increased resiliency. “Creeping criticality” — a situation in which infrastructure and processes have not been upgraded or updated to reflect the growing criticality of the applications or business processes they support — may require redundancy upgrades.
Uptime Institute expects operators to make more use of distributed resiliency in the future — especially as more workloads are designed using cloud or microservices architectures (workloads are more portable, and instances are more easily copied). But there is no sign that this is diminishing the need for site-level resiliency. The software running these distributed services is often opaque, complex and may be prone to errors of programming or configuration. Annual outage data shows these types of issues are proliferating. Further, any big component failures can cascade, making recovery difficult and expensive, with data and applications synchronized across multiple sites.
The trend for now is clear: more resiliency at every level is the least risky approach — even if it means some extra expense and duplication of effort.
Job Projections, 2019-2029, Macro shifts, Gig work and the Baby-Boomers
/in Executive/by Mark HarrisIn mid-September 2020, The US Bureau of Labor and Statistics published their updated 2019-2029 Employment Projection summary news release and the associated handbook which discusses various job roles and hiring and salary expectations over the next decade. It identified a number of easy to consume trends which caught my attention and perhaps worth your consideration.
In the report, they state that in 2019 we had 162.8 million workers in the USA which they project to grow by 6-8 million jobs (in total) over the next ten years, an annual growth of less than HALF of 1%, compared to the last decade’s growth of 1.3%. Not surprising (and due to the aging of America), 60% of those new jobs will be in the health and medical fields. You can browse through the interactive handbook and click on any number of filters and report types, but BE SURE to try the links in the “Browse Occupations” section on the bottom half of the first screen. Try “Most New Jobs” or “Highest Paid Jobs”. (Spoiler alerts: Highest-Paid Jobs has Psychiatrists listed at the top, and while Most-New Jobs is listed as “Home Health”, the second biggest increase in jobs is Fast Food workers!)
BLS talks about some notable related and supporting projections which can be summarized as follows:
The BLS offers their narrative: “The decline in labor force participation is due to the aging of the babyboom generation, a continuation of the declining trend in men’s participation, and a slight decline in women’s participation”. And throughout the report they cite the aging of the baby-boomers as a root cause.
BLS spends billions of dollars and countless resources gathering data and analyzing it and ultimately come to the same conclusion and projections that we all implicitely feel: Baby-Boomers are timing out, or as I like to say ‘Greying out’. Couple this workforce aging with the URGENT macro topics of:
and we have a crazy ride over the next 10 years to be sure. The world we knew has changed, and those changes are STRUCTURAL. We will never go back to pre-2020 life…. This is not a fad, nor a short-term manageable incident. The world has changed and the quicker each of us decides how they wish to participate in the new structure, the easier life will be for that person. And when enough people have gotten on board, life will become easier once again.
Note: While the report above is USA-centric, similiar patterns are already being seen worldwide.
Increasing data center temperatures to reduce energy costs
/in Design, Executive/by Mark HarrisPeriodically we get questions about data center temperatures and energy savings. The questions usually ask about possible energy savings in commercial data centers by raising the temperature of the data hall. Some form of this question surfaces regularly over time because it just seems to be a common sense opportunity. But there is alot to consider before running out and adjusting your “thermostat”.
The most recent question took the form of “I have followed ASHRAE for some years and have increased temperature thresholds in rooms over which I have full control. If there are cost and efficiency gains by accepting higher temperatures, is there talk of marketing this — i.e., segregated/containerized hotter rooms — to colo customers who believe they can take the heat to save on monthly recurring charges? If so, I have not seen it yet.”.
This really is a great question, because it means we are thinking about energy conservation and optimization. At first glance this seems like easy avenue to pursue. In fact many of the IT gear makers are already supporting higher inlet temperatures, some as high as 95-degrees F or more. Earlier this year, our CTO Chris Brown provided the short answer, “In general, there are cost savings and energy efficiency gains to be realized by increasing supply temperatures in data halls.” and then went on to add that, ‘BUT it is not as simple as it sounds, because alot of factors need to be considered before it’s a ‘slam-dunk'”.
So lets dive into these factors in some more detail:
So can higher temperature halls be effective? Absolutely, if they were designed to be higher. If the mechanical design accounted for the hall to be much hotter (gasp: A3 or A4 ?) and if ALL of the IT gear that was to be installed in that hall was purchased and warrantied for that range, then absolutely this works. But few if any halls are built and/or operated that way today, because doing so would severely limit the choice of gear that could be included on the raised floor.
Directionally a great discussion… it’s just the retrofit of reality discussion that gets awkward…
Infographic: Data Center Industry Staffing Shortage is Getting Worse
/in Executive, Operations/by Rhonda Ascierto, Vice President, Research, Uptime InstituteGender Diversity 2018-2020