True or false? When it comes to Tier Certification, just ask Uptime Institute
By Uptime Institute Staff
Uptime Institute’s Tier Classification System for data centers has reached the two-decade mark. Since its creation in the mid-1990s, Tiers has evolved from a shared industry terminology into the global standard for third-party validation of data center critical infrastructure.
In that time, the industry has changed, and Tiers has evolved with it, remaining as relevant and important as it was when Uptime Institute first developed and disseminated Tiers. At the same time, Uptime Institute has observed that public understanding of Tiers has been clouded by the many myths and misconceptions that have developed over the years.
Uptime Institute has long been aware that not everyone fully understands the concepts described by the Tier Standards, and yet others disagree with some of the definitions. Both these situations lead to classic misunderstandings in which individuals substitute their preferences for accurate information. Other times, however, marketers have invoked a kind of shorthand based on Tiers. While objectionable, these marketers have coined terms like Tier III plus when speaking to their potential customers. These terms have no basis in Tiers but can be especially confusing to IT, real estate, procurement personnel, and even CFOs, all of whom might lack a technical background.
Other myths develop because some industry professionals reference old, out-of-date publications and explanatory material that is no longer valid. There may be other sources of myth, but knowing that the Uptime Institute is the only source of current and reliable information about Tiers is what is really important. We conduct numerous classes during the year, write many articles, and eld numerous inquiries to keep the industry current on Tiers.
Fundamentally, Uptime Institute created the Tier Classification System to consistently evaluate various data center facilities in terms of potential site infrastructure performance, or uptime. The system comprises four Tiers; each Tier incorporates the requirements of the lower Tiers.
• Tier I: Basic Capacity
• Tier II: Redundant Capacity Components
• Tier III: Concurrently Maintainable
• Tier IV: Fault Tolerant
Data center infrastructure costs and operational complexities increase with Tier level, and it is up to the data center owner to determine the Tier that fits the business’s need.
Uptime Institute is the only organization permitted to Certify data centers against the Tier Classification System. Uptime Institute does not design, build, or operate data centers. Uptime Institute’s role is to evaluate site infrastructure, operations, and strategy.
From this experience, we have compiled and addressed many of the myths and misconceptions. You can read about some of these experiences in Uptime Institute eJournal articles such as “Avoid Failure and Delay on Capital Projects: Lessons from Tier Certification” and “Avoiding Data Center Construction Problems.” For even more information, please contact us at https://uptimeinstitute.com/contact.
Tiers does not address business requirements.
False. Tiers is a performance-based, business-case-driven data center benchmarking system. An organization’s risk tolerance determines the appropriate Tier for the business. In other words, Tiers is predicated on the business case of an individual company. Companies that fail to develop a unique business case for their facilities before developing a Tier objective are misusing Tiers and bypassing the internal dialogue that needs to occur.
Tier IV is the best.
False. An organization’s tolerance for risk determines the appropriate Tier to support the business objective. Tier IV is not the best answer for all organizations, and neither is Tier II. Owners should perform due diligence assessments of their facilities before determining a Tier objective. If no business objective is defined, then Tiers may be misused to rationalize unnecessary investment.
Tier I and Tier II are tactical solutions, usually driven more by first-cost and time-to-market than life-cycle cost and performance (uptime) requirements. Organizations selecting Tier I and Tier II solutions typically do not depend on real-time delivery of products or services for a significant part of their revenue stream. Generally, these organizations are contractually protected from damages stemming from lack of system availability.
Rigorous uptime requirements and long-term viability are usually the reason for selecting strategic solutions found in Tier III and Tier IV site infrastructure. In a Tier III facility, each and every capacity component can be taken out of service on a planned basis, without affecting the critical environment or IT processes. Tier IV solutions are even more robust, as each and every capacity component and distribution path can sustain a failure, error, or unplanned event without impacting the critical environment or IT processes.
A Tier IV solution is not better than a Tier II solution. The performance and capabilities of a data center’s infrastructure should match a business application; otherwise companies may overinvest or take on too much risk.
For example, before building a Tier II Certified Constructed Facility, which by definition does not include Concurrent Maintainability across all critical subsystems, an owner should consider whether the business can tolerate a planned or maintenance-related shutdown and how the site operations team would coordinate a site-wide shutdown for maintenance. Similarly business objectives should drive decisions to build a Tier I, Tier III, or Tier IV Certified Constructed Facility.
Component count determines Tier level.
False. Tier Certification is a performance-based evaluation of a data center’s specific infrastructure; it is not a checklist or cookbook. Unfortunately, some industry shorthand employs N terminology—where N is de ned as the number of components that are minimally required to meet the load demand—to define availability. Incorporating more equipment can be described as designing an N+1, N+2, 2N, or 2(N+1) facility. However, increasing the component count does not determine or guarantee achievement of any specific Tier level, because Tiers also includes evaluation of distribution pathways and other system elements. Therefore, it is possible to achieve Tier IV with just N+1 components, depending on how they are configured and connected to redundant distribution pathways.
Design Certification is the only Certification that matters.
False. The first step in a Tier Certification process is a Tier Certification of Design Documents. Uptime Institute Consultants review the 100% design documents, ensuring each electrical, mechanical, monitoring, and automation subsystem meets the fundamental concepts and there are no weak links in the chain. The Design Certification is intended to be a milestone so that data center owners can commence data center construction knowing that the intended design meets the Tier objective.
Tier Certification of Design Documents applies to a document package. It is intended as provisional verification until the Tier Certification of Constructed Facility. Uptime Institute has not verified the constructed environment of these facilities, and thus cannot speak to the standard(s) to which they were built. To emphasize this point, Uptime Institute implemented an expiration date on Design Certifications. All Tier Certification of Design Documents awards issued after 1 January 2014 expired two years after the award date.
During a Facility Certification, a team of Uptime Institute consultants conducts a site visit, identifying discrepancies between the design drawings and installed equipment. The consultants observe tests and demonstrations to prove Tier compliance. Fundamentally, this is the value of the Tier Certification, finding these blind spots and weak points in the chain. Uptime Institute consultants say that in almost every site visit they find that changes have been made after the Design Certification was awarded so that one or more systems or subsystems will not perform in a way that complies with Tier requirements.
More recently, Uptime Institute instituted the Tier Certification of Operational Sustainability to evaluate how operators run and manage their mission-critical facilities. Even the most robustly designed and constructed facilities may experience outages without a well-developed comprehensive management and operation program. Certification at all three levels is how data center owners can be assured they are realizing the maximum potential of their data centers.
Tier levels specify an estimated downtime per year.
False. Uptime Institute removed references to “expected downtime per year” from the Tier Standard in 2009, but they were never a part of the Tier definitions. Tier Standard: Topology is based on specific performance factors (outcomes) that demonstrate that a facility has met specific performance objectives, such as having redundant capacity components, Concurrent Maintainability (generally, the ability to remove any capacity or distribution component from service on a planned basis without impacting IT), or Fault Tolerance (generally, the ability to experience any unplanned failure in the site infrastructure without impacting IT). However, even a Tier IV data center, which is Fault Tolerant, may experience IT outages if it is not operated and managed effectively.
There are statistical tools to predict the frequency of failures and time to recover. Availability is simply the arithmetic calculation of time a site was available over total time. The number, frequency, and duration of disruptions will drive the availability result. However, caution is appropriate when using these tools. Human activity is often not considered by statistical models. In addition, the statistical prediction of a 100-year storm, for example, can obscure the possibility that several 100-year storms can happen in the same year.
Tier Certification applies only to newly built facilities.
False. Uptime Institute has Certified many existing buildings. However, the process can be more challenging when working in facilities with live loads. For best results with an existing facility, the process should begin with a Tier Gap Analysis rather than a formal Certification effort. Tier Gap Analysis provides a high-level summary review for major Tier shortfalls. This allows the owner to make an informed decision whether to proceed with a detailed, exhaustive Certification effort. Tier Certification of Constructed Facility can be performed with any load profile, including resistive load banks, live critical IT load, or a mix.
Uptime Institute Tiers is U.S.-centric.
False. Uptime Institute is currently delivering Tier Certifications in more than 85 countries. Tiers, which allows for many solutions and a variety of configurations, gives the design, engineering, and operations teams the flexibility to meet both local regulations and performance requirements. To date, there has not been a conflict between Tiers and local building codes, statutes, or jurisdictions.
TIA-942 is a guideline for Uptime Institute Tiers.
False. In 2014 Uptime Institute and the Telecommunications Industry Association (TIA) agreed on a clear separation between their respective benchmarking systems to avoid industry confusion and drive accountability. In fact, any reference to the TIA rating of a data center may not include the word Tier.
The core objective of Uptime Institute Tiers is to define performance capabilities that will deliver availability required by the data center owner. By contrast, TIA member company experts focus on the need to support the deployment of advanced communications networks. See https://uptimeinstitute.com/uptime-tia for a more detailed explanation.
Utility feeds determine Tier level.
False. According to Tier Standard: Topology, the only reliable source of power for a data center is the engine- generator plant. This is because utility power is subject to unscheduled interruption—even in places with reliable power grids. As a result, the number of utility feeds, substations, and power grids that provide public power to a data center neither predicts nor influences Tier level. As a consequence, utility power is not even required for Tiers. Most Tier Certified data centers use utility power for main operations as an economic alternative, but this decision does not affect the owner’s target Tier objective.
For Tier III and IV, the engine-generator plant must be operational at all times.
False. Tiers does not require that the engine-generator plant actually run at all times; however, data centers will typically utilize a public utility a majority of the time for cost or regulatory reasons. At the same time, the engine-generator plant must be properly configured, rated, and sized to have the capability to carry the critical load without runtime limitations. Hence, the performance requirements outlined in the Tier Standard must be met with the data center supported by engine-generator power. Meeting these criteria requires special attention to engine-generator capacity ratings and power distribution.
Uptime Institute’s Tier system is based on the U.S. Environmental Protection Agency (EPA) regulations on diesel-engine operations.
False. There is no correlation between EPA’s Tiers (or other restrictions of engine-generator operation) and Uptime Institute Tiers, except that both systems use a similar hierarchical system of nomenclature. The EPA’s limits on runtime may complicate a facility’s testing and maintenance regimens and add costs when a facility is forced to rely on backup power for an extended period. However, runtime limitations posed by local authorities do not exempt a data center from having on-site power generation rated to operate without runtime limitations at a constant load.
EPOs (emergency power off) or other systems that shut down the critical load impact Tier objectives.
False. When code or the local authority having jurisdiction (AHJ) mandate an EPO, this does not prohibit Tier compliance. At the same time, Uptime Institute does not recommend EPO installation, unless it is compelled by a local code because even Tier Certified data centers are vulnerable to outages from purposeful or accidental activation of the EPO system. Analysis of the Uptime Institute Network’s Abnormal Incident Report (AIRs) database confirms that accidental EPO activation is a recurring cause of downtime.
The Tiers Standard requires that maintenance, isolation, and/or removal can be performed on the EPO system without affecting the critical load for Tier III data centers. Tier IV data centers additionally require a Fault Tolerant EPO system.
Uptime Institute Tiers requires raised floor.
False. The choice of under floor or overhead cooling is a decision to be made by the owner based an operational preference. In Uptime Institute’s experience, a raised floor enhances operational flexibility over the long term. Yet, decisions such as raised floor or on-slab, Cold Aisle/Hot Aisle, containment of Cold Aisle/Hot Aisle, and gallery cooling can affect the efficiency of the computer room environment, but are not mandated by Uptime Institute Tiers.
Rack-based automatic transfer switches (ATS) meet the requirement for dual paths to a server (i.e., the server has one cord to the ATS, but the ATS [rack mount] has dual power feeds).
True. The Tier Standard includes a concession for equipment with odd numbers of cords (1,3,5) in the form of rack-mounted transfer switches to provide access to multiple power paths. However, Tier III and Tier IV data centers must still have multiple and independent feeds to the rack.
The Tier Standard focuses on ensuring that the facility’s infrastructure meets the requirements of the Tier objective. There are many reasons why a facility may contain single-corded IT devices or those with an odd number of power supplies, including lack of knowledge of the facility impacts, lack of options for equipment vendors, and colocation environments where facility personnel have no control over the types of IT devices within the data center. Rack-based transfer switches are most typically supplied by the IT side of the organization, so the facility’s infrastructure can meet the Tier objective. However, planned isolation or fault of these rack-based transfer switches may lead to an outage for individual racks or devices.
Tier II provides Concurrent Maintenance opportunities.
Partially true. Tier II allows for Concurrent Maintenance of capacity components, but not distribution pathways or critical elements. So a Tier II Certified facility can perform Concurrent Maintenance on engine generators, UPS, chillers, cooling towers, pumps, air conditioners, fuel tanks, water tanks, and fuel pumps
but not switchboards, panels, transfer switches, transformers, bus bars, cables, and pipes. In many cases, this limitation will require the computer room to be shutdown for planned maintenance or replacement of critical pathways and elements.
The requirement to maintain any component, pathway, or element without shutting down equipment, known as Concurrent Maintainability, defines Tier III. Many owners’ business cases, including healthcare, domestic outsourcers, and state governments, require Tier III. The list of organizations that have protected their investment with Tier Certifications may be found on Uptime Institute’s website.
A Tier III facility is Tier compliant if one of the redundant branches is inactive.
Partially true. Tier III requires active/active distribution for critical power distribution (which is defined as the output of the UPS and below). Outside of that, active/inactive is acceptable. This means that if a rack receives dual power from two separate power distributions, they must both normally be active. It is not allowable to have one feed normally disabled, nor is it Tier III compliant to have one of the power feeds directly fed from utility power while bypassing a UPS power source.
There are no active/active requirements for mechanical systems in Tier III data centers. So if there are N+1 chillers in a Tier III facility, with each chiller feeding separate A and B chilled water loops, it is permissible for one of the loops to be normally disabled, with all air conditioners normally fed from the same loop.
Facilities can’t be changed after Tier Certification of Constructed Facility.
False. Infrastructure changes must be approached using carefully developed and written procedures and processes. If the topology of a facility changes, it may no longer be Concurrently Maintainable or Fault Tolerant, so clients should have Uptime Institute review designs and construction that might affect a facility’s topology to protect their investment and Tier Certification. Tier Certifications can be revoked if unreviewed changes compromise a facility’s Concurrent Maintainability or Fault Tolerance.
A Tier IV facility must have all its cooling units operating.
Mostly false. The Tier Standard requires only that Tier IV facilities provide stable cooling to the IT and UPS environment for the time it takes for the mechanical systems to completely restart after a utility power outage and provide rated load to the data center. Tier IV data centers must also be able to maintain a stable thermal environment for the duration of the mechanical restart time and for any 15-minute period in accordance with the 2015 ASHRAE Thermal Guidelines. Tier IV facilities are also required to be active/active for all systems. This is intended to ensure that Continuous Cooling solutions are not negated by a lack of active operation of components. A lightly loaded data center or one with a very complex control system may be able to meet these requirements, without using all the available cooling units. However, there are Tier IV data center designs, especially those at full load, that would in fact require all units to run during normal operations.
Makeup air capacity counts as critical cooling capacity.
Typically false. Makeup air systems in data center applications are typically designed to meet one of three objectives (or a combination of the three):
• Provide the fresh air for occupants required by an AHJ
• Ensure positive pressure in the data hall, which will help keep contaminants out of the data center
• Aid in meeting the humidity requirements of the data center
Data centers are rarely designed in a manner that require the makeup air handler to be active in order to meet the N cooling capacity requirement. However, the existence of a makeup air handler and its operations cannot negatively impact compliance with Tiers. For example, if a makeup air handler is not sized to ASHRAE extremes in compliance with Tiers, the additional heat load from this air handler at those conditions must be considered when sizing the critical cooling system.
It is not possible to utilize diesel rotary uninterruptible power systems (DRUPS) as continuous cooling in a Tier IV facility.
False. The Tier Standard is vendor and technology neutral, which means it is possible to Tier Certify facilities that include a wide variety of innovative and new technologies, including DRUPS.
Facilities tend to deploy DRUPS, which combine a diesel engine and a rotary UPS that uses kinetic energy to eliminate batteries, which require high levels of maintenance, somewhat frequent replacement, and a lot of extra space for battery placement/storage. This design usually provides ride-through times of between 10-30 seconds, depending on the application, which is shorter than other technologies. The Tier Standard does not include a minimum ride-through time. In fact, Uptime Institute has Certified several facilities that include DRUPS technology.
DRUPS may also be used to power motor loads. That means that caution must be exercised to ensure that the DRUPS have sufficient capacity to power each and every system and subsystem, including cooling systems, which is accomplished by putting the mechanical components on a no-break bus.
Ductwork does not need to comply with the Tier requirements.
False. Tier Certification analyzes each and every system and subsystem down to the level of valve positions and panel feeds. Ductwork, just like piping systems, may need planned maintenance, replacement, or reconfiguration. As such, traditional ductwork distribution systems must meet the requirements of the Tier objective.
Uptime Institute understands that there is a lot of confusion about what “maintaining” ductwork means to meet Concurrently Maintainable requirements. But in this case, Concurrent Maintainability is about having the capability to isolate a system or part of a system to maintain, repair, upgrade, or reconfigure the data center without impacting any of the computer equipment.
Site Location affects Tier level.
False. Although a critical consideration for the life-cycle operation of the facility and in determining, evaluating, and mitigating risk to the data center, geographical location does not affect a facility’s Tier level and is not part of the Tier Standard: Topology.
Data center designers can take precautions to address the specific risks of a site. A data center sited in a high-risk earthquake zone can include equipment that has been seismically rated and certified as well as incorporate techniques that mitigate damage from seismic activity. Or if a data center has been sited in a high-risk tornado area, designers can consider wind protection measures for the exterior electrical and heat rejection equipment.
Site Location is a criterion in the Tier Certification of Operational Sustainability.
Chris Brown, Enrique Hernandez, Kevin Heslin, Julian Kudritzki, Eric Maddison, Ryan Orr, Sarah Thomas, Pitt Turner, and Rich Van Loo all contributed to this article.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/07/tiers.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-07-20 12:33:222017-07-20 12:33:22Myths and Misconceptions Regarding the Uptime Institute’s Tier Certification System
Uptime Institute recently published “Top Considerations for Addressing Data Center Facilities Management Risks,” a guide for reducing data center risks in enterprise IT organizations . The guide comprises 14 top considerations useful for designing and running an enterprise-grade data center facilities management program. The full guide is available for download on the Uptime Institute website.
Areas of discussion include:
Monitoring and Managing Staff Overtime
Managing a Critical Spares Inventory
Maintaining a Reliable Diesel Fuel Supply
Developing Emergency Operating Procedures (EOPs)
Regular Execution of Site Drills
Executing Against a Procedure-Based Control Methodology
Performing Regular Integrated and Key Systems Testing & Validation
Building & Integrating Robust Change Management Protocols
https://journal.uptimeinstitute.com/wp-content/uploads/2017/06/Uptime_Institute_Top_Considerations_for_Addressing_Facility_.gif241600Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-06-13 08:22:012017-06-13 08:22:01Top Considerations for Addressing Data Center Facilities Management Risks
Nearly perfect M&O Stamp of Approval scores prove that NRI’s data centers operations are world class
By Kevin Heslin
Nomura Research Institute, Ltd. (NRI), a ¥18 billion ($US169 million) company that provides consulting, financial IT solutions, industrial IT solutions, and IT platform services mainly to a global financial marketplace, has earned Uptime Institute Management and Operations (M&O) Stamps of Approval at two of its five facilities, receiving very high scores on both. In the process, NRI, Japan’s largest provider of critical IT services, impressed even Uptime Institute consultants, who have assessed many of the world’s best-run data centers.
Uptime Institute Consultant Steve Burgess said, “A very high M&O score translates to a quality of operations that is very proactively mitigating risk, something premium financial services companies must aggressively pursue and maintain.” Managing risk in that way is critical for NRI because of its important role in the global economy. Uptime Institute spoke to Takuhito Sakata, NRI’s managing director, Data Center Service Division.
NRI’s Tokyo 1 data center
Tell me about NRI’s operations.
TS: More than 300 financial and industry systems operate in our data centers. NRI provides the back office system that cover general securities business processes and 50% of the Tokyo Stock Exchange’s transactions are transmitted through our STAR financial back office system.
Other than financial systems, NRI operates a wide range of mission critical systems for banks, insurance companies, and many companies in other industries. Our systems have become Japan’s core infrastructure, so the cost of losing them in a disaster would be immeasurable. Without exception, our data centers are built for high capacity and availability, and NRI owns and operates the safest data centers in Japan.
Layout of NRI’s Tokyo 1 Data Center .
Please describe NRI’s data centers.
TS: NRI owns five data centers in Tokyo and Osaka. Yokohama Data Center 1(Y1DC) is the oldest of these facilities (1990), and Osaka Data Center 2 (O2DC) is the newest (2016). Tokyo Data Center 1 (T1DC) has the greatest power density (maximum 30 kW/rack). These and other differences in size, density, age, and location mean that the facilities have significant differences in operation that must be managed and supported.
The Osaka 2 data center opened just last year.
These five facilities support all our financial systems, and they align to the Payment Card Industry Data Security Standard (PCI DSS) and Sarbanes Oxley (SOX). These facilities are interconnected to provide mutual backup to each other. In fact, even the operation centers backup mutually. If one or more of the data centers are damaged, the systems are run in a remote data center and the operations also transfer to the backup. We call this concept Multi Data Center Operation (MDC).
A look at NRI’s double-deck system.
We believe that T1DC is state of the art. This facility earned an Uptime Institute Brill Award in 2015 for its patented double deck system. The double deck system separates server areas from facility areas, which helps us provide efficient power and cooling. Not only that, but the double deck system allows us to enhance our physical security and increases our structural integrity, which is important because of the risk of earthquakes in Japan.
These facilities benefit from different power utility supplies, and have redundant electric power system, including engine-generators sets, uninterruptible power systems, and standby transfer switches. The system is configured to backup the facility cooling. These and other arrangements, including a priority fuel supply contract, have enabled us to survive the Great East Japan Earthquake and the planned blackouts that followed without interruption.
Because of the ongoing risk of earthquakes, all the facilities are isolated seismically. The building is built on a huge seismic isolation rubber base and does not touch the ground directly. The building also includes a seismic damper, which is a brace that is structured to withstand shaking, so there will be no impact to IT equipment.
NRI deploys an isolation system and a bracing system to protect against potential earthquake damage.
What motivated NRI to obtain the M&O Stamp?
TS: We have been providing data center services for over 30 years, mainly to financial customers, and our service quality is now the best in Japan. We work hard to meet our customers’ rapidly changing business needs, and changing regulations and qualifications. We think we can improve our quality even further by third-party verification of our efforts.
We spent one year researching all the data center standards and certifications globally, which led us to Uptime Institute M&O Stamp of Approval. At that first (2014), no Japanese company had received the Stamp of Approval. So, we visited a number of data center sites in the U.S. that had the Stamp of Approval. We asked the owners and staffs at these sites whether the Stamp of Approval was effective and how it improved their quality of facility management. When we were convinced, we decided to submit for an M&O assessment.
How does the M&O Stamp benefit NRI?
TS: It sure helps increase the quality of the data center operation. The assessment program provides insights and advice that is shared by our five data centers, and helps us improve our operations knowledge at all of them.
NRI requires the participation of two professionals as part of many of its procedures.
When we re-assessed T1DC, we received 98.5 points (out of 100), an improvement of 1 point from the initial assessment. We are aiming for a perfect score, and the advice we receive from Uptime Institute consultants is very useful. And each time we re-assess we must show improvement as the program ratchets up the requirements.
Adding to that, M&O is also useful for creating awareness about the high quality of our data center operations. Especially in Japan, NRI received a lot of attention from the market and the media for being first to have Stamp of Approval and that we had a high score. And this awareness makes a big difference to customers touring our data centers. It is a matter of competitiveness. Some companies demand a high standard of operational readiness.
M&O is also helping us on global projects. Our data center design support service was the main project area so far, but we are being called on for our data center management consulting.
How did you prepare for the assessment process?
TS: To be honest, we didn’t prepare. We are confident of our daily regular maintenance, training, preventative checks, and routines. We wanted to evaluate our operations as they were, so we did not consciously prepare for the assessment. Instead, we viewed the results of each day’s work as a form of validation when we did well or opportunity to improve.
Of course, we knew the M&O criteria in advance, so we self-checked based on that, and we decided to go through the assessment process. For example, the M&O’s job definition form was unfamiliar to us, so we reviewed our standard to comply with global requirements.
Who was involved in the process? How did the staff benefit?
TS: We have a lot of in-house expertise, including facility management, security, and quality teams etc.). Based on these opinions, I made the decision to go ahead with the M&O. NRI created a cross-functional task force from this group to get the Stamp of Approval. As a result of M&O, our staff has become very confident about day-to-day operations. We learned about the importance of continuous improvement.
Do you find something new to apply after each assessment?
TS: That’s right. Currently we are having two data centers evaluated. The two data centers were built at different times, serve different customers, and are operated by different people. So we think it is natural that we get different results at each facilities. Those findings and advice for improvement are very helpful for us.
What are your plans for renewing the M&Os? As you know they expire every 2 years.
TS: We plan to have every data center recertified. Because conditions may change over time, we think that it is important to check on the data center every 2 years. Facilities and equipment age and staff changes so procedures may become dated or are ignored.
T1DC underwent a renewal review [Editor’s note: mandatory after 2 years] at the end of last year, and it is doing well.
What are your plans for further data center expansion? Will the M&O help with that process in any way?
TS: Global is a keyword for our company strategy, and the M&O Stamp of Approval combined with our proven track record will be part of our global business development. For this reason we plan to have the M&O Stamp of Approval at all our next-generation core data centers.
Takuhito Sakata is managing director, Data Center Service Division, NRI.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/04/NRIweb2.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-04-04 12:29:022017-06-13 07:41:12Japan’s Biggest Provider of Critical IT Services Earns M&O Stamp of Approval
This week, Uptime Institute hosted its first major event in China, Uptime Institute Symposium: Shanghai. Over 400 invite-only attendees joined Uptime Institute for a day of collaborative learning at the Grand Hyatt. The photos below illustrate some highlights from the the day.
Uptime Institute President Lee Kirby delivers the welcome keynote at Symposium: Shanghai. “The digital economy is interconnected, and you don’t want your data center to be the weak link. We need to industrialize our digital infrastructure through standards and third party assessments.”
Uptime Institute has tripled its business in China over the 5 years, with 18 Tier-certified data centers and 13 M&O data centers in China, with over 30 projects in process.
Uptime Institute Managing Director North Asia, Philip Hu discusses the process of Tier Certification. “The vast majority of even the world’s most elite data center sites do not operate as designed on day one. Tier Certification is a failsafe against a data center that doesn’t work.”
Uptime Institute Senior Director of Content and Publications Matt Stansberry unveils the results of the 2017 Data Center Industry Survey including responses from over 1500 end users worldwide.
Uptime Institute has conducted its annual survey globally for seven years. This year’s survey data included the first breakout of respondents specifically in China.
Attendees included Uptime Institute Tier Certification owners and M&O Stamp of Approval Awardees from China’s leading financial, colocation and telecom organizations.
Jack You, Cloud Architect for Huawei delivers the Diamond level Sponsor Keynote.
Uptime Institute President Lee Kirby presents Wang Hui, Director, China Carrier Service, Solutions Sales Department at Huawei with a letter of appreciation for Huawei’s commitment to Uptime Institute Symposium around the globe.
Philip Hu moderates a discussion of new technology adoption. Panelists: John Zheng, Solution Architect Director, Schneider Electric; Wu Ganxing Vice General Manager, China Building Tech Group, Data Center; Biao Leng, General Manager, Beijing Travelsky Birun Technology
Pitt Turner, Uptime Institute Network Global Executive discusses the challenges with DCIM procurement: The process takes too long; Software costs too much; Tools under-deliver; Many buyers are not achieving ROI on the purchase.
Agatha Poon, Research Director APAC at 451 Research discusses the rapid growth and adoption of cloud computing providers in China and around the world, providing cloud computing adoption statistics, key trends in the market and the competitive landscape in the region and globally.
Transforming enterprises are serious about incorporating cloud models into their business strategies. Workloads are already business-critical in the cloud.
Lee Kirby: Complex IT systems are prone to cascading failure. But calling these failures human error is a mistake. The following are management failures, not human error: Lack of staffing; Lack of training; Lack of resources; Lack of preventive maintenance.
Over 65% of enterprise IT departments use Uptime Institute’s Tier Certification and M&O Stamp of Approval as part of the vetting process for considering potential IT service providers, according to the 2016 Data Center Industry Survey.
451 Research Director Andy Lawrence: The data center in 2020 will need to be agile — reducing costs, building on demand, responding to rapid deployment needs, and innovating without service disruption.
At the end of the each Symposium Event, an expert panel asks the audience — what are you going to do differently on Monday? How will you implement what you have learned today? Speaking: Danny Cheng, Chairman & CEO, Banyano
Stay tuned to the Symposium website for updates on our events in North America (Las Vegas, September 18) and Europe (Barcelona, Dec 5-6) in 2017.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/03/ThinkstockPhotos-515811014.jpg11612581Matt Stansberryhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngMatt Stansberry2017-03-22 22:00:102017-03-23 01:56:57Highlights from Uptime Institute Symposium: Shanghai
New diesel formulations put your generators at risk
By Bernard Oegema, Pat Smyth, Brian Ponstein, William “Bill” Klein, and Martin Wesolowski
The increased use of biodiesel has increased the risk to the diesel infrastructure of data centers and other facilities first noted with the adoption of ultralow sulfur diesel (ULSD) in 2006. The U.S. Environmental Protection Agency (EPA) documented severe and rapid corrosion in the storage tank portion of the diesel infrastructure caused by changes to the chemical composition of diesel fuel mandated by the U.S., U.K., and European Community. At the same time, engine-generator OEMs noticed an increase in warranty claims on their systems, including injectors. Solving this problem will require a new generation of filters tested against the updated industry standard, Society of Automotive Engineers SAE J1488_201010, to remove emulsified water.
The following steps are recommended to minimize or eliminate potential impacts from the use of biodiesel (see the sidebar for more detail):
Check with state regulations to confirm whether biodiesel is mandated.
Check facility records to determine when the fuel was last replaced. Unfiltered older fuel will have more issues.
Create a maintenance routine for diesel fuel that will include twice-annual fuel samples and tank inspections.
Take fuel samples from main storage tanks, belly tanks, and day tanks
If the fuel fails the tests, install a fuel polishing system based on SAE J1488_201010
If the fuel is found to be acidic, drain and clean the tank, and replace the fuel.
Test a sample of the replacement fuel, and all future deliveries from the supplier.
It is important that all U.S. operators check their fuel supplies, because suppliers are not required to disclose biodiesel content of 5% or less. So a distributor may say there is no biodiesel content, when in fact there is. In addition, while a state may not have mandated the use of biodiesel, distributors have financial incentives to supply biodiesel blends.
Testing Procedures
Test for biodiesel content, water content with Karl Fisher titration (less than 200 ppm), ISO 4406 particulate content (less than 18,16,13), and TAN (Total Acid Number, less than 0.08 maximum per World Fuel Charter). Fuel samples should be taken from the bottom of the tank. Karl Fischer titration is a method to determine the trace amounts of water in a fuel sample. It can measure the amount of entrained water in fuel, as well as emulsified and free water. Other tests cannot sense entrained water.
Sample the belly tanks or day tanks attached to the generators. If possible, visually inspect the insides of these tanks. Corrosion in these tanks points to corrosion in the large storage tanks. An inexpensive fiber optic camera can assist in the inspection. Storage tanks are designed to handle the hydrostatic pressure (weight of the fuel), which is greatest on the tank bottom. Since corrosion is also concentrated on the bottom of the tank, corrosion will pit the tank wall and floor, thereby thinning the wall, increasing the likelihood of fuel leaks or tank failure.
If corrosion is found, replace the fuel. Before adding new fuel, ask for a fuel sample from the supplier’s distribution tank before delivery, and test for biodiesel content, water content with Karl Fisher titration, particulate content against ISO 4406, and TAN (Total Acid Number). If the sample fails any test, inform your supplier and request fuel within the acceptable specifications.
The EPA states, “Minimizing water presence is and has always been an important part of UST maintenance. However, diesel blended with biodiesel can hold in solution more water than diesel without a biodiesel component. This means more water is likely arriving in USTs entrained (emulsified) in fuel today since biodiesel is more common in diesel than prior to 2007.” Fuel in the large distribution storage tanks of suppliers also absorbs entrained (emulsified) water, so even new fuel could already have substantial amounts of entrained (emulsified) water.
Install a permanent fuel polishing system on the storage tank if fuel fails any of the tests. The polishing system must use the new generation of filters specifically designed to remove entrained (emulsified) water to SAE J1488_201010 standards with of a minimum 92% efficiency.
Fuel should be filtered three times a week, and all the fuel filtered within 8 hours. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE, such as Southwest Research Institute.
Corrosion resulting from acidity caused by emulsified water dramatically increases the risk of an emergency generator failure and increases the costs of maintaining the data center diesel infrastructure. Corrosion affects storage tanks, emergency backup generators, diesel engines, fittings, and piping. Removing emulsified water from diesel fuel will control acidification and corrosion; drying the fuel controls microbial growth.
Fuel experts and data center operators first noticed corrosion problems shortly after the widespread adoption of ULSD. ULSD contains various additives (less than 1% of the fuel by volume) and fuel components (more than 1%), including biodiesel. These additives and fuel components absorb water which bonds to the diesel as emulsified water. The molecules of water, biodiesel, and other additives are all polar in nature. The emulsified water supports a bloom of microbes, and some of these microbes produce acids that corrode metal in their environment.
Before 2006, LSD (low sulfur diesel) fuel contained 500 parts per million (ppm) sulfur. ULSD (15 ppm sulfur) was introduced to help fight acid rain. While LSD was phased out of the market in 2010, a 2007 research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B.B. He, 2007) makes a useful point about how much water LSD and ULSD can absorb. He notes that 100% biodiesel absorbs 15 to 25 times more emulsified water than LSD. Neither LSD nor ULSD fuels are polar so they do not absorb much emulsified water.
In addition to lubrication concerns, biodiesel was added to ULSD in the U.S. for a variety of environmental and financial reasons, as well as government concerns about national dependence on foreign oil supplies.
However, the process of scrubbing the sulfur changed the chemistry of the fuel so ULSD provided less lubrication, resulting in immediate wear to diesel engines. An additive was introduced to ULSD to restore the needed lubrication. Soon after, biodiesel was also added as a fuel component, in ever increasing percentages, 2% (B2), 5%(B5), 7% (B7), 10% (B10), and now, 20% (B20). The more biodiesel, the more emulsified water. Biodiesel blend of 5% or less does not need to be disclosed to the end user.
Before ULSD, filters could remove all freestanding water and particulate, since LSD is not polar and there was very little emulsified water. Since both biodiesel and water are polar, water bonds tightly to the fuel as emulsified water. Pre-ULSD filters cannot efficiently remove the emulsified water from biodiesel blends.
Even before the U.S. Environmental Protection Agency (EPA) detailed this rapid and severe corrosion, NFPA 110 (2013) cautioned against the use of biodiesel for standby and prime engine-generator sets. In those instances, where the use of biodiesel was unavoidable, the NFPA required its replacement annually, but with “rapid and severe corrosion,” even 12 months is too long a time to store acidified fuel. NFPA recommends using diesel that has been stored that long for other purposes.
The EPA’s July 20th 2016 final report on corrosion states, “The major finding from our research is that moderate or severe corrosion on metal components in UST [underground storage tanks] systems storing diesel fuel in the United States could be a very common occurrence… variables that were the closest to being significant predictors [of corrosion] were particulates in the fuel and entrained (emulsified) water content in the fuel.” Drying the diesel by removing all the water, both free standing and emulsified, can control corrosion. This final report has worldwide significance because countries concerned about the issue have been tracking EPA findings.
A polar molecule
SAE J1488 tests “the ability of a fuel/water separator to separate emulsified or finely dispersed water from fuels.“ Well-known organizations such as Southwest Research Institute (SwRI) can test filters for compliance to the standard and provide reports to data center managers. In 2010, SAE updated the test from J1488_199708 to J1488_201010 to address the emulsified water concerns caused by ULSD and biodiesel blends.
SAE updated J1488 in 2010 to reflect the lower interfacial tension (IFT) in ULSD and biodiesel blends. IFT is the ability of two liquids to repel each other. Since biodiesel in hydroscopic, more water is absorbed and less repelled, so its IFT is lower. The IFT of ULSD is tested at 34.5 mN/m, biodiesel at 15 mN/m (B20). The 2010 update created a standard IFT against which all filters could be compared.
A new generation of filters designed specifically to filter for SAE J1488_201010 IFT levels is required to remove emulsified water, and hence control corrosion. In June 2016, the United Kingdom adopted SAE J1488_201010 as a code requirement.
Critical Fuel Applications
Modern backup generators are high-pressure common rail (HPCR) models, with pressures of 30,000+ pounds per square inch (psi) and plans for 60,000 psi models. Emulsified water and corrosion can cause these engine generator sets to become unreliable by
Excessive emulsified water causes injector pitting.
Acidic diesel fuel causes injector pitting and corrodes other generator parts it touches. Monthly engine generator testing adds to the problem as acidic fuel remains on metal components after each test.
Excessive particulates cause injector pitting.
Deposits are left on injectors, making them sticky.
Fuel goes out of ASTM specification and the generator cannot perform at peak design performance
Biomasses that form in biodiesel clog fuel filters
Biodiesel contains less energy than ULSD, so increasing biodiesel content reduces power output a small amount.
Biodiesel freezes at a higher point than ULSD, so this needs to be addressed.
When injector pitting and corrosion are great enough, the generator will not be able to hold the required HPCR pressures and will fail. In addition, the pitting of the injectors can cause the generator to become over fueled, increasing heat, emissions, and fuel consumption. The corrosive/pitting affects injectors and spoolers on all diesel engines. The higher the psi of the engine, the greater the risk of damage and failure.
Corrosion in storage tanks, fittings, and piping dramatically increases the risk of a backup generator failing. For example, a fuel supply pipe that fails or plugs will prevent a generator from working during an emergency.
Warranty Claims
Warrantees on diesel engine-generator sets exclude damages caused by fuel-related issues. That means that vendors can deny warranty claims if the fuel does not meet the specification set forth by the vendor. The maximum emulsified water content in most warrantees is 200 parts per million (ppm), which is well below the observed amounts commonly found in tanks with biodiesel blends. Caterpillar’s warranty specifies a pre-emulsified water/corrosion water and particulate maximum of 500 ppm. Manufacturers generally enforce this requirement by requiring a fuel sample with each warranty claim.
Caterpillar’s service document SEBU6251-14 from 2014 (before corrosion was commonly recognized) reads, “The repair of any engine or after treatment components due to such contamination or due to other biodiesel impacts would not be covered under the Cat warranty for materials and/or the warranty for workmanship.” Since corrosion is such a new issue, it means Cat’s warranty could be voided the first time biodiesel is used in the engine.
Similarly, Cummins warns about biodiesel. Cummins’s service document 3379001-13 reads, “Engine damage, service issues, and/or performance issues determined by Cummins Inc. to be caused by the use of biodiesel fuel not meeting the specifications outlined in this Service Bulletin are not considered to be defects in material or workmanship and are not covered under Cummins Inc. engine warranty.”
All the other OEM warrantees include similar statements.
Sidebar: Corrosion Management Action
Removing emulsified water from the fuel is key to maintaining the integrity of diesel fuel systems. SAE J1488_201010 tested filters can remove the emulsified water, thereby restricting microbial growth, acidification, and dramatically reducing corrosion. Filters need to achieve a performance level of greater than 92% to keep water below 200 parts per million (ppm).
Below is a short checklist to help address any fuel issues, with detailed explanations following.
Check with state regulations to confirm if biodiesel is mandated.
Check your records as to when the fuel was last replaced. Unfiltered older fuel will have more issues.
Create a maintenance routine for diesel fuel that will include bi annual fuel samples and tank inspections
Take fuel samples from main storage tanks, belly tanks, and day tanks
If the fuel fails the tests, drain and clean the tank, and install a fuel polishing system based on SAE J1488_201010
If the fuel is found to be acidic, drain and clean the tank and replace the fuel
Test a sample of the replacement fuel from the supplier
While the state may not mandate biodiesel, there are financial incentives to distributors to supply biodiesel blends. Note that suppliers are not required to disclose any biodiesel content of 5% or less, so the distributor may say there is no biodiesel content, when in fact there is.
Take fuel samples from all fuel tanks. The sample should be taken from the tank bottom. Test for biodiesel content, water content with Karl Fisher titration (less than 200 ppm), ISO 4406 particulate content (less than 18,16,13), and TAN (Total Acid Number, less than .08 maximum per World Fuel Charter).
It may be quicker and easier to sample the belly tanks or day tanks attached to the generators. If possible, have a visual inspection of the inside of your belly and day tanks. If there is any corrosion in these tanks, it will point to corrosion in the large storage tanks. All storage tanks are designed to handle the hydrostatic pressure (the weight of the fuel), which is greatest on the tank bottom. Since corrosion is also focused on the bottom of the tank, corrosion will pit of the tank wall, thereby thinning the wall, increasing the likely-hood of fuel leaks or tank failure. There are inexpensive fiberoptic cameras on long hoses available on the market to assist in the inspection.
If corrosion is seen, replace the fuel. Before adding new fuel, ask for a fuel sample from the suppliers distribution tank before delivery, and test for biodiesel content, water content with Karl Fisher titration, particulate content against ISO 4406, and TAN (Total Acid Number). If the sample fails any test, inform the supplier and request fuel within the acceptable specifications.
The EPA report states, “Minimizing water presence is and has always been an important part of UST maintenance. However, diesel blended with biodiesel can hold in solution more water than diesel without a biodiesel component. This means more water is likely arriving in USTs entrained in fuel today since biodiesel is more common in diesel than prior to 2007.” The large distribution storage tanks are also absorbing entrained (emulsified) water, so new fuel could already have substantial amounts of entrained (emulsified) water.
Observed corrosion (Battelle)
Install a permanent fuel polishing system on the storage tank if fuel tests show any failures in the tests. The polishing system must use the new generation of filters specifically designed to remove emulsified water to SAE J1488_201010 standards with of a minimum 92% efficiency. The fuel should be filtered 3 times a week, and filter the complete volume of fuel in an 8-hour period. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE such as Southwest Research Institute.
Fuel Experts
While most diesel consumers did not immediately recognize the issues with corrosion, the fuel experts did. Below is a timeline of recognition of acidification caused by microbial growth, starting with the most recent and working back.
U.S. EPA recognizes entrained (emulsified) water as a key indicator of corrosion. “This research focused on better understanding a type of rapid and severe corrosion of metal components in underground storage tanks (USTs) storing diesel fuel….EPA’s research determined that corrosion is very common; it appears all USTs storing diesel could be susceptible to developing corrosion.”
BS5410-3 made it code that any filters used on critical emergency backup systems conform to SAE J1488_201010.
BS5410-3 (British Standards Institution, 2016) is the equivalent of NFPA 110. The United Kingdom has used biodiesel blends longer than in North America. “7.2.2.5 Filters: NOTE 1 Since the introduction of Biofuel (FAME) [fatty-acid methyl ester] into fuels the quality and life expectancy of fuels has been adversely affected as FAME is hygroscopic so any water in the fuel goes into suspension….Removing as much water as possible preserves the quality of the fuel and for critical standby generators only filters conforming to SAE J1488_201010 [N5] are to be used”
NFPA 110 cautions against the use of biodiesel. “A.5.1.1(1) Where possible, the purchaser of fuel for the prime mover should specify a diesel fuel that does not contain biodiesel, which can accelerate the degradation of the diesel fuel if stored longer than 6 months.”
“A.5.5.3 More important, biodiesel blends up to B5 (ASTM D 975, Standard Specification for Diesel Fuel Oils) have much shorter shelf lives than conventional diesel fuel [ultra-low sulfur diesel (ULSD)] and can accelerate degradation processes, endangering the entire diesel fuel supply. Where fuel is stored for extended periods of time (e.g., more than 12 months), it is recommended that fuels be periodically pumped out and used in other services and replaced with fresh fuel.”
The Steel Tank Institute (STI) reinforces the importance of keeping storage tanks free of water. “In a 2012 study by Battelle Labs, five of six FRP (fiber- glass reinforced plastic) tanks showed severely accelerated corrosion. The sixth tank was supposedly “clean” and intended to be the control for the study, but turned out to have corrosion, as well….In ULSD, the presence of water provides an environment where microbial life can thrive, possible influencing the corrosion of internal metal components of the UST system equipment. “
SAE updates J1488_199708 to J1488_201010 to test the efficiency of filters to remove emulsified water from the newly introduced ULSD and biodiesel blends (B20). A 92% efficiency can keep water below 200 ppm, which is the OEM manufacturers’ warranty limit. In SAE J1488_201010 testing, ULSD and biodiesel (B20) are saturated with 2500 ppm of emulsified water, which is then filtered. The water content of the filtered fuel is compared to the water content of the original saturated fuel. SAE then assigns an efficiency rating to the filter and reports its findings to the filter supplier. (SAE Technical Standards Board, October 2010).
Battelle Memorial Institute calls attention to the problem of corrosion. “Severe and rapid corrosion has been observed in systems storing and dispensing ultra low sulfur diesel (ULSD) since 2007” (Battelle Memorial Institute, 2012)
The research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B. B. He, 2007), compares the water absorption of 100% LSD to 100% biodiesel.
IBM Data Center Group Case Study
IBM has a group within the Global Technology Services division that designs, builds, migrates, and upgrades data centers for its customers. It reports that the majority of data centers in Canada have been in the Province of Ontario, which mandated the use of biodiesel with an increasing concentration, initially B2, and B4 at time of writing.
IBM was initially concerned about the reliable operation of the generators at these facilities, especially because data centers often store fuel is stored for long periods of time, unlike road or construction applications. In particular, IBM worried about the effect of the emulsified water on its engine-generator set infrastructure. Reduced engine-generator set life and increased repair costs were major concerns.
Over time, it became clear to IBM that the entire fuel storage and delivery system, metallic and non-metallic components included, is at risk due to the corrosive properties of degraded biodiesel. A major excavation required to repair or replace would be a major concern.
Fuel filters tested in systems using B2 pass the “bright and clear” test, which wasn’t deemed rigorous enough to give a true indication of the condition of biodiesel fuel. IBM has been offering their clients fuel-conditioning systems that meet or exceed a more applicable standard from the SAE J1488_201010.
Three Biodiesel Related Failures in One Day
Franklin Company in the NYC area responded to three separate fuel-related diesel generator failures in one day at three different organizations. Problems caused by ULSD can occur in many application types. Failure 3 is typical of failures at a very common type of facility. These failures give a good representation of some of the failure points data center users can experience: corrosion of tanks, failure of pipes and fittings, and biomass plugging filters.
Failure 1: Franklin Company responded to a service call with a reported interstitial alarm. A service tech removed the sensor and found a mixture of water and fuel on the sensor. The tech removed a small amount of fuel and water mixture and then reinstalled the sensor. The sensor did not go into alarm. Franklin initially thought that recent rains had worked its way past the interstitial cap.
A few weeks later, Franklin received a new service call with the same complaint. This time the tech found a significant amount of water and some fuel in the interstitial. Since the tank had not been showing any losses, Franklin thought that the outer wall was leaking groundwater and fuel into the annular space from outside. It would not be so unusual for a site to have contamination. The interstitial failed a vacuum test. The primary tank passed a second test was to confirm that it was tight. A spill was called in to the DEC for a non-tight tank system. Franklin Company contracted a third party to perform the same tests, with the same result.
As one result, Franklin excavated and checked the tank top for breaches or leaking bungs to the interstitial. All fuel was removed, the manway removed, and a confined space team entered and began cleaning a very dirty tank. The team found the steel tank had many holes on the bottom.
What happened with the tests? The annular was leaking, but only between the primary and the secondary. Tests on the primary showed “tight,” but we were really testing only the secondary tank, as the primary tank was open to the secondary. The larger amounts of water in the secondary was caused by the head pressure pushing the tank bottoms out of the bottom of the primary. The annular space on an ACT 100 is very small, so a small amount of water would migrate to the riser, showing a disproportionate amount of water versus fuel. When the tech removed water, it would look dry for an extended period of time until more water and fuel would be forced through the bottom of the tank and work its way to the annular riser.
The tank was not salvageable and had to be removed. There was no leakage to the environment as the secondary was intact and tight. Note the tank was 10 years old.
Failure 2: An underground UL 971 flexible pipe failed when B20 diesel caused the internal lining of the pipe to delaminate. The contractor said, “The pipe looked like a long sheet of white plastic wrap when we investigated the blockage.”
Failure 3: A government agency reported a slow flow of diesel fuel dispensers. A service technician dispatched to the site found that two dispensers had flow rates of only 2-3 gallons per minute. Two dispensers had no flow at all. The internal filters were removed and replaced, and the pumping unit strainers (suction style units) were removed and cleaned and then re-installed. The facility was down for two days. In this case, microbial “blooms” and excess particulates had created biofilms that plugged the filters.
Biofilm, as shown in the picture below, is a problem that can lead to the failures stated above. Biofilms concentrate the corrosive acidic effects of the fuel (Failure 1), can peel off in strips to plug pipes (Failure 2) and can disintegrate, overloading filters (Failure 3). The contact said, “WOW. Never saw anything like this. This was the consistency of boiled pig skin. It was lining the bottom of the tank. This is a tank that we could not rinse in situ. Super bugs.”
Picture Below, Biofilm from bottom of tank
Suggested Readings
Chuanfang Yang, S. L. (2007). Understanding emulsified water filtration from diesel fuels . San Antonio: 8th International Filtration Conference.
Groysman, A. (n.d.). Corrosion in Systems for Storage and Transportation of Petroleum Products and Biofuels Identification, Monitoring, and Solutions. New York, Philadelphia: Springer Science & Business Media.
PEI Petroleum Equipment Industry. (2013). Recommended Practices for Installation of Bulk Storage Plants (PEI/RP800-13). Petroleum Equipment Industry.
R.L. McCormick, T. A. (2005). Survey of the Quality and Stability of Biodiesel and Biodiesel Blends in the United States in 2004 . National Renewable Energy Laboratory, Golden, Colorado.
Steel Tank Institute. (2016). Recommended Practice For Storage Tank Maintenance R111 Revision . Steel Tank Institute , Lake Zurich, IL.
ASTSWMO Tanks Subcommittee’s Alternative Fuels Workgroup. (2013). Compatibility of UST Systems with Biofuels. Association of State and Territorial Solid Waste Management Officials (ASTSWMO), Washington, DC.
B. He, J. C. (2007). Moisture Absorption In Biodiesel and Its Petro-Diesel Blends. Food & Process Engineering Institute Division of ASABE. Moscow, Idaho: American Society of Agricultural and Biological Engineers ISSN 0883 8542.
Battelle Memorial Institute. (2012). Corrosion in Systems Storing and Dispensing Ultra Low Sulfur Diesel (ULSD), Hypotheses Investigation. Battelle Memorial Institute, Columbus, OH.
British Standards Institution. (2016). Code of practice for oil firing Part 3: Installations for furnaces, kilns, ovens, oil-fuelled standby generators and other industrial purposes. BSI Standards Limited.
NFPA110. (2013). NFPA 110 Standard for Emergency and Standby Power Systems. Quincy, MA.
Steel Tank Institute STI. (2013). Newer fuels and storage tank corrosion. Steel Tank Institute STI, Lake Zurich IL.
U.S. Environmental Protection Agency (July 2016). Investigation Of Corrosion-Influencing Factors In Underground Storage Tanks With Diesel Service. United States Environmental Protection Agency, U.S. Environmental Protection Agency Office of Underground Storage Tanks, Washington DC.
Uptime Institute. (2010). Uptime Institute Technical Paper: Biodiesel. New York.
Bernard Oegema
Bernard Oegema PE Data Centre Consultant, IBM Global Services, Data Centre Services. He has been in the critical facilities business for more than 30 years, working with various manufacturers of critical power and cooling infrastructure. Currently he is supporting IBM clients in the migration, consolidation, operation, upgrade, design and building of data centers.
Pat Smyth
Pat Smyth is Vice President of DieselPure. DieselPure is focused on fuel polishing for emergency backup generator storage. Since 2011, DieselPure has field experience in removing emulsified water from ULSD and biodiesel as per SAE J1488_201010.
Brian Ponstein
Brian Ponstein is Regional Sales Engineer, MTU Onsite Energy, a Rolls-Royce Company. At MTU, Mr. Ponstein is responsible for analyzing market needs and requirements in North America and working with engineering to provide solutions for MTU Onsite Energy’s customers.
Bill Klein
William (Bill) Klein is President of Franklin Company Contractors. He is also a New York City (NYC) Master Electrician, NYC Licensed Plumber, NYC Licensed Pump and Tank Contractor, NYC Licensed Precision Tank Tester, member of PEI, and past president of NYC Tank Installers. Mr. Klein is the third generation president of Franklin Company.
Martin Weselowski
Martin Wesolowski is Senior Associate Vice President at LiRo’s petroleum and chemical storage tank design group. He has been involved with design of petroleum and chemical UST and AST tank and remediation systems for over 23 years. Mr. Wesolowski is a technical director on LiRo’s New York City Department of Buildings (NYCDOB) Special Inspection Agency Registration.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/02/smyth.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-03-14 12:29:012017-03-16 11:29:14Reconsider Your Diesel Fuel Supply
David Meredith says that CenturyLink’s involvement, including the M&O Stamp of Approval, with Uptime Institute provides mutual benefits. Uptime Institute’s Risk Journal for IT Infrastructure is the most recent example.
CenturyLink’s commitment to operational excellence has become one of the company’s competitive advantages, and David Meredith, a senior VP at CenturyLink, has been an enthusiastic proponent of demonstrating this commitment through third-party verifications. In 2014, CenturyLink decided to pursue Uptime Institute’s Management and Operations (M&O) Stamp of Approval for all 57 of its data centers.
So Uptime Institute was delighted to see Meredith’s comments in LinkedIn about our new publication “Risk Management for IT Infrastructure.” Meredith quoted the book’s central tenet, “The handbook also argues that effective governance requires industry certifications.” The journal provides guidance on avoiding data center capital project failure and applying efficient IT principles to address sustainability risks and IT resilience during a natural disaster.
Meredith writes, “Julian Kudritzki and Matt Stansberry [the book’s editors] point out that, when the M&O Stamp of Approval is administered across an entire portfolio, it ensures consistency. CenturyLink is a perfect example of utilizing Uptime’s data center operations guidance. As the authors point out, “CenturyLink’s commitment to achieve M&O Stamp of Approval across the board shows it is willing to hold itself to a uniform set of high standards and operate with transparency.”
Meredith and CenturyLink have previously contributed to Uptime Institute publications. Please click on the following hyperlinks for some recent examples:
https://journal.uptimeinstitute.com/wp-content/uploads/2017/03/risk.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-03-09 14:25:582017-03-10 08:40:25CenturyLink Benefits from Its Commitment to Excellence
Myths and Misconceptions Regarding the Uptime Institute’s Tier Certification System
/in Design, Executive/by Kevin HeslinTrue or false? When it comes to Tier Certification, just ask Uptime Institute
By Uptime Institute Staff
Uptime Institute’s Tier Classification System for data centers has reached the two-decade mark. Since its creation in the mid-1990s, Tiers has evolved from a shared industry terminology into the global standard for third-party validation of data center critical infrastructure.
In that time, the industry has changed, and Tiers has evolved with it, remaining as relevant and important as it was when Uptime Institute first developed and disseminated Tiers. At the same time, Uptime Institute has observed that public understanding of Tiers has been clouded by the many myths and misconceptions that have developed over the years.
Uptime Institute has long been aware that not everyone fully understands the concepts described by the Tier Standards, and yet others disagree with some of the definitions. Both these situations lead to classic misunderstandings in which individuals substitute their preferences for accurate information. Other times, however, marketers have invoked a kind of shorthand based on Tiers. While objectionable, these marketers have coined terms like Tier III plus when speaking to their potential customers. These terms have no basis in Tiers but can be especially confusing to IT, real estate, procurement personnel, and even CFOs, all of whom might lack a technical background.
Other myths develop because some industry professionals reference old, out-of-date publications and explanatory material that is no longer valid. There may be other sources of myth, but knowing that the Uptime Institute is the only source of current and reliable information about Tiers is what is really important. We conduct numerous classes during the year, write many articles, and eld numerous inquiries to keep the industry current on Tiers.
Fundamentally, Uptime Institute created the Tier Classification System to consistently evaluate various data center facilities in terms of potential site infrastructure performance, or uptime. The system comprises four Tiers; each Tier incorporates the requirements of the lower Tiers.
• Tier I: Basic Capacity
• Tier II: Redundant Capacity Components
• Tier III: Concurrently Maintainable
• Tier IV: Fault Tolerant
Data center infrastructure costs and operational complexities increase with Tier level, and it is up to the data center owner to determine the Tier that fits the business’s need.
Uptime Institute is the only organization permitted to Certify data centers against the Tier Classification System. Uptime Institute does not design, build, or operate data centers. Uptime Institute’s role is to evaluate site infrastructure, operations, and strategy.
From this experience, we have compiled and addressed many of the myths and misconceptions. You can read about some of these experiences in Uptime Institute eJournal articles such as “Avoid Failure and Delay on Capital Projects: Lessons from Tier Certification” and “Avoiding Data Center Construction Problems.” For even more information, please contact us at https://uptimeinstitute.com/contact.
Tiers does not address business requirements.
False. Tiers is a performance-based, business-case-driven data center benchmarking system. An organization’s risk tolerance determines the appropriate Tier for the business. In other words, Tiers is predicated on the business case of an individual company. Companies that fail to develop a unique business case for their facilities before developing a Tier objective are misusing Tiers and bypassing the internal dialogue that needs to occur.
Tier IV is the best.
False. An organization’s tolerance for risk determines the appropriate Tier to support the business objective. Tier IV is not the best answer for all organizations, and neither is Tier II. Owners should perform due diligence assessments of their facilities before determining a Tier objective. If no business objective is defined, then Tiers may be misused to rationalize unnecessary investment.
Tier I and Tier II are tactical solutions, usually driven more by first-cost and time-to-market than life-cycle cost and performance (uptime) requirements. Organizations selecting Tier I and Tier II solutions typically do not depend on real-time delivery of products or services for a significant part of their revenue stream. Generally, these organizations are contractually protected from damages stemming from lack of system availability.
Rigorous uptime requirements and long-term viability are usually the reason for selecting strategic solutions found in Tier III and Tier IV site infrastructure. In a Tier III facility, each and every capacity component can be taken out of service on a planned basis, without affecting the critical environment or IT processes. Tier IV solutions are even more robust, as each and every capacity component and distribution path can sustain a failure, error, or unplanned event without impacting the critical environment or IT processes.
A Tier IV solution is not better than a Tier II solution. The performance and capabilities of a data center’s infrastructure should match a business application; otherwise companies may overinvest or take on too much risk.
For example, before building a Tier II Certified Constructed Facility, which by definition does not include Concurrent Maintainability across all critical subsystems, an owner should consider whether the business can tolerate a planned or maintenance-related shutdown and how the site operations team would coordinate a site-wide shutdown for maintenance. Similarly business objectives should drive decisions to build a Tier I, Tier III, or Tier IV Certified Constructed Facility.
Component count determines Tier level.
False. Tier Certification is a performance-based evaluation of a data center’s specific infrastructure; it is not a checklist or cookbook. Unfortunately, some industry shorthand employs N terminology—where N is de ned as the number of components that are minimally required to meet the load demand—to define availability. Incorporating more equipment can be described as designing an N+1, N+2, 2N, or 2(N+1) facility. However, increasing the component count does not determine or guarantee achievement of any specific Tier level, because Tiers also includes evaluation of distribution pathways and other system elements. Therefore, it is possible to achieve Tier IV with just N+1 components, depending on how they are configured and connected to redundant distribution pathways.
Design Certification is the only Certification that matters.
False. The first step in a Tier Certification process is a Tier Certification of Design Documents. Uptime Institute Consultants review the 100% design documents, ensuring each electrical, mechanical, monitoring, and automation subsystem meets the fundamental concepts and there are no weak links in the chain. The Design Certification is intended to be a milestone so that data center owners can commence data center construction knowing that the intended design meets the Tier objective.
Tier Certification of Design Documents applies to a document package. It is intended as provisional verification until the Tier Certification of Constructed Facility. Uptime Institute has not verified the constructed environment of these facilities, and thus cannot speak to the standard(s) to which they were built. To emphasize this point, Uptime Institute implemented an expiration date on Design Certifications. All Tier Certification of Design Documents awards issued after 1 January 2014 expired two years after the award date.
During a Facility Certification, a team of Uptime Institute consultants conducts a site visit, identifying discrepancies between the design drawings and installed equipment. The consultants observe tests and demonstrations to prove Tier compliance. Fundamentally, this is the value of the Tier Certification, finding these blind spots and weak points in the chain. Uptime Institute consultants say that in almost every site visit they find that changes have been made after the Design Certification was awarded so that one or more systems or subsystems will not perform in a way that complies with Tier requirements.
More recently, Uptime Institute instituted the Tier Certification of Operational Sustainability to evaluate how operators run and manage their mission-critical facilities. Even the most robustly designed and constructed facilities may experience outages without a well-developed comprehensive management and operation program. Certification at all three levels is how data center owners can be assured they are realizing the maximum potential of their data centers.
Tier levels specify an estimated downtime per year.
False. Uptime Institute removed references to “expected downtime per year” from the Tier Standard in 2009, but they were never a part of the Tier definitions. Tier Standard: Topology is based on specific performance factors (outcomes) that demonstrate that a facility has met specific performance objectives, such as having redundant capacity components, Concurrent Maintainability (generally, the ability to remove any capacity or distribution component from service on a planned basis without impacting IT), or Fault Tolerance (generally, the ability to experience any unplanned failure in the site infrastructure without impacting IT). However, even a Tier IV data center, which is Fault Tolerant, may experience IT outages if it is not operated and managed effectively.
There are statistical tools to predict the frequency of failures and time to recover. Availability is simply the arithmetic calculation of time a site was available over total time. The number, frequency, and duration of disruptions will drive the availability result. However, caution is appropriate when using these tools. Human activity is often not considered by statistical models. In addition, the statistical prediction of a 100-year storm, for example, can obscure the possibility that several 100-year storms can happen in the same year.
Tier Certification applies only to newly built facilities.
False. Uptime Institute has Certified many existing buildings. However, the process can be more challenging when working in facilities with live loads. For best results with an existing facility, the process should begin with a Tier Gap Analysis rather than a formal Certification effort. Tier Gap Analysis provides a high-level summary review for major Tier shortfalls. This allows the owner to make an informed decision whether to proceed with a detailed, exhaustive Certification effort. Tier Certification of Constructed Facility can be performed with any load profile, including resistive load banks, live critical IT load, or a mix.
Uptime Institute Tiers is U.S.-centric.
False. Uptime Institute is currently delivering Tier Certifications in more than 85 countries. Tiers, which allows for many solutions and a variety of configurations, gives the design, engineering, and operations teams the flexibility to meet both local regulations and performance requirements. To date, there has not been a conflict between Tiers and local building codes, statutes, or jurisdictions.
TIA-942 is a guideline for Uptime Institute Tiers.
False. In 2014 Uptime Institute and the Telecommunications Industry Association (TIA) agreed on a clear separation between their respective benchmarking systems to avoid industry confusion and drive accountability. In fact, any reference to the TIA rating of a data center may not include the word Tier.
The core objective of Uptime Institute Tiers is to define performance capabilities that will deliver availability required by the data center owner. By contrast, TIA member company experts focus on the need to support the deployment of advanced communications networks. See https://uptimeinstitute.com/uptime-tia for a more detailed explanation.
Utility feeds determine Tier level.
False. According to Tier Standard: Topology, the only reliable source of power for a data center is the engine- generator plant. This is because utility power is subject to unscheduled interruption—even in places with reliable power grids. As a result, the number of utility feeds, substations, and power grids that provide public power to a data center neither predicts nor influences Tier level. As a consequence, utility power is not even required for Tiers. Most Tier Certified data centers use utility power for main operations as an economic alternative, but this decision does not affect the owner’s target Tier objective.
For Tier III and IV, the engine-generator plant must be operational at all times.
False. Tiers does not require that the engine-generator plant actually run at all times; however, data centers will typically utilize a public utility a majority of the time for cost or regulatory reasons. At the same time, the engine-generator plant must be properly configured, rated, and sized to have the capability to carry the critical load without runtime limitations. Hence, the performance requirements outlined in the Tier Standard must be met with the data center supported by engine-generator power. Meeting these criteria requires special attention to engine-generator capacity ratings and power distribution.
Uptime Institute’s Tier system is based on the U.S. Environmental Protection Agency (EPA) regulations on diesel-engine operations.
False. There is no correlation between EPA’s Tiers (or other restrictions of engine-generator operation) and Uptime Institute Tiers, except that both systems use a similar hierarchical system of nomenclature. The EPA’s limits on runtime may complicate a facility’s testing and maintenance regimens and add costs when a facility is forced to rely on backup power for an extended period. However, runtime limitations posed by local authorities do not exempt a data center from having on-site power generation rated to operate without runtime limitations at a constant load.
EPOs (emergency power off) or other systems that shut down the critical load impact Tier objectives.
False. When code or the local authority having jurisdiction (AHJ) mandate an EPO, this does not prohibit Tier compliance. At the same time, Uptime Institute does not recommend EPO installation, unless it is compelled by a local code because even Tier Certified data centers are vulnerable to outages from purposeful or accidental activation of the EPO system. Analysis of the Uptime Institute Network’s Abnormal Incident Report (AIRs) database confirms that accidental EPO activation is a recurring cause of downtime.
The Tiers Standard requires that maintenance, isolation, and/or removal can be performed on the EPO system without affecting the critical load for Tier III data centers. Tier IV data centers additionally require a Fault Tolerant EPO system.
Uptime Institute Tiers requires raised floor.
False. The choice of under floor or overhead cooling is a decision to be made by the owner based an operational preference. In Uptime Institute’s experience, a raised floor enhances operational flexibility over the long term. Yet, decisions such as raised floor or on-slab, Cold Aisle/Hot Aisle, containment of Cold Aisle/Hot Aisle, and gallery cooling can affect the efficiency of the computer room environment, but are not mandated by Uptime Institute Tiers.
Rack-based automatic transfer switches (ATS) meet the requirement for dual paths to a server (i.e., the server has one cord to the ATS, but the ATS [rack mount] has dual power feeds).
True. The Tier Standard includes a concession for equipment with odd numbers of cords (1,3,5) in the form of rack-mounted transfer switches to provide access to multiple power paths. However, Tier III and Tier IV data centers must still have multiple and independent feeds to the rack.
The Tier Standard focuses on ensuring that the facility’s infrastructure meets the requirements of the Tier objective. There are many reasons why a facility may contain single-corded IT devices or those with an odd number of power supplies, including lack of knowledge of the facility impacts, lack of options for equipment vendors, and colocation environments where facility personnel have no control over the types of IT devices within the data center. Rack-based transfer switches are most typically supplied by the IT side of the organization, so the facility’s infrastructure can meet the Tier objective. However, planned isolation or fault of these rack-based transfer switches may lead to an outage for individual racks or devices.
Tier II provides Concurrent Maintenance opportunities.
Partially true. Tier II allows for Concurrent Maintenance of capacity components, but not distribution pathways or critical elements. So a Tier II Certified facility can perform Concurrent Maintenance on engine generators, UPS, chillers, cooling towers, pumps, air conditioners, fuel tanks, water tanks, and fuel pumps
but not switchboards, panels, transfer switches, transformers, bus bars, cables, and pipes. In many cases, this limitation will require the computer room to be shutdown for planned maintenance or replacement of critical pathways and elements.
The requirement to maintain any component, pathway, or element without shutting down equipment, known as Concurrent Maintainability, defines Tier III. Many owners’ business cases, including healthcare, domestic outsourcers, and state governments, require Tier III. The list of organizations that have protected their investment with Tier Certifications may be found on Uptime Institute’s website.
A Tier III facility is Tier compliant if one of the redundant branches is inactive.
Partially true. Tier III requires active/active distribution for critical power distribution (which is defined as the output of the UPS and below). Outside of that, active/inactive is acceptable. This means that if a rack receives dual power from two separate power distributions, they must both normally be active. It is not allowable to have one feed normally disabled, nor is it Tier III compliant to have one of the power feeds directly fed from utility power while bypassing a UPS power source.
There are no active/active requirements for mechanical systems in Tier III data centers. So if there are N+1 chillers in a Tier III facility, with each chiller feeding separate A and B chilled water loops, it is permissible for one of the loops to be normally disabled, with all air conditioners normally fed from the same loop.
Facilities can’t be changed after Tier Certification of Constructed Facility.
False. Infrastructure changes must be approached using carefully developed and written procedures and processes. If the topology of a facility changes, it may no longer be Concurrently Maintainable or Fault Tolerant, so clients should have Uptime Institute review designs and construction that might affect a facility’s topology to protect their investment and Tier Certification. Tier Certifications can be revoked if unreviewed changes compromise a facility’s Concurrent Maintainability or Fault Tolerance.
A Tier IV facility must have all its cooling units operating.
Mostly false. The Tier Standard requires only that Tier IV facilities provide stable cooling to the IT and UPS environment for the time it takes for the mechanical systems to completely restart after a utility power outage and provide rated load to the data center. Tier IV data centers must also be able to maintain a stable thermal environment for the duration of the mechanical restart time and for any 15-minute period in accordance with the 2015 ASHRAE Thermal Guidelines. Tier IV facilities are also required to be active/active for all systems. This is intended to ensure that Continuous Cooling solutions are not negated by a lack of active operation of components. A lightly loaded data center or one with a very complex control system may be able to meet these requirements, without using all the available cooling units. However, there are Tier IV data center designs, especially those at full load, that would in fact require all units to run during normal operations.
Makeup air capacity counts as critical cooling capacity.
Typically false. Makeup air systems in data center applications are typically designed to meet one of three objectives (or a combination of the three):
• Provide the fresh air for occupants required by an AHJ
• Ensure positive pressure in the data hall, which will help keep contaminants out of the data center
• Aid in meeting the humidity requirements of the data center
Data centers are rarely designed in a manner that require the makeup air handler to be active in order to meet the N cooling capacity requirement. However, the existence of a makeup air handler and its operations cannot negatively impact compliance with Tiers. For example, if a makeup air handler is not sized to ASHRAE extremes in compliance with Tiers, the additional heat load from this air handler at those conditions must be considered when sizing the critical cooling system.
It is not possible to utilize diesel rotary uninterruptible power systems (DRUPS) as continuous cooling in a Tier IV facility.
False. The Tier Standard is vendor and technology neutral, which means it is possible to Tier Certify facilities that include a wide variety of innovative and new technologies, including DRUPS.
Facilities tend to deploy DRUPS, which combine a diesel engine and a rotary UPS that uses kinetic energy to eliminate batteries, which require high levels of maintenance, somewhat frequent replacement, and a lot of extra space for battery placement/storage. This design usually provides ride-through times of between 10-30 seconds, depending on the application, which is shorter than other technologies. The Tier Standard does not include a minimum ride-through time. In fact, Uptime Institute has Certified several facilities that include DRUPS technology.
DRUPS may also be used to power motor loads. That means that caution must be exercised to ensure that the DRUPS have sufficient capacity to power each and every system and subsystem, including cooling systems, which is accomplished by putting the mechanical components on a no-break bus.
Ductwork does not need to comply with the Tier requirements.
False. Tier Certification analyzes each and every system and subsystem down to the level of valve positions and panel feeds. Ductwork, just like piping systems, may need planned maintenance, replacement, or reconfiguration. As such, traditional ductwork distribution systems must meet the requirements of the Tier objective.
Uptime Institute understands that there is a lot of confusion about what “maintaining” ductwork means to meet Concurrently Maintainable requirements. But in this case, Concurrent Maintainability is about having the capability to isolate a system or part of a system to maintain, repair, upgrade, or reconfigure the data center without impacting any of the computer equipment.
Site Location affects Tier level.
False. Although a critical consideration for the life-cycle operation of the facility and in determining, evaluating, and mitigating risk to the data center, geographical location does not affect a facility’s Tier level and is not part of the Tier Standard: Topology.
Data center designers can take precautions to address the specific risks of a site. A data center sited in a high-risk earthquake zone can include equipment that has been seismically rated and certified as well as incorporate techniques that mitigate damage from seismic activity. Or if a data center has been sited in a high-risk tornado area, designers can consider wind protection measures for the exterior electrical and heat rejection equipment.
Site Location is a criterion in the Tier Certification of Operational Sustainability.
Chris Brown, Enrique Hernandez, Kevin Heslin, Julian Kudritzki, Eric Maddison, Ryan Orr, Sarah Thomas, Pitt Turner, and Rich Van Loo all contributed to this article.
Top Considerations for Addressing Data Center Facilities Management Risks
/in Executive, Operations/by Kevin HeslinUptime Institute recently published “Top Considerations for Addressing Data Center Facilities Management Risks,” a guide for reducing data center risks in enterprise IT organizations . The guide comprises 14 top considerations useful for designing and running an enterprise-grade data center facilities management program. The full guide is available for download on the Uptime Institute website.
Areas of discussion include:
Japan’s Biggest Provider of Critical IT Services Earns M&O Stamp of Approval
/in Executive, Operations/by Kevin HeslinNearly perfect M&O Stamp of Approval scores prove that NRI’s data centers operations are world class
By Kevin Heslin
Nomura Research Institute, Ltd. (NRI), a ¥18 billion ($US169 million) company that provides consulting, financial IT solutions, industrial IT solutions, and IT platform services mainly to a global financial marketplace, has earned Uptime Institute Management and Operations (M&O) Stamps of Approval at two of its five facilities, receiving very high scores on both. In the process, NRI, Japan’s largest provider of critical IT services, impressed even Uptime Institute consultants, who have assessed many of the world’s best-run data centers.
Uptime Institute Consultant Steve Burgess said, “A very high M&O score translates to a quality of operations that is very proactively mitigating risk, something premium financial services companies must aggressively pursue and maintain.” Managing risk in that way is critical for NRI because of its important role in the global economy. Uptime Institute spoke to Takuhito Sakata, NRI’s managing director, Data Center Service Division.
NRI’s Tokyo 1 data center
Tell me about NRI’s operations.
TS: More than 300 financial and industry systems operate in our data centers. NRI provides the back office system that cover general securities business processes and 50% of the Tokyo Stock Exchange’s transactions are transmitted through our STAR financial back office system.
Other than financial systems, NRI operates a wide range of mission critical systems for banks, insurance companies, and many companies in other industries. Our systems have become Japan’s core infrastructure, so the cost of losing them in a disaster would be immeasurable. Without exception, our data centers are built for high capacity and availability, and NRI owns and operates the safest data centers in Japan.
Layout of NRI’s Tokyo 1 Data Center .
Please describe NRI’s data centers.
TS: NRI owns five data centers in Tokyo and Osaka. Yokohama Data Center 1(Y1DC) is the oldest of these facilities (1990), and Osaka Data Center 2 (O2DC) is the newest (2016). Tokyo Data Center 1 (T1DC) has the greatest power density (maximum 30 kW/rack). These and other differences in size, density, age, and location mean that the facilities have significant differences in operation that must be managed and supported.
The Osaka 2 data center opened just last year.
These five facilities support all our financial systems, and they align to the Payment Card Industry Data Security Standard (PCI DSS) and Sarbanes Oxley (SOX). These facilities are interconnected to provide mutual backup to each other. In fact, even the operation centers backup mutually. If one or more of the data centers are damaged, the systems are run in a remote data center and the operations also transfer to the backup. We call this concept Multi Data Center Operation (MDC).
A look at NRI’s double-deck system.
We believe that T1DC is state of the art. This facility earned an Uptime Institute Brill Award in 2015 for its patented double deck system. The double deck system separates server areas from facility areas, which helps us provide efficient power and cooling. Not only that, but the double deck system allows us to enhance our physical security and increases our structural integrity, which is important because of the risk of earthquakes in Japan.
These facilities benefit from different power utility supplies, and have redundant electric power system, including engine-generators sets, uninterruptible power systems, and standby transfer switches. The system is configured to backup the facility cooling. These and other arrangements, including a priority fuel supply contract, have enabled us to survive the Great East Japan Earthquake and the planned blackouts that followed without interruption.
Because of the ongoing risk of earthquakes, all the facilities are isolated seismically. The building is built on a huge seismic isolation rubber base and does not touch the ground directly. The building also includes a seismic damper, which is a brace that is structured to withstand shaking, so there will be no impact to IT equipment.
NRI deploys an isolation system and a bracing system to protect against potential earthquake damage.
What motivated NRI to obtain the M&O Stamp?
TS: We have been providing data center services for over 30 years, mainly to financial customers, and our service quality is now the best in Japan. We work hard to meet our customers’ rapidly changing business needs, and changing regulations and qualifications. We think we can improve our quality even further by third-party verification of our efforts.
We spent one year researching all the data center standards and certifications globally, which led us to Uptime Institute M&O Stamp of Approval. At that first (2014), no Japanese company had received the Stamp of Approval. So, we visited a number of data center sites in the U.S. that had the Stamp of Approval. We asked the owners and staffs at these sites whether the Stamp of Approval was effective and how it improved their quality of facility management. When we were convinced, we decided to submit for an M&O assessment.
How does the M&O Stamp benefit NRI?
TS: It sure helps increase the quality of the data center operation. The assessment program provides insights and advice that is shared by our five data centers, and helps us improve our operations knowledge at all of them.
NRI requires the participation of two professionals as part of many of its procedures.
When we re-assessed T1DC, we received 98.5 points (out of 100), an improvement of 1 point from the initial assessment. We are aiming for a perfect score, and the advice we receive from Uptime Institute consultants is very useful. And each time we re-assess we must show improvement as the program ratchets up the requirements.
Adding to that, M&O is also useful for creating awareness about the high quality of our data center operations. Especially in Japan, NRI received a lot of attention from the market and the media for being first to have Stamp of Approval and that we had a high score. And this awareness makes a big difference to customers touring our data centers. It is a matter of competitiveness. Some companies demand a high standard of operational readiness.
M&O is also helping us on global projects. Our data center design support service was the main project area so far, but we are being called on for our data center management consulting.
How did you prepare for the assessment process?
TS: To be honest, we didn’t prepare. We are confident of our daily regular maintenance, training, preventative checks, and routines. We wanted to evaluate our operations as they were, so we did not consciously prepare for the assessment. Instead, we viewed the results of each day’s work as a form of validation when we did well or opportunity to improve.
Of course, we knew the M&O criteria in advance, so we self-checked based on that, and we decided to go through the assessment process. For example, the M&O’s job definition form was unfamiliar to us, so we reviewed our standard to comply with global requirements.
Who was involved in the process? How did the staff benefit?
TS: We have a lot of in-house expertise, including facility management, security, and quality teams etc.). Based on these opinions, I made the decision to go ahead with the M&O. NRI created a cross-functional task force from this group to get the Stamp of Approval. As a result of M&O, our staff has become very confident about day-to-day operations. We learned about the importance of continuous improvement.
Do you find something new to apply after each assessment?
TS: That’s right. Currently we are having two data centers evaluated. The two data centers were built at different times, serve different customers, and are operated by different people. So we think it is natural that we get different results at each facilities. Those findings and advice for improvement are very helpful for us.
What are your plans for renewing the M&Os? As you know they expire every 2 years.
TS: We plan to have every data center recertified. Because conditions may change over time, we think that it is important to check on the data center every 2 years. Facilities and equipment age and staff changes so procedures may become dated or are ignored.
T1DC underwent a renewal review [Editor’s note: mandatory after 2 years] at the end of last year, and it is doing well.
What are your plans for further data center expansion? Will the M&O help with that process in any way?
TS: Global is a keyword for our company strategy, and the M&O Stamp of Approval combined with our proven track record will be part of our global business development. For this reason we plan to have the M&O Stamp of Approval at all our next-generation core data centers.
Highlights from Uptime Institute Symposium: Shanghai
/in Events, Executive/by Matt StansberryThis week, Uptime Institute hosted its first major event in China, Uptime Institute Symposium: Shanghai. Over 400 invite-only attendees joined Uptime Institute for a day of collaborative learning at the Grand Hyatt. The photos below illustrate some highlights from the the day.
Uptime Institute President Lee Kirby delivers the welcome keynote at Symposium: Shanghai. “The digital economy is interconnected, and you don’t want your data center to be the weak link. We need to industrialize our digital infrastructure through standards and third party assessments.”
Uptime Institute has tripled its business in China over the 5 years, with 18 Tier-certified data centers and 13 M&O data centers in China, with over 30 projects in process.
Uptime Institute Managing Director North Asia, Philip Hu discusses the process of Tier Certification. “The vast majority of even the world’s most elite data center sites do not operate as designed on day one. Tier Certification is a failsafe against a data center that doesn’t work.”
Uptime Institute Senior Director of Content and Publications Matt Stansberry unveils the results of the 2017 Data Center Industry Survey including responses from over 1500 end users worldwide.
Uptime Institute has conducted its annual survey globally for seven years. This year’s survey data included the first breakout of respondents specifically in China.
Attendees included Uptime Institute Tier Certification owners and M&O Stamp of Approval Awardees from China’s leading financial, colocation and telecom organizations.
Jack You, Cloud Architect for Huawei delivers the Diamond level Sponsor Keynote.
Uptime Institute President Lee Kirby presents Wang Hui, Director, China Carrier Service, Solutions Sales Department at Huawei with a letter of appreciation for Huawei’s commitment to Uptime Institute Symposium around the globe.
Philip Hu moderates a discussion of new technology adoption. Panelists: John Zheng, Solution Architect Director, Schneider Electric; Wu Ganxing
Vice General Manager, China Building Tech Group, Data Center; Biao Leng, General Manager, Beijing Travelsky Birun Technology
Pitt Turner, Uptime Institute Network Global Executive discusses the challenges with DCIM procurement: The process takes too long; Software costs too much; Tools under-deliver; Many buyers are not achieving ROI on the purchase.
Agatha Poon, Research Director APAC at 451 Research discusses the rapid growth and adoption of cloud computing providers in China and around the world, providing cloud computing adoption statistics, key trends in the market and the competitive landscape in the region and globally.
Transforming enterprises are serious about incorporating cloud models into their business strategies. Workloads are already business-critical in the cloud.
Lee Kirby: Complex IT systems are prone to cascading failure. But calling these failures human error is a mistake. The following are management failures, not human error: Lack of staffing; Lack of training; Lack of resources; Lack of preventive maintenance.
Over 65% of enterprise IT departments use Uptime Institute’s Tier Certification and M&O Stamp of Approval as part of the vetting process for considering potential IT service providers, according to the 2016 Data Center Industry Survey.
451 Research Director Andy Lawrence: The data center in 2020 will need to be agile — reducing costs, building on demand, responding to rapid deployment needs, and innovating without service disruption.
At the end of the each Symposium Event, an expert panel asks the audience — what are you going to do differently on Monday? How will you implement what you have learned today? Speaking: Danny Cheng, Chairman & CEO, Banyano
Stay tuned to the Symposium website for updates on our events in North America (Las Vegas, September 18) and Europe (Barcelona, Dec 5-6) in 2017.
Reconsider Your Diesel Fuel Supply
/in Operations/by Kevin HeslinNew diesel formulations put your generators at risk
By Bernard Oegema, Pat Smyth, Brian Ponstein, William “Bill” Klein, and Martin Wesolowski
The increased use of biodiesel has increased the risk to the diesel infrastructure of data centers and other facilities first noted with the adoption of ultralow sulfur diesel (ULSD) in 2006. The U.S. Environmental Protection Agency (EPA) documented severe and rapid corrosion in the storage tank portion of the diesel infrastructure caused by changes to the chemical composition of diesel fuel mandated by the U.S., U.K., and European Community. At the same time, engine-generator OEMs noticed an increase in warranty claims on their systems, including injectors. Solving this problem will require a new generation of filters tested against the updated industry standard, Society of Automotive Engineers SAE J1488_201010, to remove emulsified water.
The following steps are recommended to minimize or eliminate potential impacts from the use of biodiesel (see the sidebar for more detail):
It is important that all U.S. operators check their fuel supplies, because suppliers are not required to disclose biodiesel content of 5% or less. So a distributor may say there is no biodiesel content, when in fact there is. In addition, while a state may not have mandated the use of biodiesel, distributors have financial incentives to supply biodiesel blends.
Testing Procedures
Fuel should be filtered three times a week, and all the fuel filtered within 8 hours. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE, such as Southwest Research Institute.
Corrosion resulting from acidity caused by emulsified water dramatically increases the risk of an emergency generator failure and increases the costs of maintaining the data center diesel infrastructure. Corrosion affects storage tanks, emergency backup generators, diesel engines, fittings, and piping. Removing emulsified water from diesel fuel will control acidification and corrosion; drying the fuel controls microbial growth.
Fuel experts and data center operators first noticed corrosion problems shortly after the widespread adoption of ULSD. ULSD contains various additives (less than 1% of the fuel by volume) and fuel components (more than 1%), including biodiesel. These additives and fuel components absorb water which bonds to the diesel as emulsified water. The molecules of water, biodiesel, and other additives are all polar in nature. The emulsified water supports a bloom of microbes, and some of these microbes produce acids that corrode metal in their environment.
Before 2006, LSD (low sulfur diesel) fuel contained 500 parts per million (ppm) sulfur. ULSD (15 ppm sulfur) was introduced to help fight acid rain. While LSD was phased out of the market in 2010, a 2007 research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B.B. He, 2007) makes a useful point about how much water LSD and ULSD can absorb. He notes that 100% biodiesel absorbs 15 to 25 times more emulsified water than LSD. Neither LSD nor ULSD fuels are polar so they do not absorb much emulsified water.
In addition to lubrication concerns, biodiesel was added to ULSD in the U.S. for a variety of environmental and financial reasons, as well as government concerns about national dependence on foreign oil supplies.
However, the process of scrubbing the sulfur changed the chemistry of the fuel so ULSD provided less lubrication, resulting in immediate wear to diesel engines. An additive was introduced to ULSD to restore the needed lubrication. Soon after, biodiesel was also added as a fuel component, in ever increasing percentages, 2% (B2), 5%(B5), 7% (B7), 10% (B10), and now, 20% (B20). The more biodiesel, the more emulsified water. Biodiesel blend of 5% or less does not need to be disclosed to the end user.
Before ULSD, filters could remove all freestanding water and particulate, since LSD is not polar and there was very little emulsified water. Since both biodiesel and water are polar, water bonds tightly to the fuel as emulsified water. Pre-ULSD filters cannot efficiently remove the emulsified water from biodiesel blends.
Even before the U.S. Environmental Protection Agency (EPA) detailed this rapid and severe corrosion, NFPA 110 (2013) cautioned against the use of biodiesel for standby and prime engine-generator sets. In those instances, where the use of biodiesel was unavoidable, the NFPA required its replacement annually, but with “rapid and severe corrosion,” even 12 months is too long a time to store acidified fuel. NFPA recommends using diesel that has been stored that long for other purposes.
The EPA’s July 20th 2016 final report on corrosion states, “The major finding from our research is that moderate or severe corrosion on metal components in UST [underground storage tanks] systems storing diesel fuel in the United States could be a very common occurrence… variables that were the closest to being significant predictors [of corrosion] were particulates in the fuel and entrained (emulsified) water content in the fuel.” Drying the diesel by removing all the water, both free standing and emulsified, can control corrosion. This final report has worldwide significance because countries concerned about the issue have been tracking EPA findings.
A polar molecule
SAE J1488 tests “the ability of a fuel/water separator to separate emulsified or finely dispersed water from fuels.“ Well-known organizations such as Southwest Research Institute (SwRI) can test filters for compliance to the standard and provide reports to data center managers. In 2010, SAE updated the test from J1488_199708 to J1488_201010 to address the emulsified water concerns caused by ULSD and biodiesel blends.
SAE updated J1488 in 2010 to reflect the lower interfacial tension (IFT) in ULSD and biodiesel blends. IFT is the ability of two liquids to repel each other. Since biodiesel in hydroscopic, more water is absorbed and less repelled, so its IFT is lower. The IFT of ULSD is tested at 34.5 mN/m, biodiesel at 15 mN/m (B20). The 2010 update created a standard IFT against which all filters could be compared.
A new generation of filters designed specifically to filter for SAE J1488_201010 IFT levels is required to remove emulsified water, and hence control corrosion. In June 2016, the United Kingdom adopted SAE J1488_201010 as a code requirement.
Critical Fuel Applications
Modern backup generators are high-pressure common rail (HPCR) models, with pressures of 30,000+ pounds per square inch (psi) and plans for 60,000 psi models. Emulsified water and corrosion can cause these engine generator sets to become unreliable by
When injector pitting and corrosion are great enough, the generator will not be able to hold the required HPCR pressures and will fail. In addition, the pitting of the injectors can cause the generator to become over fueled, increasing heat, emissions, and fuel consumption. The corrosive/pitting affects injectors and spoolers on all diesel engines. The higher the psi of the engine, the greater the risk of damage and failure.
Corrosion in storage tanks, fittings, and piping dramatically increases the risk of a backup generator failing. For example, a fuel supply pipe that fails or plugs will prevent a generator from working during an emergency.
Warranty Claims
Warrantees on diesel engine-generator sets exclude damages caused by fuel-related issues. That means that vendors can deny warranty claims if the fuel does not meet the specification set forth by the vendor. The maximum emulsified water content in most warrantees is 200 parts per million (ppm), which is well below the observed amounts commonly found in tanks with biodiesel blends. Caterpillar’s warranty specifies a pre-emulsified water/corrosion water and particulate maximum of 500 ppm. Manufacturers generally enforce this requirement by requiring a fuel sample with each warranty claim.
Caterpillar’s service document SEBU6251-14 from 2014 (before corrosion was commonly recognized) reads, “The repair of any engine or after treatment components due to such contamination or due to other biodiesel impacts would not be covered under the Cat warranty for materials and/or the warranty for workmanship.” Since corrosion is such a new issue, it means Cat’s warranty could be voided the first time biodiesel is used in the engine.
Similarly, Cummins warns about biodiesel. Cummins’s service document 3379001-13 reads, “Engine damage, service issues, and/or performance issues determined by Cummins Inc. to be caused by the use of biodiesel fuel not meeting the specifications outlined in this Service Bulletin are not considered to be defects in material or workmanship and are not covered under Cummins Inc. engine warranty.”
All the other OEM warrantees include similar statements.
Sidebar: Corrosion Management Action
Removing emulsified water from the fuel is key to maintaining the integrity of diesel fuel systems. SAE J1488_201010 tested filters can remove the emulsified water, thereby restricting microbial growth, acidification, and dramatically reducing corrosion. Filters need to achieve a performance level of greater than 92% to keep water below 200 parts per million (ppm).
Below is a short checklist to help address any fuel issues, with detailed explanations following.
While the state may not mandate biodiesel, there are financial incentives to distributors to supply biodiesel blends. Note that suppliers are not required to disclose any biodiesel content of 5% or less, so the distributor may say there is no biodiesel content, when in fact there is.
Take fuel samples from all fuel tanks. The sample should be taken from the tank bottom. Test for biodiesel content, water content with Karl Fisher titration (less than 200 ppm), ISO 4406 particulate content (less than 18,16,13), and TAN (Total Acid Number, less than .08 maximum per World Fuel Charter).
It may be quicker and easier to sample the belly tanks or day tanks attached to the generators. If possible, have a visual inspection of the inside of your belly and day tanks. If there is any corrosion in these tanks, it will point to corrosion in the large storage tanks. All storage tanks are designed to handle the hydrostatic pressure (the weight of the fuel), which is greatest on the tank bottom. Since corrosion is also focused on the bottom of the tank, corrosion will pit of the tank wall, thereby thinning the wall, increasing the likely-hood of fuel leaks or tank failure. There are inexpensive fiberoptic cameras on long hoses available on the market to assist in the inspection.
If corrosion is seen, replace the fuel. Before adding new fuel, ask for a fuel sample from the suppliers distribution tank before delivery, and test for biodiesel content, water content with Karl Fisher titration, particulate content against ISO 4406, and TAN (Total Acid Number). If the sample fails any test, inform the supplier and request fuel within the acceptable specifications.
The EPA report states, “Minimizing water presence is and has always been an important part of UST maintenance. However, diesel blended with biodiesel can hold in solution more water than diesel without a biodiesel component. This means more water is likely arriving in USTs entrained in fuel today since biodiesel is more common in diesel than prior to 2007.” The large distribution storage tanks are also absorbing entrained (emulsified) water, so new fuel could already have substantial amounts of entrained (emulsified) water.
Observed corrosion (Battelle)
Install a permanent fuel polishing system on the storage tank if fuel tests show any failures in the tests. The polishing system must use the new generation of filters specifically designed to remove emulsified water to SAE J1488_201010 standards with of a minimum 92% efficiency. The fuel should be filtered 3 times a week, and filter the complete volume of fuel in an 8-hour period. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE such as Southwest Research Institute.
Fuel Experts
While most diesel consumers did not immediately recognize the issues with corrosion, the fuel experts did. Below is a timeline of recognition of acidification caused by microbial growth, starting with the most recent and working back.
U.S. EPA recognizes entrained (emulsified) water as a key indicator of corrosion. “This research focused on better understanding a type of rapid and severe corrosion of metal components in underground storage tanks (USTs) storing diesel fuel….EPA’s research determined that corrosion is very common; it appears all USTs storing diesel could be susceptible to developing corrosion.”
BS5410-3 made it code that any filters used on critical emergency backup systems conform to SAE J1488_201010.
BS5410-3 (British Standards Institution, 2016) is the equivalent of NFPA 110. The United Kingdom has used biodiesel blends longer than in North America. “7.2.2.5 Filters: NOTE 1 Since the introduction of Biofuel (FAME) [fatty-acid methyl ester] into fuels the quality and life expectancy of fuels has been adversely affected as FAME is hygroscopic so any water in the fuel goes into suspension….Removing as much water as possible preserves the quality of the fuel and for critical standby generators only filters conforming to SAE J1488_201010 [N5] are to be used”
NFPA 110 cautions against the use of biodiesel. “A.5.1.1(1) Where possible, the purchaser of fuel for the prime mover should specify a diesel fuel that does not contain biodiesel, which can accelerate the degradation of the diesel fuel if stored longer than 6 months.”
“A.5.5.3 More important, biodiesel blends up to B5 (ASTM D 975, Standard Specification for Diesel Fuel Oils) have much shorter shelf lives than conventional diesel fuel [ultra-low sulfur diesel (ULSD)] and can accelerate degradation processes, endangering the entire diesel fuel supply. Where fuel is stored for extended periods of time (e.g., more than 12 months), it is recommended that fuels be periodically pumped out and used in other services and replaced with fresh fuel.”
The Steel Tank Institute (STI) reinforces the importance of keeping storage tanks free of water. “In a 2012 study by Battelle Labs, five of six FRP (fiber- glass reinforced plastic) tanks showed severely accelerated corrosion. The sixth tank was supposedly “clean” and intended to be the control for the study, but turned out to have corrosion, as well….In ULSD, the presence of water provides an environment where microbial life can thrive, possible influencing the corrosion of internal metal components of the UST system equipment. “
SAE updates J1488_199708 to J1488_201010 to test the efficiency of filters to remove emulsified water from the newly introduced ULSD and biodiesel blends (B20). A 92% efficiency can keep water below 200 ppm, which is the OEM manufacturers’ warranty limit. In SAE J1488_201010 testing, ULSD and biodiesel (B20) are saturated with 2500 ppm of emulsified water, which is then filtered. The water content of the filtered fuel is compared to the water content of the original saturated fuel. SAE then assigns an efficiency rating to the filter and reports its findings to the filter supplier. (SAE Technical Standards Board, October 2010).
Battelle Memorial Institute calls attention to the problem of corrosion. “Severe and rapid corrosion has been observed in systems storing and dispensing ultra low sulfur diesel (ULSD) since 2007” (Battelle Memorial Institute, 2012)
The research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B. B. He, 2007), compares the water absorption of 100% LSD to 100% biodiesel.
IBM Data Center Group Case Study
IBM has a group within the Global Technology Services division that designs, builds, migrates, and upgrades data centers for its customers. It reports that the majority of data centers in Canada have been in the Province of Ontario, which mandated the use of biodiesel with an increasing concentration, initially B2, and B4 at time of writing.
IBM was initially concerned about the reliable operation of the generators at these facilities, especially because data centers often store fuel is stored for long periods of time, unlike road or construction applications. In particular, IBM worried about the effect of the emulsified water on its engine-generator set infrastructure. Reduced engine-generator set life and increased repair costs were major concerns.
Over time, it became clear to IBM that the entire fuel storage and delivery system, metallic and non-metallic components included, is at risk due to the corrosive properties of degraded biodiesel. A major excavation required to repair or replace would be a major concern.
Fuel filters tested in systems using B2 pass the “bright and clear” test, which wasn’t deemed rigorous enough to give a true indication of the condition of biodiesel fuel. IBM has been offering their clients fuel-conditioning systems that meet or exceed a more applicable standard from the SAE J1488_201010.
Three Biodiesel Related Failures in One Day
Franklin Company in the NYC area responded to three separate fuel-related diesel generator failures in one day at three different organizations. Problems caused by ULSD can occur in many application types. Failure 3 is typical of failures at a very common type of facility. These failures give a good representation of some of the failure points data center users can experience: corrosion of tanks, failure of pipes and fittings, and biomass plugging filters.
Failure 1: Franklin Company responded to a service call with a reported interstitial alarm. A service tech removed the sensor and found a mixture of water and fuel on the sensor. The tech removed a small amount of fuel and water mixture and then reinstalled the sensor. The sensor did not go into alarm. Franklin initially thought that recent rains had worked its way past the interstitial cap.
A few weeks later, Franklin received a new service call with the same complaint. This time the tech found a significant amount of water and some fuel in the interstitial. Since the tank had not been showing any losses, Franklin thought that the outer wall was leaking groundwater and fuel into the annular space from outside. It would not be so unusual for a site to have contamination. The interstitial failed a vacuum test. The primary tank passed a second test was to confirm that it was tight. A spill was called in to the DEC for a non-tight tank system. Franklin Company contracted a third party to perform the same tests, with the same result.
As one result, Franklin excavated and checked the tank top for breaches or leaking bungs to the interstitial. All fuel was removed, the manway removed, and a confined space team entered and began cleaning a very dirty tank. The team found the steel tank had many holes on the bottom.
What happened with the tests? The annular was leaking, but only between the primary and the secondary. Tests on the primary showed “tight,” but we were really testing only the secondary tank, as the primary tank was open to the secondary. The larger amounts of water in the secondary was caused by the head pressure pushing the tank bottoms out of the bottom of the primary. The annular space on an ACT 100 is very small, so a small amount of water would migrate to the riser, showing a disproportionate amount of water versus fuel. When the tech removed water, it would look dry for an extended period of time until more water and fuel would be forced through the bottom of the tank and work its way to the annular riser.
The tank was not salvageable and had to be removed. There was no leakage to the environment as the secondary was intact and tight. Note the tank was 10 years old.
Failure 2: An underground UL 971 flexible pipe failed when B20 diesel caused the internal lining of the pipe to delaminate. The contractor said, “The pipe looked like a long sheet of white plastic wrap when we investigated the blockage.”
Failure 3: A government agency reported a slow flow of diesel fuel dispensers. A service technician dispatched to the site found that two dispensers had flow rates of only 2-3 gallons per minute. Two dispensers had no flow at all. The internal filters were removed and replaced, and the pumping unit strainers (suction style units) were removed and cleaned and then re-installed. The facility was down for two days. In this case, microbial “blooms” and excess particulates had created biofilms that plugged the filters.
Biofilm, as shown in the picture below, is a problem that can lead to the failures stated above. Biofilms concentrate the corrosive acidic effects of the fuel (Failure 1), can peel off in strips to plug pipes (Failure 2) and can disintegrate, overloading filters (Failure 3). The contact said, “WOW. Never saw anything like this. This was the consistency of boiled pig skin. It was lining the bottom of the tank. This is a tank that we could not rinse in situ. Super bugs.”
Picture Below, Biofilm from bottom of tank
Suggested Readings
Chuanfang Yang, S. L. (2007). Understanding emulsified water filtration from diesel fuels . San Antonio: 8th International Filtration Conference.
Groysman, A. (n.d.). Corrosion in Systems for Storage and Transportation of Petroleum Products and Biofuels Identification, Monitoring, and Solutions. New York, Philadelphia: Springer Science & Business Media.
PEI Petroleum Equipment Industry. (2013). Recommended Practices for Installation of Bulk Storage Plants (PEI/RP800-13). Petroleum Equipment Industry.
R.L. McCormick, T. A. (2005). Survey of the Quality and Stability of Biodiesel and Biodiesel Blends in the United States in 2004 . National Renewable Energy Laboratory, Golden, Colorado.
Steel Tank Institute. (2016). Recommended Practice For Storage Tank Maintenance R111 Revision . Steel Tank Institute , Lake Zurich, IL.
Bibliography
European Automobile Manufacturers’ Association ACEA. (2013). World Wide Fuel Charter 5th Edition.
ASTSWMO Tanks Subcommittee’s Alternative Fuels Workgroup. (2013). Compatibility of UST Systems with Biofuels. Association of State and Territorial Solid Waste Management Officials (ASTSWMO), Washington, DC.
B. He, J. C. (2007). Moisture Absorption In Biodiesel and Its Petro-Diesel Blends. Food & Process Engineering Institute Division of ASABE. Moscow, Idaho: American Society of Agricultural and Biological Engineers ISSN 0883 8542.
Battelle Memorial Institute. (2012). Corrosion in Systems Storing and Dispensing Ultra Low Sulfur Diesel (ULSD), Hypotheses Investigation. Battelle Memorial Institute, Columbus, OH.
British Standards Institution. (2016). Code of practice for oil firing Part 3: Installations for furnaces, kilns, ovens, oil-fuelled standby generators and other industrial purposes. BSI Standards Limited.
NFPA110. (2013). NFPA 110 Standard for Emergency and Standby Power Systems. Quincy, MA.
SAE Technical Standards Board. (October 2010). Emulsified Water/Fuel Separation Test Procedure. Thompson Reuters.
Steel Tank Institute STI. (2013). Newer fuels and storage tank corrosion. Steel Tank Institute STI, Lake Zurich IL.
U.S. Environmental Protection Agency (July 2016). Investigation Of Corrosion-Influencing Factors In Underground Storage Tanks With Diesel Service. United States Environmental Protection Agency, U.S. Environmental Protection Agency Office of Underground Storage Tanks, Washington DC.
Uptime Institute. (2010). Uptime Institute Technical Paper: Biodiesel. New York.
Bernard Oegema
Bernard Oegema PE Data Centre Consultant, IBM Global Services, Data Centre Services. He has been in the critical facilities business for more than 30 years, working with various manufacturers of critical power and cooling infrastructure. Currently he is supporting IBM clients in the migration, consolidation, operation, upgrade, design and building of data centers.
Pat Smyth
Pat Smyth is Vice President of DieselPure. DieselPure is focused on fuel polishing for emergency backup generator storage. Since 2011, DieselPure has field experience in removing emulsified water from ULSD and biodiesel as per SAE J1488_201010.
Brian Ponstein
Brian Ponstein is Regional Sales Engineer, MTU Onsite Energy, a Rolls-Royce Company. At MTU, Mr. Ponstein is responsible for analyzing market needs and requirements in North America and working with engineering to provide solutions for MTU Onsite Energy’s customers.
Bill Klein
William (Bill) Klein is President of Franklin Company Contractors. He is also a New York City (NYC) Master Electrician, NYC Licensed Plumber, NYC Licensed Pump and Tank Contractor, NYC Licensed Precision Tank Tester, member of PEI, and past president of NYC Tank Installers. Mr. Klein is the third generation president of Franklin Company.
Martin Weselowski
Martin Wesolowski is Senior Associate Vice President at LiRo’s petroleum and chemical storage tank design group. He has been involved with design of petroleum and chemical UST and AST tank and remediation systems for over 23 years. Mr. Wesolowski is a technical director on LiRo’s New York City Department of Buildings (NYCDOB) Special Inspection Agency Registration.
CenturyLink Benefits from Its Commitment to Excellence
/in Executive/by Kevin HeslinDavid Meredith says that CenturyLink’s involvement, including the M&O Stamp of Approval, with Uptime Institute provides mutual benefits. Uptime Institute’s Risk Journal for IT Infrastructure is the most recent example.
CenturyLink’s commitment to operational excellence has become one of the company’s competitive advantages, and David Meredith, a senior VP at CenturyLink, has been an enthusiastic proponent of demonstrating this commitment through third-party verifications. In 2014, CenturyLink decided to pursue Uptime Institute’s Management and Operations (M&O) Stamp of Approval for all 57 of its data centers.
So Uptime Institute was delighted to see Meredith’s comments in LinkedIn about our new publication “Risk Management for IT Infrastructure.” Meredith quoted the book’s central tenet, “The handbook also argues that effective governance requires industry certifications.” The journal provides guidance on avoiding data center capital project failure and applying efficient IT principles to address sustainability risks and IT resilience during a natural disaster.
Meredith writes, “Julian Kudritzki and Matt Stansberry [the book’s editors] point out that, when the M&O Stamp of Approval is administered across an entire portfolio, it ensures consistency. CenturyLink is a perfect example of utilizing Uptime’s data center operations guidance. As the authors point out, “CenturyLink’s commitment to achieve M&O Stamp of Approval across the board shows it is willing to hold itself to a uniform set of high standards and operate with transparency.”
Meredith and CenturyLink have previously contributed to Uptime Institute publications. Please click on the following hyperlinks for some recent examples:
https://journal.uptimeinstitute.com/improving-performance-ever-changing-mission-critical-infrastructures/
https://journal.uptimeinstitute.com/executive-perspectives-colocation-wholesale-markets/