Nearly perfect M&O Stamp of Approval scores prove that NRI’s data centers operations are world class
By Kevin Heslin
Nomura Research Institute, Ltd. (NRI), a ¥18 billion ($US169 million) company that provides consulting, financial IT solutions, industrial IT solutions, and IT platform services mainly to a global financial marketplace, has earned Uptime Institute Management and Operations (M&O) Stamps of Approval at two of its five facilities, receiving very high scores on both. In the process, NRI, Japan’s largest provider of critical IT services, impressed even Uptime Institute consultants, who have assessed many of the world’s best-run data centers.
Uptime Institute Consultant Steve Burgess said, “A very high M&O score translates to a quality of operations that is very proactively mitigating risk, something premium financial services companies must aggressively pursue and maintain.” Managing risk in that way is critical for NRI because of its important role in the global economy. Uptime Institute spoke to Takuhito Sakata, NRI’s managing director, Data Center Service Division.
NRI’s Tokyo 1 data center
Tell me about NRI’s operations.
TS: More than 300 financial and industry systems operate in our data centers. NRI provides the back office system that cover general securities business processes and 50% of the Tokyo Stock Exchange’s transactions are transmitted through our STAR financial back office system.
Other than financial systems, NRI operates a wide range of mission critical systems for banks, insurance companies, and many companies in other industries. Our systems have become Japan’s core infrastructure, so the cost of losing them in a disaster would be immeasurable. Without exception, our data centers are built for high capacity and availability, and NRI owns and operates the safest data centers in Japan.
Layout of NRI’s Tokyo 1 Data Center .
Please describe NRI’s data centers.
TS: NRI owns five data centers in Tokyo and Osaka. Yokohama Data Center 1(Y1DC) is the oldest of these facilities (1990), and Osaka Data Center 2 (O2DC) is the newest (2016). Tokyo Data Center 1 (T1DC) has the greatest power density (maximum 30 kW/rack). These and other differences in size, density, age, and location mean that the facilities have significant differences in operation that must be managed and supported.
The Osaka 2 data center opened just last year.
These five facilities support all our financial systems, and they align to the Payment Card Industry Data Security Standard (PCI DSS) and Sarbanes Oxley (SOX). These facilities are interconnected to provide mutual backup to each other. In fact, even the operation centers backup mutually. If one or more of the data centers are damaged, the systems are run in a remote data center and the operations also transfer to the backup. We call this concept Multi Data Center Operation (MDC).
A look at NRI’s double-deck system.
We believe that T1DC is state of the art. This facility earned an Uptime Institute Brill Award in 2015 for its patented double deck system. The double deck system separates server areas from facility areas, which helps us provide efficient power and cooling. Not only that, but the double deck system allows us to enhance our physical security and increases our structural integrity, which is important because of the risk of earthquakes in Japan.
These facilities benefit from different power utility supplies, and have redundant electric power system, including engine-generators sets, uninterruptible power systems, and standby transfer switches. The system is configured to backup the facility cooling. These and other arrangements, including a priority fuel supply contract, have enabled us to survive the Great East Japan Earthquake and the planned blackouts that followed without interruption.
Because of the ongoing risk of earthquakes, all the facilities are isolated seismically. The building is built on a huge seismic isolation rubber base and does not touch the ground directly. The building also includes a seismic damper, which is a brace that is structured to withstand shaking, so there will be no impact to IT equipment.
NRI deploys an isolation system and a bracing system to protect against potential earthquake damage.
What motivated NRI to obtain the M&O Stamp?
TS: We have been providing data center services for over 30 years, mainly to financial customers, and our service quality is now the best in Japan. We work hard to meet our customers’ rapidly changing business needs, and changing regulations and qualifications. We think we can improve our quality even further by third-party verification of our efforts.
We spent one year researching all the data center standards and certifications globally, which led us to Uptime Institute M&O Stamp of Approval. At that first (2014), no Japanese company had received the Stamp of Approval. So, we visited a number of data center sites in the U.S. that had the Stamp of Approval. We asked the owners and staffs at these sites whether the Stamp of Approval was effective and how it improved their quality of facility management. When we were convinced, we decided to submit for an M&O assessment.
How does the M&O Stamp benefit NRI?
TS: It sure helps increase the quality of the data center operation. The assessment program provides insights and advice that is shared by our five data centers, and helps us improve our operations knowledge at all of them.
NRI requires the participation of two professionals as part of many of its procedures.
When we re-assessed T1DC, we received 98.5 points (out of 100), an improvement of 1 point from the initial assessment. We are aiming for a perfect score, and the advice we receive from Uptime Institute consultants is very useful. And each time we re-assess we must show improvement as the program ratchets up the requirements.
Adding to that, M&O is also useful for creating awareness about the high quality of our data center operations. Especially in Japan, NRI received a lot of attention from the market and the media for being first to have Stamp of Approval and that we had a high score. And this awareness makes a big difference to customers touring our data centers. It is a matter of competitiveness. Some companies demand a high standard of operational readiness.
M&O is also helping us on global projects. Our data center design support service was the main project area so far, but we are being called on for our data center management consulting.
How did you prepare for the assessment process?
TS: To be honest, we didn’t prepare. We are confident of our daily regular maintenance, training, preventative checks, and routines. We wanted to evaluate our operations as they were, so we did not consciously prepare for the assessment. Instead, we viewed the results of each day’s work as a form of validation when we did well or opportunity to improve.
Of course, we knew the M&O criteria in advance, so we self-checked based on that, and we decided to go through the assessment process. For example, the M&O’s job definition form was unfamiliar to us, so we reviewed our standard to comply with global requirements.
Who was involved in the process? How did the staff benefit?
TS: We have a lot of in-house expertise, including facility management, security, and quality teams etc.). Based on these opinions, I made the decision to go ahead with the M&O. NRI created a cross-functional task force from this group to get the Stamp of Approval. As a result of M&O, our staff has become very confident about day-to-day operations. We learned about the importance of continuous improvement.
Do you find something new to apply after each assessment?
TS: That’s right. Currently we are having two data centers evaluated. The two data centers were built at different times, serve different customers, and are operated by different people. So we think it is natural that we get different results at each facilities. Those findings and advice for improvement are very helpful for us.
What are your plans for renewing the M&Os? As you know they expire every 2 years.
TS: We plan to have every data center recertified. Because conditions may change over time, we think that it is important to check on the data center every 2 years. Facilities and equipment age and staff changes so procedures may become dated or are ignored.
T1DC underwent a renewal review [Editor’s note: mandatory after 2 years] at the end of last year, and it is doing well.
What are your plans for further data center expansion? Will the M&O help with that process in any way?
TS: Global is a keyword for our company strategy, and the M&O Stamp of Approval combined with our proven track record will be part of our global business development. For this reason we plan to have the M&O Stamp of Approval at all our next-generation core data centers.
Takuhito Sakata is managing director, Data Center Service Division, NRI.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/04/NRIweb2.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-04-04 12:29:022017-06-13 07:41:12Japan’s Biggest Provider of Critical IT Services Earns M&O Stamp of Approval
This week, Uptime Institute hosted its first major event in China, Uptime Institute Symposium: Shanghai. Over 400 invite-only attendees joined Uptime Institute for a day of collaborative learning at the Grand Hyatt. The photos below illustrate some highlights from the the day.
Uptime Institute President Lee Kirby delivers the welcome keynote at Symposium: Shanghai. “The digital economy is interconnected, and you don’t want your data center to be the weak link. We need to industrialize our digital infrastructure through standards and third party assessments.”
Uptime Institute has tripled its business in China over the 5 years, with 18 Tier-certified data centers and 13 M&O data centers in China, with over 30 projects in process.
Uptime Institute Managing Director North Asia, Philip Hu discusses the process of Tier Certification. “The vast majority of even the world’s most elite data center sites do not operate as designed on day one. Tier Certification is a failsafe against a data center that doesn’t work.”
Uptime Institute Senior Director of Content and Publications Matt Stansberry unveils the results of the 2017 Data Center Industry Survey including responses from over 1500 end users worldwide.
Uptime Institute has conducted its annual survey globally for seven years. This year’s survey data included the first breakout of respondents specifically in China.
Attendees included Uptime Institute Tier Certification owners and M&O Stamp of Approval Awardees from China’s leading financial, colocation and telecom organizations.
Jack You, Cloud Architect for Huawei delivers the Diamond level Sponsor Keynote.
Uptime Institute President Lee Kirby presents Wang Hui, Director, China Carrier Service, Solutions Sales Department at Huawei with a letter of appreciation for Huawei’s commitment to Uptime Institute Symposium around the globe.
Philip Hu moderates a discussion of new technology adoption. Panelists: John Zheng, Solution Architect Director, Schneider Electric; Wu Ganxing Vice General Manager, China Building Tech Group, Data Center; Biao Leng, General Manager, Beijing Travelsky Birun Technology
Pitt Turner, Uptime Institute Network Global Executive discusses the challenges with DCIM procurement: The process takes too long; Software costs too much; Tools under-deliver; Many buyers are not achieving ROI on the purchase.
Agatha Poon, Research Director APAC at 451 Research discusses the rapid growth and adoption of cloud computing providers in China and around the world, providing cloud computing adoption statistics, key trends in the market and the competitive landscape in the region and globally.
Transforming enterprises are serious about incorporating cloud models into their business strategies. Workloads are already business-critical in the cloud.
Lee Kirby: Complex IT systems are prone to cascading failure. But calling these failures human error is a mistake. The following are management failures, not human error: Lack of staffing; Lack of training; Lack of resources; Lack of preventive maintenance.
Over 65% of enterprise IT departments use Uptime Institute’s Tier Certification and M&O Stamp of Approval as part of the vetting process for considering potential IT service providers, according to the 2016 Data Center Industry Survey.
451 Research Director Andy Lawrence: The data center in 2020 will need to be agile — reducing costs, building on demand, responding to rapid deployment needs, and innovating without service disruption.
At the end of the each Symposium Event, an expert panel asks the audience — what are you going to do differently on Monday? How will you implement what you have learned today? Speaking: Danny Cheng, Chairman & CEO, Banyano
Stay tuned to the Symposium website for updates on our events in North America (Las Vegas, September 18) and Europe (Barcelona, Dec 5-6) in 2017.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/03/ThinkstockPhotos-515811014.jpg11612581Matt Stansberryhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngMatt Stansberry2017-03-22 22:00:102017-03-23 01:56:57Highlights from Uptime Institute Symposium: Shanghai
New diesel formulations put your generators at risk
By Bernard Oegema, Pat Smyth, Brian Ponstein, William “Bill” Klein, and Martin Wesolowski
The increased use of biodiesel has increased the risk to the diesel infrastructure of data centers and other facilities first noted with the adoption of ultralow sulfur diesel (ULSD) in 2006. The U.S. Environmental Protection Agency (EPA) documented severe and rapid corrosion in the storage tank portion of the diesel infrastructure caused by changes to the chemical composition of diesel fuel mandated by the U.S., U.K., and European Community. At the same time, engine-generator OEMs noticed an increase in warranty claims on their systems, including injectors. Solving this problem will require a new generation of filters tested against the updated industry standard, Society of Automotive Engineers SAE J1488_201010, to remove emulsified water.
The following steps are recommended to minimize or eliminate potential impacts from the use of biodiesel (see the sidebar for more detail):
Check with state regulations to confirm whether biodiesel is mandated.
Check facility records to determine when the fuel was last replaced. Unfiltered older fuel will have more issues.
Create a maintenance routine for diesel fuel that will include twice-annual fuel samples and tank inspections.
Take fuel samples from main storage tanks, belly tanks, and day tanks
If the fuel fails the tests, install a fuel polishing system based on SAE J1488_201010
If the fuel is found to be acidic, drain and clean the tank, and replace the fuel.
Test a sample of the replacement fuel, and all future deliveries from the supplier.
It is important that all U.S. operators check their fuel supplies, because suppliers are not required to disclose biodiesel content of 5% or less. So a distributor may say there is no biodiesel content, when in fact there is. In addition, while a state may not have mandated the use of biodiesel, distributors have financial incentives to supply biodiesel blends.
Testing Procedures
Test for biodiesel content, water content with Karl Fisher titration (less than 200 ppm), ISO 4406 particulate content (less than 18,16,13), and TAN (Total Acid Number, less than 0.08 maximum per World Fuel Charter). Fuel samples should be taken from the bottom of the tank. Karl Fischer titration is a method to determine the trace amounts of water in a fuel sample. It can measure the amount of entrained water in fuel, as well as emulsified and free water. Other tests cannot sense entrained water.
Sample the belly tanks or day tanks attached to the generators. If possible, visually inspect the insides of these tanks. Corrosion in these tanks points to corrosion in the large storage tanks. An inexpensive fiber optic camera can assist in the inspection. Storage tanks are designed to handle the hydrostatic pressure (weight of the fuel), which is greatest on the tank bottom. Since corrosion is also concentrated on the bottom of the tank, corrosion will pit the tank wall and floor, thereby thinning the wall, increasing the likelihood of fuel leaks or tank failure.
If corrosion is found, replace the fuel. Before adding new fuel, ask for a fuel sample from the supplier’s distribution tank before delivery, and test for biodiesel content, water content with Karl Fisher titration, particulate content against ISO 4406, and TAN (Total Acid Number). If the sample fails any test, inform your supplier and request fuel within the acceptable specifications.
The EPA states, “Minimizing water presence is and has always been an important part of UST maintenance. However, diesel blended with biodiesel can hold in solution more water than diesel without a biodiesel component. This means more water is likely arriving in USTs entrained (emulsified) in fuel today since biodiesel is more common in diesel than prior to 2007.” Fuel in the large distribution storage tanks of suppliers also absorbs entrained (emulsified) water, so even new fuel could already have substantial amounts of entrained (emulsified) water.
Install a permanent fuel polishing system on the storage tank if fuel fails any of the tests. The polishing system must use the new generation of filters specifically designed to remove entrained (emulsified) water to SAE J1488_201010 standards with of a minimum 92% efficiency.
Fuel should be filtered three times a week, and all the fuel filtered within 8 hours. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE, such as Southwest Research Institute.
Corrosion resulting from acidity caused by emulsified water dramatically increases the risk of an emergency generator failure and increases the costs of maintaining the data center diesel infrastructure. Corrosion affects storage tanks, emergency backup generators, diesel engines, fittings, and piping. Removing emulsified water from diesel fuel will control acidification and corrosion; drying the fuel controls microbial growth.
Fuel experts and data center operators first noticed corrosion problems shortly after the widespread adoption of ULSD. ULSD contains various additives (less than 1% of the fuel by volume) and fuel components (more than 1%), including biodiesel. These additives and fuel components absorb water which bonds to the diesel as emulsified water. The molecules of water, biodiesel, and other additives are all polar in nature. The emulsified water supports a bloom of microbes, and some of these microbes produce acids that corrode metal in their environment.
Before 2006, LSD (low sulfur diesel) fuel contained 500 parts per million (ppm) sulfur. ULSD (15 ppm sulfur) was introduced to help fight acid rain. While LSD was phased out of the market in 2010, a 2007 research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B.B. He, 2007) makes a useful point about how much water LSD and ULSD can absorb. He notes that 100% biodiesel absorbs 15 to 25 times more emulsified water than LSD. Neither LSD nor ULSD fuels are polar so they do not absorb much emulsified water.
In addition to lubrication concerns, biodiesel was added to ULSD in the U.S. for a variety of environmental and financial reasons, as well as government concerns about national dependence on foreign oil supplies.
However, the process of scrubbing the sulfur changed the chemistry of the fuel so ULSD provided less lubrication, resulting in immediate wear to diesel engines. An additive was introduced to ULSD to restore the needed lubrication. Soon after, biodiesel was also added as a fuel component, in ever increasing percentages, 2% (B2), 5%(B5), 7% (B7), 10% (B10), and now, 20% (B20). The more biodiesel, the more emulsified water. Biodiesel blend of 5% or less does not need to be disclosed to the end user.
Before ULSD, filters could remove all freestanding water and particulate, since LSD is not polar and there was very little emulsified water. Since both biodiesel and water are polar, water bonds tightly to the fuel as emulsified water. Pre-ULSD filters cannot efficiently remove the emulsified water from biodiesel blends.
Even before the U.S. Environmental Protection Agency (EPA) detailed this rapid and severe corrosion, NFPA 110 (2013) cautioned against the use of biodiesel for standby and prime engine-generator sets. In those instances, where the use of biodiesel was unavoidable, the NFPA required its replacement annually, but with “rapid and severe corrosion,” even 12 months is too long a time to store acidified fuel. NFPA recommends using diesel that has been stored that long for other purposes.
The EPA’s July 20th 2016 final report on corrosion states, “The major finding from our research is that moderate or severe corrosion on metal components in UST [underground storage tanks] systems storing diesel fuel in the United States could be a very common occurrence… variables that were the closest to being significant predictors [of corrosion] were particulates in the fuel and entrained (emulsified) water content in the fuel.” Drying the diesel by removing all the water, both free standing and emulsified, can control corrosion. This final report has worldwide significance because countries concerned about the issue have been tracking EPA findings.
A polar molecule
SAE J1488 tests “the ability of a fuel/water separator to separate emulsified or finely dispersed water from fuels.“ Well-known organizations such as Southwest Research Institute (SwRI) can test filters for compliance to the standard and provide reports to data center managers. In 2010, SAE updated the test from J1488_199708 to J1488_201010 to address the emulsified water concerns caused by ULSD and biodiesel blends.
SAE updated J1488 in 2010 to reflect the lower interfacial tension (IFT) in ULSD and biodiesel blends. IFT is the ability of two liquids to repel each other. Since biodiesel in hydroscopic, more water is absorbed and less repelled, so its IFT is lower. The IFT of ULSD is tested at 34.5 mN/m, biodiesel at 15 mN/m (B20). The 2010 update created a standard IFT against which all filters could be compared.
A new generation of filters designed specifically to filter for SAE J1488_201010 IFT levels is required to remove emulsified water, and hence control corrosion. In June 2016, the United Kingdom adopted SAE J1488_201010 as a code requirement.
Critical Fuel Applications
Modern backup generators are high-pressure common rail (HPCR) models, with pressures of 30,000+ pounds per square inch (psi) and plans for 60,000 psi models. Emulsified water and corrosion can cause these engine generator sets to become unreliable by
Excessive emulsified water causes injector pitting.
Acidic diesel fuel causes injector pitting and corrodes other generator parts it touches. Monthly engine generator testing adds to the problem as acidic fuel remains on metal components after each test.
Excessive particulates cause injector pitting.
Deposits are left on injectors, making them sticky.
Fuel goes out of ASTM specification and the generator cannot perform at peak design performance
Biomasses that form in biodiesel clog fuel filters
Biodiesel contains less energy than ULSD, so increasing biodiesel content reduces power output a small amount.
Biodiesel freezes at a higher point than ULSD, so this needs to be addressed.
When injector pitting and corrosion are great enough, the generator will not be able to hold the required HPCR pressures and will fail. In addition, the pitting of the injectors can cause the generator to become over fueled, increasing heat, emissions, and fuel consumption. The corrosive/pitting affects injectors and spoolers on all diesel engines. The higher the psi of the engine, the greater the risk of damage and failure.
Corrosion in storage tanks, fittings, and piping dramatically increases the risk of a backup generator failing. For example, a fuel supply pipe that fails or plugs will prevent a generator from working during an emergency.
Warranty Claims
Warrantees on diesel engine-generator sets exclude damages caused by fuel-related issues. That means that vendors can deny warranty claims if the fuel does not meet the specification set forth by the vendor. The maximum emulsified water content in most warrantees is 200 parts per million (ppm), which is well below the observed amounts commonly found in tanks with biodiesel blends. Caterpillar’s warranty specifies a pre-emulsified water/corrosion water and particulate maximum of 500 ppm. Manufacturers generally enforce this requirement by requiring a fuel sample with each warranty claim.
Caterpillar’s service document SEBU6251-14 from 2014 (before corrosion was commonly recognized) reads, “The repair of any engine or after treatment components due to such contamination or due to other biodiesel impacts would not be covered under the Cat warranty for materials and/or the warranty for workmanship.” Since corrosion is such a new issue, it means Cat’s warranty could be voided the first time biodiesel is used in the engine.
Similarly, Cummins warns about biodiesel. Cummins’s service document 3379001-13 reads, “Engine damage, service issues, and/or performance issues determined by Cummins Inc. to be caused by the use of biodiesel fuel not meeting the specifications outlined in this Service Bulletin are not considered to be defects in material or workmanship and are not covered under Cummins Inc. engine warranty.”
All the other OEM warrantees include similar statements.
Sidebar: Corrosion Management Action
Removing emulsified water from the fuel is key to maintaining the integrity of diesel fuel systems. SAE J1488_201010 tested filters can remove the emulsified water, thereby restricting microbial growth, acidification, and dramatically reducing corrosion. Filters need to achieve a performance level of greater than 92% to keep water below 200 parts per million (ppm).
Below is a short checklist to help address any fuel issues, with detailed explanations following.
Check with state regulations to confirm if biodiesel is mandated.
Check your records as to when the fuel was last replaced. Unfiltered older fuel will have more issues.
Create a maintenance routine for diesel fuel that will include bi annual fuel samples and tank inspections
Take fuel samples from main storage tanks, belly tanks, and day tanks
If the fuel fails the tests, drain and clean the tank, and install a fuel polishing system based on SAE J1488_201010
If the fuel is found to be acidic, drain and clean the tank and replace the fuel
Test a sample of the replacement fuel from the supplier
While the state may not mandate biodiesel, there are financial incentives to distributors to supply biodiesel blends. Note that suppliers are not required to disclose any biodiesel content of 5% or less, so the distributor may say there is no biodiesel content, when in fact there is.
Take fuel samples from all fuel tanks. The sample should be taken from the tank bottom. Test for biodiesel content, water content with Karl Fisher titration (less than 200 ppm), ISO 4406 particulate content (less than 18,16,13), and TAN (Total Acid Number, less than .08 maximum per World Fuel Charter).
It may be quicker and easier to sample the belly tanks or day tanks attached to the generators. If possible, have a visual inspection of the inside of your belly and day tanks. If there is any corrosion in these tanks, it will point to corrosion in the large storage tanks. All storage tanks are designed to handle the hydrostatic pressure (the weight of the fuel), which is greatest on the tank bottom. Since corrosion is also focused on the bottom of the tank, corrosion will pit of the tank wall, thereby thinning the wall, increasing the likely-hood of fuel leaks or tank failure. There are inexpensive fiberoptic cameras on long hoses available on the market to assist in the inspection.
If corrosion is seen, replace the fuel. Before adding new fuel, ask for a fuel sample from the suppliers distribution tank before delivery, and test for biodiesel content, water content with Karl Fisher titration, particulate content against ISO 4406, and TAN (Total Acid Number). If the sample fails any test, inform the supplier and request fuel within the acceptable specifications.
The EPA report states, “Minimizing water presence is and has always been an important part of UST maintenance. However, diesel blended with biodiesel can hold in solution more water than diesel without a biodiesel component. This means more water is likely arriving in USTs entrained in fuel today since biodiesel is more common in diesel than prior to 2007.” The large distribution storage tanks are also absorbing entrained (emulsified) water, so new fuel could already have substantial amounts of entrained (emulsified) water.
Observed corrosion (Battelle)
Install a permanent fuel polishing system on the storage tank if fuel tests show any failures in the tests. The polishing system must use the new generation of filters specifically designed to remove emulsified water to SAE J1488_201010 standards with of a minimum 92% efficiency. The fuel should be filtered 3 times a week, and filter the complete volume of fuel in an 8-hour period. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE such as Southwest Research Institute.
Fuel Experts
While most diesel consumers did not immediately recognize the issues with corrosion, the fuel experts did. Below is a timeline of recognition of acidification caused by microbial growth, starting with the most recent and working back.
U.S. EPA recognizes entrained (emulsified) water as a key indicator of corrosion. “This research focused on better understanding a type of rapid and severe corrosion of metal components in underground storage tanks (USTs) storing diesel fuel….EPA’s research determined that corrosion is very common; it appears all USTs storing diesel could be susceptible to developing corrosion.”
BS5410-3 made it code that any filters used on critical emergency backup systems conform to SAE J1488_201010.
BS5410-3 (British Standards Institution, 2016) is the equivalent of NFPA 110. The United Kingdom has used biodiesel blends longer than in North America. “7.2.2.5 Filters: NOTE 1 Since the introduction of Biofuel (FAME) [fatty-acid methyl ester] into fuels the quality and life expectancy of fuels has been adversely affected as FAME is hygroscopic so any water in the fuel goes into suspension….Removing as much water as possible preserves the quality of the fuel and for critical standby generators only filters conforming to SAE J1488_201010 [N5] are to be used”
NFPA 110 cautions against the use of biodiesel. “A.5.1.1(1) Where possible, the purchaser of fuel for the prime mover should specify a diesel fuel that does not contain biodiesel, which can accelerate the degradation of the diesel fuel if stored longer than 6 months.”
“A.5.5.3 More important, biodiesel blends up to B5 (ASTM D 975, Standard Specification for Diesel Fuel Oils) have much shorter shelf lives than conventional diesel fuel [ultra-low sulfur diesel (ULSD)] and can accelerate degradation processes, endangering the entire diesel fuel supply. Where fuel is stored for extended periods of time (e.g., more than 12 months), it is recommended that fuels be periodically pumped out and used in other services and replaced with fresh fuel.”
The Steel Tank Institute (STI) reinforces the importance of keeping storage tanks free of water. “In a 2012 study by Battelle Labs, five of six FRP (fiber- glass reinforced plastic) tanks showed severely accelerated corrosion. The sixth tank was supposedly “clean” and intended to be the control for the study, but turned out to have corrosion, as well….In ULSD, the presence of water provides an environment where microbial life can thrive, possible influencing the corrosion of internal metal components of the UST system equipment. “
SAE updates J1488_199708 to J1488_201010 to test the efficiency of filters to remove emulsified water from the newly introduced ULSD and biodiesel blends (B20). A 92% efficiency can keep water below 200 ppm, which is the OEM manufacturers’ warranty limit. In SAE J1488_201010 testing, ULSD and biodiesel (B20) are saturated with 2500 ppm of emulsified water, which is then filtered. The water content of the filtered fuel is compared to the water content of the original saturated fuel. SAE then assigns an efficiency rating to the filter and reports its findings to the filter supplier. (SAE Technical Standards Board, October 2010).
Battelle Memorial Institute calls attention to the problem of corrosion. “Severe and rapid corrosion has been observed in systems storing and dispensing ultra low sulfur diesel (ULSD) since 2007” (Battelle Memorial Institute, 2012)
The research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B. B. He, 2007), compares the water absorption of 100% LSD to 100% biodiesel.
IBM Data Center Group Case Study
IBM has a group within the Global Technology Services division that designs, builds, migrates, and upgrades data centers for its customers. It reports that the majority of data centers in Canada have been in the Province of Ontario, which mandated the use of biodiesel with an increasing concentration, initially B2, and B4 at time of writing.
IBM was initially concerned about the reliable operation of the generators at these facilities, especially because data centers often store fuel is stored for long periods of time, unlike road or construction applications. In particular, IBM worried about the effect of the emulsified water on its engine-generator set infrastructure. Reduced engine-generator set life and increased repair costs were major concerns.
Over time, it became clear to IBM that the entire fuel storage and delivery system, metallic and non-metallic components included, is at risk due to the corrosive properties of degraded biodiesel. A major excavation required to repair or replace would be a major concern.
Fuel filters tested in systems using B2 pass the “bright and clear” test, which wasn’t deemed rigorous enough to give a true indication of the condition of biodiesel fuel. IBM has been offering their clients fuel-conditioning systems that meet or exceed a more applicable standard from the SAE J1488_201010.
Three Biodiesel Related Failures in One Day
Franklin Company in the NYC area responded to three separate fuel-related diesel generator failures in one day at three different organizations. Problems caused by ULSD can occur in many application types. Failure 3 is typical of failures at a very common type of facility. These failures give a good representation of some of the failure points data center users can experience: corrosion of tanks, failure of pipes and fittings, and biomass plugging filters.
Failure 1: Franklin Company responded to a service call with a reported interstitial alarm. A service tech removed the sensor and found a mixture of water and fuel on the sensor. The tech removed a small amount of fuel and water mixture and then reinstalled the sensor. The sensor did not go into alarm. Franklin initially thought that recent rains had worked its way past the interstitial cap.
A few weeks later, Franklin received a new service call with the same complaint. This time the tech found a significant amount of water and some fuel in the interstitial. Since the tank had not been showing any losses, Franklin thought that the outer wall was leaking groundwater and fuel into the annular space from outside. It would not be so unusual for a site to have contamination. The interstitial failed a vacuum test. The primary tank passed a second test was to confirm that it was tight. A spill was called in to the DEC for a non-tight tank system. Franklin Company contracted a third party to perform the same tests, with the same result.
As one result, Franklin excavated and checked the tank top for breaches or leaking bungs to the interstitial. All fuel was removed, the manway removed, and a confined space team entered and began cleaning a very dirty tank. The team found the steel tank had many holes on the bottom.
What happened with the tests? The annular was leaking, but only between the primary and the secondary. Tests on the primary showed “tight,” but we were really testing only the secondary tank, as the primary tank was open to the secondary. The larger amounts of water in the secondary was caused by the head pressure pushing the tank bottoms out of the bottom of the primary. The annular space on an ACT 100 is very small, so a small amount of water would migrate to the riser, showing a disproportionate amount of water versus fuel. When the tech removed water, it would look dry for an extended period of time until more water and fuel would be forced through the bottom of the tank and work its way to the annular riser.
The tank was not salvageable and had to be removed. There was no leakage to the environment as the secondary was intact and tight. Note the tank was 10 years old.
Failure 2: An underground UL 971 flexible pipe failed when B20 diesel caused the internal lining of the pipe to delaminate. The contractor said, “The pipe looked like a long sheet of white plastic wrap when we investigated the blockage.”
Failure 3: A government agency reported a slow flow of diesel fuel dispensers. A service technician dispatched to the site found that two dispensers had flow rates of only 2-3 gallons per minute. Two dispensers had no flow at all. The internal filters were removed and replaced, and the pumping unit strainers (suction style units) were removed and cleaned and then re-installed. The facility was down for two days. In this case, microbial “blooms” and excess particulates had created biofilms that plugged the filters.
Biofilm, as shown in the picture below, is a problem that can lead to the failures stated above. Biofilms concentrate the corrosive acidic effects of the fuel (Failure 1), can peel off in strips to plug pipes (Failure 2) and can disintegrate, overloading filters (Failure 3). The contact said, “WOW. Never saw anything like this. This was the consistency of boiled pig skin. It was lining the bottom of the tank. This is a tank that we could not rinse in situ. Super bugs.”
Picture Below, Biofilm from bottom of tank
Suggested Readings
Chuanfang Yang, S. L. (2007). Understanding emulsified water filtration from diesel fuels . San Antonio: 8th International Filtration Conference.
Groysman, A. (n.d.). Corrosion in Systems for Storage and Transportation of Petroleum Products and Biofuels Identification, Monitoring, and Solutions. New York, Philadelphia: Springer Science & Business Media.
PEI Petroleum Equipment Industry. (2013). Recommended Practices for Installation of Bulk Storage Plants (PEI/RP800-13). Petroleum Equipment Industry.
R.L. McCormick, T. A. (2005). Survey of the Quality and Stability of Biodiesel and Biodiesel Blends in the United States in 2004 . National Renewable Energy Laboratory, Golden, Colorado.
Steel Tank Institute. (2016). Recommended Practice For Storage Tank Maintenance R111 Revision . Steel Tank Institute , Lake Zurich, IL.
ASTSWMO Tanks Subcommittee’s Alternative Fuels Workgroup. (2013). Compatibility of UST Systems with Biofuels. Association of State and Territorial Solid Waste Management Officials (ASTSWMO), Washington, DC.
B. He, J. C. (2007). Moisture Absorption In Biodiesel and Its Petro-Diesel Blends. Food & Process Engineering Institute Division of ASABE. Moscow, Idaho: American Society of Agricultural and Biological Engineers ISSN 0883 8542.
Battelle Memorial Institute. (2012). Corrosion in Systems Storing and Dispensing Ultra Low Sulfur Diesel (ULSD), Hypotheses Investigation. Battelle Memorial Institute, Columbus, OH.
British Standards Institution. (2016). Code of practice for oil firing Part 3: Installations for furnaces, kilns, ovens, oil-fuelled standby generators and other industrial purposes. BSI Standards Limited.
NFPA110. (2013). NFPA 110 Standard for Emergency and Standby Power Systems. Quincy, MA.
Steel Tank Institute STI. (2013). Newer fuels and storage tank corrosion. Steel Tank Institute STI, Lake Zurich IL.
U.S. Environmental Protection Agency (July 2016). Investigation Of Corrosion-Influencing Factors In Underground Storage Tanks With Diesel Service. United States Environmental Protection Agency, U.S. Environmental Protection Agency Office of Underground Storage Tanks, Washington DC.
Uptime Institute. (2010). Uptime Institute Technical Paper: Biodiesel. New York.
Bernard Oegema
Bernard Oegema PE Data Centre Consultant, IBM Global Services, Data Centre Services. He has been in the critical facilities business for more than 30 years, working with various manufacturers of critical power and cooling infrastructure. Currently he is supporting IBM clients in the migration, consolidation, operation, upgrade, design and building of data centers.
Pat Smyth
Pat Smyth is Vice President of DieselPure. DieselPure is focused on fuel polishing for emergency backup generator storage. Since 2011, DieselPure has field experience in removing emulsified water from ULSD and biodiesel as per SAE J1488_201010.
Brian Ponstein
Brian Ponstein is Regional Sales Engineer, MTU Onsite Energy, a Rolls-Royce Company. At MTU, Mr. Ponstein is responsible for analyzing market needs and requirements in North America and working with engineering to provide solutions for MTU Onsite Energy’s customers.
Bill Klein
William (Bill) Klein is President of Franklin Company Contractors. He is also a New York City (NYC) Master Electrician, NYC Licensed Plumber, NYC Licensed Pump and Tank Contractor, NYC Licensed Precision Tank Tester, member of PEI, and past president of NYC Tank Installers. Mr. Klein is the third generation president of Franklin Company.
Martin Weselowski
Martin Wesolowski is Senior Associate Vice President at LiRo’s petroleum and chemical storage tank design group. He has been involved with design of petroleum and chemical UST and AST tank and remediation systems for over 23 years. Mr. Wesolowski is a technical director on LiRo’s New York City Department of Buildings (NYCDOB) Special Inspection Agency Registration.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/02/smyth.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-03-14 12:29:012017-03-16 11:29:14Reconsider Your Diesel Fuel Supply
David Meredith says that CenturyLink’s involvement, including the M&O Stamp of Approval, with Uptime Institute provides mutual benefits. Uptime Institute’s Risk Journal for IT Infrastructure is the most recent example.
CenturyLink’s commitment to operational excellence has become one of the company’s competitive advantages, and David Meredith, a senior VP at CenturyLink, has been an enthusiastic proponent of demonstrating this commitment through third-party verifications. In 2014, CenturyLink decided to pursue Uptime Institute’s Management and Operations (M&O) Stamp of Approval for all 57 of its data centers.
So Uptime Institute was delighted to see Meredith’s comments in LinkedIn about our new publication “Risk Management for IT Infrastructure.” Meredith quoted the book’s central tenet, “The handbook also argues that effective governance requires industry certifications.” The journal provides guidance on avoiding data center capital project failure and applying efficient IT principles to address sustainability risks and IT resilience during a natural disaster.
Meredith writes, “Julian Kudritzki and Matt Stansberry [the book’s editors] point out that, when the M&O Stamp of Approval is administered across an entire portfolio, it ensures consistency. CenturyLink is a perfect example of utilizing Uptime’s data center operations guidance. As the authors point out, “CenturyLink’s commitment to achieve M&O Stamp of Approval across the board shows it is willing to hold itself to a uniform set of high standards and operate with transparency.”
Meredith and CenturyLink have previously contributed to Uptime Institute publications. Please click on the following hyperlinks for some recent examples:
https://journal.uptimeinstitute.com/wp-content/uploads/2017/03/risk.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-03-09 14:25:582017-03-10 08:40:25CenturyLink Benefits from Its Commitment to Excellence
Luxconnect obtains different Tier Certifications to meet the needs of the changing market demands
By Christine De Ridder
LuxConnect, a multi-tenant, multi-tier data center and dark fiber network operator based in Luxembourg, developed an innovative strategy using Uptime Institute Tiers to differentiate the services and pricing it offers to key customer groups. In doing so, LuxConnect became the first—and so far only—facility to offer multiple Tier levels in the same data center.
The exterior of DC1.3, Luxconnect’s green data center.
“LuxConnect’s multi-tier approach to Tier Certification enables it to offer customers different rooms with different Tier Objectives in the same building.” Said Phil Collerton, managing director, Uptime Institute EMEA.” This gives the client the opportunity to use different rooms to house different types of applications, both mission critical and noncritical, while also benefiting from the flexible SLAs and pricing models that this strategy allows LuxConnect to offer.”
A Tier IV Certified Constructed Facility server room ready to host servers.
LuxConnect is a private limited company owned by the Luxembourg State and founded in 2006 with several missions such as strengthening the Luxembourg’s position on the European Internet map and providing state-of-the-art IT environment.
Roger Lampach, CEO, said, “Among our goals was to promote and facilitate high-end ICT investments into Luxembourg and offer white space in high-quality data centers. And that is what we achieved.”
Today LuxConnect has two sites and four buildings with a total server space of 14,700 square meters. LuxConnect has three independent data centres, DC1.1, DC1.2 and DC1.3, on its Bettembourg ICT Campus in the south of the country. The DC2 facility is located in Bissen. The two sites are interconnected via multiple routes using dark fiber to guarantee customers data center business continuity.
Luxconnect’s low-voltage distribution room.
LuxConnect designed, planned, and built these data centers in less than 10 years. The newest facility, DC1.3, launched in August 2015. Three of the four buildings have been Certified by the Uptime Institute, including Tier IV Certification of Design Documents for DC1.1 and Tier II Certification of Design Documents in DC2. DC1.3 has Tier II and IV Certifications of Design Documents and Constructed Facility in the same facility, which is a unique configuration.
Cold water distribution
Lampach said, “We know that we have to convince our visitors and potential customers that we are professional and that we know what to do and how to do it. Our visitors are usually very impressed by the quality and high level of our five-star data centers.”
LuxConnect’s mix of Tier Certifications is an integral part of meeting its customer’s demands. Having multiple Tier II facilities enables LuxConnect to attract gamers who require high availability and low latency but are very cost sensitive. Other verticals, such as banks, require the Fault Tolerant Tier IV Certification of Constructed Facilities.
Luxconnect received Tier II and Tier IV Certifications for spaces in the same facility, making it unique.
With support from the government, LuxConnect is able to expand as soon as market demand appears. Lampach said, “We start building when we see that there is not enough capacity in the market and without having any contracts. It’s not like a profit-driven company would do, but as our primary shareholder wants outside companies to come at any moment, we have to anticipate.”
The cornerstone of this approach is DC 1.3, which has both Tier II and Tier IV Certifications of Constructed Facility on the same floor in the same building, which is a unique approach.
The Tier II spaces are mostly for the gamers, according to Lampach, “The gamers have two installations, one called front end where the gaming company connects and the other one where our client analyzes customer behavior and determines which new products to present their customers. These companies pushed us to consider offering Tier II spaces. And we began the idea by achieving Tier II Certification of Design Documents in DC2.”
Uptime Institute presents Tier Certifications to Luxconnect.
From the beginning, LuxConnect designed its data centers following Uptime Institute’s Tier standards and specifications. The company’s two project managers are Uptime Institute Accredited Tier Designers. Lampach said: “Having the Tier Certifications has been a real benefit for us. It’s also a concern for us because we go to the market through our business partners. At the beginning, they don’t really understand the differences between Tier II, III, and IV and they think downtime will never happen or they decide to take the risk.”
Lampach feels that the Tier Certification of Constructed Facility process provided the project team with an opportunity to better understand the behavior of the different systems, which helps them to assure business continuity. The Operations staff also benefited from the Tier Certification process with a very intensive workweek that was instructive on the technical and human level.
“At the end of the week, Operations could really see how the electrical and mechanical systems reacted during the multiple demonstrations. They feel more comfortable working with the systems because they know exactly how to proceed for maintenance without impacting business continuity.” The staff was also able to witness how the systems react during different failure scenarios.
Rich Van Loo, Uptime Institute’s senior vice president, Facility Management Services, said, “The operations assessments helped to insure LuxConnect gets the most availability out of each data center regardless of Tier while still allowing flexibility to its customers. It positions them to be a true leader for IT services in the region.
LuxConnect is justifiably proud of its performance during the Certification process, as it had just two minor issues, both of which the company was able to resolve during the testing week. The minor nature of these changes testifies to the quality of LuxConnect’s design, construction, and commissioning processes.
Christine De Ridder
Christine De Ridder worked for several companies, including Schneider Electric and Siemens, before joining LuxConnect in 2012 as manager, Data Centre Projects. She became an Accredited Tier Designer in 2013. In addition, she speaks five languages and has an engineering degree in electromechanics.
Tier Certification of Operational Sustainability enhances ENTEL’s services to customers
By Kevin Heslin
ENTEL began operations in 1964 as a provider of national and international long distance telephone services to companies in Chile. Today it is a consolidated provider of integrated telecommunication services and information technologies services, meeting the needs of corporations and large companies through tailored solutions, providing value, experience, and quality of service.
The company began operations in the aftermath of an earthquake that severely damaged the Chilean telecommunications network. More recently it added more services. Towards the end of 1995, ENTEL started to provide Internet connection services and in 1997 it introduced the first commercial network with ATM technology in Latin America, which evolved into the current Multiservice IP network in order to offer broadband solutions, guarantee quality of service, and add value to its clients. Today ENTEL offers Information Technology (IT) services, which vary according to the industrial segment and business model of each client, allowing competitive efficiencies and advantages that differentiate ENTEL from traditional telecommunications services.
As part of its offerings, ENTEL also offers a number of data center services to more than 3,000 clients. It owns and operates a network of five data centers that are certified under ISO 9001-2000 and ISO 27001 standards. In addition, two of them are Tier III Gold Certified Constructed Facility and one has the M&O Stamp of Approval.
Data Center Infrastructure
ENTEL has five data centers located in Amunátegui, Pedro de Valdivia, Ñuñoa, Longovilo, and Ciudad De Los Valles that are linked through high availability and high capacity IP/MPLS/DWDM These data centers have in excess of 7,500 square meters (m2) of data hall space with plans to expand to 11,675 m2. From these data centers, ENTEL offers IT outsourcing services, from server hosting to more complex services involving operation and exploitation of the platforms that support clients’ business processes. These data center services meet the needs of companies’ most vital applications, improving security and protection of critical data and reducing considerably their infrastructure investments. ENTEL’s IT strategy is based on traditional data center services and growth in cloud services and permanent innovation.
ENTEL first offered data center services in 2003 and is now Chile’s largest provider. To this end, ENTEL has staffed its facilities with more than 120 professionals and management devoted to implementation and operation of data center Infrastructure projects
Thanks to its team of specialists with extensive experience, the IT challenges faced by ENTEL have been met entirely in-house. Of particular note are the Ciudad de los Valles 1 and 2 facilities, each offering 2,000 m2 of white space. Ciudad de los Valles 1 (CDLV1) entered production in May 2010, and Ciudad de los Valles 2 (CDLV2) entered production in March 2013 (see The Uptime Institute Journal, vol. 2, p. 64). Both are Tier III Gold Certified Constructed Facilities (see Figure 1). Both received Tier Certification for Operational Sustainability in October 2015 (see Table 1). Josué Ramírez, Uptime Institute, director of Business Development LATAM, “With these Certifications, ENTEL shows it commitment to seeking excellence in its operations to provide better services to its clients and to contribute to the development of knowledge and professionalism in the region of Latin America.”
Figure 1. CDLV 1 and 2 are both Tier III Gold Certified Constructed Facilities.
Rich Van Loo, Uptime Institute, VP Facility Management Services, said, “The Tier Certification and Operational assessments have had a broad impact on not only the data center management divisions, but the company environment as a whole. Operational procedures not only reduce risk but help improve consistency and efficiency in those operations. ENTEL is looking to expand this philosophy to all their data centers.“
Table 1
ENTEL began construction of a third facility in 2015 at Ciudad de los Valles. This will add an additional 2,000 m2 of white space. The new facility incorporates free-cooling technologies that achieve better energy efficiency and lower total costs of ownership for clients.
Tier Certification of Operational Sustainability
To differentiate itself from its competitors, ENTEL became the first to Tier Certify its facilities. As a result, ENTEL decided to further differentiate itself by earning Uptime Institute’s Tier Certification of Operational Sustainability for CDLV 1 and 2. This decision was based on client demand for excellence of service, risk mitigation, and assurance of operational continuity.
ENTEL believes that
Risk management should be approached as a team, leaving no room for assumptions or improvisation
Planning and later exhaustive review of activities should have an integral perspective, which allows risk to be controlled, mitigated, and contained
Planning activities and teamwork generate a virtuous circle of shared knowledge and learning
Empirical verification that continuous improvement is key to mitigating human error in Operations
Figure 2. The Tier Certification of Operational Sustainability verifies that ENTEL maximizes the potential of its facilities and differentiates it from its competitors.
The Tier Certification of Operational Sustainability (TCOS) was an excellent way to reach these goals and why a new internal organization was dedicated to defining data center operations activities and a second team was tasked to leading the effort to earn the Tier Certification of Operational Sustainability.
In addition, ENTEL created an Infrastructure Change Control Board (CCI), specific to Datacenter Infrastructure Management, with the purpose of establishing planning and control of activities in each data center. This Board is a consulting body. It meets periodically and manages, reviews, and approves infrastructure management projects to data centers.
Scope of the Infrastructure Change Control Board (CCI)
Although ENTEL had previously adopted the ITIL model and established a Change Advisory Board (CAB) that validates and approves all activities in the IT and data center infrastructure environment, it was determined to create the CCI because of the degree of specialization of infrastructure work and the risk associated with it. Its focus is to raise risk points associated to high-impact activities and ascertain control points, e.g. cells, transformers, generators, UPS, towers and chillers, where the system redundancy could be lost.
A summary of the type of work evaluated and documentation to be reviewed follows (see Table 2):
Table 2.
The CCI is intended to ensure that the good practices characteristic of Tier Certification of Operational Sustainability are respected and that all documentation and instructions are kept thoroughly valid, that they are applied, and are subject to continuous improvement.
The Process
In order to increase ENTEL’s familiarity with the Tier Certification of Operational Sustainability process, ENTEL and Uptime Institute agreed to carry it the process in four phases (see Figure 3).
Figure 3. ENTEL views operational sustainability as a four-phase continuous process.
Important Decisions
Eventually, Uptime Institute and ENTEL adjusted the process in light of the initial progress, the maturity level of the team, and the imperative work of assuring operational continuity:
The Operations team was split in two groups, both reporting to a single manager in charge of coordinating activities
Operation, maintenance and tests teams were responsible for day-to-day activities
The Operations team was also entrusted with adjusting existing procedures/instructions, for training the specialist staff, and with leading the internal change within the datacenter.
ENTEL also
Retained an external consultant to structure its methodology and develop a map for sustainable training.
Implemented a maintenance management system
Defined a support team to manage eventual setbacks for weekly follow-up, progress control, and resolution
Assured that activities the processes, procedures/instructions, and methodologies Operational Sustainability would be adopted and applied
ENTEL believes that it received many benefits from earning the Tier Certification of Operational Sustainability, including
Improved “standard operation” processes
Continuous improvement of processes and procedures
Formal training, sustainable over time
Shared lessons learned at all five data centers
The knowledge of the Certification methodology and good practices stayed in house, which ENTEL can replicate at all its data centers and eventually to other areas of the organization (see Figure 4). ENTEL has worked to share this information company wide, with the additional notable achievement of Uptime Institute’s M&O Stamp of Approval at its Amunategui facility in December 2016.
Figure 4. The staff at ENTEL is charged with promoting Operational Sustainability across the whole organization.
https://journal.uptimeinstitute.com/wp-content/uploads/2017/02/UI2.jpg5941500Kevin Heslinhttps://journal.uptimeinstitute.com/wp-content/uploads/2022/12/uptime-institute-logo-r_240x88_v2023-with-space.pngKevin Heslin2017-02-17 14:08:202017-02-22 11:19:33ENTEL Achieves Uptime Institute Tier Certification of Operational Sustainability
Japan’s Biggest Provider of Critical IT Services Earns M&O Stamp of Approval
/in Executive, Operations/by Kevin HeslinNearly perfect M&O Stamp of Approval scores prove that NRI’s data centers operations are world class
By Kevin Heslin
Nomura Research Institute, Ltd. (NRI), a ¥18 billion ($US169 million) company that provides consulting, financial IT solutions, industrial IT solutions, and IT platform services mainly to a global financial marketplace, has earned Uptime Institute Management and Operations (M&O) Stamps of Approval at two of its five facilities, receiving very high scores on both. In the process, NRI, Japan’s largest provider of critical IT services, impressed even Uptime Institute consultants, who have assessed many of the world’s best-run data centers.
Uptime Institute Consultant Steve Burgess said, “A very high M&O score translates to a quality of operations that is very proactively mitigating risk, something premium financial services companies must aggressively pursue and maintain.” Managing risk in that way is critical for NRI because of its important role in the global economy. Uptime Institute spoke to Takuhito Sakata, NRI’s managing director, Data Center Service Division.
NRI’s Tokyo 1 data center
Tell me about NRI’s operations.
TS: More than 300 financial and industry systems operate in our data centers. NRI provides the back office system that cover general securities business processes and 50% of the Tokyo Stock Exchange’s transactions are transmitted through our STAR financial back office system.
Other than financial systems, NRI operates a wide range of mission critical systems for banks, insurance companies, and many companies in other industries. Our systems have become Japan’s core infrastructure, so the cost of losing them in a disaster would be immeasurable. Without exception, our data centers are built for high capacity and availability, and NRI owns and operates the safest data centers in Japan.
Layout of NRI’s Tokyo 1 Data Center .
Please describe NRI’s data centers.
TS: NRI owns five data centers in Tokyo and Osaka. Yokohama Data Center 1(Y1DC) is the oldest of these facilities (1990), and Osaka Data Center 2 (O2DC) is the newest (2016). Tokyo Data Center 1 (T1DC) has the greatest power density (maximum 30 kW/rack). These and other differences in size, density, age, and location mean that the facilities have significant differences in operation that must be managed and supported.
The Osaka 2 data center opened just last year.
These five facilities support all our financial systems, and they align to the Payment Card Industry Data Security Standard (PCI DSS) and Sarbanes Oxley (SOX). These facilities are interconnected to provide mutual backup to each other. In fact, even the operation centers backup mutually. If one or more of the data centers are damaged, the systems are run in a remote data center and the operations also transfer to the backup. We call this concept Multi Data Center Operation (MDC).
A look at NRI’s double-deck system.
We believe that T1DC is state of the art. This facility earned an Uptime Institute Brill Award in 2015 for its patented double deck system. The double deck system separates server areas from facility areas, which helps us provide efficient power and cooling. Not only that, but the double deck system allows us to enhance our physical security and increases our structural integrity, which is important because of the risk of earthquakes in Japan.
These facilities benefit from different power utility supplies, and have redundant electric power system, including engine-generators sets, uninterruptible power systems, and standby transfer switches. The system is configured to backup the facility cooling. These and other arrangements, including a priority fuel supply contract, have enabled us to survive the Great East Japan Earthquake and the planned blackouts that followed without interruption.
Because of the ongoing risk of earthquakes, all the facilities are isolated seismically. The building is built on a huge seismic isolation rubber base and does not touch the ground directly. The building also includes a seismic damper, which is a brace that is structured to withstand shaking, so there will be no impact to IT equipment.
NRI deploys an isolation system and a bracing system to protect against potential earthquake damage.
What motivated NRI to obtain the M&O Stamp?
TS: We have been providing data center services for over 30 years, mainly to financial customers, and our service quality is now the best in Japan. We work hard to meet our customers’ rapidly changing business needs, and changing regulations and qualifications. We think we can improve our quality even further by third-party verification of our efforts.
We spent one year researching all the data center standards and certifications globally, which led us to Uptime Institute M&O Stamp of Approval. At that first (2014), no Japanese company had received the Stamp of Approval. So, we visited a number of data center sites in the U.S. that had the Stamp of Approval. We asked the owners and staffs at these sites whether the Stamp of Approval was effective and how it improved their quality of facility management. When we were convinced, we decided to submit for an M&O assessment.
How does the M&O Stamp benefit NRI?
TS: It sure helps increase the quality of the data center operation. The assessment program provides insights and advice that is shared by our five data centers, and helps us improve our operations knowledge at all of them.
NRI requires the participation of two professionals as part of many of its procedures.
When we re-assessed T1DC, we received 98.5 points (out of 100), an improvement of 1 point from the initial assessment. We are aiming for a perfect score, and the advice we receive from Uptime Institute consultants is very useful. And each time we re-assess we must show improvement as the program ratchets up the requirements.
Adding to that, M&O is also useful for creating awareness about the high quality of our data center operations. Especially in Japan, NRI received a lot of attention from the market and the media for being first to have Stamp of Approval and that we had a high score. And this awareness makes a big difference to customers touring our data centers. It is a matter of competitiveness. Some companies demand a high standard of operational readiness.
M&O is also helping us on global projects. Our data center design support service was the main project area so far, but we are being called on for our data center management consulting.
How did you prepare for the assessment process?
TS: To be honest, we didn’t prepare. We are confident of our daily regular maintenance, training, preventative checks, and routines. We wanted to evaluate our operations as they were, so we did not consciously prepare for the assessment. Instead, we viewed the results of each day’s work as a form of validation when we did well or opportunity to improve.
Of course, we knew the M&O criteria in advance, so we self-checked based on that, and we decided to go through the assessment process. For example, the M&O’s job definition form was unfamiliar to us, so we reviewed our standard to comply with global requirements.
Who was involved in the process? How did the staff benefit?
TS: We have a lot of in-house expertise, including facility management, security, and quality teams etc.). Based on these opinions, I made the decision to go ahead with the M&O. NRI created a cross-functional task force from this group to get the Stamp of Approval. As a result of M&O, our staff has become very confident about day-to-day operations. We learned about the importance of continuous improvement.
Do you find something new to apply after each assessment?
TS: That’s right. Currently we are having two data centers evaluated. The two data centers were built at different times, serve different customers, and are operated by different people. So we think it is natural that we get different results at each facilities. Those findings and advice for improvement are very helpful for us.
What are your plans for renewing the M&Os? As you know they expire every 2 years.
TS: We plan to have every data center recertified. Because conditions may change over time, we think that it is important to check on the data center every 2 years. Facilities and equipment age and staff changes so procedures may become dated or are ignored.
T1DC underwent a renewal review [Editor’s note: mandatory after 2 years] at the end of last year, and it is doing well.
What are your plans for further data center expansion? Will the M&O help with that process in any way?
TS: Global is a keyword for our company strategy, and the M&O Stamp of Approval combined with our proven track record will be part of our global business development. For this reason we plan to have the M&O Stamp of Approval at all our next-generation core data centers.
Highlights from Uptime Institute Symposium: Shanghai
/in Events, Executive/by Matt StansberryThis week, Uptime Institute hosted its first major event in China, Uptime Institute Symposium: Shanghai. Over 400 invite-only attendees joined Uptime Institute for a day of collaborative learning at the Grand Hyatt. The photos below illustrate some highlights from the the day.
Uptime Institute President Lee Kirby delivers the welcome keynote at Symposium: Shanghai. “The digital economy is interconnected, and you don’t want your data center to be the weak link. We need to industrialize our digital infrastructure through standards and third party assessments.”
Uptime Institute has tripled its business in China over the 5 years, with 18 Tier-certified data centers and 13 M&O data centers in China, with over 30 projects in process.
Uptime Institute Managing Director North Asia, Philip Hu discusses the process of Tier Certification. “The vast majority of even the world’s most elite data center sites do not operate as designed on day one. Tier Certification is a failsafe against a data center that doesn’t work.”
Uptime Institute Senior Director of Content and Publications Matt Stansberry unveils the results of the 2017 Data Center Industry Survey including responses from over 1500 end users worldwide.
Uptime Institute has conducted its annual survey globally for seven years. This year’s survey data included the first breakout of respondents specifically in China.
Attendees included Uptime Institute Tier Certification owners and M&O Stamp of Approval Awardees from China’s leading financial, colocation and telecom organizations.
Jack You, Cloud Architect for Huawei delivers the Diamond level Sponsor Keynote.
Uptime Institute President Lee Kirby presents Wang Hui, Director, China Carrier Service, Solutions Sales Department at Huawei with a letter of appreciation for Huawei’s commitment to Uptime Institute Symposium around the globe.
Philip Hu moderates a discussion of new technology adoption. Panelists: John Zheng, Solution Architect Director, Schneider Electric; Wu Ganxing
Vice General Manager, China Building Tech Group, Data Center; Biao Leng, General Manager, Beijing Travelsky Birun Technology
Pitt Turner, Uptime Institute Network Global Executive discusses the challenges with DCIM procurement: The process takes too long; Software costs too much; Tools under-deliver; Many buyers are not achieving ROI on the purchase.
Agatha Poon, Research Director APAC at 451 Research discusses the rapid growth and adoption of cloud computing providers in China and around the world, providing cloud computing adoption statistics, key trends in the market and the competitive landscape in the region and globally.
Transforming enterprises are serious about incorporating cloud models into their business strategies. Workloads are already business-critical in the cloud.
Lee Kirby: Complex IT systems are prone to cascading failure. But calling these failures human error is a mistake. The following are management failures, not human error: Lack of staffing; Lack of training; Lack of resources; Lack of preventive maintenance.
Over 65% of enterprise IT departments use Uptime Institute’s Tier Certification and M&O Stamp of Approval as part of the vetting process for considering potential IT service providers, according to the 2016 Data Center Industry Survey.
451 Research Director Andy Lawrence: The data center in 2020 will need to be agile — reducing costs, building on demand, responding to rapid deployment needs, and innovating without service disruption.
At the end of the each Symposium Event, an expert panel asks the audience — what are you going to do differently on Monday? How will you implement what you have learned today? Speaking: Danny Cheng, Chairman & CEO, Banyano
Stay tuned to the Symposium website for updates on our events in North America (Las Vegas, September 18) and Europe (Barcelona, Dec 5-6) in 2017.
Reconsider Your Diesel Fuel Supply
/in Operations/by Kevin HeslinNew diesel formulations put your generators at risk
By Bernard Oegema, Pat Smyth, Brian Ponstein, William “Bill” Klein, and Martin Wesolowski
The increased use of biodiesel has increased the risk to the diesel infrastructure of data centers and other facilities first noted with the adoption of ultralow sulfur diesel (ULSD) in 2006. The U.S. Environmental Protection Agency (EPA) documented severe and rapid corrosion in the storage tank portion of the diesel infrastructure caused by changes to the chemical composition of diesel fuel mandated by the U.S., U.K., and European Community. At the same time, engine-generator OEMs noticed an increase in warranty claims on their systems, including injectors. Solving this problem will require a new generation of filters tested against the updated industry standard, Society of Automotive Engineers SAE J1488_201010, to remove emulsified water.
The following steps are recommended to minimize or eliminate potential impacts from the use of biodiesel (see the sidebar for more detail):
It is important that all U.S. operators check their fuel supplies, because suppliers are not required to disclose biodiesel content of 5% or less. So a distributor may say there is no biodiesel content, when in fact there is. In addition, while a state may not have mandated the use of biodiesel, distributors have financial incentives to supply biodiesel blends.
Testing Procedures
Fuel should be filtered three times a week, and all the fuel filtered within 8 hours. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE, such as Southwest Research Institute.
Corrosion resulting from acidity caused by emulsified water dramatically increases the risk of an emergency generator failure and increases the costs of maintaining the data center diesel infrastructure. Corrosion affects storage tanks, emergency backup generators, diesel engines, fittings, and piping. Removing emulsified water from diesel fuel will control acidification and corrosion; drying the fuel controls microbial growth.
Fuel experts and data center operators first noticed corrosion problems shortly after the widespread adoption of ULSD. ULSD contains various additives (less than 1% of the fuel by volume) and fuel components (more than 1%), including biodiesel. These additives and fuel components absorb water which bonds to the diesel as emulsified water. The molecules of water, biodiesel, and other additives are all polar in nature. The emulsified water supports a bloom of microbes, and some of these microbes produce acids that corrode metal in their environment.
Before 2006, LSD (low sulfur diesel) fuel contained 500 parts per million (ppm) sulfur. ULSD (15 ppm sulfur) was introduced to help fight acid rain. While LSD was phased out of the market in 2010, a 2007 research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B.B. He, 2007) makes a useful point about how much water LSD and ULSD can absorb. He notes that 100% biodiesel absorbs 15 to 25 times more emulsified water than LSD. Neither LSD nor ULSD fuels are polar so they do not absorb much emulsified water.
In addition to lubrication concerns, biodiesel was added to ULSD in the U.S. for a variety of environmental and financial reasons, as well as government concerns about national dependence on foreign oil supplies.
However, the process of scrubbing the sulfur changed the chemistry of the fuel so ULSD provided less lubrication, resulting in immediate wear to diesel engines. An additive was introduced to ULSD to restore the needed lubrication. Soon after, biodiesel was also added as a fuel component, in ever increasing percentages, 2% (B2), 5%(B5), 7% (B7), 10% (B10), and now, 20% (B20). The more biodiesel, the more emulsified water. Biodiesel blend of 5% or less does not need to be disclosed to the end user.
Before ULSD, filters could remove all freestanding water and particulate, since LSD is not polar and there was very little emulsified water. Since both biodiesel and water are polar, water bonds tightly to the fuel as emulsified water. Pre-ULSD filters cannot efficiently remove the emulsified water from biodiesel blends.
Even before the U.S. Environmental Protection Agency (EPA) detailed this rapid and severe corrosion, NFPA 110 (2013) cautioned against the use of biodiesel for standby and prime engine-generator sets. In those instances, where the use of biodiesel was unavoidable, the NFPA required its replacement annually, but with “rapid and severe corrosion,” even 12 months is too long a time to store acidified fuel. NFPA recommends using diesel that has been stored that long for other purposes.
The EPA’s July 20th 2016 final report on corrosion states, “The major finding from our research is that moderate or severe corrosion on metal components in UST [underground storage tanks] systems storing diesel fuel in the United States could be a very common occurrence… variables that were the closest to being significant predictors [of corrosion] were particulates in the fuel and entrained (emulsified) water content in the fuel.” Drying the diesel by removing all the water, both free standing and emulsified, can control corrosion. This final report has worldwide significance because countries concerned about the issue have been tracking EPA findings.
A polar molecule
SAE J1488 tests “the ability of a fuel/water separator to separate emulsified or finely dispersed water from fuels.“ Well-known organizations such as Southwest Research Institute (SwRI) can test filters for compliance to the standard and provide reports to data center managers. In 2010, SAE updated the test from J1488_199708 to J1488_201010 to address the emulsified water concerns caused by ULSD and biodiesel blends.
SAE updated J1488 in 2010 to reflect the lower interfacial tension (IFT) in ULSD and biodiesel blends. IFT is the ability of two liquids to repel each other. Since biodiesel in hydroscopic, more water is absorbed and less repelled, so its IFT is lower. The IFT of ULSD is tested at 34.5 mN/m, biodiesel at 15 mN/m (B20). The 2010 update created a standard IFT against which all filters could be compared.
A new generation of filters designed specifically to filter for SAE J1488_201010 IFT levels is required to remove emulsified water, and hence control corrosion. In June 2016, the United Kingdom adopted SAE J1488_201010 as a code requirement.
Critical Fuel Applications
Modern backup generators are high-pressure common rail (HPCR) models, with pressures of 30,000+ pounds per square inch (psi) and plans for 60,000 psi models. Emulsified water and corrosion can cause these engine generator sets to become unreliable by
When injector pitting and corrosion are great enough, the generator will not be able to hold the required HPCR pressures and will fail. In addition, the pitting of the injectors can cause the generator to become over fueled, increasing heat, emissions, and fuel consumption. The corrosive/pitting affects injectors and spoolers on all diesel engines. The higher the psi of the engine, the greater the risk of damage and failure.
Corrosion in storage tanks, fittings, and piping dramatically increases the risk of a backup generator failing. For example, a fuel supply pipe that fails or plugs will prevent a generator from working during an emergency.
Warranty Claims
Warrantees on diesel engine-generator sets exclude damages caused by fuel-related issues. That means that vendors can deny warranty claims if the fuel does not meet the specification set forth by the vendor. The maximum emulsified water content in most warrantees is 200 parts per million (ppm), which is well below the observed amounts commonly found in tanks with biodiesel blends. Caterpillar’s warranty specifies a pre-emulsified water/corrosion water and particulate maximum of 500 ppm. Manufacturers generally enforce this requirement by requiring a fuel sample with each warranty claim.
Caterpillar’s service document SEBU6251-14 from 2014 (before corrosion was commonly recognized) reads, “The repair of any engine or after treatment components due to such contamination or due to other biodiesel impacts would not be covered under the Cat warranty for materials and/or the warranty for workmanship.” Since corrosion is such a new issue, it means Cat’s warranty could be voided the first time biodiesel is used in the engine.
Similarly, Cummins warns about biodiesel. Cummins’s service document 3379001-13 reads, “Engine damage, service issues, and/or performance issues determined by Cummins Inc. to be caused by the use of biodiesel fuel not meeting the specifications outlined in this Service Bulletin are not considered to be defects in material or workmanship and are not covered under Cummins Inc. engine warranty.”
All the other OEM warrantees include similar statements.
Sidebar: Corrosion Management Action
Removing emulsified water from the fuel is key to maintaining the integrity of diesel fuel systems. SAE J1488_201010 tested filters can remove the emulsified water, thereby restricting microbial growth, acidification, and dramatically reducing corrosion. Filters need to achieve a performance level of greater than 92% to keep water below 200 parts per million (ppm).
Below is a short checklist to help address any fuel issues, with detailed explanations following.
While the state may not mandate biodiesel, there are financial incentives to distributors to supply biodiesel blends. Note that suppliers are not required to disclose any biodiesel content of 5% or less, so the distributor may say there is no biodiesel content, when in fact there is.
Take fuel samples from all fuel tanks. The sample should be taken from the tank bottom. Test for biodiesel content, water content with Karl Fisher titration (less than 200 ppm), ISO 4406 particulate content (less than 18,16,13), and TAN (Total Acid Number, less than .08 maximum per World Fuel Charter).
It may be quicker and easier to sample the belly tanks or day tanks attached to the generators. If possible, have a visual inspection of the inside of your belly and day tanks. If there is any corrosion in these tanks, it will point to corrosion in the large storage tanks. All storage tanks are designed to handle the hydrostatic pressure (the weight of the fuel), which is greatest on the tank bottom. Since corrosion is also focused on the bottom of the tank, corrosion will pit of the tank wall, thereby thinning the wall, increasing the likely-hood of fuel leaks or tank failure. There are inexpensive fiberoptic cameras on long hoses available on the market to assist in the inspection.
If corrosion is seen, replace the fuel. Before adding new fuel, ask for a fuel sample from the suppliers distribution tank before delivery, and test for biodiesel content, water content with Karl Fisher titration, particulate content against ISO 4406, and TAN (Total Acid Number). If the sample fails any test, inform the supplier and request fuel within the acceptable specifications.
The EPA report states, “Minimizing water presence is and has always been an important part of UST maintenance. However, diesel blended with biodiesel can hold in solution more water than diesel without a biodiesel component. This means more water is likely arriving in USTs entrained in fuel today since biodiesel is more common in diesel than prior to 2007.” The large distribution storage tanks are also absorbing entrained (emulsified) water, so new fuel could already have substantial amounts of entrained (emulsified) water.
Observed corrosion (Battelle)
Install a permanent fuel polishing system on the storage tank if fuel tests show any failures in the tests. The polishing system must use the new generation of filters specifically designed to remove emulsified water to SAE J1488_201010 standards with of a minimum 92% efficiency. The fuel should be filtered 3 times a week, and filter the complete volume of fuel in an 8-hour period. A letter confirming SAE J1488_201010 test results of greater than 92% must be supplied by an accredited testing organization authorized by SAE such as Southwest Research Institute.
Fuel Experts
While most diesel consumers did not immediately recognize the issues with corrosion, the fuel experts did. Below is a timeline of recognition of acidification caused by microbial growth, starting with the most recent and working back.
U.S. EPA recognizes entrained (emulsified) water as a key indicator of corrosion. “This research focused on better understanding a type of rapid and severe corrosion of metal components in underground storage tanks (USTs) storing diesel fuel….EPA’s research determined that corrosion is very common; it appears all USTs storing diesel could be susceptible to developing corrosion.”
BS5410-3 made it code that any filters used on critical emergency backup systems conform to SAE J1488_201010.
BS5410-3 (British Standards Institution, 2016) is the equivalent of NFPA 110. The United Kingdom has used biodiesel blends longer than in North America. “7.2.2.5 Filters: NOTE 1 Since the introduction of Biofuel (FAME) [fatty-acid methyl ester] into fuels the quality and life expectancy of fuels has been adversely affected as FAME is hygroscopic so any water in the fuel goes into suspension….Removing as much water as possible preserves the quality of the fuel and for critical standby generators only filters conforming to SAE J1488_201010 [N5] are to be used”
NFPA 110 cautions against the use of biodiesel. “A.5.1.1(1) Where possible, the purchaser of fuel for the prime mover should specify a diesel fuel that does not contain biodiesel, which can accelerate the degradation of the diesel fuel if stored longer than 6 months.”
“A.5.5.3 More important, biodiesel blends up to B5 (ASTM D 975, Standard Specification for Diesel Fuel Oils) have much shorter shelf lives than conventional diesel fuel [ultra-low sulfur diesel (ULSD)] and can accelerate degradation processes, endangering the entire diesel fuel supply. Where fuel is stored for extended periods of time (e.g., more than 12 months), it is recommended that fuels be periodically pumped out and used in other services and replaced with fresh fuel.”
The Steel Tank Institute (STI) reinforces the importance of keeping storage tanks free of water. “In a 2012 study by Battelle Labs, five of six FRP (fiber- glass reinforced plastic) tanks showed severely accelerated corrosion. The sixth tank was supposedly “clean” and intended to be the control for the study, but turned out to have corrosion, as well….In ULSD, the presence of water provides an environment where microbial life can thrive, possible influencing the corrosion of internal metal components of the UST system equipment. “
SAE updates J1488_199708 to J1488_201010 to test the efficiency of filters to remove emulsified water from the newly introduced ULSD and biodiesel blends (B20). A 92% efficiency can keep water below 200 ppm, which is the OEM manufacturers’ warranty limit. In SAE J1488_201010 testing, ULSD and biodiesel (B20) are saturated with 2500 ppm of emulsified water, which is then filtered. The water content of the filtered fuel is compared to the water content of the original saturated fuel. SAE then assigns an efficiency rating to the filter and reports its findings to the filter supplier. (SAE Technical Standards Board, October 2010).
Battelle Memorial Institute calls attention to the problem of corrosion. “Severe and rapid corrosion has been observed in systems storing and dispensing ultra low sulfur diesel (ULSD) since 2007” (Battelle Memorial Institute, 2012)
The research paper “Moisture Absorption in Biodiesel and its Petro-diesel Blends” (B. B. He, 2007), compares the water absorption of 100% LSD to 100% biodiesel.
IBM Data Center Group Case Study
IBM has a group within the Global Technology Services division that designs, builds, migrates, and upgrades data centers for its customers. It reports that the majority of data centers in Canada have been in the Province of Ontario, which mandated the use of biodiesel with an increasing concentration, initially B2, and B4 at time of writing.
IBM was initially concerned about the reliable operation of the generators at these facilities, especially because data centers often store fuel is stored for long periods of time, unlike road or construction applications. In particular, IBM worried about the effect of the emulsified water on its engine-generator set infrastructure. Reduced engine-generator set life and increased repair costs were major concerns.
Over time, it became clear to IBM that the entire fuel storage and delivery system, metallic and non-metallic components included, is at risk due to the corrosive properties of degraded biodiesel. A major excavation required to repair or replace would be a major concern.
Fuel filters tested in systems using B2 pass the “bright and clear” test, which wasn’t deemed rigorous enough to give a true indication of the condition of biodiesel fuel. IBM has been offering their clients fuel-conditioning systems that meet or exceed a more applicable standard from the SAE J1488_201010.
Three Biodiesel Related Failures in One Day
Franklin Company in the NYC area responded to three separate fuel-related diesel generator failures in one day at three different organizations. Problems caused by ULSD can occur in many application types. Failure 3 is typical of failures at a very common type of facility. These failures give a good representation of some of the failure points data center users can experience: corrosion of tanks, failure of pipes and fittings, and biomass plugging filters.
Failure 1: Franklin Company responded to a service call with a reported interstitial alarm. A service tech removed the sensor and found a mixture of water and fuel on the sensor. The tech removed a small amount of fuel and water mixture and then reinstalled the sensor. The sensor did not go into alarm. Franklin initially thought that recent rains had worked its way past the interstitial cap.
A few weeks later, Franklin received a new service call with the same complaint. This time the tech found a significant amount of water and some fuel in the interstitial. Since the tank had not been showing any losses, Franklin thought that the outer wall was leaking groundwater and fuel into the annular space from outside. It would not be so unusual for a site to have contamination. The interstitial failed a vacuum test. The primary tank passed a second test was to confirm that it was tight. A spill was called in to the DEC for a non-tight tank system. Franklin Company contracted a third party to perform the same tests, with the same result.
As one result, Franklin excavated and checked the tank top for breaches or leaking bungs to the interstitial. All fuel was removed, the manway removed, and a confined space team entered and began cleaning a very dirty tank. The team found the steel tank had many holes on the bottom.
What happened with the tests? The annular was leaking, but only between the primary and the secondary. Tests on the primary showed “tight,” but we were really testing only the secondary tank, as the primary tank was open to the secondary. The larger amounts of water in the secondary was caused by the head pressure pushing the tank bottoms out of the bottom of the primary. The annular space on an ACT 100 is very small, so a small amount of water would migrate to the riser, showing a disproportionate amount of water versus fuel. When the tech removed water, it would look dry for an extended period of time until more water and fuel would be forced through the bottom of the tank and work its way to the annular riser.
The tank was not salvageable and had to be removed. There was no leakage to the environment as the secondary was intact and tight. Note the tank was 10 years old.
Failure 2: An underground UL 971 flexible pipe failed when B20 diesel caused the internal lining of the pipe to delaminate. The contractor said, “The pipe looked like a long sheet of white plastic wrap when we investigated the blockage.”
Failure 3: A government agency reported a slow flow of diesel fuel dispensers. A service technician dispatched to the site found that two dispensers had flow rates of only 2-3 gallons per minute. Two dispensers had no flow at all. The internal filters were removed and replaced, and the pumping unit strainers (suction style units) were removed and cleaned and then re-installed. The facility was down for two days. In this case, microbial “blooms” and excess particulates had created biofilms that plugged the filters.
Biofilm, as shown in the picture below, is a problem that can lead to the failures stated above. Biofilms concentrate the corrosive acidic effects of the fuel (Failure 1), can peel off in strips to plug pipes (Failure 2) and can disintegrate, overloading filters (Failure 3). The contact said, “WOW. Never saw anything like this. This was the consistency of boiled pig skin. It was lining the bottom of the tank. This is a tank that we could not rinse in situ. Super bugs.”
Picture Below, Biofilm from bottom of tank
Suggested Readings
Chuanfang Yang, S. L. (2007). Understanding emulsified water filtration from diesel fuels . San Antonio: 8th International Filtration Conference.
Groysman, A. (n.d.). Corrosion in Systems for Storage and Transportation of Petroleum Products and Biofuels Identification, Monitoring, and Solutions. New York, Philadelphia: Springer Science & Business Media.
PEI Petroleum Equipment Industry. (2013). Recommended Practices for Installation of Bulk Storage Plants (PEI/RP800-13). Petroleum Equipment Industry.
R.L. McCormick, T. A. (2005). Survey of the Quality and Stability of Biodiesel and Biodiesel Blends in the United States in 2004 . National Renewable Energy Laboratory, Golden, Colorado.
Steel Tank Institute. (2016). Recommended Practice For Storage Tank Maintenance R111 Revision . Steel Tank Institute , Lake Zurich, IL.
Bibliography
European Automobile Manufacturers’ Association ACEA. (2013). World Wide Fuel Charter 5th Edition.
ASTSWMO Tanks Subcommittee’s Alternative Fuels Workgroup. (2013). Compatibility of UST Systems with Biofuels. Association of State and Territorial Solid Waste Management Officials (ASTSWMO), Washington, DC.
B. He, J. C. (2007). Moisture Absorption In Biodiesel and Its Petro-Diesel Blends. Food & Process Engineering Institute Division of ASABE. Moscow, Idaho: American Society of Agricultural and Biological Engineers ISSN 0883 8542.
Battelle Memorial Institute. (2012). Corrosion in Systems Storing and Dispensing Ultra Low Sulfur Diesel (ULSD), Hypotheses Investigation. Battelle Memorial Institute, Columbus, OH.
British Standards Institution. (2016). Code of practice for oil firing Part 3: Installations for furnaces, kilns, ovens, oil-fuelled standby generators and other industrial purposes. BSI Standards Limited.
NFPA110. (2013). NFPA 110 Standard for Emergency and Standby Power Systems. Quincy, MA.
SAE Technical Standards Board. (October 2010). Emulsified Water/Fuel Separation Test Procedure. Thompson Reuters.
Steel Tank Institute STI. (2013). Newer fuels and storage tank corrosion. Steel Tank Institute STI, Lake Zurich IL.
U.S. Environmental Protection Agency (July 2016). Investigation Of Corrosion-Influencing Factors In Underground Storage Tanks With Diesel Service. United States Environmental Protection Agency, U.S. Environmental Protection Agency Office of Underground Storage Tanks, Washington DC.
Uptime Institute. (2010). Uptime Institute Technical Paper: Biodiesel. New York.
Bernard Oegema
Bernard Oegema PE Data Centre Consultant, IBM Global Services, Data Centre Services. He has been in the critical facilities business for more than 30 years, working with various manufacturers of critical power and cooling infrastructure. Currently he is supporting IBM clients in the migration, consolidation, operation, upgrade, design and building of data centers.
Pat Smyth
Pat Smyth is Vice President of DieselPure. DieselPure is focused on fuel polishing for emergency backup generator storage. Since 2011, DieselPure has field experience in removing emulsified water from ULSD and biodiesel as per SAE J1488_201010.
Brian Ponstein
Brian Ponstein is Regional Sales Engineer, MTU Onsite Energy, a Rolls-Royce Company. At MTU, Mr. Ponstein is responsible for analyzing market needs and requirements in North America and working with engineering to provide solutions for MTU Onsite Energy’s customers.
Bill Klein
William (Bill) Klein is President of Franklin Company Contractors. He is also a New York City (NYC) Master Electrician, NYC Licensed Plumber, NYC Licensed Pump and Tank Contractor, NYC Licensed Precision Tank Tester, member of PEI, and past president of NYC Tank Installers. Mr. Klein is the third generation president of Franklin Company.
Martin Weselowski
Martin Wesolowski is Senior Associate Vice President at LiRo’s petroleum and chemical storage tank design group. He has been involved with design of petroleum and chemical UST and AST tank and remediation systems for over 23 years. Mr. Wesolowski is a technical director on LiRo’s New York City Department of Buildings (NYCDOB) Special Inspection Agency Registration.
CenturyLink Benefits from Its Commitment to Excellence
/in Executive/by Kevin HeslinDavid Meredith says that CenturyLink’s involvement, including the M&O Stamp of Approval, with Uptime Institute provides mutual benefits. Uptime Institute’s Risk Journal for IT Infrastructure is the most recent example.
CenturyLink’s commitment to operational excellence has become one of the company’s competitive advantages, and David Meredith, a senior VP at CenturyLink, has been an enthusiastic proponent of demonstrating this commitment through third-party verifications. In 2014, CenturyLink decided to pursue Uptime Institute’s Management and Operations (M&O) Stamp of Approval for all 57 of its data centers.
So Uptime Institute was delighted to see Meredith’s comments in LinkedIn about our new publication “Risk Management for IT Infrastructure.” Meredith quoted the book’s central tenet, “The handbook also argues that effective governance requires industry certifications.” The journal provides guidance on avoiding data center capital project failure and applying efficient IT principles to address sustainability risks and IT resilience during a natural disaster.
Meredith writes, “Julian Kudritzki and Matt Stansberry [the book’s editors] point out that, when the M&O Stamp of Approval is administered across an entire portfolio, it ensures consistency. CenturyLink is a perfect example of utilizing Uptime’s data center operations guidance. As the authors point out, “CenturyLink’s commitment to achieve M&O Stamp of Approval across the board shows it is willing to hold itself to a uniform set of high standards and operate with transparency.”
Meredith and CenturyLink have previously contributed to Uptime Institute publications. Please click on the following hyperlinks for some recent examples:
https://journal.uptimeinstitute.com/improving-performance-ever-changing-mission-critical-infrastructures/
https://journal.uptimeinstitute.com/executive-perspectives-colocation-wholesale-markets/
Luxembourg Colo Provides Multi-Tier Options
/in Design, Executive/by Kevin HeslinLuxconnect obtains different Tier Certifications to meet the needs of the changing market demands
By Christine De Ridder
LuxConnect, a multi-tenant, multi-tier data center and dark fiber network operator based in Luxembourg, developed an innovative strategy using Uptime Institute Tiers to differentiate the services and pricing it offers to key customer groups. In doing so, LuxConnect became the first—and so far only—facility to offer multiple Tier levels in the same data center.
The exterior of DC1.3, Luxconnect’s green data center.
“LuxConnect’s multi-tier approach to Tier Certification enables it to offer customers different rooms with different Tier Objectives in the same building.” Said Phil Collerton, managing director, Uptime Institute EMEA.” This gives the client the opportunity to use different rooms to house different types of applications, both mission critical and noncritical, while also benefiting from the flexible SLAs and pricing models that this strategy allows LuxConnect to offer.”
A Tier IV Certified Constructed Facility server room ready to host servers.
LuxConnect is a private limited company owned by the Luxembourg State and founded in 2006 with several missions such as strengthening the Luxembourg’s position on the European Internet map and providing state-of-the-art IT environment.
Roger Lampach, CEO, said, “Among our goals was to promote and facilitate high-end ICT investments into Luxembourg and offer white space in high-quality data centers. And that is what we achieved.”
Today LuxConnect has two sites and four buildings with a total server space of 14,700 square meters. LuxConnect has three independent data centres, DC1.1, DC1.2 and DC1.3, on its Bettembourg ICT Campus in the south of the country. The DC2 facility is located in Bissen. The two sites are interconnected via multiple routes using dark fiber to guarantee customers data center business continuity.
Luxconnect’s low-voltage distribution room.
LuxConnect designed, planned, and built these data centers in less than 10 years. The newest facility, DC1.3, launched in August 2015. Three of the four buildings have been Certified by the Uptime Institute, including Tier IV Certification of Design Documents for DC1.1 and Tier II Certification of Design Documents in DC2. DC1.3 has Tier II and IV Certifications of Design Documents and Constructed Facility in the same facility, which is a unique configuration.
Cold water distribution
Lampach said, “We know that we have to convince our visitors and potential customers that we are professional and that we know what to do and how to do it. Our visitors are usually very impressed by the quality and high level of our five-star data centers.”
LuxConnect’s mix of Tier Certifications is an integral part of meeting its customer’s demands. Having multiple Tier II facilities enables LuxConnect to attract gamers who require high availability and low latency but are very cost sensitive. Other verticals, such as banks, require the Fault Tolerant Tier IV Certification of Constructed Facilities.
Luxconnect received Tier II and Tier IV Certifications for spaces in the same facility, making it unique.
With support from the government, LuxConnect is able to expand as soon as market demand appears. Lampach said, “We start building when we see that there is not enough capacity in the market and without having any contracts. It’s not like a profit-driven company would do, but as our primary shareholder wants outside companies to come at any moment, we have to anticipate.”
The cornerstone of this approach is DC 1.3, which has both Tier II and Tier IV Certifications of Constructed Facility on the same floor in the same building, which is a unique approach.
The Tier II spaces are mostly for the gamers, according to Lampach, “The gamers have two installations, one called front end where the gaming company connects and the other one where our client analyzes customer behavior and determines which new products to present their customers. These companies pushed us to consider offering Tier II spaces. And we began the idea by achieving Tier II Certification of Design Documents in DC2.”
Uptime Institute presents Tier Certifications to Luxconnect.
From the beginning, LuxConnect designed its data centers following Uptime Institute’s Tier standards and specifications. The company’s two project managers are Uptime Institute Accredited Tier Designers. Lampach said: “Having the Tier Certifications has been a real benefit for us. It’s also a concern for us because we go to the market through our business partners. At the beginning, they don’t really understand the differences between Tier II, III, and IV and they think downtime will never happen or they decide to take the risk.”
Lampach feels that the Tier Certification of Constructed Facility process provided the project team with an opportunity to better understand the behavior of the different systems, which helps them to assure business continuity. The Operations staff also benefited from the Tier Certification process with a very intensive workweek that was instructive on the technical and human level.
“At the end of the week, Operations could really see how the electrical and mechanical systems reacted during the multiple demonstrations. They feel more comfortable working with the systems because they know exactly how to proceed for maintenance without impacting business continuity.” The staff was also able to witness how the systems react during different failure scenarios.
Rich Van Loo, Uptime Institute’s senior vice president, Facility Management Services, said, “The operations assessments helped to insure LuxConnect gets the most availability out of each data center regardless of Tier while still allowing flexibility to its customers. It positions them to be a true leader for IT services in the region.
LuxConnect is justifiably proud of its performance during the Certification process, as it had just two minor issues, both of which the company was able to resolve during the testing week. The minor nature of these changes testifies to the quality of LuxConnect’s design, construction, and commissioning processes.
Christine De Ridder
Christine De Ridder worked for several companies, including Schneider Electric and Siemens, before joining LuxConnect in 2012 as manager, Data Centre Projects. She became an Accredited Tier Designer in 2013. In addition, she speaks five languages and has an engineering degree in electromechanics.
ENTEL Achieves Uptime Institute Tier Certification of Operational Sustainability
/in Operations/by Kevin HeslinTier Certification of Operational Sustainability enhances ENTEL’s services to customers
By Kevin Heslin
ENTEL began operations in 1964 as a provider of national and international long distance telephone services to companies in Chile. Today it is a consolidated provider of integrated telecommunication services and information technologies services, meeting the needs of corporations and large companies through tailored solutions, providing value, experience, and quality of service.
The company began operations in the aftermath of an earthquake that severely damaged the Chilean telecommunications network. More recently it added more services. Towards the end of 1995, ENTEL started to provide Internet connection services and in 1997 it introduced the first commercial network with ATM technology in Latin America, which evolved into the current Multiservice IP network in order to offer broadband solutions, guarantee quality of service, and add value to its clients. Today ENTEL offers Information Technology (IT) services, which vary according to the industrial segment and business model of each client, allowing competitive efficiencies and advantages that differentiate ENTEL from traditional telecommunications services.
As part of its offerings, ENTEL also offers a number of data center services to more than 3,000 clients. It owns and operates a network of five data centers that are certified under ISO 9001-2000 and ISO 27001 standards. In addition, two of them are Tier III Gold Certified Constructed Facility and one has the M&O Stamp of Approval.
Data Center Infrastructure
ENTEL has five data centers located in Amunátegui, Pedro de Valdivia, Ñuñoa, Longovilo, and Ciudad De Los Valles that are linked through high availability and high capacity IP/MPLS/DWDM These data centers have in excess of 7,500 square meters (m2) of data hall space with plans to expand to 11,675 m2. From these data centers, ENTEL offers IT outsourcing services, from server hosting to more complex services involving operation and exploitation of the platforms that support clients’ business processes. These data center services meet the needs of companies’ most vital applications, improving security and protection of critical data and reducing considerably their infrastructure investments. ENTEL’s IT strategy is based on traditional data center services and growth in cloud services and permanent innovation.
ENTEL first offered data center services in 2003 and is now Chile’s largest provider. To this end, ENTEL has staffed its facilities with more than 120 professionals and management devoted to implementation and operation of data center Infrastructure projects
Thanks to its team of specialists with extensive experience, the IT challenges faced by ENTEL have been met entirely in-house. Of particular note are the Ciudad de los Valles 1 and 2 facilities, each offering 2,000 m2 of white space. Ciudad de los Valles 1 (CDLV1) entered production in May 2010, and Ciudad de los Valles 2 (CDLV2) entered production in March 2013 (see The Uptime Institute Journal, vol. 2, p. 64). Both are Tier III Gold Certified Constructed Facilities (see Figure 1). Both received Tier Certification for Operational Sustainability in October 2015 (see Table 1). Josué Ramírez, Uptime Institute, director of Business Development LATAM, “With these Certifications, ENTEL shows it commitment to seeking excellence in its operations to provide better services to its clients and to contribute to the development of knowledge and professionalism in the region of Latin America.”
Figure 1. CDLV 1 and 2 are both Tier III Gold Certified Constructed Facilities.
Rich Van Loo, Uptime Institute, VP Facility Management Services, said, “The Tier Certification and Operational assessments have had a broad impact on not only the data center management divisions, but the company environment as a whole. Operational procedures not only reduce risk but help improve consistency and efficiency in those operations. ENTEL is looking to expand this philosophy to all their data centers.“
Table 1
ENTEL began construction of a third facility in 2015 at Ciudad de los Valles. This will add an additional 2,000 m2 of white space. The new facility incorporates free-cooling technologies that achieve better energy efficiency and lower total costs of ownership for clients.
Tier Certification of Operational Sustainability
To differentiate itself from its competitors, ENTEL became the first to Tier Certify its facilities. As a result, ENTEL decided to further differentiate itself by earning Uptime Institute’s Tier Certification of Operational Sustainability for CDLV 1 and 2. This decision was based on client demand for excellence of service, risk mitigation, and assurance of operational continuity.
ENTEL believes that
Figure 2. The Tier Certification of Operational Sustainability verifies that ENTEL maximizes the potential of its facilities and differentiates it from its competitors.
The Tier Certification of Operational Sustainability (TCOS) was an excellent way to reach these goals and why a new internal organization was dedicated to defining data center operations activities and a second team was tasked to leading the effort to earn the Tier Certification of Operational Sustainability.
In addition, ENTEL created an Infrastructure Change Control Board (CCI), specific to Datacenter Infrastructure Management, with the purpose of establishing planning and control of activities in each data center. This Board is a consulting body. It meets periodically and manages, reviews, and approves infrastructure management projects to data centers.
Scope of the Infrastructure Change Control Board (CCI)
Although ENTEL had previously adopted the ITIL model and established a Change Advisory Board (CAB) that validates and approves all activities in the IT and data center infrastructure environment, it was determined to create the CCI because of the degree of specialization of infrastructure work and the risk associated with it. Its focus is to raise risk points associated to high-impact activities and ascertain control points, e.g. cells, transformers, generators, UPS, towers and chillers, where the system redundancy could be lost.
A summary of the type of work evaluated and documentation to be reviewed follows (see Table 2):
Table 2.
The CCI is intended to ensure that the good practices characteristic of Tier Certification of Operational Sustainability are respected and that all documentation and instructions are kept thoroughly valid, that they are applied, and are subject to continuous improvement.
The Process
In order to increase ENTEL’s familiarity with the Tier Certification of Operational Sustainability process, ENTEL and Uptime Institute agreed to carry it the process in four phases (see Figure 3).
Figure 3. ENTEL views operational sustainability as a four-phase continuous process.
Important Decisions
Eventually, Uptime Institute and ENTEL adjusted the process in light of the initial progress, the maturity level of the team, and the imperative work of assuring operational continuity:
ENTEL also
ENTEL believes that it received many benefits from earning the Tier Certification of Operational Sustainability, including
The knowledge of the Certification methodology and good practices stayed in house, which ENTEL can replicate at all its data centers and eventually to other areas of the organization (see Figure 4). ENTEL has worked to share this information company wide, with the additional notable achievement of Uptime Institute’s M&O Stamp of Approval at its Amunategui facility in December 2016.
Figure 4. The staff at ENTEL is charged with promoting Operational Sustainability across the whole organization.