Australian Colo Provider Achieves High Reliability Using Innovative Techniques

NEXTDC deploys new isolated parallel diesel rotary uninterruptible
 power supply systems and other innovative technologies

By Jeffrey Van Zetten

NEXTDC’s colocation data centers in Australia incorporate innovation in engineering design, equipment selection, commissioning, testing, and operation. This quality-first philosophy saw NEXTDC become one of 15 organizations globally to win a 2015 Brill Award for Efficient IT. NEXTDC’s B1 Brisbane, M1 Melbourne, S1 Sydney, C1 Canberra, and P1 Perth data centers (see Figure 1-3), have a combined 40-megawatt (MW) IT load (see Figure 4).

In order to accomplish the business goals NEXTDC established, it set the following priorities:

1.   High reliability, so that clients can trust NEXTDC facilities with their mission critical IT equipment

2.   The most energy-efficient design possible, especially where it can also assist reliability and total cost of ownerships, but not at the detriment of high reliability

3.   Efficient total cost of ownership and day one CapEx by utilizing innovative design and technology

4.   Capital efficiency and scalability to allow flexible growth and cash flow according to demand

5.   Speed to market, as NEXTDC was committed to build and open five facilities within just a few years using a small team across the entire 5,000-kilometer-wide continent, to be the first truly national carrier-neutral colocation provider in
the Australian market

6.   Flexible design suitable for the different regions and climates of Australia ranging
from subtropical to near desert.

NEXTDC put key engineering design decisions for these facilities through rigorous engineering decision matrices that weighed and scored the risks, reliability, efficiency, maintenance, total cost of ownership, day one CapEx, full final day CapEx, scalability, vendor local after-sales support, delivery, and references. The company extensively examined all the possible alternative designs to obtain accurate costing and modeling. NEXTDC Engineering and management worked closely to ensure the design would be in accord with the driving brief and the mandate on which the company is based.

NEXTDC also carefully selected the technology providers, manufacturers, and contractors for its projects. This scrutiny was critical, as the quality of local support in Australia can vary from city to city and region to region. NEXTDC paid as much attention to the track record of after-sales service teams as to the initial service or technology.

“Many companies promote innovative technologies; however, we were particularly interested in the after-sales support and the track record of the people who would support the technology,” said Mr. Van Zetten. “We needed to know if they were a stable and reliable team and had in-built resilience and reliability, not only in their equipment, but in their personnel.”

NEXTDC’s Perth and Sydney data centers successfully achieved Uptime Institute Tier III Certification of Design Documents (TCDD) and Tier III Certification of Constructed Facilities (TCCF) using Piller’s isolated parallel (IP) diesel rotary uninterruptible power supply (DRUPS) system. A very thorough and exhaustive engineering analysis was performed on all electrical system design options and manufacturers available, including static uninterruptible power supply (UPS) designs with distributed redundant and block redundant distribution, along with the more innovative options such as the IP DRUPS solution. Final scale and capacity was a key design input for making the final decision, and indeed for smaller scale data centers a more traditional static UPS design is still favored by NEXTDC. For facilities larger than 5 MW, the IP DRUPS allows NEXTDC to:

•   Eliminate batteries, which fail after 5 to 7 years, causing downtime and loss of redundancy and can cause
hydrogen explosions

•   Eliminate the risks of switching procedures, as human error causes most failures

•   Maintain power to both A & B supplies without switching even if more than one engine-generator set or UPS is
out of service

•   Eliminate problematic static switches.

NEXTDC benefits because:

•   If a transformer fails, only the related DRUPS engine generator needs to start. The other units in parallel can all remain on mains [editor’s note: incoming utility] power.

•   Electrically decoupled cabinet rotary UPS are easier to maintain, providing less down time and more long-term reliability, which reduces the total cost of ownership.

•   The N+1 IP DRUPS maintain higher loaded UPS/engine generators to reduce risk of cylinder glazing/damage at low and growing loads.

•   Four levels of independently witnessed, loaded integrated systems testing were applied to verify the performance.

•   The IP topology shares the +1 UPS capacity across the facility and enables fewer UPS to run at higher loads for better efficiency.

•   The rotary UPSs utilize magnetic-bearing helium-gas enclosures for low friction optimal efficiency.

•   The IP allows scalable installation of engine generators and UPS.

For example, the 11.5-MW S1 Sydney data center is based on 12+1 1,336-kilowatt (kW) continuous-rated Piller DRUPS with 12+1 IP power distribution boards. The facility includes sectionalized fire-segregated IP and main switchboard rooms. This design ensures that a failure of any one DRUPS, IP, or main switchboard does not cause a data center failure. The return ring IP bus is also fire segregated.

Differential protection also provides a level of Fault Tolerance. Because the design is scalable, NEXTDC is now increasing the system to a 14+1 DRUPS and IP design to increase the final design capacity from 11.5 to 13.8 MW of IT load to meet rapid growth. 

All NEXTDC stakeholders, especially those with significant operational experience, were particularly keen to eliminate the risks associated with batteries, static switches, and complex facilities management switching procedures. The IP solution successfully eliminated these risks with additional benefits for CapEx and OpEx efficiency.

From a CapEx perspective, the design allows a common set of N+1 DRUPS units to be deployed based on actual IT load for the entire facility (see Figure 5). From an OpEx perspective, the advantage is the design is always able to operate in a N+1 configuration across the entire facility to match actual IT load, so the load is maintained at a higher percentage and thus at efficiencies approaching 98%, whereas low loaded UPS in a distributed redundant design, for example, can often have actual efficiencies of less than 95%. Operating engine-generator sets at higher loads also reduces the risks of engine cylinder glazing and damage, further reducing risks and maintenance costs (see Figure 6).

NEXTDC repeated integrated systems tests four times. Testing, commissioning, and tuning are the keys to delivering a successful project. Each set of tests—by the subcontractors, NEXTDC Engineering, independent commissioning agent engineers, and those required for Uptime Institute TCCF—helped to identify potential improvements, which were immediately implemented (see Figure 7). 

In particular, the TCCF review identified some improvements that NEXTDC could make to Piller’s software logic so that the control became truly distributed, redundant, and Concurrently Maintainable. This improvement ensured that even the complete failure of any panel in the entire system would not cause loss of N IP and main switchboards, even if the number of DRUPS is fewer than the number of IP main switchboards installed. This change improves CapEx efficiency without adding risks. The few skeptics we had regarding Uptime Institute Tier Certification became believers once they saw the professionalism, thoroughness, and helpfulness of Uptime Institute professionals on site.

From an operational perspective, NEXTDC found that eliminating static switches and complex switching procedures for the facilities managers also reduced risk and delivered optimal uptime in reality.

MECHANICAL SYSTEMS
The mechanical system designs and equipment were also run through equally rigorous engineering decision matrices, which assessed the overall concept designs and supported the careful selection of individual valves, testing, and commissioning equipment. 

For example, the final design of the S1 facility includes 5+1 2,700-kW (cooling), high-efficiency, water-cooled, magnetic oil-free bearing, multi-compressor chillers in an N+1 configuration and received Uptime Institute Tier III Design Documents and Constructed Facility Certifications. The chillers are supplemented by both water-side and air-side free-cooling economization with Cold Aisle containment and electronically commutable (EC) variable-speed CRAC fans. Primary/secondary pump configurations are utilized, although, a degree of primary variable flow control is implemented for significant additional energy savings. Furthermore, NEXTDC implemented extreme oversight on testing and commissioning and continues to work with the facilities management teams to carefully tune and optimize the systems. This reduces not only energy use but also wear on critical equipment, extending equipment life, reducing maintenance, and increasing long-term reliability.

The entire mechanical plant is supported by the IP DRUPS for continuously available compressor cooling even in the event of a mains power outage. This eliminates the risks associated with buffer tanks and chiller/compressor restarts that occur on most conventional static-UPS-supported data centers and is a common cause of facility outage.

The central cooling plant achieved its overall goals because of the following additional key design decisions:

•   Turbocor magnetic oil-free bearing, low-friction compressors developed in Australia provide both reliability and efficiency (see Figure 8).

•   Multi-compressor chillers provide redundancy within the chillers and
improved part load operation.

•   Single compressors can be replaced while the chiller keeps running.

•   N+1 chillers are utilized to increase thermal transfer area for better part-load coefficient of performance
(COP) and Fault Tolerance, as the +1 chiller is already on-line and operating should one chiller fail.
•   Variable-speed drive, magnetic-bearing, super-low-friction chillers provide leading COPs.

•   Variable number of compressors can optimize COPs.

•   Seasonal chilled water temperature reset enables even higher COPs and greater free economization in winter.

•   Every CRAC is fitted with innovative pressure-independent self-balancing characterized control valves
(PICCV) to ensure no part of system is starved of chilled water with scalable dynamic staged expansions, and also ensure minimal flow per IT power to minimize pumping energy.

•   Variable speed drives (VSDs) are installed on all pumps for less wear and reduced failure.

•   100% testing, tuning, commissioning and independent witnessing of all circuits, and minimization of pump
∆P for reduced wear.

•   High ∆T and return water temperatures optimize water-side free cooling.

•   High ∆T optimizes seasonal water reset free-cooling economization.

The cooling systems utilize evaporative cooling, which takes advantage of Australia’s climate, with return water precooling heat exchangers that remove load from the chiller compressors for more efficient overall plant performance. The implementation of the water-side and air-side free economization systems is a key to the design.

Very early smoke detection apparatus (VESDA) air-quality automatic damper shutdown is designed and tested along the facility’s entire façade. Practical live witness testing was performed with smoke bombs directed at the façade intakes and utilized a crane to simulate the worst possible case bush-fire event, with a sudden change of wind direction to ensure that false discharges of the gas suppression could be mitigated.

The free-cooling economization systems provide the following benefits to reliability and efficiency (see Figures 9-12):

•   Two additional cooling methods provide backup 
in addition to chillers for 
most of the year

•   Reduced running time on chillers and pumps extend the life and reduce failure and maintenance.

•   The design is flexible to use either water side or air side depending on geographic locations and outside air quality.

•   Actual results have proven a reduced total cooling plant energy.

•   Reduced loads on chillers provide even better chiller COPs at partial loads.

•   Reduced pump energy is achieved when air-side economization is utilized.

WHITE SPACE

NEXTDC’s designer consultants specified raised floor in the first two data rooms in the company’s M1 Melbourne facility (the company’s first builds) as a means of supplying cold air to the IT gear. A Hot Aisle containment system prevents intermixing and returns hot air to the CRACs via chimneys and an overhead return hot air plenum and back to the CRACs. This system minimizes fan speeds, reducing wear and maintenance. Containment also makes it simpler to run the correct number of redundant fans, which provides a high level of redundancy, and due to fan laws, reduces fan wear and maintenance. At NEXTDC, containment means higher return air temperatures, which enables more air-side economization and energy efficiency and an innovative, in-house floor grille management tool that minimizes fan energy according to IT load (see Figure 13). 

For all later builds, however, NEXTDC chose Cold Aisle containment to eliminate the labor costs and time to build overhead plenum and chimneys for the Hot Aisle containment system, which reduced payback and return on investment. NEXTDC now specifies Cold Aisle containment in all its data centers.

The common-sense implementation of containment has proved to be worthwhile and enabled genuine energy savings. Operational experience suggested that containment alone saves only a part of the total possible energy savings. To capture even more savings, NEXTDC Engineering developed a program that utilizes the actual contracted loads and data from PDU branch circuit monitoring to automatically calculate the ideal floor grille balance for each rack. This intelligent system tuning saved an additional 60% from NEXTDC’s CRAC fan power by increasing air-side ∆T and reducing airflow rates (see Figure 14).

NEXTDC also learned not to expect mechanical subcontractors to have long-term operational expenses and energy bills as their primary concern. NEXTDC installed pressure/temperature test points across all strainers and equipment and specified that all strainers had to be cleaned prior to commissioning. During the second round of tests, NEXTDC Engineering found that secondary pump differential pressures and energy were almost double what they theoretically should be. Using its own testing instruments, NEXTDC Engineering determined that some critical strainers on the index circuit had in fact been dirtied with construction debris—the contractors had simply increased the system differential pressure setting to deliver the correct flow rates and specified capacity. After cleaning the relevant equipment, the secondary pump energy was almost halved. NEXTDC would have paid the price for the next 20 years had these thorough checks not been performed.

Similarly, primary pumps and the plant needed to be appropriately tuned and balanced based on actual load, as the subcontractors had a tendency to setup equipment to ensure capacity but not for minimal energy consumption. IT loads are very stable, so it is possible to adjust the primary flow rates and still maintain N+1 redundancy, thanks to pump laws—with massive savings on pump energy. The system was designed with pressure independent self-balancing control valves and testing and commissioning sets to ensure scalable, efficient, flow distribution, and high water-side ∆Ts to enable optimal use of water-side free-cooling economization. The challenge then was personally witnessing all flow tests to ensure that the subcontractors had correctly adjusted the equipment. Another lesson learned was that a single flushing bypass left open by a contractor can seriously reduce the return water temperature and prevent the water-side economization system from operating entirely if not tracked down and resolved during commissioning. Hunting down all such incorrect bypasses helped to increase the return water temperature by almost 11ºF (6ºC) for a massive improvement in economization.

Operational tuning through the first year, with the Engineering and facilities management teams comparing actual trends to the theoretical design model provided savings exceeding even NEXTDC’s optimistic hopes. Creating clear and simple procedures with the facilities management teams and running carefully overseen trended trials was critical before rolling out these initiatives nationally. 

Every single tuning initiative implemented nationally after the facilities go-live date is trended, recorded, and collated into a master national energy savings register. Figure 15 provides just a few examples. Tuning has so far yielded a 24% reduction in the power consumption for mechanical plant with still conservative safety factors. Over time, with additional trend data and machine learning, power consumption is still expected to considerably improve on this via continuous improvement. NEXTDC expects a further 10–20% saving as NEXTDC is on target to operate Australia’s first National Australian Built Environment Rating System (NABERS) 5-star-rated mega data centers.

The design philosophy didn’t end with the electrical and mechanical cooling systems, but also applied to the hydraulics and fire protection systems:

•   Rainwater collection is implemented on site to supply cooling towers, which provides additional hours of water most of the year.

•   The water tanks are scalable.

•   Rainwater collection minimizes mains water consumption.

•   VESDA laser optical early detection developed in Australia and licensed internationally was utilized.

•   The design completely eliminated water-based sprinkler systems from within the critical IT equipment data
halls, instead utilizing IG55 inert-gas suppression, so that IT equipment can continue to run even if a single server
has an issue (see Figure 16). Water-based pre-action sprinklers risk catastrophic damage to IT equipment that is
not suffering from an over-heating or fire event, risking unnecessary client IT outages.

•   The gas suppression system is facility staff friendly, unlike alternatives that dangerously deplete oxygen levels in
the data hall.

•   The design incorporates a fully standby set of gas suppression bottle banks onsite.

•   The gas suppression bottle banks are scalable.

•   The IG55 advanced gas suppression is considered one of the world’s most environmentally friendly gas
suppression systems.

The design of NEXTDC’s data centers is one of the fundamental reasons IT, industrial, and professional services companies are choosing NEXTDC as a colocation data center partner for the region. This has resulted in very rapid top and bottom line financial growth, leading to profitability and commercial success in just a few years. NEXTDC was named Australia’s fastest-growing communications and technology company at Deloitte Australia’s 2014 Technology Fast 50 awards. 

Mr. Van Zetten said, “What we often found was that when innovation was primarily sought to provide improved resilience and reliability, it also provided improved energy efficiency, better total cost of ownership, and CapEx efficiency. The IP power distribution system is a great example of this. Innovations that were primarily sought for energy efficiency and total cost of ownership likewise often provided higher reliability. The water-side and air-side economization free cooling are great examples. Not only do they reduce our power costs, they also provide legitimate alternative cooling redundancy for much of the year and reduce wear and maintenance on chillers, which improves overall reliability for the long term. 

 “Cold Aisle containment, which was primarily also sought to reduce fan energy, eliminates client problems associate with air mixing and bypassing, thus providing improved client IT reliability.”

---

Jeffrey Van Zetten has been involved with NEXTDC since it was founded in 2010 as Australia’s first national data center company. He is now responsible for the overall design, commissioning, Uptime Institute Tier III Certification process, on-going performance, and energy tuning for the B1 Brisbane, M1 Melbourne, S1 Sydney, C1 Canberra, and P1 Perth data centers. Prior to joining NEXTDC, Mr. Van Zetten was based in Singapore as the Asia Pacific technical director for a leading high-performance, buildings technology company. While based in Singapore, he was also the lead engineer for a number of successful energy-efficient high tech and mega projects across Asia Pacific, such as the multi-billion dollar Marina Bay Sands. Mr. Van Zetten has experience in on-site commissioning and troubleshooting data center and major projects throughout Asia, Australia, Europe, North America, and South America.