Thinking Ahead Can Prevent the Mid-Life Energy Crisis in Data Centers

Turning back the clock at Barclays Americas’ data centers

By Jan van Halem and Frances Cabrera

Barclays is a major global financial services provider engaged in personal banking, credit cards, corporate and investment banking, and wealth and investment management with an extensive international presence in Europe, the Americas, Africa, and Asia. Barclays has two major data centers in the northeastern United States with production environments that support the Americas region’s operations (see Figure 1). Barclays Corporate Real Estate Solutions (CRES) Engineering team manages the data centers, in close partnership with the Global Technology and Information Services (GTIS) team.

Both of Barclays Americas’ data centers are reaching 7 years old, but not showing their age—at least not energy wise. For the last 3 years, Barclays Americas’ engineering and IT teams have been on an energy-efficiency program to ensure that the company’s data center portfolio continues operating even more efficiently than originally commissioned.

By 2013, Barclays Americas’ two data centers had reduced their energy consumption by 8,000 (megawatt-hours (MWh) or 8%, which equates to 3,700 tons of carbon emissions avoided. In addition, the power usage effectiveness (PUE) of the largest data center dropped from an annual average of 1.63 to 1.54–earning it an Energy Star certification for 2013.

The Barclays Americas team pinpointed the following strategies for a three-pronged attack on energy inefficiency:

• Airflow management
• Enhancement of cooling operations
• Variable frequency drive (VFD) installations on computer room air
  conditioning (CRAC) units and pumps

The goals were to:

• Reduce total cooling
• Enhance heat transfer efficiencies
• Reduce cooling losses (i.e., short cycling of air)

The team found considerable savings without having to make large capital investments or use complex solutions that would threaten the live data center environment (see Figure 2). The savings opportunities were all identified, tested, implemented, and validated using in-house resources with collaboration between the engineering and IT teams.

Background

In 2010, Barclays launched a Climate Action Programme, which has since been expanded into its 2015 Citizenship Plan. The program included a carbon reduction target of 4% by the end of 2013 from 2010 levels. The target spurred action among the regions to identify and implement energy conservation measures and created a favorable culture for the funding of initiatives. Due to energy initiatives like the data center programs put in place in the Americas, Barclays reduced its carbon emissions globally by 12% in 2012, meeting the target two years early. Barclays is now setting its sights on a further carbon reduction of 10% by 2015.

From the beginning, it was understood that any new data center energy initiatives in the Americas must take into account both IT and building operation considerations. CRES and GTIS worked together on research, testing, and mocking up these initiatives. Their findings were shared with the central teams at headquarters so that case studies could be developed to ensure that the successes can be used at other sites around the world.

Blowing Back the Years

Initially, the team focused on airflow management. The team successfully reduced CRAC unit fan speeds at DC 1 from 70% to 40% by raising temperature set points as recommended by ASHRAE TC9.9, replacing unneeded perforated tiles with solid tiles, and installing Cold Aisle containment. All these changes were implemented with no impact to equipment operation.

The first step, and the easiest, was to evaluate the initial fan speed of the CRAC units at DC 1, and reduce it from 70% to 58%, with no effect on performance. When the DCs were commissioned they were briefly operating at 55°F (15°C), which was quickly increased to 65°F (18°C). By 2010, the temperature set points in Cold Aisles were raised to 75°F (24°C) from 65°F (18°C) in both DC 1 and DC 2 as recommended by updated ASHRAE standard for data center cooling.

Next, Barclays’ teams turned to the perforated tiles. Perforated tiles in DC 1 were replaced with solid tiles. This reduced CRAC fan speeds from 58% to 50%, with no impact on Cold Aisle temperature conditions and equipment operation.

Finally, in 2011, the PDU aisles were retrofitted with containment doors, which further improved airflow efficiency. After the site teams researched various options and recommended a commercial product, an in-house crew completed the installation. The team opted to use an in-house personnel to avoid having external contractors working in a live environment, which meant that the IT operations team felt comfortable having trusted engineering staff in the server rooms. The engineering staff used its expertise and common sense to install the doors with no disturbance to the end users.

The team chose not to extend the containment above the aisles. Using data from wireless temperature and humidity sensors located throughout the aisles, the team found that it could still achieve ~80% of the projected energy savings without having the containment reaching the ceiling and avoid the additional cost and potential issues with local fire codes.

With the doors installed, the teams continue monitoring temperatures through the sensors and can control airflow by adjusting CRAC fan speeds,which regulates the amount of supply air in the Cold Aisle to minimize the bypass overflow and ensure efficient air distribution. As a result, return air temperature increased, driving efficiency and allowing further fan speed reductions of 50% to 40%. Again, these reductions were achieved with no impact to operating conditions. Based on these successes, Cold Aisle containment was installed at DC 2 in 2014.

A curve based on manufacturer’s data and direct power readings for the CRAC units enabled the teams to calculate the power reductions associated with volume flow rate reductions As a result, they calculated that the airflow initiatives resulted in 3.6 gigawatt-hours (GWh) of energy annually across DC 1 and DC 2 (see Figure 3).

Staying Cool

DC 1’s primary cooling system is a water-cooled plant consisting of two 1,250-ton chillers and five cooling towers. DC 1 was designed to utilize free cooling technology that feeds condenser water from cooling towers into a heat exchanger. The heat exchanger then cools the building’s chilled water. In 2011, the CRES Engineering team embarked on an initiative to maximize the use of free cooling.

After reviewing operational parameters such as temperature, water flow, and cooling tower fan speeds, Barclays team made the following adjustments:

• Adding additional units to the series to slow down the water pumped through the heat exchanger. Making
  the water go through two units instead of one increased the efficiency of the heat exchange.
• Running two cooling tower unit fans at half the speed of one fan to reduce power demand as the result of
  analyzing data from an electrical power monitoring system (EPMS). As a result, the volume of
  condenser water was divided among multiple cooling tower units instead of all going into one.
• Increasing chilled water temperature 4°F (7°C) from 2011 to today, expanded the period of time that
  free cooling is possible (see Figure 4).

Of the three strategies, it is hardest to directly measure and attribute energy savings to enhancing cooling operations, as the changes impact several parts of the cooling plant. Barclays used the EPMS to track power readings throughout the systems, particularly the cooling tower units. The EPMS enables PUE trending and shows the reduction of DC 1’s PUE overtime. Since 2011, it’s dropped 5% to an annual average of 1.54.

Driving Back Inefficiency

In 2013 the teams began an intensive review of VFD technology. They found that considerable energy savings were to be obtained by installing VFDs on several pieces of equipment such as air handlers in plant rooms and condenser water pumps in both DC 1 and DC 2. The VFDs control the speed of an existing AC induction motor. By reducing the speed on air handlers, the unit load can be adjusted to the existing heat load (see Figure 5).

The team focused on (36) 30-ton units throughout DC 1 that would yield positive energy and cost savings. Utility rebates further enhanced the business case. The VFDs were installed towards the end of 2013, and the business case applied to DC 2 for further installations in 2014.

To calculate savings, direct power readings were taken at the CRAC units at intervals of 2 hertz (Hz) from 60 Hz to 30 Hz. As shown in Figure 6, reducing CRAC frequency from 60 Hz to 56 Hz reduced power demand by 19%. In addition, the fan motor will release less heat to the air further reducing cooling load.

Additional maintenance cost savings are achieved through the extension of the filter replacement cycle. The VFDs allow less air to go through the system increasing the life span of the system filters. Once fully implemented the VFD installations will save over 3.9 GWh of energy.

Data as the Fountain of Youth

Comprehensive monitoring systems in place across the two data centers provided data accessible by both the GTIS and CRES teams, enabling them to make the best, data-driven decisions. The use of the sites’ EPMS and branch circuit monitoring system (BCMS) enable the teams to pinpoint areas with the greatest energy-saving potential and work together to trial and implement initiatives.

Barclays uses the EPMS as a tool to monitor, measure, and optimize the performance of the electrical loads. It monitors critical systems such as HVAC equipment, electrical switchgear, etc. A dashboard displays trend data. For example, the team can trend the total UPS load and the total building load, which yields PUE, on a continuous, real-time basis.

In addition to the EPMS, the CRES and GTIS teams also use the BCMS to track energy use by server cabinet, remote power panel, and power distribution unit. This system is used for capacity planning and load balancing. In addition, the monitored cabinet loads are used to optimize the airflow in the Cold Aisles.

Conclusion

With the right level of collaboration between CRES and GTIS, the right data, and the right corporate environmental targets, the Barclays Americas’ team was able to find energy and cost savings in their data centers. By executing airflow management, enhanced cooling, and VFD strategies in the existing data centers, the team applied the latest standards and best practices to keep energy consumption at levels typical of new data centers. At 8 GWh lighter, with an Energy Star certification and a PUE that keeps dropping—these data centers are not showing their energy age.

---

 

Jan van Halem is vice president, Data Center Engineering at Barclays. He joined Barclays’ engineering team in 2004. Mr. van Halem has more than 20 years experience and a strong knowledge of critical data center systems and mechanical operations. At Barclays, he is responsible for the mechanical and electrical systems of the company’s major data centers in the Americas. In addition, he has provided engineering design to new construction and expansion projects in the region.

Before joining Barclays, Mr. van Halem was with real estate organizations and held positions in facility management, construction management, and project management. Mr. van Halem has a BS degree in Marine Engineering from the Maritime Institute De Ruyter in Flushing, the Netherlands. He served as a marine engineer for 8 years in the Dutch Merchant Marine.

Frances Cabrera, LEED AP, is vice president, Environmental Management at Barclays. She joined Barclays’ environmental management team in 2011. Ms. Cabrera oversees Barclays Americas’ environmental programs, both resource saving and compliance, to support the region’s ISO certified management system. With the collaboration of the corporate real estate team, the region has achieved multiple LEED and Energy Star certifications. She’s also part of the firm’s global center of excellence for environment, where she works across regions to measure and support the firm’s green IT achievements.

Before joining Barclays, Ms. Cabrera ran the ISO 14001 systems in North and South America for Canon USA and worked at various manufacturing companies in Rochester, NY, integrating environmental considerations into their operations and meeting regulations. Ms. Cabrera has a BS degree in Environmental Technology and a MS degree in Environmental, Health, and Safety Management from the Rochester Institute of Technology.