Russia’s First Tier IV Certification of Design Documents

Next Step: Preparing for Facility Certification

By Alexey Karpov

Mordovia Republic-Technopark Mordovia Data Center (Technopark Data Center) is one of the most significant projects in Mordovia (see Figure 1). The facility is a mini-city that includes research organizations, industry facilities, business centers, exhibition centers, schools, a residential village, and service facilities. One of the key parts of the project is a data center intended to provide information, computing, and telecommunication services and resources to residents of Technopark-Mordovia, public authorities, business enterprises of the region, and the country as a whole. The data processing complex will accommodate institutions primarily engaged in software development, as well as companies whose activities are connected with the information environment and the creation of information resources and databases using modern technologies.

The data center offers colocation and hosting services, hardware maintenance, infrastructure as a service (IaaS) through a terminal access via open and secure channels, and access to Groupware software based on a SaaS model. As a result, Technopark Data Center will minimize the residents’ costs to conduct research, manage general construction and design projects, and interact with consumers in the early stages of production through outsourcing of information and telecommunication functions and collective use of expensive software and hardware complexes. Mordovia created and helped fund the project to help enterprises develop and promote innovative products and technologies. About 30 leading science and technology centers cooperate with Technopark-Mordovia, conduct research, and introduce into production new and innovative technologies, products, and materials because of the support of the Technopark Data Center (see Figure 2).

Why Design Certification?
Technopark Data Center is the largest and most powerful computing center in Mordovia. Its designers understood that the facility would eventually serve many of the government’s most significant social programs. In addition, the data center would also be used to test and run Electronic Government programs, which are currently in development. According to Alexey Romanov, Director of Gosinform, the state operator of Technopark-Mordovia, “Our plan is to attract several groups of developers to become residents. They will use the computing center as a testing ground for developing programs such as Safe City, medical services for citizens, etc. Therefore, we are obliged to provide the doctors with round the clock online access to clinical records, as well as provide the traffic police with the same access level to the management programs of the transport network in the region.”

To meet these requirements, Technoserv followed the provisions of Uptime Institute requirements for engineering infrastructure (Data Center Site Infrastructure Tier Standard: Topology). As a result, all engineering systems are designed to fully meet requirements for Uptime Institute Tier IV Certification of Design Documents for redundancy, physical separation, and maintenance of equipment and distribution lines (see Figures 3 and 4).

Meeting these requirements enables Mordovia to achieve significant savings, as the Technopark Data Center makes possible an overall data center plan that makes use of lower reliability regional centers. Though not Tier Certified by the Uptime Institute, these regional data centers are built to follow redundant components requirements, which reduces capital costs. Meanwhile, the central data center provides backup in case one of the regional data centers experiences downtime.

The Technopark Data Center is the core of all IT services in Mordovia. The regional data centers are like “access terminals” in this environment, so the government reasoned that it was not necessary to build them to meet high reliability requirements.

The Data Center Specification
The Technopark Data Center is a 1,900-kW facility that can house about 110 racks, with average consumption of 8-9 kW per rack. Power is supplied from four independent sources: two independent feeds from the city’s electricity system and diesel generator sets with 2N redundancy.

Main characteristics:
(See Table 1)

The data center building is a multi-story structure. Servers occupy the first floor: computing resources are placed in three areas, and various types of IT equipment (basic computing, telecommunications, and storage systems) are placed in different rooms. The administrative block and call-center are on the second floor.

Chillers, pumping stations, chilled water storage tanks, and UPS batteries, etc. are located in the basement and technical floors. Transformers and diesel generators are located in a separate area adjoining the data center. Diesel fuel tanks are located in two deepened areas at opposite sides of the building.

The data center design includes several energy-saving technologies, which enables the facility to be very energy efficient by Russian standards (PUE <1.45). For example, the cooling system includes a free-cooling mode, and all power and cooling equipment operate in modes intended to provide maximum efficiency. Other energy efficiency details include:

• Computing equipment is installed in a Cold Aisle/Hot Aisle configuration, with containment of the Hot Aisles. In-row cooling further improves energy efficiency.
• The cooling system utilizes efficient chillers with screw compressors and water-cooled condensers. The dry cooling towers installed on the roof refrigerate the condensers of the chillers in the summer. In the winter, these cooling towers help provide free cooling. Calculations for the design of the cooling system and air conditioning were performed according to ASHRAE standards.
• All elements of the engineered systems, as well as the systems themselves, are integrated into a single BMS. This BMS controls all the necessary functions of the equipment and interconnected subsystems and quickly localizes faults and limits the consequences of emergencies. Technoserv utilizes a distributed architecture in which each component has a dedicated controller that feeds information back to a single BMS. If the BMS servers fail, the individual controllers maintain autonomous control of the facility.The BMS also collects and processes exhaustive amounts of information about equipment, issues reports, and archives data. A control room is provided at the facility for operators, where they can monitor the operation of all elements of the engineering infrastructure.

From a security standpoint, the data center is organized into three access levels:

• Green areas provide open admission for users and to the showroom.
• Blue areas are restricted to Technopark Data Center residents performing their own IT projects.
• Red areas are open only to data center staff.

Three independent fiberoptic lines, each having a capacity of 10 Gbits per second, ensure uninterrupted and high capacity data transmission to users of Technopark Data Center’s network infrastructure. Russia’s key backbone operators (Rostelecom, Transtelekom, and Megaphone) were selected as Technopark Data Center’s telecom partners because of their well-connected and powerful infrastructure in Russia.

The data center also includes a monitoring and dispatching system. The system is based on three software products: EMC Ionix (monitoring the availability of all components of the IT infrastructure), EMC APG (accumulation of statistics and performance analysis), VMware vCenter Operations Enterprise (intelligent performance monitoring and capacity of objects the virtual environments VMware), and integration modules specially designed by Technoserv.

Challenges

As noted previously, the data center was designed to achieve the highest levels of reliability. There are some data centers in Russia that perform critical national tasks, but none of those facilities require the highest levels of reliability. This reality made the task seem more daunting to everyone who worked on it. Technoserv had to do something that had never been done in Russia and do so in a limited time. Technoserv managed to accomplish this feat in less than two years.

During the Uptime Institute’s Design Certification process, Technoserv stayed in close contact with Uptime Institute subject matter experts. As a result, Technoserv was able to develop solutions as problems emerged. The company is also proud of the qualifications of Technoserv specialists, who have extensive experience in designing and building data centers and who provided the basis for the successful completion of this project.

The technical challenge was also significant. Meeting Tier IV Design Documents requirements can require a large number of redundant elements, the close relationship of mechanical and electrical systems, and testing to demonstrate that emergencies can be addressed without human intervention or damage to IT equipment.

It was necessary to account for all developments in the space and then properly develop BMS hardware that would meet these potential challenges. In addition, the automation system should also work with no loss of functionality in the event of a fault of the BMS system. Design and implementation of algorithms for the BMS demanded involvement of the automation division of Technoserv and almost 6 months of hard work.

It was important to limit the noise from the engineering equipment, as the data center is located in a residential area. Noise insulation measures required examination of the normative and regulatory documents. Knowledge of local codes was key!

Lessons Learned
Technoserv also learned again that there no minor details in a high-tech data center. For example, a topcoat applied to the floor during construction caused the floor to oxidize actively. Only after numerous measurements and testing did Technoserv find that the additive in the coating composition had entered into an electrochemical reaction with the metal supports that formed sulfuric acid and caused an electric potential on the racks of the raised floor.

The data center is currently operational. Technoserv plans to complete the Tier IV Certification of Constructed Facility process.

Alexey Karpov is head of the Data Center Construction Department at Technoserv. Having more than 10 years experience in designing and building data centers, Mr. Karpov is an Accredited Tier Designer, Certified Data Centre Design Professional, and Certified Data Centre Management Professional. VTB Bank, recognized as the largest infrastructure project in Russia in 2010, and the data center for Bashneft are two large-scale projects completed under his guidance. Technoserv, Russia’s largest system integrator, was founded in 1992.Technoserve installs, develops, and outsources IT infrastructure and develops communications, engineering, and information security systems as well as power systems and application platforms. According to RA Expert, a leading Russian analytical agency, Technoserv is a leader in providing IT services in Russia. Business volumes confirm the company’s leadership in the Russian IT market; total revenues for the entire Technoserv group of companies exceeded 43 billion rubles in fiscal year 2012.