Lessons from the Tier Certification of Constructed Facility

Telecom company Entel achieves first two Tier III Certification of Constructed Facility Awards in Chile

Empresa Nacional de Telecomunicaciones S.A. (Entel) is the largest telecommunications company in Chile. The company reported US$3.03 billion in annual revenue in December 2012, and an EBITDA around 40%. In Chile, Entel holds a leading position in traditional data services, growing through the integration of IT services, with significant experience.

The company also offers mobile and wireline services (including data center and IT services, Internet, local telephony, long distance and related services) and call center operations in Peru. Its standardized service offering to all companies is based on cloud computing.

To deliver these services, Entel has developed the largest mobile-fixed wire network in Chile, with 70 gigabyte per second (Gbps) of capacity. The company serves 9.3 million mobile customers and offers an MPLS-IP network of wide and national coverage with quality of service (QoS) with 8.8 Gbps peak traffic in 2011. The company also provides international network connectivity (internet peak traffic of 17.7 Gbps in 2011).

Entel is also a large data center infrastructure provider, with more than 13,800 square meters (m²) in seven interconnected data centers (see Figure 1):

Ciudad de Los Valles
Longovilo (three facilities)
Amunátegui
Ñuñoa
Pedro de Valdivia

These facilities host more than 7,000 servers and 2,000 managed databases. Entel currently plans to add 4,000 msup>2 in two steps at its Ciudad de los Valles facilities. As part of its IT services, Entel also provides an operational continuity service to more than 80,000 PCs and POS and terminals countrywide. Its service platform and processes are modeled under the ITIL framework, and Entel has SAS-70/II and COPC certifications.

Finally, Entel also provides processing services, virtualization, on-demand computing, SAP outsourcing and other services. Entel has seen rapid growth in demand for IaaS platforms, that it meets with robust and multiple platform offerings (iSeries, p Series, Sun Solaris and x86) and different tiers of storage. Finally, the company has a 38% market share of managed services to large financial service institutions.

Entel has defined its corporate strategy to enable it to continue to lead the traditional telco market as well as increase coverage and capacity by deploying a fiber optic access network (GPON). The company also wants to increase its market share in O/S, data center, and virtual and on-demand services and expand to other countries in the region, leveraging its experience in Chile.

Entel chose to invest in its own data centers for three reasons, two of which related directly to competitive advantage:

Investing in its own infrastructure would help Entel develop its commercial offering to corporate clients in Chile. Entel believed that having control over its basic infrastructure would enable it guarantee the operational continuity of customers.
Entel also wanted to add white space when it felt more capacity was needed. It feels this flexibility helped it win additional large deals.
The country does not have enough experienced personnel with the experience to manage/coordinate facilities like Entel’s. The company finds it a challenge to coordinate and supervise subcontractors responsible for various systems.

Figure 1. General Scope of Entel’s Operations

As part of this process, Entel determined that it would work with the Uptime Institute to earn Tier Certification for Constructed Facility at its Ciudad de Los Valles Data Center. As a result, Entel learned several important lessons about its operations and also decided to obtain the Uptime Institute’s Tier Certification of Operational Sustainability.

Project Master Plan

The initial phase of the project included the construction of a 2,000 m² of floor space for servers. A second phase was recently inaugurated that added another 2,000 m² of floor space for additional equipment. The complete project is intended to reach a total growth of 8,000 m² distributed in four buildings, each one with two floors of 1,000 m², and with a total of 26,000 m² of building space. (The general layout of the data centers is shown in Figure 2.)

Figure 2. General layout of the Entel facilities

The data centers had to meet four key requirements:

1. It had to house the most demanding technical and IT equipment: 1,500 kilograms (kg)/m² in IT spaces and more than 3,000 kg/m² in battery spaces, with special support for heavier equipment such as transformers, diesel generators, and chillers.

2. It had to achieve Uptime Institute Tier III requirements, meaning that the design includes two independent and separate power and cooling distribution paths, each able to support 100% of the capacity to allow Concurrent Maintainability so that all the critical elements can be replaced or maintained without impact in the service.

3. The building had to have sufficient capacities to meet high electrical and cooling demand (see Figure 3).

4. Service areas such as UPS and meet-me rooms had to be outside of the server rooms.

The structural design of the facility also had to address the threat of earthquake.

Entel decided to certify its data centers to meet its commercial commitments. Due to contractual agreements, Entel had to certify the infrastructure design and its facilities with an internationally recognized institution. In addition Entel wanted to validate its design and construction for audit processes and as a way to differentiate itself from competitors.

Figure 3. Comparison of two of Entel’s facilities.

The first step was to choose a standard to follow. In the end, Entel decided to certify Ciudad de Los Valles according to Tier Standards by the Uptime Institute because it is the most recognized body in the Chilean market and customers were already requesting it. As a result Entel’s facility was the first in Chile to earn Tier Certification of Constructed Facility.

Preparation

Ciudad de Los Valles Data Center is a multi-tenant data center that Entel uses to provide housing and hosting services, so its customers also had to be directly involved in project planning, and their authorization and approvals were important. At the time the Tier Certification of Design Documents began, the facility was in production, with important parts of its server rooms at 100% capacity. Modifications to the infrastructure had to be done without service disruptions or incident.

Tier III Certification of Constructed Facility testing had to be done at 100% of the electrical and air conditioning approved design load so coordination was extremely challenging.

As part of the design review, the Uptime Institute consultants recommended some important improvements:

Separate common electrical feeds from the electrical generators
Additional breakers to isolate bus bars serving critical loads
Separate water tanks (chilled water tanks from building tanks)
Redundant electrical feeders for air handling units

Additionally, it was essential that all the servers and communication equipment have redundant electrical feeders to avoid incidents during performance tests.

Risks and Mitigation

Recognizing that Tier Certification testing would be challenging, Entel developed an approval process to help it meet what it considered four main challenges:

Meeting the Tier Certification timeline with a range of stakeholders to inform and coordinate
Preventing incidents due to activities necessary to test the infrastructure
Avoiding delays caused by coordination problems and obtaining approvals and permissions to modify the infrastructure
The possible existence of unidentified/undocumented single-corded servers and telecom equipment

The approval process included a high-level committee, including commercial and technical executives to ensure communications. In addition every technical activity had to be approved in advance, including a detailed work plan, timing, responsibilities, checkpoints, mitigation measures and backtracking procedures. And for the most important activities, the plan was presented to the most important customers.

The Certification process proved to be an excellent opportunity to improve procedures and test them. The process also made it possible to do on-the-job training and check the data center facilities at full load.

In addition, Entel completed the Tier Certification process of both design and facility without experiencing any incidents. However changes made to meet Uptime Institute Tier III requirements led Entel to exceed its project budget. In addition, unidentified/undocumented single-corded equipment delayed work and the eventual completion of the project. In order to proceed, Entel had to visually check each rack in the eight data halls rack to identify the single-corded loads. Once the loads had been identified, Entel installed static transfer switches (STS) to protect the affected racks, which required coordinating the disconnection of the affected equipment and its reconnection to the associated STS.

Lessons Learned

Entel learned a number of important lessons as a result of Tier Certification of two data centers.

The most important conclusion is that is absolutely possible to Tier Certify an ‘in-service data center’ with good planning and validation, approval and execution procedures. At the Ciudad de Los Valles 1 Data Center, Entel learned the importance of having a good understanding of the Tier Certification process scope, timing and testing, and the necessity of having good server room equipment inventories.

As a result of Tier Certification of Ciudad de Los Valles 2 Data Center, Entel can attest that is easier to certify a new data center with construction, commissioning and certification all taking place in order before the facility goes live.

Next Steps

Entel’s next goal is to obtain the Uptime Institute’s Operational Sustainability Certification. For a service provider like Entel, robust infrastructure is not enough:

A good operation regime is as important as the design, construction and testing
It’s critical to have detailed maintenance plans and testing of the infrastructure equipment
Entel wants its own people to control operations
Entel needs well trained people

Entel is preparing for this Certification by adapting its staffing plan, maintenance procedures, training and other programs to meet Uptime Institute requirements.

In addition to its other ‘firsts’ in terms of Tier III Certification of Constructed Facility—twice over—Entel seeks to be the first data center to achieve Tier III Gold Certification.

Mr. Juan Miquel Durán, electrical engineer, joined Entel Chile in September 2011. He has more than 20 years of experience, with strong knowledge of data centers, switching and telecommunication services.

He is responsible for Facilities operation and maintenance for Ciudad de los Valles TIER III Certified Data Centers. Mr. Duran is also responsible Entel’s Longovilo Data Centers.

He participated in the certification process team, and he has strong experience in data center operations (Negotiation and implementation of long terms housing contracts, according international standards), as well as in infrastructure mission critical data center projects planning and design, including tender formulation, designing, projects evaluation, management for different companies.