Avoid Failure and Delay on Capital Projects: Lessons from Tier Certification

How insights from Tier Certification of Constructed Facilities avoids unforeseen costs and errors for all new projects

By Kevin Heslin

Uptime Institute’s Tier Classification System has been a part of the data center industry lexicon for 20 years. Since its creation in the mid-1990s, the system has evolved into the global standard for third-party validation of data center critical infrastructure. At the same time, misconceptions about the Tier program confuse the industry and make it harder for a data center team to understand and prepare sufficiently for the rigors of the Tier process. Fortunately, Uptime Institute provides guidance and plenty of opportunity for clients to prepare for a Tier Certification well before the on-site review of functionality demonstrations.

This article discusses the Tier Certification process in more detail, specifically explaining the on-site demonstration portion of a Tier Certification. This is a highly rigorous process that requires preparation.

Uptime Institute consultants notice that poor preparation for a Tier Certification of Constructed Facility signals that serious problems will be uncovered during the facility demonstration. Resolving problems at this late stage may require a second site visit or even expensive mechanical and electrical changes. These delays may also cause move-in dates to be postponed. In other situations, the facility may not qualify for the Certification at the specified and expected design load.

TIER CERTIFICATION–HOW IT WORKS
Tier Certification is a performance-based evaluation of a data center’s specific infrastructure and not a checklist or cookbook. The process ensures that the owner’s facility has been constructed as designed and verifies that it is capable of meeting the defined availability requirements. Even the best laid plans can go awry, and common construction phase practices or value engineering proposals can compromise the design intent of a data center (See “Avoiding Data Center Construction Problems” by Keith Klesner, The Uptime Institute Journal, vol 5, p. 6).

With multiple vendors, subcontractors, and typically more than 50 different disciplines involved in any data center project—structural, electrical, HVAC, plumbing, fuel pumps, networking, and more—it would be remarkable if there were no errors or unintended risks introduced during the construction process.

The Tier Certification typically starts with a company deploying new data center capacity. The data center owner defines a business requirement to achieve a specific Tier (I-IV) level to ensure the asset meets performance capacity, effectiveness, and reliability requirements. Only Uptime Institute can certify that a data center design and/or constructed facility does, in fact, meet the owner’s Tier objective.

The first step is the Tier Certification of Design Documents. Uptime Institute consultants review 100% of the design documents, ensuring that each and every electrical, mechanical, monitoring, and automation subsystem meets the Tier objective requirements. Uptime Institute reports its findings to the owner, noting any Tier deficiencies. It is then up to the owner’s team to revise the drawings to address those deficiencies.

After Uptime Institute consultants determine that the revised design drawings correct the deficiencies, the Tier Certification of Design Documents foil may be awarded.

Immediately after the Design Certification is awarded, the Facility Certification begins when Uptime Institute consultants begin devising a list of demonstrations to be performed by site staff upon the conclusion of Level 5 Commissioning. Uptime Institute consultants provide the demonstrations to the client in advance and are available to answer questions throughout the construction process. In addition, Uptime Institute has created an instructional presentation for each Tier Level, so that owners can better understand the process and how to prepare for the site visit.

Fortunately, Uptime Institute provides guidance and plenty of opportunity for clients to prepare for a Facility Certification well before the on-site review of functionality demonstrations. Uptime Institute Senior Vice President, Tier Standards, Chris Brown said, “From my perspective a successful Facility Certification is one in which the clients are well prepared and all systems meet Tier requirements the first time. They have all their documents and planning done. And, pretty much everything goes without a hitch.”

Planning for the Facility Certification begins early in the construction process. Clients must schedule adequate time for Level 5 Commissioning and the site visit, ensuring that construction delays and cost overruns do not compromise time for either the commissioning or Certification process. They will also want to make sure that vendors and contractors will have the right personnel on site during the Certification visit.

Tier Certification of Constructed Facility:

• Ensures that a facility has been constructed as designed

• Verifies a facility’s capability to meet the defined availability requirements

• Follows multiple mechanical and electrical criteria as defined in the Tier Standard: Topology

• Seamlessly integrates into the project schedule

• Ensures that deficiencies in the design are identified, solved, and tested before operations commence

• Includes live system demonstrations under real-world conditions, validating performance according to the
facility’s Tier objective

• Conducts demonstrations specifically tailored to the exact topology of the data center

A team of Uptime Institute consultants will visit the site for 3-5 days, generally a little more than a month after construction of critical systems ends, allowing plenty of time for commissioning. While on site, the team identifies discrepancies between the design drawings and installed equipment. They also observe the results of each demonstration.

PREPARATIONS FOR TIER CERTIFICATION DEMONSTRATIONS
The value of Tier Certification comes from finding and eliminating blind spots, points of failure, and weak points before a site is fully operational and downtime results.

In an ideal scenario, data center facilities teams would have successfully completed Level 5 Commissioning (Integrated System Testing). The commissioning testing would have already exercised all of the components as though the data center were operational. That means that load banks simulating IT equipment are installed in the data center to fully test the power and cooling. The systems are operated together to ensure that the data center operates as designed in all specified maintenance and failure scenarios.

Everything should be running exactly as specified, and changes from the design should be minimal and insignificant. In fact, facility personnel should be familiar with the facility and its operation and may even have practiced the demonstrations during commissioning.

“Clients who are prepared have fully commissioned the site and they have put together the MOPs and SOPs (operating procedures) for the work that is required to actually accomplish the demonstrations,” said Chris Brown, Uptime Institute Senior Vice President, Tier Standards. “They have the appropriate technical staff on site. And, they have practiced or rehearsed the demonstrations before we get there.

“Because we typically come in for site visit immediately following Level 5 Commissioning, we ask for a commissioning schedule, and that will give us an indication of how rigorous commissioning has been and give us an idea of whether this is going to be a successful facility certification or not.”

Once on site, Uptime Institute consultants observe the Operations team as it performs the different demonstrations, such as interrupting the incoming utility power supply to see that the UPS carries the critical load until the engine generators come on line and provide power to the data center and that the cooling system maintains the critical load during the transition.

Other demonstrations include removing redundant capacity components from service and showing that, for example, N engine generators can support the data center critical load or that the redundant chilled water distribution loop can be removed from service while still sufficiently cooling the data center.

“Some of these demonstrations take time, especially those involving the cooling systems,” Brown said.

The client must prepare properly because executing the demonstrations in a timely manner requires that the client:

1. Create a schedule and order of demonstrations

2. Create all procedures required to complete the functionality demonstrations

3. Provide necessary technical support personnel

a. Technicians

b. Installers

c. Vendors

4. Provide, install, and operate load banks

5. Direct on-site technical personnel

6. Provide means to measure the presence of voltage on electrical devices during isolation by qualified personnel.

Even though the assessment evaluates the physical topology and equipment, organizations need proper staffing, training, and documentation in place to ensure the demonstrations are performed successfully.

AVOID TIER CERTIFICATION HORROR STORIES
Fundamentally, the Tier Certification process is a binary exercise. The data center facility is compliant with its stated Tier objective, or it is not. So, why then, do Uptime Institute consultants sometimes experience a sinking feeling upon arriving at a site and meeting the team?

Frequently, within the first moments at the facility, it is obvious to the Uptime Institute consultant that nothing is as it should be and the next several days are going to be difficult ones.

Uptime Institute consultants can often tell at first glance whether the client has met the guidance to prepare for the Tier Certification. In fact, the consultants will often know a great deal from the number and tone of questions regarding the demonstrations list.

“You can usually tell if the client has gone through the demonstration list,” said Uptime Institute Consultant Eric Maddison. “I prepare the demonstration list from schematics, so they are not listed in order. It is the client’s obligation to sequence them logically. So it is a good sign if the client has sequenced the demonstrations logically and prepared a program with timing and other information on how they are to perform the demonstration.”

Similarly, Uptime Institute consultants can see whether technicians and appropriate load banks are available. “You can walk into a data hall and see how the load banks are positioned and get a pretty good sense for whether there will be a problem or not,” Brown said, “If the facility has a 1,200-kilowatt load and they only have two large load banks, that’s a dead giveaway that the load banks are too large and not in the right place.”

Similarly, a few quick conversations will tell whether a client has arranged appropriate staffing for the demonstrations. The facility tour provides plenty of opportunity to observe whether the staff on site is knowledgeable about the facility and has practiced the demonstrations. The appropriate level of staff varies from site to site with the complexity of the data center. But, the client needs to have people on site who understand the overall design and can operate each individual piece of equipment as well as respond to unexpected failures or performance issues.

Brown recalled one facility tour that included 25 people, but only two had any familiarity with the site. He said, “Relying on a headcount is not possible; we need to understand who the people are and their backgrounds.

“We start with an in-brief meeting the morning of the first day for every Tier Certification and that usually takes about an hour, and then we do a tour of the facility, which can take anywhere from 2-4 hours, depending on the size of the facility. By the time those are finished, we have a pretty good idea of whether they are in the A, B, C, or D range of preparation. You start to worry if no one can answer the questions you pose, or if they don’t really have a plan or schedule for demonstrations. You can also get a good comfortable feeling if everybody has the answers, and they know where the equipment is and how it works.”

During this first day, the consultants will also determine who the client liaison is, meet the whole client team, and go over the goals of the program and site visit. During the site tour, the consultants will familiarize themselves with the facility, verify it looks as expected, check that valves and breakers are in place, and confirm equipment nameplates.

Having the proper team on site requires a certain amount of planning. Brown said that owners often inadvertently cause a big challenge by letting contractors and vendors go before the site visit. “They’ve done the construction, they’ve done the operations, and they’ve done the commissioning,” he said, “But then the Operations team comes in cold on Monday after everyone else has left. The whole brain trust that knows how the facility systems work and operate just left the site. And there’s a stack of books and binders 2-feet long for the Operations guys to read, good luck.”

Brown described a much better scenario in which the Uptime Institute consultants arrive on site and find, “The Operations team ready to go and the processes and procedures are already written. They are ready to take over on Day 1 and remain in place for 20 years. They can roll through the demonstrations. And they have multiple team members checking each other, just like they were doing maintenance for 5 years together.

“During the demonstrations, they are double checking process and procedures. ‘Oh wait,’ they’ll say, ‘we need to do this.’ They will update the procedure and do the demonstration again. In this case, the site is going to be ready to handover and start working the Monday after we leave. And everybody’s there still trying to finish up that last little bit of understanding the data center, and see that the process and procedures have all the information they need to effectively operate for years to come.”

In these instances, the owner would have positioned the Operations staff in place early in the project, whether that role is filled by a vendor or in-house staff.

In one case, a large insurer staffed a new facility by transferring half the staff from an existing data center and hiring the rest. That way, it had a mix of people who knew the company’s systems and knew how things worked. They were also part of the commissioning process. The experienced people could then train the new hires.

WHAT CAUSES POOR PERFORMANCE DURING TIER CERTIFICATION?
So with all these opportunities to prepare, why do some Tier Certifications become difficult experiences?

Obviously, lack of preparation tops the list. Brown said, “Often, there’s hubris. ‘I am a savvy owner, and therefore, the construction team knows what they are doing, and they’ll get it right the first time.’” Similarly, clients do not always understand that the Tier Certification is not a rubber stamp. Maddison recalled more than one instance when he arrived on site only to find the site unprepared to actually perform any of the demonstrations.

Failure to understand the isolation requirements, incomplete drawings, design changes made after the Design Certification, and incomplete or abbreviated commissioning can cause headaches too. Achieving complete isolation of equipment, in particular, has proven to be challenging for some owners.

Sometimes clients aren’t familiar with all the features of a product, but more often the client will open the breakers and think it is isolated. Similar situations can arise when closing valves to confirm that a chilled water system is Concurrently Maintainable or Fault Tolerant.

Incomplete or changed drawings are a key source of frustration. Brown recalled one instance when the Tier Certified Design Documents did not show a generator control. And, since each and every system must meet the Tier requirement, the “new” equipment posed a dilemma, as no demonstration had been devised.

More common, though, are changes to design drawings. Brown said, “During every construction there are going to be changes from what’s designed to what’s built; sometimes there are Tier implications. And, these are not always communicated to us, so we discover them on the site visit.”

Poor commissioning will also cause site visit problems. Brown said, “A common view is that commissioning is turning everything on to see if it works. They’ll say, ‘Well we killed the utility power and the generator started. We have some load in there.’ They just don’t understand that load banking only the generator isn’t commissioning.”

He cites an instance in which the chillers weren’t provided the expected cooling. It turned out that cooling tubes were getting clogged by dirt and silt. “There’s no way they did commissioning—proper commissioning—or they would have noticed that.”

Construction delays can compromise commissioning. As Brown said, “Clients expect to have a building handed over on a certain date. Of course there are delays in construction, delays in commissioning, and delays because equipment didn’t work properly when it was first brought in. For any number of reasons the schedule can be pushed. And the commissioning just gets reduced in scope to meet that same schedule.”

Very often, then, the result is that owner finds that the Tier Certification cannot be completed because not everything has been tested thoroughly and things are not working as anticipated.

If the commissioning is compromised or the staff is not prepared, can the facility pass the Tier Certification anyway?

The short answer is yes.

But often these problems are indications of deeper issues that indicate a facility may not meet the performance requirements of the owner, require expensive mechanical and electrical changes, or cause move-in dates to be postponed–or some combination of all three. In these instances, failing the Tier Certification at the specified and expected time may be the least of the owner’s worries.

In these instances and many others, Uptime Institute consultants can—and do—offer useful advice and solutions. Communication is the key because each circumstance is different. In some cases, the solutions, including multiple site visits, can be expensive. In other cases, however, something like a delayed client move-in date can be worked out by rescheduling a site visit.

In addition, Maddison points out the well-prepared clients can manage the expected setbacks that might occur during the site visit. “Often more compartmentalization is needed or more control strategy. Some clients work miracles. They move control panels or put in new breakers overnight.”

But, successful outcomes like that must be planned. Asking questions, preparing Operations teams, scheduling and budgeting for Commissioning, and understanding the value of the Tier Certification can lead to a smooth process and a facility that meets or exceeds client requirements.

The aftermath of a smooth site visit should almost be anticlimactic. Uptime Institute consultants will prepare a Tier Certification Memorandum documenting any remaining issues, listing any documents to be updated, and describing the need for possible follow-on site visits.

And then finally, upon successful completion of the process, the owner receives the Tier Certification of Constructed Facility letter, an electronic foil, and plaque attesting to the performance of the facility.

---

Uptime Institute’s Chris Brown, Eric Maddison, and Ryan Orr contributed to this article.

---

Kevin Heslin is Chief Editor and Director of Ancillary Projects at Uptime Institute. In these roles, he supports Uptime Institute communications and education efforts. Previously, he served as an editor at BNP Media, where he founded Mission Critical, a commercial publication dedicated to data center and backup power professionals. He also served as editor at New York Construction News and CEE and was the editor of LD+A and JIES at the IESNA. In addition, Heslin served as communications manager at the Lighting Research Center of Rensselaer Polytechnic Institute. He earned the B.A. in Journalism from Fordham University in 1981 and a B.S. in Technical Communications from Rensselaer Polytechnic Institute in 2000.