To challenge established industry assumptions regarding data center cooling, Intel IT conducted a proof of concept (PoC) test that used an air economizer to cool production servers with 100 percent outside air at temperatures of up to 90 degrees Fahrenheit (F). With this approach, we could use an economizer to provide nearly all data center cooling, substantially reducing power consumption. This could potentially reduce annual operating costs by up to USD 2.87 million for a 10-megawatt (MW) data center.
We ran the PoC in a dry, temperate climate over 10 months using about 900 production blade servers, divided equally between two side-by-side compartments, as shown in Figure 1.
One used standard air conditioning, the other an air economizer.
Servers in the economizer compartment were subjected to considerable variation in temperature and humidity as well as poor air quality; however, there was no significant increase in server failures. If subsequent investigation confirms these promising results, we anticipate using this approach in future high-density data centers.
Intel Information Technology
Reducing Data Center Cost
with an Air Economizer
Profile: Air Economizer PoC
900 heavily utilized production servers in a high-density data center
100% air exchange at up to 90 degrees F, with no humidity control and minimal air filtration
67% estimated power savings using economizer 91% of the time—an estimated annual savings of approximately USD 2.87 million in
a 10-MW data center •
Figure 1. Proof of concept (PoC) data center environment.
Server RacksServer RacksTemperature and Humidity SensorHot AirCold AirTraditional Air-ConditionedCompartmentHot air is cooled and recirculatedAir Economizer CompartmentHot air is flushed outside, andoutside air is drawn inAir ConditioningUnitAirEconomizer
Data center power consumption is soaring, driven by increasing demand for computing capacity. In a typical data center, 60 to 70 percent of data center power may be used for facilities power and data center cooling.
At Intel, our data centers need to support the rapid growth in computing capacity required to design increasingly complex semiconductors. At the same time, we are trying to minimize data center power consumption and operating costs.
Our strategy is based on high-performance, high-density data centers containing thousands of blade servers. These blades deliver considerable computing capacity, but they also generate substantial heat. We supply cooling air to the blades at 68 degrees F; as the air passes over the blades, the air temperature rises by 58 degrees F, resulting in an exit temperature of 126 degrees F. This means that we need to cool the air by 58 degrees F before recirculating it. The air conditioning units required to do this consume a considerable amount of electricity.
Air economizers represent one potential way to reduce data center power consumption and cooling cost. Instead of cooling and recirculating the hot air from the servers, air economizers simply expel the hot air outdoors and draw in outside air to cool the IT equipment.
The current industry assumption is that the usefulness of air economizers is limited by the need to supply cooling air at a relatively low temperature. The implication is that air economizers can only be used at times when the outside air is relatively cool. There is also concern about variation in humidity, because the humidity of outside air can change rapidly. A third area of concern is particulate counts in outside air.
We decided to challenge these assumptions by employing an economizer to cool a high-density production environment using a much wider range of outside air temperatures—up to 90 degrees F. We reasoned that this might be feasible because server manufacturers specify that their products can operate in temperatures as high as 98 degrees F. We also wanted to push the accepted limits of humidity and air quality.
If we were successful, we would be able to use air economizers for most of the year in dry, temperate climates. This could drastically reduce power consumption and cooling costs while improving Intel’s environmental footprint.
Proof of Concept
We conducted a large PoC test using approximately 900 production design servers at a data center located in a temperate desert climate with generally low relative humidity. We began the PoC in October 2007, and continued the test for 10 months until August 2008.
We set up the PoC in a 1,000-square-foot (SF) trailer that was originally installed to provide temporary additional computing capacity and divided the trailer into two compartments of approximately 500 SF each. To minimize the cost of the PoC, we used low-cost, warehouse-grade direct expansion (DX) air conditioning equipment. Temperature and humidity sensors were installed to monitor the conditions in each compartment.
We cooled one compartment with a traditional approach, using a DX unit to recirculate hot air and provide cooling at all times.
For the other compartment, we used essentially the same air-conditioning equipment, but with modifications that enabled it to operate as an economizer by expelling hot air to the outdoors and drawing in 100 percent outside air for cooling.
Because one of our goals was to test the acceptable limits of operating temperature, we configured the cooling equipment in the economizer compartment to supply air at temperatures ranging from 65 degrees F to 90 degrees F. We designed the system to use only the economizer until the supply air exceeded the 90 degree F maximum, at which point we began using the chiller to cool the air to 90 degrees F. If the temperature dropped below 65 degrees F, we warmed the supply air by mixing it with hot return air from the servers.
We made no attempt to control humidity. We also wanted to test the limits of air quality, so we applied minimal filtering to incoming air, using a standard household air filter that removed only large particles from the incoming air but permitted fine dust to pass through.
Each room contained eight racks. Each rack contained four blade servers with 14 blades each, for a total of 448 blades per compartment. This represented a power density of more than
200 watts per square foot (WPSF).
During the PoC, we used the servers to run large production batch silicon design workloads, resulting in very high server utilization rates
of about 90 percent.
We measured server failure rates in each compartment and compared them with the failure rates that we experienced during the same period within our main data center at the same location.
During the PoC, the servers in the economizer compartment were subjected to wide variation in environmental conditions, as shown in Figure 2.
The temperature of the supply air varied from 64 degrees F to
more than 92 degrees F. This variation slightly exceeded our set points partly due to the slow response of our low-cost air conditioning units.
Humidity varied from 4 percent to more than 90 percent and
changed rapidly at times.
The servers and the interior of the compartment became covered
in a layer of dust.
Total power consumption of the trailer was approximately 500 kilowatts (KW) when using air conditioning in both compartments. When using the economizer, the DX cooling load in the economizer compartment was reduced from 111.78 KW to 28.6 KW, representing a 74 percent reduction in energy consumption.
Server Failure Rates
Despite the dust and variation in humidity and temperature, there was only a minimal difference between the 4.46 percent failure rate in the economizer compartment and the 3.83 percent failure rate in our main data center over the same period. The failure rate in the trailer compartment with DX cooling was 2.45 percent, actually lower than in the main data center.
We estimated the average annual power savings we could achieve in a data center that uses an economizer. To do this, we used historical weather data for the data center location, summarized in Table 1. Analysis of the data indicated that during an average year, the temperature is below our 90 degrees F maximum 91 percent of the time.
Based on our 74 percent measured decrease in power consumption when using the economizer during the PoC, and assuming that we could rely on the economizer 91 percent of the year, we could potentially save
Figure 2. Temperature and humidity variation in the air economizer compartment from January through May 2008.
05101520253035%0102030405060708090100º FPercent HumidityTemperature in Degrees Fahrenheit (° F)TemperatureHumidityJanuaryFebruaryMarchAprilMay
Table 1. Average Annual Temperatures at the Proof of Concept (PoC) Test Location
48 Degrees Fahrenheit (° F)
approximately 67 percent of the total power used annually for cooling compared with a traditional data center cooling approach. This translates into approximately 3,500 kilowatt hours (KWH) per KW of overall data center power consumption, based on an assumption that 60 percent of data center power typically is used for mechanical cooling systems.
This would result in an estimated annual cost reduction of approximately USD 143,000 for a small 500-KW data center, based on electricity costs of 0.08 per KWH. In a larger 10-MW data center, the estimated annual cost reduction would be approximately USD 2.87 million.
In addition, we could avoid certain capital expenditures in new data centers because we would require less cooling equipment. Even when the outside air temperature exceeded our maximum supply air temperature, we would only have to cool the air to our specified maximum rather than to the 68 degrees F used in our traditional data center approach. Reducing the complexity and cost of the cooling system also reduces the number of failure modes, increasing overall resiliency.
We observed no consistent increase in server failure rates as a result of the greater variation in temperature and humidity, and the decrease in air quality, in the trailer. This suggests that existing assumptions about the need to closely regulate these factors bear further scrutiny.
Air economizers seem particularly suited to temperate climates with low humidity. A data center equipped with an air economizer could substantially reduce Intel’s environmental footprint by reducing consumption of both power and water. In dry climates, traditional air-conditioned data centers typically include evaporative cooling using water towers as a pre-cooling stage. With an economizer, this would not typically be used, potentially saving up to 76 million gallons of water annually in a 10-MW data center.
We plan to further test for possible hardware degradation using a server aging analysis that compares systems used in the economizer compartment, in the air-conditioned compartment, and in our main data center.
If subsequent investigation confirms our promising PoC results, we expect to include air economizers in future data center designs. A possible next step would be a 1-MW demonstration data center using the equipment designed for the PoC.
This paper is for informational purposes only. THIS DOCUMENT IS PROVIDED “AS IS” WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. Intel disclaims all liability, including liability for infringement of any proprietary rights, relating to use of information in this specification. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted herein.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2008 Intel Corporation. All rights reserved.
Printed in USA Please Recycle
Don Atwood is a regional data center manager with Intel Information Technology.
John G. Miner is a senior systems engineer with Intel
Intel IT’s Eight-year Data Center
This proof of concept is part of Intel IT’s enterprise-wide, eight-year data center efficiency strategy, begun in 2007. The goals of the strategy are to transform our global data center environment, increasing efficiency and business responsiveness while substantially reducing cost.
Key focus areas of the strategy include:
• Standardizing processes and design specifications by using a strategic, long-range planning approach.
• Accelerating server refresh, while implementing grid computing and server virtualization to increase utilization.
• Consolidating data centers, concentrating computing resources into fewer large, highly efficient hub data centers.
• Using green computing with power-efficient servers and energy-saving data center design, as well as developing our commitment to industry initiatives.