p. 1

data center projects establishing a floor plan by neil rasmussen wendy torell white paper #144

[close]

p. 2

executive summary a floor plan strongly affects the power density capability and electrical efficiency of a data center despite this critical role in data center design many floor plans are established through incremental deployment without a central plan once a poor floor plan has been deployed it is often difficult or impossible to recover the resulting loss of performance this paper provides structured floor plan guidelines for defining room layouts and for establishing it equipment layouts within existing rooms ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 2

[close]

p. 3

introduction a data center floor plan includes the layout of the boundaries of the room or rooms and the layout of it equipment within the room most users do not understand how critical the floor layout is to the performance of a data center or they only understand its importance after a poor layout has compromised the deployment the floor plan either determines or strongly affects the following characteristics of a data center · · · · · the number of rack locations that are possible in the room the achievable power density the complexity of the power and cooling distribution systems the predictability of temperature distribution in the room the electrical power consumption of the data center many users do not appreciate these effects during data center planning and do not establish the floor layout early enough as a result many data centers unnecessarily provide suboptimal performance the purpose of this paper is to explain how floor plans affect these characteristics and to prescribe an effective method for developing a floor layout specification role of the floor plan in the system planning sequence floor plans must be considered and developed at the appropriate point in the data center design process considering floor plans during the detailed design phase is typical but simply too late in the process floor plans should instead be considered to be part of the preliminary specification and determined before detailed design begins figure 1 illustrates where floor planning fits into the system planning sequence apc white paper #142 data center projects system planning explains this planning sequence in greater detail it is not necessary for a floor layout to comprehend the exact location of specific it devices this paper will show that effective floor plans only need to consider the location of equipment racks or other cabinets and to target power densities these preliminary floor layouts do not require knowledge of specific it equipment for most users it is futile to attempt to specify particular it equipment locations in advance ­ in fact racks may ultimately house equipment that is not even available on the market at the time the data center is designed ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 3

[close]

p. 4

figure 1 ­ the floor plan is a key input in the system planning sequence lit y itica · cr city pa an · ca h pl ow t · gr the floor plan is created after the system concept is developed and becomes an input to user requirements nce fere · reesign d om · ro 3 identify needs s nce fere pre ints stra con for more about this planning sequence see apc white paper #142 data center projects system planning complete system specification detailed design ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 4

[close]

p. 5

the reasons that floor plans must be considered early as part of the preliminary specification and not left until the later detailed design include · density is best specified at the row level so rows must be identified before a density specification can be created for more information on specifying densities see apc white paper #120 guidelines for specification of data center power density · · · phasing plans are best specified using rows or groups of rows so rows must be identified before an effective phasing plan can be created the floor grid for a raised floor and the ceiling grid for a suspended ceiling should be aligned to the rack enclosures so rows must be identified before those grids can be located criticality or availability can optionally be specified differently for different zones of the data center ­ rows must be identified before a multi-tier criticality plan can be created density and phasing plans are a key part of any data center project specification and both require a row layout detailed design can only commence after density phasing and criticality have been specified therefore a floor plan must be established early in the specification phase of a project after system concept but well before detailed design see figure 1 floor planning concepts a data center floor plan has two components the structural layout of the empty room and the equipment layout of what will go in the room note that for many projects the room is pre-existing and the only option is to lay out the equipment within the room a key rule of data center design is that there is a potentially huge advantage in efficiency and density capacity if planners can lay out the room boundaries at the outset wherever possible an attempt should be made to influence the structural room layout using the principles established in this paper structural layout the room layout includes the location of walls doors support columns windows viewing windows and key utility connections if the room has a raised floor the height of the raised floor and the location of access ramps or lifts are also part of the structural layout if the room has a raised floor or a suspended ceiling the index points for the floor or ceiling grid are critical design variables and must also be included in the structural layout for purposes of this paper room measurements will be described in units of tiles where a tile width is equal to 2 feet 600 mm or one standard rack enclosure width equipment layout the equipment layout shows the footprint of it equipment and the footprint of power and cooling equipment it equipment can usually be defined as rack locations without regard for the specific devices in the cabinets ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 5

[close]

p. 6

but other equipment such as tape libraries or large enterprise servers may have form factors that are different from typical racks and must be called out explicitly in addition it equipment in a layout must be characterized by its airflow path in the case of typical it racks the airflow is front-to-back but some devices have other airflow patterns such as front-to-top power and cooling equipment must also be accounted for in equipment layouts but many new power and cooling devices are either rack mountable or designed to integrate into rows of racks which simplifies the layout the effects of floor plans on data center performance several important data center characteristics are affected by floor plans to understand effective floor layout methods it is important to understand the consequences number of rack locations the floor layout can have a dramatic affect on the number of rack locations that are possible in the room although on average the number of it rack locations possible can be estimated by dividing the room area by 28 sq ft rack 2.6 sq meters rack the actual number of racks for a particular data center can vary greatly from this typical value 1 the basic principle of floor planning is to maximize the number of rack locations possible small variations in the location of walls existing it devices air conditioners and power distribution units can have a surprisingly large impact on the number of possible rack locations this effect is magnified when high power densities are required for this reason a careful and systematic approach to floor planning is essential achievable power density the floor plan can have a major impact on the achievable power density with certain cooling architectures a poor layout can decrease the permissible power for a given rack by over 50 this is a huge performance compromise in a modern data center where new technologies have power densities that are already stressing the capabilities of data center design in many data centers users may want to establish zones of different power density these density zones will be defined by the equipment layout the floor plan is therefore a critical tool to describe and specify density for data centers complexity of distribution systems the floor plan can have a dramatic affect on the complexity of the power and cooling distribution systems in general longer rows and rows arranged in the patterns described in this paper simplify power and cooling distribution problems reduce their costs and increase their reliability 1 apc white paper #120 guidelines for specification of data center power density ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 6

[close]

p. 7

predictability of cooling performance in addition to impacting the density capability of a data center the floor plan can also significantly affect the ability to predict density capability it is a best practice to know in advance what density capability is available at a given rack location and not to simply deploy equipment and hope for the best as is a common current practice an effective floor plan in combination with row-oriented cooling technologies allows simple and reliable prediction of cooling capacity design tools such as apc infrastruxure designer can automate the process during the design cycle and when layouts follow standard methods off-the-shelf operating software such as apc infrastruxure manager can allow users to monitor power and cooling capacities in real time electrical efficiency most users are surprised to learn that the electrical power consumption of a data center is greatly affected by the equipment layout this is because the layout has a large impact on the effectiveness of the cooling distribution system this is especially true for traditional perimeter cooling techniques for a given it load the equipment layout can reduce the electrical power consumption of the data center significantly by affecting the efficiency of the air conditioning system · the layout affects the return temperature to the crac units with a poor layout yielding a lower return air temperature a lower return temperature reduces the efficiency of the crac units · the layout affects the required air delivery temperature of the crac units with a poor layout requiring a colder supply for the same it load a lower crac supply temperature reduces the efficiency of the crac units and causes them to dehumidify the air which in turn increases the need for energy-consuming humidification · the layout affects the amount of crac airflow that must be used in mixing the data center air to equalize the temperature throughout the room a poor layout requires additional mixing fan power which decreases efficiency and may require additional crac units which draw even more electrical power a conservative estimate is that billions of kilowatt hours of electricity have been wasted due to poor floor plans in data centers this loss is almost completely avoidable basic principles of equipment layout the existence of the rack as the primary building block for equipment layouts permits a standardized floor planning approach the basic principles are summarized as follows · · control the airflow using a hot-aisle/cold-aisle rack layout provide access ways that are safe and convenient ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 7

[close]

p. 8

· · · align the floor or ceiling tile systems with the equipment minimize isolated it devices and maximize row lengths plan the complete equipment layout in advance even if future plans are not defined once these principles are understood an effective floor planning method becomes clear control of airflow using hot-aisle/cold-aisle rack layout the use of the hot-aisle/cold-aisle rack layout method is well known and the principles are described in other documents such as ashrae tc9.9 mission critical facilities thermal guidelines for data processing environments 2004 and a white paper from the uptime institute titled alternating cold and hot aisles provides more reliable cooling for server farms the basic principle is to maximize the separation between it equipment exhaust air and intake air by establishing cold aisles where only equipment intakes are present and establishing hot aisles where only equipment hot exhaust air is present the goal is to reduce the amount of hot exhaust air that is drawn into the equipment air intakes the basic hot-aisle/coldaisle concept is shown in figure 2 figure 2 ­ basic hot-aisle/cold-aisle data center equipment layout plan in the figure the rows represent the it equipment enclosures racks the racks are arranged such that the adjacent rows face back to back forming the hot aisles the benefits of the hot-aisle/cold-aisle arrangement become dramatic as the power density increases when compared to random arrangements or arrangements where racks are all lined up in the same direction the hot-aisle/cold-aisle approach allows for a power density increase up to 100 or more without hot spots if the appropriate arrangement of crac units is used because all cooling architectures except for fully enclosed rack-based cooling benefit dramatically from hot-aisle/cold-aisle layout this method is a principal design strategy for any floor layout ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 8

[close]

p. 9

provide access ways that are safe and convenient it is a legal requirement and common sense to provide appropriate access ways around equipment the hot-aisle/cold-aisle system creates natural hallways or aisles that are well defined since an effective data center is based on row layouts with aisles that serve as access ways it is important to identify and understand the impact of column locations a column could consume up to three rack locations if it falls within a row of racks ­ or even worse could result in the elimination of a complete row of racks if it obstructs an aisle figures 3 and 4 illustrate the possible impact of a column on the number of usable rack locations the room in figure 3a fits 40 rack locations when no columns are present when a column exists but aligns with a row of equipment racks as figure 3b shows only one rack may be impacted 2 when a column aligns with an aisle a more significant impact can occur figure 4a shows the column in the aisle occasionally this may be determined to be an acceptable obstruction if the local authority having jurisdiction ahj believes that shifting the equipment rows and possibly eliminating available rack positions is an unreasonable accommodation according to local disabilities acts figure 4b illustrates an attempt at recreating sufficient aisles but represents a bad alternative since the hot-aisle/cold-aisle layout is severely impacted and cooling becomes less predictable figure 4c shows what happens to the rack count when the rows must be shifted so that the column is no longer in the aisle note that the rack count has drastically dropped from 40 to 29 a decrease of over 25 the last scenario figure 4d shows an alternate configuration of racks aligned north-south rather than east-west this is called the axis of the layout this configuration accommodates 35 rack locations which is more favorable than shifting rows later sections of this paper further explain the considerations in rotating the layout axis by 90º to obtain optimal positioning 3 it is a common practice to adjust the equipment layout to place columns within rows where they consume potential equipment locations keeping support columns out of aisle-ways is a severe constraint on maximizing equipment locations because it dictates areas that cannot be used as aisle-ways as figures 3 and 4 illustrate the shifting of equipment rows to accommodate column locations can cause the loss of an entire row of equipment when that row becomes trapped against a wall or other obstacle therefore the careful location of equipment rows relative to the columns is of primary concern in the floor layout figures 3 and 4 are based on room layouts with 4-foot cold aisles and 3-foot hot aisles and 4-foot perimeter clearance disabilities acts across the globe provide guidelines for acceptable aisle-ways the local authority having jurisdiction ahj determines if reasonable accommodations have been made on a case-by-case basis some examples of these disability acts can be found at the following sites http www.ada.gov/adastd94.pdf http www.dwp.gov.uk/employers/dda and http www.comlaw.gov.au/comlaw/legislation/actcompilation1.nsf/0/896cf5a0e01ca785ca25705700098a96 file/dis abilitydiscrimination1992_wd02.pdf 3 2 ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 9

[close]

p. 10

figure 3 ­ sample impact of columns on number of rack locations when column aligns with row 3a no column this example room with no columns allows 40 rack locations assuming 4-ft cold aisle 3-ft hot aisle and a 4-ft perimeter 40 rack locations cold aisle 3b column aligns with row of racks 39 rack locations cold aisle hot aisle hot aisle cold aisle cold aisle one rack location is eliminated in this example depending on the size of the column and location of the column within the row of racks as many as three rack locations could be eliminated hot aisle hot aisle cold aisle cold aisle figure 4 ­ sample impact of columns on number of rack locations when column aligns with aisle 4a column partially obstructs aisle 38 rack locations cold aisle 4b hot-aisle cold-aisle layout is impacted 34 rack locations cold aisle keeping the column as an aisle-way obstacle has smallest impact of rack locations however this practice is often not accepted by ahjs hot aisle hot aisle cold aisle hot aisle x cold aisle hot aisle cold aisle 35 rack locations eliminating several racks in the middle of rows creates an environment where air mixing occurs cold aisle 4c row of equipment is eliminated 29 rack locations 4d rotation of rows to align with column rotating the rows 90º creates a smaller impact on the number of rack locations shifting the rows in this example means the loss of entire row of racks hot aisle cold aisle hot aisle cold aisle ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 10

[close]

p. 11

align the floor and/or ceiling tiles with the equipment in many data centers the floor and ceiling tile systems are used as part of the air distribution system in a raised floor data center it is essential that the floor grille align with racks if the racks and the floor grid do not align airflow can be significantly compromised it is also beneficial to align any ceiling tile grid with the floor grid this means the floor grid should not be designed or installed until after the equipment layout is established and the grid should be aligned or indexed to the equipment layout according to the row layout options unfortunately specifiers and designers often miss this simple and no-cost optimization opportunity the result is that either 1 the grid is misaligned with the racks with a corresponding reduction in efficiency and density capability or 2 the racks are aligned to the grid but a suboptimal layout results limiting the number of racks that can be accommodated pitch ­ the measurement of row spacing the row length in a hot-aisle/cold aisle layout is adjustable in increments of rack width which provides significant flexibility however the spacing between aisles has much less flexibility and is a controlling constraint in the equipment layout the measurement of row-to-row spacing is called pitch the same term that is used to describe the repeating center-to-center spacing of such things as screw threads sound waves or studs in a wall the pitch of a data center row layout is the distance from one mid-cold-aisle to the next mid-cold-aisle figure 5 figure 5 ­ pitch of a row layout pitch figure 6 shows the four standard pitches used in data center floor layouts each pitch is defined by a number of tiles where a tile is 2 ft 600 mm wide note that in all four pitches the equipment racks are aligned to tiles in the cold aisles this is because in a raised-floor environment with perimeter-based cooling full perforated tiles are needed in the cold-aisles for air delivery ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 11

[close]

p. 12

figure 6 ­ the four standard pitches of row layouts pitch a ­ compact pitch b ­ wide hot-aisle pitch c ­ wide cold-aisle pitch d ­ wide hot-aisle cold-aisle pitch a shows the most compact geometry for a row pair with a 7-tile 14 foot overall width the most commonly used building block for data center row layouts however in some situations wider pitches are necessary once a reference concept design has been selected for a data center figure 7 provides guidelines for when to use each of the four pitches for instance the flowchart illustrates that pitch b provides 50 more raised floor cooling capacity in the cold aisle for higher density applications with raised floor air distribution pitch c or d on the other hand may be needed when using racks with specialized cooling plenums fixed to the back ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 12

[close]

p. 13

figure 7 ­ choosing the appropriate row-to-row pitch start no will the data center be on a raised floor yes no does the design concept require cooling accessories on the back of the racks no does the design concept require cooling accessories on the back of the racks yes yes yes no is the data center supporting high density applications yes does the design concept use rack containment no does the design concept require double perforated tiles for higher density yes does the design concept use in-row cooling with hot aisle containment no no yes pitch a pitch b pitch c pitch d compact 14 ft 4 ft cold aisle 3 ft hot aisle wide cold aisle 16 ft 6 ft cold aisle 3 ft hot aisle wide hot aisle 16 ft 4 ft cold aisle 5 ft hot aisle wide hot cold aisle 18 ft 6 ft cold aisle 5 ft hot aisle ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 13

[close]

p. 14

special spacing for hard floor environments nonstandard row pitches which are incrementally smaller than the four standard ones can be advantageous for installations with a hard non-raised floor the four pitches in figure 6 align cold-aisle racks with floor tiles to facilitate air delivery through the raised floor however when a raised floor is not used that cooling constraint goes away for example consider a high-density scenario where back-of-therack cooling accessories increase rack depth from 42 inches to 50 inches using the minimum row spacing of a 3-foot hot aisle and 4-foot cold aisle the smallest allowable pitch would be 15 feet 4 inches 8 inch increase x 2 rows of racks when a layout on a hard floor using the four standard pitches is a problem ­ i.e you lose an additional row by a couple of feet or less ­ the pitch can be compressed as long as aisle widths remain the minimum of 3 feet for the hot aisle and 4 feet for the cold aisle an effective data center floor plan should be deployed in row pairs using the basic spacing pitches described above however we will show that various barriers and constraints may interfere with the optimal layout minimize isolated it devices and maximize row lengths the control of airflow by separating hot and cold air as described above is compromised at the end of a row where hot air can go around the side of the end rack and return to it equipment air intakes on the back therefore the theoretical ideal design of a data center is to have no row ends ­ i.e rows of infinite length conversely the worst case situation would be rows of one-rack length ­ i.e isolated single racks in addition the ability to effectively implement redundancy is improved with longer rows the goal of row layout is to maximize row length consistent with the goals of providing safe and convenient access ways in general a layout that provides longer row lengths is preferred and a row layout that generates short rows of 1-3 racks should be avoided special considerations for wide racks standard-width racks 2 ft or 600 mm conveniently align with the width of raised-floor tiles when underfloor cables must be distributed to such a rack a hole is typically created in the tile directly below the rack to run the cables if that particular rack is then re-located or removed the tile is simply replaced with a new one wide racks that do not align with the standard raised floor tile width are creating a new challenge because a rack may occupy two or even three tiles if such a rack is removed no longer can the tile simply be replaced with a new one since the tile is partially underneath the neighboring rack as well these issues can be avoided altogether by overhead power and data cable distribution plan the complete floor layout in advance the first phase of equipment deployment often constrains later deployments for this reason it is essential to plan the complete floor layout in advance ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 14

[close]

p. 15

basic principles of structural room layouts many data centers are fit-outs of existing space in these cases the structural layout of the room is fixed and cannot be specified for some data centers wall relocation or other modifications may be possible for data centers in new construction considerable options are available regarding wall locations it is a basic and generally unappreciated principle of data center design that the ability to locate walls can greatly improve the performance of the data center therefore when it is possible the layout of the room boundaries should be chosen based on the standard data center design principles described in this section readers who have no room layout flexibility may choose to skip this section standardized room dimensions there are preferred room dimensions for data centers based on the pitch chosen given an area or room that is rectangular in shape free of the constraints imposed by support columns described earlier the preferred length and width are established as follows · · one dimension of the room should be a multiple of the hot-aisle/cold-aisle pitch plus a peripheral access-way spacing of approximately 2-4 tiles the other dimension of the room is flexible and will impact the length of the rows of racks when one of the dimensions of the room is not optimal the performance of the room can be dramatically reduced particularly if the room is smaller the most obvious problem is that the number of equipment racks may be lower than expected because some space cannot be used the second and less obvious problem is that when the ideal layout cannot be achieved the power density and electrical efficiency of the system is reduced to understand the effect of room dimension on the number of racks consider a room with a fixed length of 28 feet and a variable width in such a room the length of a row would be 10 racks allowing for 2 tiles 4 feet at each row-end for access clearance the number of racks that could fit in this room will vary as a function of the width of the room as shown in figure 8 figure 8 shows that the number of installable racks jumps at certain dimensions as new rows fit into the room furthermore the chart shows that certain numbers of racks are preferred because the even row number permits a complete additional hot-aisle/cold-aisle pair to be installed the preferred width dimensions are indicated by the arrows for the pitch the most compact pitch a in this case and perimeter clearances 2 tiles defined ©2007 american power conversion all rights reserved no part of this publication may be used reproduced photocopied transmitted or stored in any retrieval system of any nature without the written permission of the copyright owner www.apc.com wp144 rev 2007-0 15

[close]