rack power question

Hopefully this classifies as on-topic...

I am discussing with some investors the possible setup of
new datacenter 
space.

Obviously we want to be able to fill a rack, and in order to
do so, we 
need to provide enough power to each rack.

Right now we are in spreadsheet mode evaluating different
scenarios.

Are there cases where more than 6000W per rack would be
needed?

(We are not worried about cooling due to the special
circumstances of 
the space.)

Would someone pay extra for > 7KW in a rack?  What would
be the maximum 
you could ever see yourself needing in order to power all
42U ?

Cordially

Patrick Giagnocavo
patrickzill.net

> Obviously we want to be able to fill a rack, and in
order to do so, we
> need to provide enough power to each rack.

Which is the hardest part of designing and running a
datacenter.


> Are there cases where more than 6000W per rack would be
needed?

Yes. We are seeing 10kw rack, and requests for 15 to 20kw
are starting
to come in. Think blade.


> (We are not worried about cooling due to the special
circumstances of
> the space.)

You've already lost.



> Would someone pay extra for > 7KW in a rack?  What
would be the
maximum
> you could ever see yourself needing in order to power
all 42U ?

These days, there is not (or should not) be a connection
between rack
pricing and what you charge for power. 

As for how much, ask HP or IBM or whoever how many blades
they can shove
in 42U.

On Sat, 22 Mar 2008 22:02:49 EDT, Patrick Giagnocavo said:

> Obviously we want to be able to fill a rack, and in
order to do so, we 
> need to provide enough power to each rack.
> 
> Right now we are in spreadsheet mode evaluating
different scenarios.
> 
> Are there cases where more than 6000W per rack would be
needed?
> 
> (We are not worried about cooling due to the special
circumstances of 
> the space.)

Ooh. Please share.  You're the first case I've seen in a
*long* time where
getting the BTU's *out* of the rack wasn't more of a
challenge than getting
the watts *into* the rack.  Once you're talking about
"datacenter" sized
spaces, those BTU's add up, and there's plenty of current
spaces where the
limit isn't the power feed into the building, it's the room
on the roof for
the chillers....

On Sat, 22 Mar 2008, Patrick Giagnocavo wrote:
> Would someone pay extra for > 7KW in a rack?  What
would be the maximum you 
> could ever see yourself needing in order to power all
42U ?

As you recognize, its not an engineering question; its an
economic 
question.  Notice how Google's space/power philosphy changed
between
leveraging other people's space/power, and now that they own
their own 
space/power.

Existing equipment could exceed 20kW in a rack, and some
folks are
planning for equipment exceeding 30kW in a rack.

But things get more interesting when you look at the total
economics
of a data center.  8kW/rack is the new "average,"
but that includes
a lot of assumptions.  If someone else is paying, I want it
and more.
If I'm paying for it, I discover I can get by with less.

PG> Date: Sat, 22 Mar 2008 22:02:49 -0400
PG> From: Patrick Giagnocavo

PG> Hopefully this classifies as on-topic...
PG>
PG> I am discussing with some investors the possible
setup of new
PG> datacenter space.

You might also try the isp-colo.com list.


PG> Are there cases where more than 6000W per rack would
be needed?

It depends how one differentiates between "want"
and "need".


PG> (We are not worried about cooling due to the special
circumstances
PG> of the space.)

ixp.aq? 


PG> Would someone pay extra for > 7KW in a rack?

They should.  If they need more than 6kW, their alternative
is to pay
for a second rack, which hardly would be free.


PG> What would be the maximum you could ever see yourself
needing in
PG> order to power all 42U ?

1. For colo, think 1U dual-core servers with 3-4 HDD;
2. For routers, Google: juniper t640 kw.


HTH,
Eddy
--
Everquick Internet - http://www.everquick.net/
A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/

Bandwidth, consulting, e-commerce, hosting, and network
building
Phone: +1 785 865 5885 Lawrence and [inter]national
Phone: +1 316 794 8922 Wichita
____________________________________________________________
____________
DO NOT send mail to the following addresses:
davidcbrics.com -*- jfconmaapaqintc.net -*- sameverquick.net
Sending mail to spambait addresses is a great way to get
blocked.
Ditto for broken OOO autoresponders and foolish AV software
backscatter.

At 10:02 PM -0400 3/22/08, Patrick Giagnocavo wrote:
>Hopefully this classifies as on-topic...
>
>I am discussing with some investors the possible setup
of new datacenter space.
>
>Obviously we want to be able to fill a rack, and in
order to do so, we need to provide enough power to each
rack.
>
>Right now we are in spreadsheet mode evaluating
different scenarios.
>
>Are there cases where more than 6000W per rack would be
needed?

10K per rack | ~ 400 watts/sqft is a common design point
being used
by the large scale colocation/reit players.  It's quite
possible to exceed
with blade servers or high-density storage (Hitachi, EMC,
etc) but it'd
take unusual business models today to exceed that on every
rack.

>(We are not worried about cooling due to the special
circumstances of the space.)

So, even presuming an abundance of cold air right outside
the facility,
you are still going to move the equipment generated heat to
chillers
or cooling towers.  It is quite likely that your HVAC plant
will could be
your effective limit in ability to add power drops.

>Would someone pay extra for > 7KW in a rack?  What
would be the maximum you could ever see yourself needing in
order to power all 42U ?

Again, you can find single rack, 30" deep storage
arrays/controllers
that will exceed 20KW, but the hope is that you've got a
cabinet or
two of less dense equipment surrounding them.  Best thing to
do is
fine someone in the particular market segment you're aiming
for and
ask them for some averages and trends, since it's going to
vary widely
depending on webhosting/enterprise data center/content
behemoth.

/John

This greatly depends on what you want to do with the space. 
If you're 
putting in co-lo space by the square footage footprint then
your 
requirements will be much less.  If you expect a large
percentage of it 
to be leased out to an enterprise then you should expect the
customers 
to use every last U in a cabinet before leasing the next
cabinet in the 
row.  Ie your power usage will be immense.

I did something similar about 2 years ago.  We were moving a
customer 
from one facility to another.  We mapped out each cabinet
including 
server models.  I looked up maximum power consumption for
each model 
including startup consumption.  The heaviest loaded cabinet
specced out 
at 12,000w.  The cabinet was full of old 1U servers.  New 1U
servers are 
the worst-case scenario by far.  12k is rather low IMHO. 
Some industry 
analysts estimate that the power requirements for
high-density 
applications scale as high as 40kw.

http://www.servertechblog.com/pages/2007/01/cabi
net_level_p.html

There are a few things to remember.  Code only permits you
to load a 
circuit to 80% of its maximum-rated capacity.  The remaining
20% is the 
safety margin required by the NEC.  Knowing this that means
that the 
12Kw specified above require 7x 20a 120v circuits or 5x 30a
120v circuits.

You can get 20a and 30a horizontal PDUs for both 120v and
240v.  There 
are also 208v options.  You can also get up to 40a vertical
PDUs.  One 
word of caution about the vertical PDUs.  If your cabinets
aren't deep 
enough in the rear (think J Lo) the power cabling will get
in the way of 
the rails and other server cabling.  There are others but
they are less 
common.

Also remember that many of the larger servers (such as the
Dell 6850s or 
6950s) are 240v and will require a pair of dedicated
circuits (20a or 30a).

I would also recommend that you look into in-row power
distribution 
cabinets like the Liebert FDC.  This means shorter home-runs
for the 
large number of circuits you'll be putting in (saving your a
bundle in 
copper too).  It also means less under-floor wiring to work
around, 
making future changes much easier.  Changes in distribution
cabinets are 
also much easier, safer and less prone to accidents/mistakes
than they 
are in distribution panels.

Grounding is a topic that is worthy of its own book. 
Consult an 
electrician used to working with data centers.  Don't
overlook this 
critical thing.  Standby power sources fall into this topic
as well. 
How many 3-phase generators are you going to need to keep
your UPSs hot?

I'm curious what your cooling plans are.  I would encourage
you to 
consider geothermal cooling though.  The efficiencies that
geothermal 
brings to the table are worth you time to investigate.

Best of luck,
  Justin


Patrick Giagnocavo wrote:
> 
> Hopefully this classifies as on-topic...
> 
> I am discussing with some investors the possible setup
of new datacenter 
> space.
> 
> Obviously we want to be able to fill a rack, and in
order to do so, we 
> need to provide enough power to each rack.
> 
> Right now we are in spreadsheet mode evaluating
different scenarios.
> 
> Are there cases where more than 6000W per rack would be
needed?
> 
> (We are not worried about cooling due to the special
circumstances of 
> the space.)
> 
> Would someone pay extra for > 7KW in a rack?  What
would be the maximum 
> you could ever see yourself needing in order to power
all 42U ?
> 
> Cordially
> 
> Patrick Giagnocavo
> patrickzill.net
> 
> 

On Sat, 22 Mar 2008, Patrick Giagnocavo wrote:

> I am discussing with some investors the possible setup
of new datacenter 
> space.
>
> Obviously we want to be able to fill a rack, and in
order to do so, we need 
> to provide enough power to each rack.
>
> Right now we are in spreadsheet mode evaluating
different scenarios.
>
> Are there cases where more than 6000W per rack would be
needed?

Is this just for servers, or could there be network gear in
the racks as 
well?  We normally deploy our 6509s with 6000W AC power
supplies these 
days and and I do have some that can draw close to or over
3000W on a 
continuous basis.  A fully populated 6513 with power hungry
blades 
could eat 6000W.

It's been awhile since I've tumbled the numbers, but I think
a 42U rack 
full of 1U servers or blade servers could chew through 6000W
and still be 
hungry.  Are you also taking into account a worst-case
situation, i.e. 
everything in the rack powering on at the same time, such as
after a 
power outage?

> (We are not worried about cooling due to the special
circumstances of the 
> space.)
>
> Would someone pay extra for > 7KW in a rack?  What
would be the maximum you 
> could ever see yourself needing in order to power all
42U ?

I don't know what you mean by 'extra', but I'd imagine that
if someone 
needs 7KW or more in a rack, then they'd be prepared to pay
for the amount 
of juice they use.  This also means deploying a
metering/monitoring 
solution so you can track how much juice your colo customers
use and bill 
them accordingly.

Power consumption, both direct (by the equipment itself) and
indirect 
(cooling required to dissipate the heat generated by said
equipment) is a 
big issue in data center environments these days.  Cooling
might not be an 
issue in your setup, but it is a big headache for most large

enterprise/data center operators.

jms

On Sun, 23 Mar 2008, Justin M. Streiner wrote:

> and and I do have some that can draw close to or over
3000W on a continuous 
> basis.  A fully populated 6513 with power hungry blades
could eat 6000W.

Easily.   The HP blades I have right now are 14 servers in
10u, 6-7,000W. 
Breaker on it needs to be for over 10,000W. (30x208x1.73 for
30A 3 phase)

With our 1u servers, we're able to get about 12 or so in a
rack with a 20A 
208V single phase (my exact budget numbers are behind a vpn
I don't feel 
like firing up...;) plus a pair of switches.

At 370W (peak), i'd need 15540W to power 42 of them, 27,972W
at the 
breaker (I prefer 75 to 80% of breakered capacity vs the
NEC's 85%). 
Works out to something like 4 30A 208v single phase circuits
and 1 20A. 
So 29120W at the breaker.

That's a lot of hot air. ;)


...david


---
david raistrick        http:
//www.netmeister.org/news/learn2quote.html
draisicantclick.org             http://www.expita.c
om/nomime.html