Out of Memory during Sorts

Can someone lead me in the right direction as to configuring
settings 
for large sorting operations > 1M rows.  I keep getting
out of memory 
exceptions during the sort phase.  Here are my current
settings.  I have 
2G heap space on each box.

Dennis

<property>
  <name>io.sort.factor</name>
  <value>20</value>
  <description>
  The number of streams to merge at once while sorting
  files.  This determines the number of open file handles.
  </description>
</property>

<property>
  <name>io.sort.mb</name>
  <value>200</value>
  <description>
  The total amount of buffer memory to use while sorting
  files, in megabytes.  By default, gives each merge stream
1MB, which
  should minimize seeks.
  </description>
</property>

<property>
  <name>io.file.buffer.size</name>
  <value>8192</value>
  <description>
  The size of buffer for use in sequence files.
  The size of this buffer should probably be a multiple of
hardware
  page size (4096 on Intel x86), and it determines how much
data is
  buffered during read and write operations.
  </description>
</property>

<property>
  <name>io.bytes.per.checksum</name>
  <value>4096</value>
  <description>
  The number of bytes per checksum.  Must not be larger than
  io.file.buffer.size.
  </description>
</property>

Okay, I changed io.sort.factor to 100 and now it worked. 
Anybody have 
any idea why?  I also restarted via stop-all and start-all. 
Maybe 
memory was released?

Dennis

Dennis Kubes wrote:
> Can someone lead me in the right direction as to
configuring settings 
> for large sorting operations > 1M rows.  I keep
getting out of memory 
> exceptions during the sort phase.  Here are my current
settings.  I 
> have 2G heap space on each box.
>
> Dennis
>
> <property>
>  <name>io.sort.factor</name>
>  <value>20</value>
>  <description>
>  The number of streams to merge at once while sorting
>  files.  This determines the number of open file
handles.
>  </description>
> </property>
>
> <property>
>  <name>io.sort.mb</name>
>  <value>200</value>
>  <description>
>  The total amount of buffer memory to use while sorting
>  files, in megabytes.  By default, gives each merge
stream 1MB, which
>  should minimize seeks.
>  </description>
> </property>
>
> <property>
>  <name>io.file.buffer.size</name>
>  <value>8192</value>
>  <description>
>  The size of buffer for use in sequence files.
>  The size of this buffer should probably be a multiple
of hardware
>  page size (4096 on Intel x86), and it determines how
much data is
>  buffered during read and write operations.
>  </description>
> </property>
>
> <property>
>  <name>io.bytes.per.checksum</name>
>  <value>4096</value>
>  <description>
>  The number of bytes per checksum.  Must not be larger
than
>  io.file.buffer.size.
>  </description>
> </property>
>

On Jun 11, 2006, at 8:07 AM, Dennis Kubes wrote:

> Can someone lead me in the right direction as to
configuring settings 
> for large sorting operations > 1M rows.  I keep
getting out of memory 
> exceptions during the sort phase.  Here are my current
settings.  I 
> have 2G heap space on each box.

In general, it would help to have stack traces so that we
could tell 
what part of the sort was running out of memory. What was
your jvm heap 
size set to? It is interesting that increasing the sort
factor got you 
around the problem.

-- Owen

Unfortunately I don't have the stack traces anymore.  The
java heap size 
was set to 2G.  When I upped the sort factor I get alot more
IO, but it 
finishes and doesn't run out of memory.

Dennis

Owen O'Malley wrote:
>
> On Jun 11, 2006, at 8:07 AM, Dennis Kubes wrote:
>
>> Can someone lead me in the right direction as to
configuring settings 
>> for large sorting operations > 1M rows.  I keep
getting out of memory 
>> exceptions during the sort phase.  Here are my
current settings.  I 
>> have 2G heap space on each box.
>
> In general, it would help to have stack traces so that
we could tell 
> what part of the sort was running out of memory. What
was your jvm 
> heap size set to? It is interesting that increasing the
sort factor 
> got you around the problem.
>
> -- Owen
>

On Jun 11, 2006, at 8:07 AM, Dennis Kubes wrote:

> Can someone lead me in the right direction as to
configuring settings 
> for large sorting operations > 1M rows.  I keep
getting out of memory 
> exceptions during the sort phase.  Here are my current
settings.  I 
> have 2G heap space on each box.

Please check the value of
"mapred.child.java.opts". It controls the 
options for heap sized allocated to the Task.  It defaults
to 200m, 
which seems really low given that the default io.sort.mb is
100m (and 
yours is set to 200m). Try increasing the max heap size to
1024m or so.

-- Owen