Cross Platform Administration and Deployment for Nutch and Hadoop

All,

We are starting to design and develop a framework in Python
that will 
better automate different pieces of Nutch and Hadoop
administration and 
deployment and we wanted to get community feedback.

We first want to replace the DFS and MapReduce startup
scripts with 
python alternatives.  All of the features below are assumed
to be 
executed from a central location (for example the namenode).
 These 
script would allow expanded functionality that would
include:

1) Start, stop, restart of individual DFS and MapReduce
nodes.  (This 
would be able to start and stop the namenode and jobtracker
as well but 
would first check to see if data/task nodes were running and
take 
appropriate action.)
2) Start, stop, restart dfs or map reduce cluster
independently.
3) Start, stop, restart the entire dfs and map reduce
cluster.
4) Allow for individual data/job nodes to have different
deployment 
locations.  This is necessary for a heterogeneous OS
cluster.
5) Allow cross platform and heterogeneous OS clusters.
6) Get detailed status of individual nodes or all nodes. 
This would 
include items such as disk space, cpu usage, etc.
7) Reboot or shutdown machines.  Again this would take into
account 
running services.

Next we would like to split the tools for nutch (such as
crawl, 
invertlinks, etc.) and the tools for hadoop (dfs, job, etc.)
into their 
own individual python scripts that would allow the following
functionality.

1) Dynamic configuration of variables or resetting of config
directory. 
  This might need to be enhanced with changes to the
configuration 
classes in Hadoop (don know yet).
2) Dynamically set other variables such as java heap space
and log file 
directories.

We already have a script that automates a continual fetching
process in 
user defined blocks of number of urls.  This script handles
the entire 
process of injecting, generating, fetching, updating db,
merging 
segments and crawl databases and looping and doing the next
fetch and so 
on until a stop command is given.

Next, we want to create python script that will automate
deployment to 
various nodes and perform maintenance tasks.  Unless
otherwise stated 
the scripts would be able to deploy to different deployment
locations 
configured per machine and allow deployment to an individual
machine, a 
list of machines, or all machines in the cluster.  It would
also allow 
either the backing up or removal of old items.  This would
include:

1) Deploy new release, all code and files.
2) Deploy all lib files.
3) Deploy all conf.
4) Deploy a single file.
5) Deploy all bin files.
6) Deploy a single plugin.
7) Deploy the Nutch job and jar files.
8) Deploy all plugins
9) Remove all log files or archive to a given location.

Finally we would like to automate search index deployment
and 
administration.   Unless otherwise stated the scripts would
be able to 
deploy to target different locations configured per machine
and allow 
targeting to an individual machine, a list of machines, or
all machines 
in the cluster. This functionality would include:

1) Configure a cluster of search servers.
2) Deploy, remove, and redeploy index pieces (parts of an
index) to 
search servers.
3) Start, Stop, and restart search servers.

We would have detailed help screens and and fully documented
scripts 
using a common framework of scripts.  If it was designed
correctly we 
could setup job streams that did automatic crawls,
re-crawls, 
integrations, indexing, and deployments to search servers. 
All of which 
would be needed for the ongoing operation of a web search
engine.

There is a catch and that is that this functionality would
require 
python to be installed on at least the controller node.
This would be a push to the machines and it would be
implemented in 
python using pexpect and probably implementing commands
through ssh, etc.

If you have thoughts on this please let us know and we will
see if we 
can integrate the requests into the development.

Dennis Kubes