lustre join file design doc

It would be nice to have an executive summary of what a join
file is,
and the reasons behind it...  It sure would make a whole lot
more sense
to readers of the document.  IE you would know what a join
file is for
instead of trying to figure it out as you read the document.

Evan 

-----Original Message-----
From: lustre-devel-bouncesclusterfs.com
[mailto:lustre-devel-bouncesclusterfs.com] On Behalf Of
wangdi
Sent: Thursday, April 27, 2006 7:50 AM
To: lustre-develclusterfs.com
Subject: [Lustre-devel] lustre join file design doc

hi guys

This is the design doc of lustre join file. If you are
interested in it,
you can have a look.

thanks
wangdi

--
Cheers
Wangdi
___________________________________________________
Wangdi       Cluster File Systems, Inc
                http://www.clusterfs.com
          Tel:+86-10-6554-1806

Fu Hu Plaza D-12-A
No. 8, Chao Yang Men Bei Da Street Dongcheng District
Beijing, P.R.C.
100027
                        Tel: +86(10)6554-1806
        e-mail:Wang Di <wangdiclusterfs.com>
___________________________________________________

_______________________________________________
Lustre-devel mailing list
Lustre-develclusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-devel

On Apr 28, 2006  11:01 -0700, Felix, Evan J wrote:
> It would be nice to have an executive summary of what a
join file is,
> and the reasons behind it...  It sure would make a
whole lot more sense
> to readers of the document.  IE you would know what a
join file is for
> instead of trying to figure it out as you read the
document.

A file join is currently implemented to emulate a feature in
a Cray?
filesystem, where a series of existing files are
"joined" together
to produce a single, larger file.  This is essentially
concatenation
done smartly, without moving any data around:

	"join f1 f2 f3 f4 f5" == "cat f1 f2 f3 f4
f5 > tt; mv tt f1"

I personally have little idea of what the demand for this
functionality
is, as it stands by itself.  From a Lustre POV, however, the
same
functionality could be used to implement a variety of
interesting and
arbitrary striping patterns, for complex data formats like
HDF5 where
some parts of the file are accessed by all clients (and want
wide striping)
and others are file-per-process (and want 1 or 2 stripes).

It could also be used as the basis for the case where an
existing file
is out of space on the OST(s) that it is currently using, so
a second
set of objects would be "joined" at the end of
the file so it could
continue to grow.  That wouldn't help the case where a file
is sparse
and being written in the middle, but that is a less common
case.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

_______________________________________________
Lustre-devel mailing list
Lustre-develclusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-devel