examples of (large) Gentoo clusters

    From: "Daniel van Ham Colchete"
<daniel.colchetegmail.com>
    Date: Mon, 4 Dec 2006 21:55:12 -0200
[...]    
	    Lustre uses a lot of kernel features that if not
enabled will
    cause the kernel to crash.
[...]    
I don't think that's true.

I've been running lustre on assorted kernels, mostly under
gentoo dists, for
some months, and found that once you get past the
installation issues, it's
pretty trouble free.

Now, note the caveats there: The installation issues are
non-trivial, mostly
because lustre is very intrusive into the vfs layer.  This
causes no end of
headaches integrating with various other peoples' kernel
patches, due to
collisions with the other peoples' patches to vfs.  That
statement is as true
of gentoo as it is of any kernel other than the few for
which they supply
canned patchsets.  But I've never seen anything in there
that contitutes using
a kernel feature which causes the kernel to crash if not
enabled.  The closest
thing I've seen to that is if you muff the patch merging and
end up with an
inconsistent patchset, that generally leads to a crash :-}

Lustre 1.6 (at least the client end) doesn't even really
*require* all those
kernel patches, ie they do support the idea of a patchless
client.  The issue
is that lustre changes the logic involved in various kinds
of fs operations,
including anything related to lookups, so as to
short-circuit much of the work
involved when it figures out that it can do so.  Running the
client without
the patches will work, but it won't give you the performance
that you'd get
with the patches.  So odds are anybody who's interested in
running lustre in
the first place probably wants the patches too.

Lustre also is not restricted to precompiled kernels, their
build script
contains patchsets for things other than their recommended
redhat and suse
ones.  We routinely compile it up for all kinds of
experimental kernels with
no trouble.  Again, once you've gotten over the hurdle of
getting the kernel
patches integrated, the rest of it behaves reasonably well. 
The reason cfs
advocates the small number of kernels they do is because
they know what a pain
it is to draw outside the lines, and they try to steer
people away from that.

We at sicortex are planning on rolling out a gentoo-based
cluster that depends
heavily on lustre, so we've spent a fair bit of time banging
on it.  I'm
pretty sure we understand it at this point.  We'll know for
sure soon :-}
-- 
gentoo-clustergentoo.org mailing list

"John R. Dunning" writes:
> 
> Lustre 1.6 (at least the client end) doesn't even
really *require* all those
> kernel patches, ie they do support the idea of a
patchless client.  The issue
> is that lustre changes the logic involved in various
kinds of fs operations,
> including anything related to lookups, so as to
short-circuit much of the wor
> k
> involved when it figures out that it can do so. 
Running the client without
> the patches will work, but it won't give you the
performance that you'd get
> with the patches.  So odds are anybody who's interested
in running lustre in
> the first place probably wants the patches too.

I hadn't realized that the patchless client was potentially
lower-performance than a patched client.  Are you sure about
that?
How much of a difference do you think it is?
Are you using version 1.6 or 1.4?

> 
> We at sicortex are planning on rolling out a
gentoo-based cluster that depend
> s
> heavily on lustre, so we've spent a fair bit of time
banging on it.  I'm
> pretty sure we understand it at this point.  We'll know
for sure soon :-}

Do you get support from CFS?  It seems pretty clear that you
do not.
What kernel versions do you use?

-bryan

-- 
gentoo-clustergentoo.org mailing list

On 12/5/06, John R. Dunning <jrdsicortex.com> wrote:
>     From: "Daniel van Ham Colchete"
<daniel.colchetegmail.com>
>     Date: Mon, 4 Dec 2006 21:55:12 -0200
> [...]
>             Lustre uses a lot of kernel features that
if not enabled will
>     cause the kernel to crash.
> [...]
> I don't think that's true.
>
> I've been running lustre on assorted kernels, mostly
under gentoo dists, for
> some months, and found that once you get past the
installation issues, it's
> pretty trouble free.
>
> Now, note the caveats there: The installation issues
are non-trivial, mostly
> because lustre is very intrusive into the vfs layer. 
This causes no end of
> headaches integrating with various other peoples'
kernel patches, due to
> collisions with the other peoples' patches to vfs. 
That statement is as true
> of gentoo as it is of any kernel other than the few for
which they supply
> canned patchsets.  But I've never seen anything in
there that contitutes using
> a kernel feature which causes the kernel to crash if
not enabled.  The closest
> thing I've seen to that is if you muff the patch
merging and end up with an
> inconsistent patchset, that generally leads to a crash
:-}

Well, my first Lustre test was crashing on every 'write'
operation.
Them I enabled LVM and it worked. I'm using only the vanilla
2.6.12.6
kernel with the lastest 1.4 release.

I have another machine with the same kernel that crashes
everytime I
try to use Lustre over the network, either as a client or as
a server.
Locally it works perfectly. But I'm still trying to learn it
and I
think I still have to spend plenty of time studying it .

> Lustre 1.6 (at least the client end) doesn't even
really *require* all those
> kernel patches, ie they do support the idea of a
patchless client.
That's a very good point.

> We at sicortex are planning on rolling out a
gentoo-based cluster that depends
> heavily on lustre, so we've spent a fair bit of time
banging on it.  I'm
> pretty sure we understand it at this point.  We'll know
for sure soon :-}

Question: would you use Lustre 1.6 now or you would wait
until the
official version is out?

Question: do you expect in upgrade incopability between the
current
1.6 beta and next betas or the official version?

Best regards,
Daniel Colchete
-- 
gentoo-clustergentoo.org mailing list

"Daniel van Ham Colchete" writes:
> 
> Question: do you expect in upgrade incopability between
the current
> 1.6 beta and next betas or the official version?
 
According to the person that I talked to at CFS, the beta is
pretty much
finished, and they are waiting for tester feedback before
releasing it as
1.6.  However, I just learned that they pushed the release
back from January
to March, so perhaps that means another beta release
inbetween.  I dont
know.

-bryan

-- 
gentoo-clustergentoo.org mailing list