The Future of the ALFS project

George Makrydakis wrote:

> Provided there is time on my side, I am available to
start coding for
> your project in C/C++/C#/Java (preference for C++ if in
binary), I will
> in any case complete the bash - based
"parsing" approach, for it may
> prove handy as a tool elsewhere in the project (support
scripts). Also
> with it goes a work for table structures in bash.

Selecting the proper language for a project is an art.  What
is desired
is to make things easy to code and maintain as well as
provide for
efficient execution.

I have nothing against what George has done, but quite
frankly, what he
he has done is proven that bash is a Touring Complete (TC)
language.
What that means is that anything you can do in one TC
language, you can
do in another.  I don't see any advantages over the jhalfs
methodology.
 In fact, the books are written in xml using docbook.  The
purpose of
xslt is to transform xml into another form.  That's exactly
what jhalfs
does.

Lets look at the issues.

1.  Is it easy to code and maintain?  Any interpreted
language that is
understood by the developers is reasonably easy to code and
maintain.
This would include bash, perl, php, python, and several
others
languages.  C, C++, Ada, FORTRAN, Objective-C, and others
are compiled
languages that can be difficult for less expert users to
understand and
modify. Java is an intermediate language, somewhat like the
original
Pascal, in that is creates an intermediate file that is then
used by a
specialized interpreter.   All are are equivalent in
functionality.

The only issue as I see it is whether it is desired to have
transparency
in the process actually extracting the code from the xml. 
xsltproc does
not offer much transparency here.  The only way to use it is
to compare
the input and the output.  The transformation of the .xsl
files with
xsltproc uses a significantly different programming paradigm
and is thus
not transparent to most people.

2.  Does the process use excessive resources such as time,
memory, disk
space, etc?  nALFS requires a "profile" that is
not automated.  This is
not desirable.  jhalfs takes a couple of minutes.  Compared
to the build
time of an LFS system, or even a typical BLFS application,
this is
negligible.  The other resources such as memory and disk
space are also
negligible compared the the applications we are building and
so are non
issues.

In my mind, I prefer the jhalfs process.  It works now and
is relatively
easy to understand and maintain.  The only difficult part,
the .xsl code
used to do the transformation, does not need frequent
updates.  The bash
scripts are easy to understand and modify and the input and
output are
also easy to understand.

To me the ALFS project is essentially complete for LFS.  It
can be
extended to HLFS, CLFS, and BLFS using the same techniques
if desired,
but it does not need the same level of effort as the other
projects
where there is a constant churn of updated packages.  It
only needs to
be updated if there is a significant change to the LFS book
structure.

  -- Bruce
-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

Bruce Dubbs wrote:
> George Makrydakis wrote:
> 
>> Provided there is time on my side, I am available
to start coding for
>> your project in C/C++/C#/Java (preference for C++
if in binary), I will
>> in any case complete the bash - based
"parsing" approach, for it may
>> prove handy as a tool elsewhere in the project
(support scripts). Also
>> with it goes a work for table structures in bash.
> 
> Selecting the proper language for a project is an art. 
What is desired
> is to make things easy to code and maintain as well as
provide for
> efficient execution.

You realize that George's comments above were about coding
for alfs, 
right? (At least, I think they were...)

> I have nothing against what George has done, but quite
frankly, what he
> he has done is proven that bash is a Touring Complete
(TC) language.
> What that means is that anything you can do in one TC
language, you can
> do in another.  I don't see any advantages over the
jhalfs methodology.
>  In fact, the books are written in xml using docbook. 
The purpose of
> xslt is to transform xml into another form.  That's
exactly what jhalfs
> does.
> 
> Lets look at the issues.

I agree, mostly. Still, if I could get my C parser
finished... I'm so 
close it hurts! I just need to get entities parsed and add
the ability 
to parse the extra xpointer flags - this is a must for CLFS.
See http://wiki.linuxfromscratch.org/alfs/browser
/alfs-POC/src/parser.c

The two main benefits are speed improvements and the ability
to drop a 
dependency.

Speed: Compare 1-2 seconds parsing time to 1-2 minutes. Yes,
as you have 
pointed out Bruce, when doing an entire jhalfs build, 1-2
minutes is 
negligible. However, when you're developing, or testing an
LFS build and 
you use jhalfs several times to parse the book, that 1-2
minute wait 
becomes annoying. At least it does for me.

Dependencies: This is probably the more important issue. How
many people 
like to be *required* to add libxml2 and libxslt just to
parse the book 
so they can automate the build?

In any case, this is all a side issue. The how of the parser
isn't 
currently as important as other jhalfs issues, or getting
alfs off the 
ground.

[snip]

> 
> To me the ALFS project is essentially complete for LFS.
 It can be
> extended to HLFS, CLFS, and BLFS using the same
techniques if desired,
> but it does not need the same level of effort as the
other projects
> where there is a constant churn of updated packages. 
It only needs to
> be updated if there is a significant change to the LFS
book structure.

There are yet some functionality improvements that need to
be made, IMO. 
It should be more modular, and the code needs to be less
dependent on 
the specifics of the LFS book. I'd be very happy if jhalfs
could take 
the place of nALFS as the current ALFS tool.

Did I read you right, though, that you don't think a new
alfs tool is 
necessary?

--
JH
-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

El Martes, 28 de Febrero de 2006 19:20, Jeremy Huntwork
escribió:

>
> Did I read you right, though, that you don't think a
new alfs tool is
> necessary?

I see some different tasks here:

1) *LFS commands generation. nALFS uses hand-edited XML
profiles to do that. 
jhalfs extract automatically the commands using XSL, but
work is in progress 
to use a pure bash parser. Plus, a C parser is also under
development.

2) Automatized *LFS builds. That is what currently do nALFS
when using an *LFS 
profile. Has been proved that jhalfs can do that task also,
at least for 
{C,H}LFS builds. IMHO, having a way to automatize the build
extracting the 
commands directly from the book's sources make useless to
maintain a set of 
*LFS profiles.

3) Administrative tasks. Using local profiles nALFS can be
used to automatize 
several system administration tasks. jhalfs could do that
also creating a 
separate module.

4) Remote builds/administration. That is what is supossed
that the new alfs 
server/client tool should to do, using nALFS, jhalfs or
anything else as a 
backend.

Based on that, I think that jhalfs is a working nALFS
replacement for 1) and 
2), and maybe 3). But 4) will require a new different tool.


-- 
Manuel Canales Esparcia
Usuario de LFS nº2886:       http://www.linuxfroms
cratch.org
LFS en castellano: http://www.escomp
oslinux.org/lfs-es http://www.lfs-es.com
TLDP-ES:                           http://es.tldp.org
-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

Jeremy Huntwork wrote:

> Did I read you right, though, that you don't think a
new alfs tool is
> necessary?

Yes, that is what I was saying.  However, your point about
dependencies
is something I didn't think about.  For developers, it is a
non-issue.
I can't imagine an LFS developer not having libxml2 or
libxslt on their
system.  It may be an issue for non-developer users, but
IMO, not a
major one.  There are no required dependencies of these
packages and the
total build time is less than 2 SBU.

If you want to practice coding in C (or any other language)
and use ALFS
as a vehicle, then that's great.  I just don't think the
project needs
it.  What it needs is polishing and enhancing an already
excellent
tool--jhalfs.

The time for jhalfs is mostly in the xsltproc parsing.  How
often does
the book change that you have to run jhalfs.  Certainly not
several
times a day.  In any case:

[root~]# time jhalfs
mkdir: created directory `/mnt/lfs/jhalfs'
`/usr/share/jhalfs/functions' ->
`/mnt/lfs/jhalfs/functions'
mkdir: created directory `/mnt/lfs/jhalfs/logs'
Downloading the LFS Book, version development... done
Extracting commands... done
Creating Makefile... done

real    0m22.100s
user    0m14.705s
sys     0m1.352s

Is all that effort really needed to improve a 22 second
process that
runs at most a few time a day?  I'll also say that I do
have a 3GHz
system and slower systems will take proportionally longer.

Please also note that I'm *not* saying to abandon the idea
of a compiled
system.  If you have an itch, scratch it.  I *am* saying
that if I was
running a commercial enterprise where I was paying a
programmer, I
wouldn't do it because it is not an effective use of time.

  -- Bruce

-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

M.Canales.es wrote:
> El Martes, 28 de Febrero de 2006 19:20, Jeremy Huntwork
escribió:
> 3) Administrative tasks. Using local profiles nALFS can
be used to automatize 
> several system administration tasks. jhalfs could do
that also creating a 
> separate module.
> 
> 4) Remote builds/administration. That is what is
supossed that the new alfs 
> server/client tool should to do, using nALFS, jhalfs or
anything else as a 
> backend.
> 
> Based on that, I think that jhalfs is a working nALFS
replacement for 1) and 
> 2), and maybe 3). But 4) will require a new different
tool.

Excellent. Thanks Manuel. I was going to reply to Bruce with
something 
similar, but you've hit it first. 

There were many requests for features to be implemented in
alfs. The 
alfs-SRS gives a good background of what was discussed.

http://www.linuxfromscratch.org/alfs/view/alfs-s
rs/alfs-srs.html

jhalfs wasn't meant to even attempt to cover all that was
asked, so if 
we *are* going to include the requested features, something
more needs done.

However, I've been doing some thinking about this,
especially in 
connection with the managing of multiple systems (which
includes remote 
building). The proposed model would have been one 'client'
(client in 
the sense that it initiates remote connections to alfs
daemons) which 
sends out commands for the daemons to process and run - the
daemons 
build LFS on the remote machines. The idea was to be able to
manage from 
one machine the building of LFS onto several machines.

But, when managing several machines this way, is it really
necessary to 
have *each* of them build LFS individually? Let's say that
all your 
client machines are i686 (which is very likely). Would not
building LFS 
in the exact same way, with the exact same commands, on each
of those 
machines produce *exactly* the same code? (kernel
individualization not 
counted)

If so, in a situation like this, wouldn't it be more
efficient to have 
one machine build the master system, so to speak and then
copy it over 
to each of the others? Perhaps even create your own RPM
repository 
(don't cringe!) that you build, according to LFS
instructions and then 
deploy to the remaining clients.

I don't know - but the more I think about it, the more I
feel that such 
a setup would be more useful than what was originally
suggested.

--
JH
-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

Jeremy Huntwork wrote:

> But, when managing several machines this way, is it
really necessary to
> have *each* of them build LFS individually? Let's say
that all your
> client machines are i686 (which is very likely). Would
not building LFS
> in the exact same way, with the exact same commands, on
each of those
> machines produce *exactly* the same code? (kernel
individualization not
> counted)
> 
> If so, in a situation like this, wouldn't it be more
efficient to have
> one machine build the master system, so to speak and
then copy it over
> to each of the others? Perhaps even create your own RPM
repository
> (don't cringe!) that you build, according to LFS
instructions and then
> deploy to the remaining clients.

Yup.  I did that quite successfully on several identical
systems.  Just
tar it up and untar into a fresh partition.  A couple of
config files
need to be updated for IP address, host name, etc, but
overall it is a
very efficient process.

  -- Bruce
-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

Jeremy Huntwork wrote:
> client machines are i686 (which is very likely). Would
not building LFS 
> in the exact same way, with the exact same commands, on
each of those 
> machines produce *exactly* the same code?

s/code/binaries/

but I think you knew what I meant. ;)

--
JH
-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page

I think that you have misunderstood my purposes. I have
never implied or suggested that a pure bash approach should
be or must be or (whatever) be followed 
religiously to a point where an automated method for
building LFS should exclusively rely upon it. I just used it
because your work seemed challenging towards 
that direction. In more than one occasions I have said that
this was simply an experiment, nothing more, nothing less.
Having more books based on LFS will 
eventually lead to such a break - up from the main LFS/BLFS
core that I doubt it should be necessary to have ONE tool
for building everything. In any case, some 
clarifications must be made regarding
intentions/directions/whatsoever:

1. jhalfs suits its purpose. When I originally started
writing down the xml - "parsing" script it was
mainly out of curiosity, and not an actual *need*. The 
reason I do not like / use the nALFS way is the way it
works: you actually use a re-written part of the book,
simply rewiring xml source from the LFS book. Bash 
alone can get everything included within (<element>,
</element>) pair in about 30 - 40 s within the ENTIRE
book (ie also the chapters not necessary for building 
it). Do check previous posts in the mailing list and you
will understand how. Some argued about not being able to
have the <screen ...> ability too so that 
nodump elements are also taken under consideration. That is
complete now, so what? Next you will ask about
xpointer/xpath like features. This can be done too. 
In the previous post I made, there was a semi complete
"parsing" method concept, that actually did one
simple thing: print out in the screen either data 
contained within the (<,>) character pair (grammar
specific for xml) separately from data contained within
(>,<) pair, which actually conceptually solves all 
issues you have with TRAC too (I read the code you posted in
your newest post Jeremy, i do not see any <,> and
>,< pairs divided there, this could help you out, 
I think). "Parsing" is then applied for any
"data" contained withing (<,>) pair, where
everything is easier because of the way the XML reading keys
work. This 
an interesting way to go with any parser to be used (if
anything like this should be coded for the alfs project
instead of reusing code from a third party 
source). If bash does it for the entire book in about 1
(one) min, why shouldn't C do it much faster... So to end
this definitely, please read the latest script 
post, i believe that it has conceptually what it takes. That
said, I am done with this. I intend to use x2sh concepts
within personal bash scripts I use for 
working with xml based logs. Scripting is only a quick and
dirty way to do simple things. It is not meant for running
Deep Space 9 on its own.

2. In a few words: bash may be T-Complete (heh heh, :P) as a
language, but it is _NOT_ suitable for the general purpose
of this project at all. It would make 
the bash script so cryptically coded that newbies would be
scared off anyway, while the more advanced users would not
play with bash because of its _evident_ 
limitations. The jhalfs method is an insanely smart
configure - like script and nothing more. This is why it
serves its purpose so well. Sticks to KISS - like 
principles. No one can beat that. Had it been a script -
based approach only, I would have initially started this
"script" based approach in Ruby, but bash is 
good too. "Porting" code rather than concepts to
another language is like duplicating the work, the bugs, the
debugging process...

3. I am not going to enter the best - language - in - the
world war here. It is counter - productive. If professional
- level project management is applied to 
the ALFS project, you will end up stemming another
distribution, further parting away from the project's
initial scope (ie educational). An object oriented 
approach would be the best to follow, so whatever you will
end up choosing, if it does not have OOP mentality, you will
end up reinventing it within it.

4. Suppose you have what you need right now... LFSxml ->
bash scripts -> installed binaries. It would be very
tempting NOT to make a self - hosting distribution 
out of this, especially since skills and sufficient
motivation are present. Is this what you are actually after?

5. It seems only right to follow the DIY method in here too:
talking much about a project is not the same as moving the
project forward. I will just start to 
code things for myself in C++ from a clean start; anything
else but conceptual work would prove an obstacle more than a
helping hand.

Thank you,

George Makrydakis

(gmak)

-- 
http://linuxfromscratch.org/mailman/listinfo/alfs-discu
ss
FAQ: http://www.linux
fromscratch.org/faq/
Unsubscribe: See the above information page