> At Sat, 6 Oct 2007 15:55:14 -0600, John E Hein wrote:
> Subject: Re: Scripting language choice (Was: Scripting the Amanda core)
>>
>> Yes. But please pick the scripting language on their own merits, not
>> on backward-looking considerations
>
> Indeed!
>
>
> In fact that probably lobs both Python and Perl so far out the window so
> quickly that they'll go into orbit or further.
>[...fine thoughts snipped...]

It seems to me that the question of merits obscures the real issue, which
is whether to redesign AMANDA to be rewritten in a script language or
merely to expose its capabilities to a scripting language, while keeping
the main functionality in C. The latter approach doesn't preclude a
gradual shift toward the former.

As for the choice of PERL, python, I think a more obvious choice is Bourne
shell. That is, as AMANDA is a collection of executables already,
splitting those up even further might allow some disentanglement.

Another way to look at it is in terms of an API. Why force users to choose
a scripting language, when the work of maintenance is on the backend?
Define a scripting interface, and let the scripting languages and C use
it. As more and more of the core of AMANDA is rewritten to use the API,
it can be moved to script, if that seems desirable.

Finally, every problem defines a language that solves it. Why not roll
our own? I posit, however, that for AMANDA that language would closely
resemble a subset of sh.

On 10/7/07, Greg A. Woods < woods%40weird.com">woodsweird.com> wrote:
&gt; If "we're" talking about something for scripting _within_ (i.e. in and
> "embedded" sense) an application then you want a truly easily embeddable
> interpreter, too, regardless of the language.

The proposal is actually to do the opposite -- embed Amanda's
low-level and efficiency-critical stuff (written in C) *in* a
higher-level language. So "main()" is written in ${LANGUAGE}, and
takes care of the argument parsing, operation planning, error
recovery, etc., while calling C functions to take care of the details.

So far in this discussion, I've seen many reasons *not* to use
${LANGUAGE}, but few arguments *for* ${LANGUAGE}. There have been
several affirmations of Perl as preferred language (Paul's, for
example).

Assuming the extra burden that Python et al. would bring (as a new
dependency), I think we need a positive benefit for these other
languages that justifies the additional burden.

Dustin

--
Storage Software Engineer
http://www.zmanda.com

On 10/8/07, lheal%40uiuc.edu">lhealuiuc.edu < lheal%40uiuc.edu">lhealuiuc.edu&gt; wrote:
&gt; As for the choice of PERL, python, I think a more obvious choice is Bourne
&gt; shell. That is, as AMANDA is a collection of executables already,
> splitting those up even further might allow some disentanglement.

The problem with shells (beyond the obvious portability headaches) is
that they don't allow linked-in extensions. There are several
operations that Amanda must perform that require it to e.g., hold open
an OS file descriptor for the duration. To script that in shell would
require starting a long-running process (to hold open the descriptor)
and communicating with that process using some kind of line-oriented
protocol.

Also, shells will not bring the various other benefits of a
higher-level language: sophisticated data structures and types, native
support for Unicode, support for asynchronous operation, etc.

> Another way to look at it is in terms of an API. Why force users to choose
&gt; a scripting language, when the work of maintenance is on the backend?
> Define a scripting interface, and let the scripting languages and C use
> it. As more and more of the core of AMANDA is rewritten to use the API,
>; it can be moved to script, if that seems desirable.

I'm not quite sure what you mean here, but a few things strike me as inaccurate.

First, we have a number of APIs defined already: changer, device,
security, etc.

Second, I fully expect that the bulk of maintenance and development
will be on the higher-level stuff. If you look over the
amanda-hackers discussions of the last few months, or the Amanda
wishlist, you'll see that few of the changes are backend-specific
(e.g., "support control of hardware encryption on DLT8000 drives&quot;),
but rather are about the higher-level operations Amanda performs:
roaming backups, multi-tape backups, device-to-device dump migration,
Unicode support, etc.

This is, IMHO, the best reason to use a scripting language. All of
these tasks will require a significant rewrite of a portion of Amanda,
and all of them are much more easily accomplished in a high-level
language than in C.

Dustin

--
Storage Software Engineer
http://www.zmanda.com

At Mon, 8 Oct 2007 15:00:23 +0200, Ragnar KjŽørstad wrote:
Subject: Re: Scripting language choice (Was: Scripting the Amanda core)
&gt;
> On Sun, Oct 07, 2007 at 08:58:19PM -0400, Greg A. Woods wrote:
&gt; > http://en.wikipedia.org/wiki/Scripting_language#Extension.2Fembeddable_language
> >
> > (and no neither Perl nor Python, despite their proponents claims, and
> > depsite the appearance of Python in the wikipedia entry, can be
> > effectively used as embedded scripting languages)
>
> Out of curiosity; why not?

Actually most of that should be obvious from the various published
descriptions and goals of embeddable scripting languages, some of which
I pointed to in my post.

> I've used python as a plugin-language for C projects before, and found
&gt; it to work out quite well.

Both Python and Perl require a huge infrastructure in order to deliver
what most programmers familiar with those languages will desire, often
even for what might seem to be the most trivial purposes on the surface.

This is a problem more of expectations than anything else of course.

It might be particularly of issue though for installing the client.

If users don't mind dragging in the whole Python universe, for example,
just to install the client in such a way that they might at some future
date be able to script some custom actions for the client, then Python
might be OK. On the other hand I would want to install the client
without any other runtime dependencies outside of itself (and I wouldn't
want the whole Python universe embedded within either -- if it were to
be something like that it needs to be done in such a way that the Python
install is completely stand-alone as well, i.e. so that it can be used
as the Python environment for any other purpose too).

Ideally though you want a very small and light language which has had a
very well designed and well integrated set of built-in APIs that give
high-level interfaces to the internals of the application. This is
because many scripting needs will most likely simply require wrapping
some new algorithms around existing functionality. A good embeddable
language will allow the user to do that by writing the algorithm in
simple control structure statements, etc., and without requiring _any_
other library files, etc. However a good language will allow any level
of complexity to be reached in the add-on code. A very good example of
this is how lisp is embedded in many versions of emacs. In some
examples the basic editor works all by itself, as one stand-alone
executable. However it can load anything from simple setq statements to
full blown new-purpose applications such as mail readers and spreadsheets.

--
Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack < woods%40robohack.ca">woodsrobohack.ca>
Planix, Inc. < woods%40planix.com">woodsplanix.com> Secrets of the Weird < woods%40weird.com">woodsweird.com>

I'd like to refocus this conversation a little bit. In my mind, there
are two reasons to script the Amanda core:
1. It will facilitate implementation of new features that active
developers are working on
2. It will significantly lower the barrier to entry by new developers

The important question, then, is not which language has more merit,
but which language will bring the project more productive development?

So, which language would best facilitate your own contribution to
Amanda development? "Leave it in C" is a valid option here.

Which language do you think would attract new developers to the project?

Dustin

--
Storage Software Engineer
http://www.zmanda.com

At Mon, 8 Oct 2007 09:43:20 -0500 (CDT), lheal%40uiuc.edu">lhealuiuc.edu wrote:
Subject: Re: Scripting language choice (Was: Scripting the Amanda core)
&gt;
> It seems to me that the question of merits obscures the real issue, which
&gt; is whether to redesign AMANDA to be rewritten in a script language or
> merely to expose its capabilities to a scripting language, while keeping
> the main functionality in C. The latter approach doesn't preclude a
> gradual shift toward the former.

I was going to mention that too.

For example there's already rdiff-backup, which is implemented in
Python and which in many ways does far more than Amanda does now. With
the addition of off-line storage device handling rdiff-backup would be a
far superior backup tool to Amanda in many (most?) ways.

> Another way to look at it is in terms of an API. Why force users to choose
&gt; a scripting language, when the work of maintenance is on the backend?
> Define a scripting interface, and let the scripting languages and C use
> it. As more and more of the core of AMANDA is rewritten to use the API,
>; it can be moved to script, if that seems desirable.

That has to be done regardless -- and to do it in such a way that any
scripting language can use the API would only be a small amount of
additional work I'd think.

> Finally, every problem defines a language that solves it. Why not roll
>; our own? I posit, however, that for AMANDA that language would closely
> resemble a subset of sh.

Very true too!

--
Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack < woods%40robohack.ca">woodsrobohack.ca>
Planix, Inc. < woods%40planix.com">woodsplanix.com> Secrets of the Weird < woods%40weird.com">woodsweird.com>

At Mon, 8 Oct 2007 10:05:22 -0500, Dustin J. Mitchell wrote:
Subject: Re: Scripting language choice (Was: Scripting the Amanda core)
&gt;
> On 10/7/07, Greg A. Woods < woods%40weird.com">woodsweird.com> wrote:
&gt; > If "we're" talking about something for scripting _within_ (i.e. in and
> > "embedded" sense) an application then you want a truly easily embeddable
> > interpreter, too, regardless of the language.
>
> The proposal is actually to do the opposite -- embed Amanda's
> low-level and efficiency-critical stuff (written in C) *in* a
> higher-level language. So "main()" is written in ${LANGUAGE}, and
> takes care of the argument parsing, operation planning, error
&gt; recovery, etc., while calling C functions to take care of the details.

You see that's what rdiff-backup is -- it takes a few bits that need
high efficiency and plugs them into the interpreter (e.g. librsync) and
then the rest of the application is just a Python program.

That's where it's good to have the full compliment of a big general
purpose scripting language like Python.

However if the idea is to add script-ability to an existing application
like Amanda then one needs to embed an interpreter into the existing
application (and then decide how and what to expose to the embedded
language).

And this is where a nice to have a tight small little functional
language implementation that's been designed explicitly for embedding
into an existing application.

>; Assuming the extra burden that Python et al. would bring (as a new
> dependency), I think we need a positive benefit for these other
&gt; languages that justifies the additional burden.

Python wouldn't be a "new&quot; dependency -- just a different one.

As far as languages go, Python is so far up and above and beyond Perl
that you can't even see the poor old stinker in the dust. Perl needs to
die, quietly.

--
Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack < woods%40robohack.ca">woodsrobohack.ca>
Planix, Inc. < woods%40planix.com">woodsplanix.com> Secrets of the Weird < woods%40weird.com">woodsweird.com>

On 10/8/07, Greg A. Woods < woods%40weird.com">woodsweird.com> wrote:
&gt; At Mon, 8 Oct 2007 10:05:22 -0500, Dustin J. Mitchell wrote:
&gt; Subject: Re: Scripting language choice (Was: Scripting the Amanda core)
&gt; >
>; > On 10/7/07, Greg A. Woods < woods%40weird.com">woodsweird.com> wrote:
&gt; > > If "we're" talking about something for scripting _within_ (i.e. in and
> > > "embedded" sense) an application then you want a truly easily embeddable
> > > interpreter, too, regardless of the language.
> >
>; > The proposal is actually to do the opposite -- embed Amanda's
> > low-level and efficiency-critical stuff (written in C) *in* a
> > higher-level language. So "main()" is written in ${LANGUAGE}, and
> > takes care of the argument parsing, operation planning, error
&gt; > recovery, etc., while calling C functions to take care of the details.
>
>; You see that's what rdiff-backup is -- it takes a few bits that need
>; high efficiency and plugs them into the interpreter (e.g. librsync) and
> then the rest of the application is just a Python program.
>
>; That's where it's good to have the full compliment of a big general
> purpose scripting language like Python.

Then let's truncate the embedding discussion, since what you've
described above is the proposal befor eus. In fact, it would be great
to see other projects begin to use Amanda as a library, similar to
rdiff-backup's use of librsync.

Dustin

--
Storage Software Engineer
http://www.zmanda.com

At Mon, 8 Oct 2007 14:48:06 -0500, Dustin J. Mitchell wrote:
Subject: Re: Scripting language choice (Was: Scripting the Amanda core)
&gt;
> Then let's truncate the embedding discussion, since what you've
&gt; described above is the proposal befor eus. In fact, it would be great
&gt; to see other projects begin to use Amanda as a library, similar to
> rdiff-backup's use of librsync.

A more realistic comparison for what Amanda might need in library form
would be some compression library.

I.e. I don't see why the proposal isn't just to rewrite Amanda from the
ground up in something like Python. There's really nothing whatsoever
at all to be gained from trying to keep any of the current logic or
other goo from the existing C code in some object library form. Really.

Most of that code is probably long past its best before date anyway
(most code is the day it is written) -- it really needs rewriting
regardless of whether the implementation language is changed in the
process or not.

Even the lowest level device interfacing can probably be best done
directly in any suitable high-level scripting language.

That's my whole point about rdiff-backup. You can easily write whole
systems level applications directly in a suitable high-level interpreted
language with only VERY minimal need for interfacing to compiled code.

--
Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack < woods%40robohack.ca">woodsrobohack.ca>
Planix, Inc. < woods%40planix.com">woodsplanix.com> Secrets of the Weird < woods%40weird.com">woodsweird.com>

On 10/8/07, Greg A. Woods < woods%40weird.com">woodsweird.com> wrote:
&gt; Most of that code is probably long past its best before date anyway
&gt; (most code is the day it is written) -- it really needs rewriting
> regardless of whether the implementation language is changed in the
> process or not.

This is a *great* point -- it's a popular myth on this list that the
existing Amanda code is well-tested or correct. It doesn't take a
very deep reading of a typical Amanda source file to find a potential
segfault or other incorrect operation -- usually on an edge case that
isn't triggered by typical use.

> Even the lowest level device interfacing can probably be best done
>; directly in any suitable high-level scripting language.

I disagree here, though -- Amanda is a performance-sensitive
application, responsible for moving lots of data. This is the weak
point of most popular scripting languages, because they try to treat
data as strings, and are fond of copying strings many times as part of
their normal operation. Amanda already copies data way too much in
the *C* version -- a failing we'd like to remedy, not exacerbate

Secondarily, Amanda has a number of compatibility-sensitive areas,
notably the security API and the sendbackup/dumper protocol, where a
reimplementation would carry a serious risk of subtle
incompatibilities. Obviously, having network protocols specified by
implementation is less than ideal, but it's what we've got.

Finally, there are some aspects of driving tape devices that are not
easily available from scripting languages.

Dustin

--
Storage Software Engineer
http://www.zmanda.com