Re: commit: ldap/servers/slapd/back-perl SampleLDAP.pm

Hello Hallvard,

Am 19.06.2007 um 17:22 schrieb Hallvard B Furuseth:
> Dagobert Michelsen writes:
>> - each Perl backend database has a separate Perl
interpreter
>> - each connection gets an interpreter cloned from
the specific
>>    backend. Interpreters are recycled in a pool for
performance
>> - concurrent requests to different interpreters can
run
>>    concurrently in separate threads
>
> Is there a max number of Perl interpreters per
backend?

I suppose you mean "per database". In my
understanding there
are different backends coded in slapd/back-* where the
back-perl
I am working on is one backend which can be instantiated in
slapd.conf. There may be more databases configured for the
Perl backend.

There is exactly one interpreter in use for each active
connection
and each configured back-perl database because allocation is
done
on connection_init.

The maximum number of interpreters is the maximum count of
connections (dtblsize) multiplied with the number of
back-perl
databases.

> What happens if there are more long-lived connections
than that?

As this is the maximum this can never be the case.

> Does a connection release its Perl interpreter to the
pool if
> a few requests on the connection use the Perl database
and then
> the rest of the requests use a non-Perl database?

Perl interpreters are allocated when a connection to
OpenLDAP
is made and released to the pool when the connection is
terminated.

This method has some advantages:
- As access to the Perl interpreter must be exclusive there
is
   no contention because no two operations can be active for
one
   connection at the same time
- Data from a connection is automatically available in all
method
   invocations for this connection as the calls are all
directed to
   the same Perl interpreter

The disadvange is that there is no data sharing between
connections.
Implementing data sharing with this model is quite complex
and
involves attaching (Perl-)magic to shared variables which
implements
mutex'ed access between interpreters.

Currently Perl interpreters are never freed once they are
released.
Releasing of more than, say, 5 idle interpreters may be
implemented
as an optimization. I look into that if I have a fully
working
prototype.

> Are you using threads created by Perl, or
ldap_pvt_thread_create(),
> or are you somehow giving the interpreters short-lived
tasks via
> ldap_pvt_thread_pool_submit()?  There are some slapd
features which
> can only be used if you do the latter, since that
creates a
> "thread context" which holds thread-specific
variables.

I don't do any explicit thread management. All threads have
been set up by OpenLDAP when the backend entrypoints are
called,
so I assume that this is done right 

>> - argument passing is done in a style similar to
Net::LDAP,
>>    parsing is no longer needed as attributes and
values are
>>    separately stored
>
> You might not even need to do that.  Could wrap an
Entry in a class
> which translates slapd attributes to Perl lists on
demand.

This is possible, however not trivial: The C data
structures
would be implemented as blessed scalars containing the C
pointer.
Access method could then convert the data on demand.
However,
the C pointer must be some kind of "weak" pointer
which is
only valid in the context of this method invocation.
Otherwise
the scalar with the C reference could be stored in the
interpreter
and deferenced later in another method entry where the C
data
has already been freed. This all is a bit complex and can
be
done later without API change if proven useful.

As the method usually does something with the parameters the
data
is needed anyway, so savings here should be minimal if not
negative.
Another thing is access to seldom used data like connection
parameters or operation details where transformation to
your
proposed structure would indeed be useful.

>> - switching to old-backend-compatible-mode is done
when
>>    no connection_init method is found
>
> That sounds like a hack, one might not be interested in
defining a
> connection_init method anyway.

In my current implementation it is mandatory to implement
connection_init. The returned value must be a reference to
an object on which the methods for the operations are
called
(like search, compare, etc.)

If you have a usecase for an application without
connection_init
I would like to hear it.

>> The new backend is not 100% compatible to the old
backend
>> due to the lack of data sharing. This is a complex
issue
>> but maybe solvable with the help of the mod_perl
code.
>> Alternatively the code could fall back to a single
>> interpreter when compatibility mode is on.
>
> Unless the incompatibilities are quite small:
>
> I think it would be a bad idea to automatically invoke
an "almost-
> compatible" mode.  Then a number of old Perl
programs can break
> mysteriously, and maybe munge whatever data they are
maintaining
> before it is discovered.
>
> I suggest you make the new database fail to configure
if provided  
> an old
> Perl program and vice versa.

Ok. I see the importance now and make the changes
accordingly.
To summarize:
- Exit with fault if old Perl-module is used with new
backend
- Provide compatibility-module which must be enabled
explicitly
- Exit with fault if new Perl-module is used with old
backend

> I also suggest you do not let compatibility get in the
way of good
> design decisions for the new code.  If good design
leads to bad
> compatibility, we could always put the old back-perl
in
> contrib/back-oldperl or something.

Including the old backend is something I really would like
to
avoid as this means adapting autoconf-stuff and be
confusing.

>> It is still not possible to do all things from Perl
which
>> can be done in C. Exposing more datastructures and
functions
>> to Perl may even make it possible to write overlays
in Perl
>> in some distant future.
>
> Both are possible, in particular if you provide Perl
types as wrappers
> around the basic C types instead of translating them to
arrays or  
> hashes
> before delivering them to Perl.  (If you've translated
them it's a bit
> of a problem to call a number of C functions because
you've lost the
> original data structure which you would pass _to_ the C
function.

This is true but otherwise you need "weak"
references or induce
the possibility of subtle bugs with dangling pointers. If
this
feature is provided it must be as safe as possible in my
opinion.

> Or the user has made updates in Perl structures which
are not  
> reflected in
> the C structures they came from.  I looked into doing
this with Python
> once, but punted since the Python C API #includes a
header with Python
> autoconf variables which conflict with slapd's autoconf
variables.

Do you have some design notes of this left which you can
send me?

When I have something which is usable for testing I'll
post.


Best regards

   -- Dagobert


BTW: For my personal use I set up OpenGrok for browsing the
      OpenLDAP sources which I really enjoy. Feel free to
have
      a look at http://openldap.balt
ic-online.de

-- 
Dagobert Michelsen (Leiter IT)          Baltic Online
Computer GmbH
Firmensitz:   Alter Markt 1-2, 24103 Kiel, Telefon: +49 431
54003-0
Geschäftsführer:        Erik Cickovskis, Amtsgericht Kiel,
HRB 3756
"Of course computer servers don't need thrust, since
they generally
don't go anywhere."  -- Comment in TR on new HP server
turbine fans