ExtUtils::CBuilder: are we not there yet?

The included patch makes it possible to build *some* XS
extensions using
Module::Builder.  However, the tests of CBuilder and
Module::Builder
do not pass.

The tests not passing are just tips of the iceberg; CBuilder
is not yet
close enough to be a robust PerlExtension-DLL builder
environment.
Below are the reasons via the current API can't work (in
random order)
[in my patch I use some heuristics to circumvent these
limitation; the
heuristics might work in simplest cases only]:

  0) Documentation states that CBuilder is used for building
Perl
     extension DLLs.  However, the CBuilder test suite
attempts to test
     building *standalone* DLLs.

	Possible solutions:

	  a) include a test with perl-extension.c (pre-built with
xsubpp,
	     or manually); one may even try to load it from Perl!

	  b) Document that CBuilder may be used to build standalone
DLLs
	     with "simplest linking requirements"; this
might be indicated
	     by omitting `module_name' key in the argument hash of
link() etc.

  1) DynaLoader is given the module name, and constructs the
DLL basename
     via this module name.  Therefore, module builder should
imitate
     this process, and construct the DLL basename via the
`module_name'
     field.  However, the default link() implementation
ignores
     `module_name' completely, and tries to construct the
DLL basename via
     the names of object files (?!).

  1') The actual name of the built DLL should be a
combination of the given
      output directory, and DLL basename calculated as in
"1".  However,
      there is no way to specify the output directory to
link()...

  2) Similarly, there should be a way to find the name of
constructed DLL
     without actually building it.  Currently, the only way
to "try to do
     it" is via lib_file() method; but it has no way to
inspect the
     `module_name' field, and/or output directory, so
lib_file() can't
     be used for this purpose at all.

  3) The structure of linker command line should be

	link_command link_args1 OBJECTS link_args2 LIBS link_args3
TARGET

     (with TARGET put in appropriate place).  However, the
method link() is
     not given LIBS at all!  As a result, one can't put
link_args2 in proper
     position.

  4) It is not documented where the "EXPORT FILE"
(constructed on the
     pre_link() stage) should be created: in current
directory, in the
     directory of .OBJ files, in the directory of DLL, some
other place?

   4') If "0" allows generic DLLs, then one may
want to construct a
       "link library" for this DLL (this is not
needed for Perl Extension
       DLLs, since one never links against them - one loads
the only
       "exported" function from the DLL via
dlsym() instead).

       Same questions arise: what should be the basename of
this library,
       and in which directory should it be created...

   5) I do not see it documented whether prelink() is called
for
      builds of executables, and if so, how to detect it
from the method...

Hope this helps,
Ilya

============================================================
======

This patch "works around" major problems of
CBuilder, AND leaves only
the following failures for Module::Builder:

Failed Test Stat Wstat Total Fail  Failed  List of Failed
------------------------------------------------------------
-------------------
t/ppm.t        1   256    12    1   8.33%  3
t/tilde.t      1   256    11    1   9.09%  3
t/xs.t       255 65280    22    2   9.09%  22
2 tests and 12 subtests skipped.
Failed 3/21 test scripts, 85.71% okay. 3/644 subtests
failed, 99.53% okay.

The chunk for Base.pm just adds comments which field means
what; I
also omitted the following chunk, which shows the problem
with trying
to use lib_file():

    --- ./t/01-basic.t-pre	Sat Mar 25 11:37:28 2006
    +++ ./t/01-basic.t	Thu Dec 21 16:57:32 2006
     -45,7 +45,11  ok 1;
     my ($lib, temps) = $b->link(objects =>
$object_file,
				  module_name => 'compilet');
     $lib =~ tr/"'//d;
    -ok $lib_file, $lib;
    +if ($^O eq 'os2') {
    +  ok 1;	# lib_file() not applicable: name based on
module name, not OBJ name
    +} else {
    +  ok $lib_file, $lib;
    +}

     for ($source_file, $object_file, $lib_file) {
       tr/"'//d;


--- ./lib/ExtUtils/CBuilder/Base.pm-pre	Sat Mar 25 11:37:28
2006
+++ ./lib/ExtUtils/CBuilder/Base.pm	Thu Dec 21 16:29:14 2006
 -167,9
+167,9  sub prelink {
     DL_FUNCS => $args     || {},
     FUNCLIST => $args || [],
     IMPORTS  => $args   || {},
-    NAME     => $args,
-    DLBASE   => $args,
-    FILE     => $args,
+    NAME     => $args,		# Name of the Perl
module
+    DLBASE   => $args,		# Basename of DLL file
+    FILE     => $args,		# Dir + Basename of
symlist file
     VERSION  => (defined $args ?
$args : '0.0'),
   );
   
--- ./lib/ExtUtils/CBuilder/Platform/os2.pm-pre	Sat Mar 25
11:37:28 2006
+++ ./lib/ExtUtils/CBuilder/Platform/os2.pm	Fri Dec 22
14:58:12 2006
 -12,26
+12,60  sub need_prelink 
 sub prelink {
   # Generate import libraries (XXXX currently near .DEF;
should be near DLL!)
   my $self = shift;
-  my res = $self->SUPER::prelink(_);
+  my %args = _;
+
+  my res = $self->SUPER::prelink(%args);
   die "Unexpected number of DEF files" unless
res
== 1;
   die "Can't find DEF file in the output"
-    unless $res[0] =~ m,^(.*?)([^\/]+).def$,si;
-  my $libname = "$2$self->";
+    unless $res[0] =~ m,^(.*).def$,si;
+  my $libname =
"$1$self->";	# Put .LIB file
near .DEF file
   $self->do_system('emximp', '-o', $libname, $res[0]) or
die "emxexp: res=$?";
   return (res, $libname);
 }
 
 sub _do_link {
+  my $self = shift;
+  my ($how, %args) = _;
+  if ($how eq 'lib_file'
+      and (defined $args and length
$args)) {
+
+    # DynaLoader::mod2fname() is a builtin func
+    my $lib = DynaLoader::mod2fname([split /::/,
$args]);
+
+    # Now know the basename, find directory parts via
lib_file, or objects
+    my $objs = ( (ref $args) ? $args :
[$args] );
+    my $near_obj = $self->lib_file($objs);
+    my $ref_file = ( defined $args ?
$args : $near_obj );
+    my $lib_dir = ($ref_file =~ m,(.*)[/\],s ?
"$1/" : '' );
+    my $exp_dir = ($near_obj =~ m,(.*)[/\],s ?
"$1/" : '' );
+
+    $args = $1 if $near_obj =~ m,(.*).,s; # put
ExportList near OBJ
+    $args =
"$lib_dir$lib.$self->";	# DLL
file
+
+    # XXX _do_link does not have place to put libraries?
+    push $objs, $self->perl_inc() .
"/libperl$self->";
+    $args = $objs;
+  }
   # Some 'env' do exec(), thus return too early when run
from ksh;
   # To avoid 'env', remove (useless) shrpenv
-  my $self = shift;
   local $self-> = '';
-  return $self->SUPER::_do_link(_);
+  return $self->SUPER::_do_link($how, %args);
 }
 
-sub extra_link_args_after_prelink {	# Add .DEF file to the
link line
+sub extra_link_args_after_prelink {
+  # Add .DEF file to the link line
   my ($self, %args) = _;
-  grep /.def$/i, {$args};
+
+  my DEF = grep /.def$/i, {$args};
+  die "More than one .def files created by `prelink'
stage" if DEF > 1;
+  # XXXX No "$how" argument here, so how to test
for dynamic link?
+  die "No .def file created by `prelink' stage"
+    unless DEF or not {$args};
+
+  my after_libs = ($OS2::is_aout ? ()
+      : $self->perl_inc() .
"/libperl_override$self->");
+  # , "-L", "-lperl"
+  (after_libs, DEF);
 }
 
 sub link_executable {

On Dec 22, 2006, at 7:27 PM, Ilya Zakharevich wrote:

> The included patch makes it possible to build *some* XS
extensions  
> using
> Module::Builder.  However, the tests of CBuilder and
Module::Builder
> do not pass.

Are you speaking about a specific platform or a specific
perl version  
or something?  I build a lot of XS extensions using M::B and
 
CBuilder, so I assume you mean specifically the OS/2
platform?

BTW, it's Module::Build, not Module::Builder - it represents
"a  
build", and I hate class names that end with -er. 


> The tests not passing are just tips of the iceberg;
CBuilder is not  
> yet
> close enough to be a robust PerlExtension-DLL builder
environment.

Certainly true.  Right now it's still basically just a bunch
of  
command-line crap that I chopped out of ExtUtils::MakeMaker
a couple  
years ago and tried to make more generic in interface. 
Certainly  
your help toward improving things is welcome - I think you
do  
understand the goal of the module well.


> Below are the reasons via the current API can't work
(in random order)
> [in my patch I use some heuristics to circumvent these
limitation; the
> heuristics might work in simplest cases only]:
>
>   0) Documentation states that CBuilder is used for
building Perl
>      extension DLLs.

Yes, that's its goal.

>   However, the CBuilder test suite attempts to test
>      building *standalone* DLLs.

This is just because I thought that would satisfice for
testing  
purposes.  On the platforms I have access to, it seems to,
but I  
guess you're saying it doesn't?

BTW, what's a "standalone DLL"?  Isn't that an
oxymoron?


> 	Possible solutions:
>
> 	  a) include a test with perl-extension.c (pre-built
with xsubpp,
> 	     or manually); one may even try to load it from
Perl!

What is perl-extension.c?

If this is a major issue, I think probably the best solution
would be  
to build & use a small XS module in the tests.


>
>   1) DynaLoader is given the module name, and
constructs the DLL  
> basename
>      via this module name.  Therefore, module builder
should imitate
>      this process, and construct the DLL basename via
the  
> `module_name'
>      field.  However, the default link() implementation
ignores
>      `module_name' completely, and tries to construct
the DLL  
> basename via
>      the names of object files (?!).

That should be fixed, then.  Probably it's like that because
I didn't  
understand what was actually happening and that
"worked" for whatever  
subset I've tested it on.


>   1') The actual name of the built DLL should be a
combination of  
> the given
>       output directory, and DLL basename calculated as
in "1".   
> However,
>       there is no way to specify the output directory
to link()...

It's not just the directory portion of the 'lib_file'
argument?


>   2) Similarly, there should be a way to find the name
of  
> constructed DLL
>      without actually building it.  Currently, the only
way to "try  
> to do
>      it" is via lib_file() method; but it has no
way to inspect the
>      `module_name' field, and/or output directory, so
lib_file() can't
>      be used for this purpose at all.
>
>   3) The structure of linker command line should be
>
> 	link_command link_args1 OBJECTS link_args2 LIBS
link_args3 TARGET
>
>      (with TARGET put in appropriate place).  However,
the method  
> link() is
>      not given LIBS at all!  As a result, one can't put
link_args2  
> in proper
>      position.

When you speak of the structure of the command here, do you
mean  
specifically on OS/2?

>
>   4) It is not documented where the "EXPORT
FILE" (constructed on the
>      pre_link() stage) should be created: in current
directory, in the
>      directory of .OBJ files, in the directory of DLL,
some other  
> place?

Sorry, what's the export file?  Is it some Mksymlists()
argument I  
don't know about or something?

>
>    4') If "0" allows generic DLLs, then one
may want to construct a
>        "link library" for this DLL (this is
not needed for Perl  
> Extension
>        DLLs, since one never links against them - one
loads the only
>        "exported" function from the DLL via
dlsym() instead).


I do not think we need to allow generic DLLs.  If people
want to  
build generic DLLs, let them type the commands to do so. 
This module  
is for building DLLs to link with perl modules.

>    5) I do not see it documented whether prelink() is
called for
>       builds of executables, and if so, how to detect
it from the  
> method...

I don't know much about prelink() and when it's needed,
actually - it  
was written to support platforms I don't have runtime access
to, and  
I just kind of took the patches on faith.  Do you have
suggestions  
for what it should be doing or how it should be used?


As for the patch; I've committed the comments to Base.pm,
that's a no- 
brainer.  I'm inclined to commit the os2.pm stuff as-is
without  
review, or if you'd prefer we can see what other people on
p5p say.

Thanks for all your contributions, as you can probably see
I'm not  
really the best person to shepherd this module, but I think
it had to  
be written, so I wrote it.

  -Ken

On Fri, Dec 22, 2006 at 11:00:31PM -0600, Ken Williams
wrote:

> >The included patch makes it possible to build
*some* XS extensions using
> >Module::Builder.  However, the tests of CBuilder
and Module::Builder
> >do not pass.
> 
> Are you speaking about a specific platform or a
specific perl version or 
> something?  I build a lot of XS extensions using M::B
and CBuilder, so I assume 
> you mean specifically the OS/2 platform?

Time to time I happen to reread what I write, and start to
wonder: am
I dislexic, or what?  Of course I meant OS/2; it would make
wonders if
I rememebered to mention it too.  

> BTW, it's Module::Build, not Module::Builder - it
represents "a build", and I 
> hate class names that end with -er. 

> >  0) Documentation states that CBuilder is used for
building Perl
> >     extension DLLs.

> Yes, that's its goal.

Since it is much easier to build generic DLLs, why not
support it too,
when it does not include an additional work?    E.g., I
support
Math::Pari, which is an interface to a library which
sometimes is a
bitch to compile standalone.  Since I make a DLL with all
the
functions in the library *anyway*, why not allow to *also*
build one
which does not link with Perl DLLs?

> >  However, the CBuilder test suite attempts to test
> >     building *standalone* DLLs.

> This is just because I thought that would satisfice for
testing purposes.  On 
> the platforms I have access to, it seems to, but I
guess you're saying it 
> doesn't?

No, it does not.  Moreover, you do not even check whether it
load!

> BTW, what's a "standalone DLL"?  Isn't that
an oxymoron?

Probably.  But DLLs "used from a random program",
and DLL "used from
Perl" are very different beasts, with very different
requirements.

> >	Possible solutions:
> >
> >	  a) include a test with perl-extension.c
(pre-built with xsubpp,
> >	     or manually); one may even try to load it
from Perl!

> What is perl-extension.c?

Make "a trivial" perl-extension.xs, and run xs2c
(sp?) on it.

> If this is a major issue, I think probably the best
solution would
> be to build & use a small XS module in the tests.

IMO, this is too much extra dependencies.  Cut out the .xs
crap, and
check the simple stuff - the roadway from C to a DLL.

> >  1') The actual name of the built DLL should be a
combination of the given
> >      output directory, and DLL basename calculated
as in "1".  However,
> >      there is no way to specify the output
directory to link()...

> It's not just the directory portion of the 'lib_file'
argument?

It is how it is done in my patch.  However, this completely
ignores
the basename of lib_file, which may be
"surprising" given the current
docs.  It is better to have fewer surprises; why not allow
lib_file_dir, and allow "the directory portion of the
'lib_file'
argument" if no lib_file_dir is given?

> >  3) The structure of linker command line should be
> >
> >	link_command link_args1 OBJECTS link_args2 LIBS
link_args3 TARGET
> >
> >     (with TARGET put in appropriate place). 
However, the method link() is
> >     not given LIBS at all!  As a result, one can't
put link_args2 in proper
> >     position.

> When you speak of the structure of the command here, do
you mean
> specifically on OS/2?

In fact, I tried to find a description which would match an
odd dozen
of very different linkers I had access to...

> >  4) It is not documented where the "EXPORT
FILE" (constructed on the
> >     pre_link() stage) should be created: in
current directory, in the
> >     directory of .OBJ files, in the directory of
DLL, some other place?

> Sorry, what's the export file?  Is it some Mksymlists()
argument I
> don't know about or something?

It is the file created by Mksymlists().  (Its principal
purpose is to
describe which symbols in DLL are visible from outside world
["exported"]; thus the name.)

> >   4') If "0" allows generic DLLs, then
one may want to construct a
> >       "link library" for this DLL (this
is not needed for Perl Extension
> >       DLLs, since one never links against them -
one loads the only
> >       "exported" function from the DLL
via dlsym() instead).

> I do not think we need to allow generic DLLs.  If
people want to
> build generic DLLs, let them type the commands to do
so.

How would they *find* the commands they need to type?  And
how much
work it is to support this?  I described above the
situation:

   a) a Perl distribution with a subdirectory;

   b) subdirectory contains code of a library (as in xs
tutorials);

   c) toplevel directory contains .xs file with glue
interface to a
      library;

   d) To build Perl-extension DLL, you link together the
library, the
      glue code, and the principal Perl-proper DLL; all you
need to
      "export" is one function, which will
bootstrap the glue code
      (but you can add exporting of arbitrary functions in
the library
      via EXPORT keyword of Mksymlists()).

What one needs to do to support a "generic" DLL
which would export
just functions from the library?  *Remove* part
"c" (but it is
*already* outside of scope of this module!), and *remove*
linking with
Perl DLL (and exporting the bootstrap function).

Is it too much work to support this if 'module_name' is
undefined?
 
Take into account, especially, that what is removed is a
NOOP
anyway on legacy platforms (those with "russian
roulette" approach to
dynalinking, when you do not have export lists, and DO NOT
link with
any library).  

> >   5) I do not see it documented whether prelink()
is called for
> >      builds of executables, and if so, how to
detect it from the method...

> I don't know much about prelink() and when it's needed,
actually - it was 
> written to support platforms I don't have runtime
access to, and I just kind of 
> took the patches on faith.  Do you have suggestions for
what it should be doing 
> or how it should be used?

"Sane" approach to DLL linking is: linking of/with
DLL has the same
semantic as static linking, period.  You KNOW at link time
which name
mentioned in any .OBJ file is resolved to which library.  To
know
this when linking WITH a DLL, you need to know which names
should be
resolved to addresses in this DLL.

Hencefore, when you were linking this DLL, you needed to
SPECIFY
SEPARATELY which symbols of DLL should be visible from
outside world.
This is the purpose of "symbol list file".

Dually, when you link WITH a DLL, you need to know this list
too.
This information can be either extracted from a DLL file
directly, or
from accompanying .a (or .lib, or whatever) file.

E.g., Perl-proper DLL, perl5.so, may be accompanied by
perl.a (which
contains no code, but essentially just lists the functions
in the perl
library in the format understood by the linker).  When you
link an
executable which mentions, e.g., Perl_svpvn(), you do it
like this:

  ld -o my_perl.exe my_perlmain.o -L /usr/local/lib -lperl
-lotherlib

The linker can see that Perl_svpvn() is mentioned in perl.a,
and makes
a record in executable that perl.so should be loaded when
this
executable is run, and this part of executable code should
be patched
to resolve to the address of function Perl_svpvn() in
perl.so.

Dually, when you comple perl.so, you need to mention which
symbols in
the object files which comprise Perl proper should be
visible from
outside world.

  Note that linking against multiple DLLs, each with its own
list of
  "exported symbols", is very similar to
distinction between static
  and extern symbols in C; it is just happens on higher step
of the
  hierarchy (DLLs are aggregates of object files, and an
application
  is an aggregate of DLLs).

If you link a Perl Extension DLL, you need to do both: you
need to
"export" the name of glue code bootstrap function
(e.g.,
boot_Module__Build), and you need to "import" Perl
core functions
mentioned in your glue code.

Therefore, you need to both:

   a) specify an export list file which mentions
boot_Module__Build;

   b) allow linker to "import" Perl core functions
from
      /usr/local/lib/perl.a (by mentioning it on the linker
line *at a
      proper location* - order IS as imporant as with static
linking,
      and has the SAME semantic).

-------

I hope this wouldn't just mud your understanding of things
linking of
DLL may involve...

> As for the patch; I've committed the comments to
Base.pm, that's a no-brainer.  
> I'm inclined to commit the os2.pm stuff as-is without
review, or if you'd 
> prefer we can see what other people on p5p say.

I definitely welcome whatever contribution can be made! 
Especially if
it can simplify life for the future users of this
extension...

A lot of thanks on your other comments (if whatever part of
your
message is not mentioned, this means I agree with it , and for
your
work over this module.  I'm convinced it is *very* close now
to be a
more robust and convenient replacement for MakeMaker.

Best wishes for holidays,
Ilya