Function Binding Redux (long post)

Okay.  So I've been spending the last few weeks attempting
to figure out 
how one would use a "natural" syntax to make a
function call in the Java 
Langbinding, that is, if you have a Langbinding class
"X" and a function 
"f" defined as a member, how can you write code
such as this:

   X x = new X(...);
   ...
   SomeClass sc = x.f(y, z, 42);

The short answer seems to be "not easily."  At the
very least, at the 
center of any solution needs to be some kind of function
dispatcher on 
the Java side that will take in the arguments, pass them to
the 
Langbinding module, and get the return.  Whether it's as
simple as:

   public <ReturnValue> ReturnValue call(Object ...
args) { ... }

or something involving more encapsulation, something akin to
this needs 
to dispatch the function call.

So here's the solutions so far:

1.  Use the function dispatcher directly.

This is ok, but not great, and it's certainly a step away
from natural. 
  It's at least not entirely procedural, because you can
define a 
wrapper object and make a call on it:

   import boost.langbinding.Instance;
   ...
   Instance x = new Instance("X", arg1, arg2,
arg3);
   x.call("f", arg4, arg5);

2a. Generate a Java source file where the methods wrap the
function 
dispatcher.

Not entirely appealing, because it's a build-time solution,
not a 
runtime solution.  But it would enable the use of
"natural" syntax very 
easily.

2b. Generate the same, except directly in bytecode, and put
the result 
in a class file.

Same advantages as (2a), except without the compile time,
but the 
drawback that we'd have to maintain bytecode generation
(probably not a 
big deal, though).

3.  Do the same as (2b), but load the class at runtime with
a special 
class loader.  This is (from what I understand) the crux of
Rene's idea, 
and the most initially appealing.

I've been trying to think of ways to make this work, but
there are some 
major compromises:

   1.  Without wrangling class loaders at startup, it's
impossible to use
       the class name in source code without it having been
loaded first.
       This means that you have to load the class, then use
it through
       its Class interface.  This means using reflection,
which is an
       entirely procedural API.  It's pretty gross.  If the
class can
       implement an interface though, (I'm referring to the
Java kind of
       interface), then you could just cast it, because a
class can
       implement an interface that's already been loaded by
another class
       loader with no problem.

   2.  You could wrangle class loaders at startup, or some
major
       milestone in your program's execution.  That is, you
could come up
       with a "launcher" that set up the class
loader chain, and invoke
       "main" from your main app class.  This
would enable you to use
       your generated classes naturally, because the main
class would be
       loaded with the special class loader in the loader
chain, and all
       of its dependent classes (i.e., the rest of the
program) would be
       loaded with the same loader chain.

       Unfortunately, this means that you can't have your
app classes
       visible from the launcher's classpath.  You actually
have to take
       steps to *prevent* the default loaders from finding
your app
       classes.  This is inconvenient, and may simply be
impossible for
       some users.  They may not have control over how their
classes are
       started.

   3.  Hack the JVM startup so that it uses a different
class loader.
       This still has the problem of accessibility, and
probably the
       problem of portability, because not all JVMs may
implement an app
       class loader override.

I thought of another alternative.  If we're willing to
compromise on 
natural syntax, we can go one better than compromise (1)
above.  I got 
this idea from looking at the Java scripting API.  In the
scripting API, 
you can pass an interface class to your script object, and
have it 
return an object defined in the script as an instance of the
interface.

We could do something like this:

   Instance x = new Instance("X", ctorArg1,
ctorArg2);
   Runnable r = x.asInterface(Runnable.class);

   new Thread(r).start();

or even:

   WrappedClass x = new WrappedClass("X");
   Runnable r = x.asInterface(Runnable.class, ctorArg1,
ctorArg2);

   new Thread(r).start();

This would presumably throw an exception if there wasn't a
function void 
run() that could be dispatched to.

It seems that it should be possible to generate a class that
implements 
an interface on the fly (I think that implemented interfaces
are just 
markers in the class data anyway).

What do you think?

-- 

John Moeller
fishcorngmail.com


------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

on Fri Jul 20 2007, John Moeller
<fishcorn-AT-gmail.com> wrote:

> Okay.  So I've been spending the last few weeks
attempting to figure out 
> how one would use a "natural" syntax to make
a function call in the Java 
> Langbinding, that is, if you have a Langbinding class
"X" and a function 
> "f" defined as a member, how can you write
code such as this:
>
>    X x = new X(...);
>    ...
>    SomeClass sc = x.f(y, z, 42);

Is this Java or C++ code?

In the domain of language binding, if you want to be
understood, you
have to be constantly vigilant for these ambiguities,
especially when
the two languages have similar syntaxes.

I didn't read further in detail, so you might want to
revisit the
whole post and look for other instances of this issue.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-cons
ulting.com

The Astoria Seminar ==> http://www.astoriasemin
ar.com


------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

David Abrahams wrote:
> on Fri Jul 20 2007, John Moeller
<fishcorn-AT-gmail.com> wrote:
> 
>> Okay.  So I've been spending the last few weeks
attempting to figure out 
>> how one would use a "natural" syntax to
make a function call in the Java 
>> Langbinding, that is, if you have a Langbinding
class "X" and a function 
>> "f" defined as a member, how can you
write code such as this:
>>
>>    X x = new X(...);
>>    ...
>>    SomeClass sc = x.f(y, z, 42);
> 
> Is this Java or C++ code?
> 
> In the domain of language binding, if you want to be
understood, you
> have to be constantly vigilant for these ambiguities,
especially when
> the two languages have similar syntaxes.
> 
> I didn't read further in detail, so you might want to
revisit the
> whole post and look for other instances of this issue.

Fair enough.  All of the code in the post is Java code.

-- 

John Moeller
fishcorngmail.com


------------------------------------------------------------
-------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

on Fri Jul 20 2007, John Moeller
<fishcorn-AT-gmail.com> wrote:

> 2a. Generate a Java source file where the methods wrap
the function 
> dispatcher.
>
> Not entirely appealing, because it's a build-time
solution, not a 
> runtime solution.  

Not necessarily.  If Java can compile and load source files
dynamically (it can, can't it?), I don't see any reason why
you can't
generate this source file on demand at runtime.  I guess
that's where
the "special class loader" you meantion below
would come in.

> But it would enable the use of "natural"
syntax very easily.
>
> 2b. Generate the same, except directly in bytecode, and
put the result 
> in a class file.
>
> Same advantages as (2a), except without the compile
time, but the 
> drawback that we'd have to maintain bytecode generation
(probably not a 
> big deal, though).

Agreed, probably not a big deal at all.  I'd be surprised if
details
of the function-calling convention ever changed enough to
require
maintenance.

> 3.  Do the same as (2b), but load the class at runtime
with a
> special class loader.  This is (from what I understand)
the crux of
> Rene's idea, and the most initially appealing.
>
> I've been trying to think of ways to make this work,
but there are some 
> major compromises:
>
>    1.  Without wrangling class loaders at startup, 

I don't know what you mean, or anything about what's
involved here.
That's not a criticism; I'm just letting you know I'm not
even a Java
n00b, much less an Xpert.

>        it's impossible to use the class name in source
code without
>        it having been loaded first.  This means that
you have to
>        load the class, then use it through its Class
interface.
>        This means using reflection, which is an
entirely procedural
>        API.  It's pretty gross.  If the class can
implement an
>        interface though, (I'm referring to the Java
kind of
>        interface), then you could just cast it, because
a class can
>        implement an interface that's already been
loaded by another
>        class loader with no problem.

I don't see how the "implement an interface" thing
helps you very
much.  You'd still need to generate Java code for the
interface,
right?

>    2.  You could wrangle class loaders at startup, or
some major
>        milestone in your program's execution.  That is,
you could come up
>        with a "launcher" that set up the
class loader chain, and invoke
>        "main" from your main app class.  This
would enable you to use
>        your generated classes naturally, because the
main class would be
>        loaded with the special class loader in the
loader chain, and all
>        of its dependent classes (i.e., the rest of the
program) would be
>        loaded with the same loader chain.

I'm way out of my depth now.

>        Unfortunately, this means that you can't have
your app classes
>        visible from the launcher's classpath.  You
actually have to take
>        steps to *prevent* the default loaders from
finding your app
>        classes.  This is inconvenient, and may simply
be impossible for
>        some users.  They may not have control over how
their classes are
>        started.
>
>    3.  Hack the JVM startup so that it uses a different
class loader.
>        This still has the problem of accessibility, and
probably the
>        problem of portability, because not all JVMs may
implement an app
>        class loader override.
>
> I thought of another alternative.  If we're willing to
compromise on
> natural syntax,

I don't recommend that.

> we can go one better than compromise (1) above.  I got
this idea
> from looking at the Java scripting API.  In the
scripting API, you
> can pass an interface class to your script object, and
have it
> return an object defined in the script as an instance
of the
> interface.
>
> We could do something like this:
>
>    Instance x = new Instance("X", ctorArg1,
ctorArg2);
>    Runnable r = x.asInterface(Runnable.class);
>
>    new Thread(r).start();
>
> or even:
>
>    WrappedClass x = new WrappedClass("X");
>    Runnable r = x.asInterface(Runnable.class, ctorArg1,
ctorArg2);
>
>    new Thread(r).start();

Is this just a compromise on the construction syntax, or
something
more/worse?

> This would presumably throw an exception if there
wasn't a function void 
> run() that could be dispatched to.
>
> It seems that it should be possible to generate a class
that
> implements an interface on the fly (I think that
implemented
> interfaces are just markers in the class data anyway).
>
> What do you think?

Too ignorant to have much of an opinion; I hope someone with
an iota
of Java-fu will have some useful feedback for you.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-cons
ulting.com

The Astoria Seminar ==> http://www.astoriasemin
ar.com


------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

David Abrahams wrote:
> on Fri Jul 20 2007, John Moeller
<fishcorn-AT-gmail.com> wrote:
> 
>> 2a. Generate a Java source file where the methods
wrap the function 
>> dispatcher.
>>
>> Not entirely appealing, because it's a build-time
solution, not a 
>> runtime solution.  
> 
> Not necessarily.  If Java can compile and load source
files
> dynamically (it can, can't it?), I don't see any reason
why you can't
> generate this source file on demand at runtime.  I
guess that's where
> the "special class loader" you meantion below
would come in.

It can only load and link classes dynamically.  The source
is compiled 
to bytecode as a separate build step.

> [snip]
> 
> I don't know what you mean, or anything about what's
involved here.
> That's not a criticism; I'm just letting you know I'm
not even a Java
> n00b, much less an Xpert.

That's fine, you're an expert in plenty else.    Here was
where I was 
hoping that Rene or his business partner would weigh in
(when they had 
time, of course).

>>        ...
>>        API.  It's pretty gross.  If the class can
implement an
>>        interface though, (I'm referring to the Java
kind of
>>        interface), then you could just cast it,
because a class can
>>        implement an interface that's already been
loaded by another
>>        class loader with no problem.
> 
> I don't see how the "implement an interface"
thing helps you very
> much.  You'd still need to generate Java code for the
interface,
> right?

Right, but not necessarily in the context of the classes
that you are 
using.  As I'm sure you know, a Java "interface"
is essentially just a 
pure virtual base class.  Because of that abstraction, you
can 
polymorphically use a class that you generate that
implements the 
interface.  It doesn't matter *how* you load the class or
define it.

So you can use the interface to help protect yourself from
the vagaries 
of class loading, implementation, the Reflection API, etc. 
That's 
mostly what I was getting at.

>> ...
>> I thought of another alternative.  If we're willing
to compromise on
>> natural syntax,
> 
> I don't recommend that.

Ok, this is where I wanted your (Dave's) input.  That is, I
wanted to 
know how far I could push the usage model.  Say I had a
Langbinding 
class definition (C++):

   class_<X>("X") [
     def("isFoo", &X::isFoo),
     def("getBar", &X::getBar)
   ];

In Python, it's straightforward to *use* it naturally, i.e.
(and you'll 
have to forgive me, I don't speak Python):

   X x;
   if (x.isFoo())
     return x.getBar("stuff");

Whereas in Java, it would require some gymnastics and/or
extra 
installation (both on the client's part) to get such a thing
to compile. 
  The alternative is to use a wrapper API or Reflection.

>> [snip]
>>    WrappedClass x = new
WrappedClass("X");
>>    Runnable r = x.asInterface(Runnable.class,
ctorArg1, ctorArg2);
>>
>>    new Thread(r).start();
> 
> Is this just a compromise on the construction syntax,
or something
> more/worse?

If you had an interface FooBarable defined as:

   public interface FooBarable {
     public boolean isFoo();
     public Object getBar(String request);
   }

[Note:  I'm assuming here that FooBarable is an interface
*not* 
generated by Langbinding; that the client creates this
interface or uses 
another provided by a library because it matches a subset of
the method 
interface of X, which the client must be familiar with to
use X in the 
first place.  I'm beginning to suspect that I didn't
communicate this 
clearly.]

In Java code, you could "inform" the Java backend
of Langbinding that X 
really did implement FooBarable, and get an X as an instance
of the 
class.  This would prompt the internals of the Java backend
to crank out 
a bytecode representation of X that implemented FooBarable:

   Class XClass = WrappedClass.asInterface(FooBarable,
"X");
   FooBarable fb = WrappedClass.construct(XClass, ca1,
ca2);
   if (fb.isFoo())
     return fb.getBar("stuff");

So, getting to your question, it would be a compromise of
the 
construction syntax, but probably no more.  Additionally,
Java 
programmers are generally used to "interface
programming."

As for iterator/collection support, I was thinking of
offering an 
interface that just hotwired the iterators into a Java
Collection.  That 
way, it could be used with a "foreach" loop in
Java.  This could be done 
without the aforementioned user gymnastics, as Collection is
an 
interface provided by the Java API.

However, not all of the binding "goodies"
available in Python or Lua are 
available in Java.  As an example, overriding field/property
access as 
you do in Boost.Python would be impossible in a Java
binding.  You'd 
have to settle for getters/setters.  I was going to bring
this issue up 
in a different post, though.

-- 

John Moeller
fishcorngmail.com


------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

John Moeller wrote:
> David Abrahams wrote:
>> on Fri Jul 20 2007, John Moeller
<fishcorn-AT-gmail.com> wrote:
>>
>>> 2a. Generate a Java source file where the
methods wrap the function 
>>> dispatcher.
>>>
>>> Not entirely appealing, because it's a
build-time solution, not a 
>>> runtime solution.  
>> Not necessarily.  If Java can compile and load
source files
>> dynamically (it can, can't it?), I don't see any
reason why you can't
>> generate this source file on demand at runtime.  I
guess that's where
>> the "special class loader" you meantion
below would come in.
> 
> It can only load and link classes dynamically.  The
source is compiled 
> to bytecode as a separate build step.

Although that depends on the Java runtime one is running in.
The JDK 
runtime comes with the compiler in it so it's possible to
"eval" source 
code. There are probably numerous limitations though.



-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software
.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

John Moeller wrote:

> 2b. Generate the same, except directly in bytecode, and
put the result 
> in a class file.
> 
> Same advantages as (2a), except without the compile
time, but the 
> drawback that we'd have to maintain bytecode generation
(probably not a 
> big deal, though).

There are bytecode generation libraries out there that would
make this 
even less of a big deal.

> 3.  Do the same as (2b), but load the class at runtime
with a special 
> class loader.  This is (from what I understand) the
crux of Rene's idea, 
> and the most initially appealing.

Yes, that's what I meant. But using a class loader is only
an idea. More 
briefly I would want to create the Java "code" and
have it bound to the 
native code when the library is loaded. It would work
something like:

When the library is loaded (System.loadLibrary) during the
JNI_OnLoad 
<http://tinyurl.com/2bj
zmz>.
   * It binds the Java native methods to the C++ functions
with 
RegisterNatives <http://tinyurl.com/2bj
zmz>.

That allows for the distribution of only the shared library
(although it 
might be possible to work something equivalent for the
embedded use 
case). All users need to do is call System.loadLibrary. What
I don't 
know is how the Java compiler could be made to work without
actual 
*.class files.

> I've been trying to think of ways to make this work,
but there are some 
> major compromises:

Hm, I think your three points aren't relevant to what I just
mentioned 
above so I'm refraining from comment until proven otherwise




-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software
.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

Rene Rivera wrote:
> What I don't 
> know is how the Java compiler could be made to work
without actual 
> *.class files.

Thinking a bit more about this... It should be easy to
create a test for 
this. Just:

* Create a Java source file which declares a native class.
* Compile that to the *.class equivalent.
* Write a library that implements the basic algo I
mentioned, but is 
hardwired to:
     * Load in the contents of the *.class file.
     * Bind a method from the Java class to an arbitrary C++
function.
* Create a second Java class that subclasses, or otherwise
uses, the 
library class and that does the System.loadLibrary before
defining the 
class.
* Try and compile the new class.


-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software
.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

John Moeller wrote:
> I think I see where you're going, but it won't work. 
At the point where 
> you "subclass or otherwise use" and then
compile, the compiler will 
> error out.  If the compiler can't find the dependent
class, the target 
> class will fail to compile.  I tried this.

Yea, I see that now that I've spent today trying out the
test I 
mentioned. But it turns out it's worse...

> Additionally, as I explain in more detail in my other
response, you 
> can't, in class D, try to load class C explicitly and
then use it 
> implicitly.  In the process of loading class D, the JVM
can try to load 
> class C and any other class that D depends on, before
it even runs one 
> speck of code from class D.

So on the suggestion from JW (John Welch), I tried an
experiment. He 
suggested a rather devious possible solution; Create an
empty stub 
*.java and corresponding *.class that loads the native
class:

====java
package boost;

public class A
{
     static {
        
java.lang.System.out.println("stub/boost/A/static_init.
..");
        
java.lang.System.loadLibrary("boost_A.dll");
     }
}
====

And as part of the loadLibrary() redefine the same class to
the 
native+JNI version (see attached test code). Hence when one
uses the A 
class, it can be loaded, but one is freed from having to
declare all the 
redundant Java members. But, and it's a big one, it seems
the Java 
compiler *does not load* a class in order to use it during
compilation. 
Rather it seems to read the .class file directly. Which
seems to me a 
rather serious case of false advertising from Sun.

> Unless, that is, you use class C 
> explicitly, through Reflection.  Which is a gross way
to use a class.

Yes, that's fairly gross. Given your experiments, and mine,
this only 
leaves your suggestion of interfaces or of a launcher that
puts in a 
custom class loader.


-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software
.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo

------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding

on Wed Jul 25 2007, Rene Rivera
<grafikrobot-AT-gmail.com> wrote:

> Grr... Not only did I forget the attachment. When I put
it in the SF 
> mailman program cheerfully informs me they are blocking
attachments.

That's odd; we have content filtering turned *off* on this
list.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-cons
ulting.com

The Astoria Seminar ==> http://www.astoriasemin
ar.com


------------------------------------------------------------
-------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and
a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Boost-langbinding mailing list
Boost-langbindinglists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/boost
-langbinding