Re: new method dispatch rule (matz' proposal)

This is a late response to the very long thread that started
back in 
January with this message:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/rub
y/ruby-core/9996

As I understand it, the problem Matz is seeking to address
is that 
private methods are inherited by and may be overridden by
subclasses. 
This means that a subclass writer can inadvertently alter
the behavior 
of its superclass simply by defining a method that has the
same name as 
an internal helper method of the superclass.  The only way
to defend 
against this is for subclass writers to be familiar with the
internal 
implementation of the superclass.

Matz's proposal was to resolve this problem by altering the
method 
resolution algorithm.  If I understand correctly, the
problems were:

1) It was kind of complicated
2) The method lookup algorithm depended on whether a method
was called 
functionally without an object prefix or in OO form with an
object 
prefix.  (That is called as foo() or o.foo())
3) There were backward compatibility issues

Part of the discussion was about the question of whether to
alter the 
semantics of private methods, or add a new "local"
or "personal" 
visibility level for methods that were truely local to the
defining 
class and could not be inherited or overridden.

With all that as review, here are my thoughts on the
matter.

1) It is reasonable to expect private methods (or local
methods if we 
call them that) to be looked up and dispatched differently
than 
non-private methods.

2) The problem, though, is that since Ruby is a dynamic
language, the 
interpreter can't tell whether a method is private or not
until it has 
looked it up.

3) Therefore, solutions like Matz's proposal depend on the
method 
invocation syntax (functional style versus object-oriented
style) to 
determine which method lookup algorithm is to be used.  This
leads to 
the confusing and hard-to-justify situation in which the
invocation 
"foo" might do something different than the
invocation "self.foo".

I propose, therefore, that some new syntax be introduced
(avoiding 
backward incompatibility) either for making local method
names obviously 
different from other method names.  Or that a  new method
invocation 
syntax be introduced and that this new syntax be used when
we want to 
invoke a local method defined in (or inherited by) the
caller without 
considering any methods defined further down the class
hierarchy.

I don't grok parse.y well enough to know whether any syntax
I would 
propose would actually result in an unambiguous grammar, but
my 
suggestion is that local methods have an  prefix
just like instance 
variables do.  Then the interpreter knows from the name of
the method 
that it is to be dispatched differently.

In fact, I might actually go further than this and propose
that local 
methods are not actually methods, but are syntactic sugar
for lambdas.

In Ruby 1.8, we can write code like this:

class Test
   def initialize(greeting)
     greeting = greeting
     greeter = lambda { |x| puts "#greeting
#" }
   end

   def greet(x)
     greeter[x]
   end
end

t = Test.new("hello")
t.greet("world")

In this code, the instance variable greeter refers to a local
function 
that is completely hidden from subclasses and cannot be
altered.

Perhaps Ruby 1.9 could take this idiom and add syntax sugar
to make the 
definition and invocation of local functions more like the
definition 
and invocation of regular methods.

   David Flanagan

On 6/13/07, David Flanagan <daviddavidflanagan.com>
wrote:
> This is a late response to the very long thread that
started back in
> January with this message:
>
> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/rub
y/ruby-core/9996
>
> As I understand it, the problem Matz is seeking to
address is that
> private methods are inherited by and may be overridden
by subclasses.
> This means that a subclass writer can inadvertently
alter the behavior
> of its superclass simply by defining a method that has
the same name as
> an internal helper method of the superclass.  The only
way to defend
> against this is for subclass writers to be familiar
with the internal
> implementation of the superclass.
>
> Matz's proposal was to resolve this problem by altering
the method
> resolution algorithm.  If I understand correctly, the
problems were:
>
> 1) It was kind of complicated
> 2) The method lookup algorithm depended on whether a
method was called
> functionally without an object prefix or in OO form
with an object
> prefix.  (That is called as foo() or o.foo())
> 3) There were backward compatibility issues
>
> Part of the discussion was about the question of
whether to alter the
> semantics of private methods, or add a new
"local" or "personal"
> visibility level for methods that were truely local to
the defining
> class and could not be inherited or overridden.
>
> With all that as review, here are my thoughts on the
matter.
>
> 1) It is reasonable to expect private methods (or local
methods if we
> call them that) to be looked up and dispatched
differently than
> non-private methods.
>
> 2) The problem, though, is that since Ruby is a dynamic
language, the
> interpreter can't tell whether a method is private or
not until it has
> looked it up.
>
> 3) Therefore, solutions like Matz's proposal depend on
the method
> invocation syntax (functional style versus
object-oriented style) to
> determine which method lookup algorithm is to be used. 
This leads to
> the confusing and hard-to-justify situation in which
the invocation
> "foo" might do something different than the
invocation "self.foo".
>
> I propose, therefore, that some new syntax be
introduced (avoiding
> backward incompatibility) either for making local
method names obviously
> different from other method names.  Or that a  new
method invocation
> syntax be introduced and that this new syntax be used
when we want to
> invoke a local method defined in (or inherited by) the
caller without
> considering any methods defined further down the class
hierarchy.
>
> I don't grok parse.y well enough to know whether any
syntax I would
> propose would actually result in an unambiguous
grammar, but my
> suggestion is that local methods have an  prefix
just like instance
> variables do.  Then the interpreter knows from the name
of the method
> that it is to be dispatched differently.
>
> In fact, I might actually go further than this and
propose that local
> methods are not actually methods, but are syntactic
sugar for lambdas.
>
> In Ruby 1.8, we can write code like this:
>
> class Test
>    def initialize(greeting)
>      greeting = greeting
>      greeter = lambda { |x| puts "#greeting
#" }
>    end
>
>    def greet(x)
>      greeter[x]
>    end
> end
>
> t = Test.new("hello")
> t.greet("world")
>
> In this code, the instance variable greeter
refers to a local function
> that is completely hidden from subclasses and cannot be
altered.

I strongly disagree here. While I think it is good to
protect private
methods against inadvertently overriding them, I consider it
harmful
to completely hide them from subclasses. If I really want to
change
something I should be able to do so (i.e. via reflection).
In
languages with strict privacy concepts like Java it is a
pain to
extend a class in ways other than intended by the author of
that
class.

-- henon

>
> Perhaps Ruby 1.9 could take this idiom and add syntax
sugar to make the
> definition and invocation of local functions more like
the definition
> and invocation of regular methods.
>
>    David Flanagan
>
>

Meinrad,

I favor leaving private methods exactly as they currently
are, but 
adding a new mechanism or visibility level that a class
writer can use 
to ensure that local helper methods are not overwritten. 
Surely the 
author of a class ought to have the right to decide whether
subclassers 
are allowed to meddle!

     David

Meinrad Recheis wrote:
> I strongly disagree here. While I think it is good to
protect private
> methods against inadvertently overriding them, I
consider it harmful
> to completely hide them from subclasses. If I really
want to change
> something I should be able to do so (i.e. via
reflection). In
> languages with strict privacy concepts like Java it is
a pain to
> extend a class in ways other than intended by the
author of that
> class.
> 
> -- henon

Perhaps an underscore:

  class X
    def _foo
      puts "I'm local to X."
    end
  end

I suggest that b/c the idea of local instance vars in the
form of _a
has also been given consideration.

Though, I sort of like the idea of:

  def foo
    ...
  end

But maybe that only makes sense if all instance vars are
local and
accessors are required to communicate between class and
subclass.

T.

David Flanagan wrote:
> In Ruby 1.8, we can write code like this:
> 
> class Test
>   def initialize(greeting)
>     greeting = greeting
>     greeter = lambda { |x| puts "#greeting
#" }
>   end
> 
>   def greet(x)
>     greeter[x]
>   end
> end
> 
> t = Test.new("hello")
> t.greet("world")
> 
> In this code, the instance variable greeter
refers to a local function 
> that is completely hidden from subclasses and cannot be
altered.

class Test2 < Test; end
t2 = Test2.new("hello")
t2.greet("world") #=> "hello world"

How is this hidden from subclasses? Or did I miss
something?

But this touches on an idea I was thinking about recently...
how about 
giving a warning when a subclasses overrides a method and
doesn't use 
"super"? IMHO in 99% of cases if you override a
method you should call 
super. In the other 1% of cases you could use an idiom like
"super if 
false" which makes it very clear when reading the code
that we are 
(dangerously) skipping the normal inheritance chain. It
solves the same 
problem as the new method dispatch rule but for *both*
public and 
private methods, while still leaving the programmer freedom
to override 
private methods if he knows what he's doing. Comments?

Daniel

On 6/13/07, David Flanagan <daviddavidflanagan.com>
wrote:
> Meinrad,
>
> I favor leaving private methods exactly as they
currently are, but
> adding a new mechanism or visibility level that a class
writer can use
> to ensure that local helper methods are not
overwritten.  Surely the
> author of a class ought to have the right to decide
whether subclassers
> are allowed to meddle!
>
>      David
>
Hmm, but if there is something about a superclass which
cannot be
changed (not even by using some backdoor) the re-usability
of that
class is poor because it might not be possible to adapt the
class to
specific needs. It is impossible for the author to predict
all
possible use cases. That's why everything should be allowed.
Nobody
wants that others decide what he needs or needs not.

I agree with you that prevention of *accidental* overwriting
of
private methods is a good thing though.

-- henon

Daniel DeLorme wrote:
> David Flanagan wrote:
>> In Ruby 1.8, we can write code like this:
>>
>> class Test
>>   def initialize(greeting)
>>     greeting = greeting
>>     greeter = lambda { |x| puts "#greeting
#" }
>>   end
>>
>>   def greet(x)
>>     greeter[x]
>>   end
>> end
>>
>> t = Test.new("hello")
>> t.greet("world")
>>
>> In this code, the instance variable greeter
refers to a local 
>> function that is completely hidden from subclasses
and cannot be altered.
> 
> class Test2 < Test; end
> t2 = Test2.new("hello")
> t2.greet("world") #=> "hello
world"
> 
> How is this hidden from subclasses? Or did I miss
something?
> 
> But this touches on an idea I was thinking about
recently... how about 
> giving a warning when a subclasses overrides a method
and doesn't use 
> "super"? IMHO in 99% of cases if you override
a method you should call 
> super. In the other 1% of cases you could use an idiom
like "super if 
> false" which makes it very clear when reading the
code that we are 
> (dangerously) skipping the normal inheritance chain. It
solves the same 
> problem as the new method dispatch rule but for *both*
public and 
> private methods, while still leaving the programmer
freedom to override 
> private methods if he knows what he's doing. Comments?
That ends up looking a lot like a magical side-effect to me.
 Besides, I 
don't think the assumption (that you should often be calling
super) 
necessarily valid.

What it sounds to me like you're pining for is the
"override" keyword 
from C#.  I guess an equivalent implementation in Ruby might
be for 
methods defined with "def" to pop a warning when
they redefine a 
superclass method, and to introduce a "redef"
keyword to do specifically 
that.

Not sure I like the look of it (introducing new keywords,
bad), but I 
think it would fit the bill here.

-- 
Alex

On 6/13/07, David Flanagan <daviddavidflanagan.com>
wrote:
> This is a late response to the very long thread that
started back in
> January with this message:
>
> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/rub
y/ruby-core/9996
>
> As I understand it, the problem Matz is seeking to
address is that
> private methods are inherited by and may be overridden
by subclasses.
> This means that a subclass writer can inadvertently
alter the behavior
> of its superclass simply by defining a method that has
the same name as
> an internal helper method of the superclass.  The only
way to defend
> against this is for subclass writers to be familiar
with the internal
> implementation of the superclass.

I would argue that there's really no technical defense
against the
issues raised by inheriting from classes outside of your
control, or
having classes outside of your control subclass your class.
Inheritance is implementation sharing and it inherently
breaks
encapsulation.  That's why it's almost always better to
delegate to
classes you 'buy' rather than subclassing them.

Back when I was evangelizing OOP for IBM, I used to talk
about this
and say that inheritance was like sex, it's much more
socially
acceptable when it's done between committed consensual
adults.

This isn't to say that such subclassing is always bad, just
that it
requires care.

The case isn't that subclass writers can inadvertently alter
the
behavior of a superclass by overriding a private helper
method.
Without actually opening up the superclass and changing it,
the
superclass and it's instances will still work.  What gets
broken is
the subclass and its instances, and I'd argue that that's no
different
than any other bug in the subclass, and it's not limited to
inadvertantly changing a PRIVATE method. Of course private
methods are
more likely to be undocumented and missed by a drive-by
subclasser.

And if the concern about breaking classes is real, then the
solution
would need to include doing away with the ability to open
existing
classes. That would really mean that Ruby wouldn't be Ruby
anymore
IMHO.

My sense is that if the changes were to be implemented they
would just
provide a false sense of security, better to avoid
introducing further
complexity, I'd argue for better "inheritance
education" which
advocated delegation over promiscuous inheritance, and
"safe-inheritance" practices using the existing
mechanisms.

> Matz's proposal was to resolve this problem by altering
the method
> resolution algorithm.  If I understand correctly, the
problems were:
>
> 1) It was kind of complicated
> 2) The method lookup algorithm depended on whether a
method was called
> functionally without an object prefix or in OO form
with an object
> prefix.  (That is called as foo() or o.foo())
> 3) There were backward compatibility issues
>
> Part of the discussion was about the question of
whether to alter the
> semantics of private methods, or add a new
"local" or "personal"
> visibility level for methods that were truely local to
the defining
> class and could not be inherited or overridden.
>
> With all that as review, here are my thoughts on the
matter.
>
> 1) It is reasonable to expect private methods (or local
methods if we
> call them that) to be looked up and dispatched
differently than
> non-private methods.

Personally, my instincts say no.

> 2) The problem, though, is that since Ruby is a dynamic
language, the
> interpreter can't tell whether a method is private or
not until it has
> looked it up.

And here I think that we run into the tensions between the
concepts of
static and dynamic languages. Introducing static features
can have
some unpleasant effects.  There are already areas where Ruby
falls
down slightly on properly handling dynamic changes, for
example the
double module inclusion problem where a change to module A
to include
module B isn't seen by modules/classes which have already
included
module A:
http://eigenclass.org/hiki/The+double+inclusion+problem

It seems to me that this is just an implementation issue,
fixing it
would involve a little more internal bookkeeping so that the
method
lookup chain of mdules including A when A was changed.

Or the issues involved in handling "re-including"
a module which is
already included by an ancestor:
http://talk
likeaduck.denhaven2.com/articles/2006/10/09/a-subtle-change-
to-mixin-semantics-in-ruby-1-9

Ruby 1.8 ignores the 're-inclusion' at one point 1.9 altered
this
behavior, then it went back, I haven't re-built 1.9 in a
while so I'm
not sure where the matter lies now.

The 1.8 semantics on module 're-inclusion' always seemed
wrong to me,
I'm not sure I fully understand the rationale, I haven't
thought it
through but I thnk that it might be wrapped up in a
relationship with
the double inclusion problem.

These edge-cases are really small flaws which make Ruby just
a little
brittle and less dynamic than it could be.  I think of them
like the
stress hardening which happens when you bend a piece of
metal back and
forth.  Do it enough and the metal breaks at those hardened
points.
Fixing them would be like annealing the metal to get back
its original
malleability.

Getting back  to my main point my fear is that implementing
the kind
of tinkering with method lookup being proposed will 
increase the
number of these edge cases.

> 3) Therefore, solutions like Matz's proposal depend on
the method
> invocation syntax (functional style versus
object-oriented style) to
> determine which method lookup algorithm is to be used. 
This leads to
> the confusing and hard-to-justify situation in which
the invocation
> "foo" might do something different than the
invocation "self.foo".
>
> I propose, therefore, that some new syntax be
introduced (avoiding
> backward incompatibility) either for making local
method names obviously
> different from other method names.  Or that a  new
method invocation
> syntax be introduced and that this new syntax be used
when we want to
> invoke a local method defined in (or inherited by) the
caller without
> considering any methods defined further down the class
hierarchy.

IMHO, Ruby already has enough syntax, of course that comes
from my
Smalltalk background.


> I don't grok parse.y well enough to know whether any
syntax I would
> propose would actually result in an unambiguous
grammar, but my
> suggestion is that local methods have an  prefix
just like instance
> variables do.  Then the interpreter knows from the name
of the method
> that it is to be dispatched differently.
>
> In fact, I might actually go further than this and
propose that local
> methods are not actually methods, but are syntactic
sugar for lambdas.

Which means that the same thing should be possible with DSL
like
metaprogramming and without syntax changes to the language.

> In Ruby 1.8, we can write code like this:
>
> class Test
>    def initialize(greeting)
>      greeting = greeting
>      greeter = lambda { |x| puts "#greeting
#" }
>    end
>
>    def greet(x)
>      greeter[x]
>    end
> end
>
> t = Test.new("hello")
> t.greet("world")
>
> In this code, the instance variable greeter
refers to a local function
> that is completely hidden from subclasses and cannot be
altered.

Actually, it can since it's an instance variable.

class TestSub < Test
     def initialize
          super
          greeter = 
      end
end

or even

class TestSub2 < Test
    def set_greeter(&b)
        greeter = lambda(&b)
    end
end


> Perhaps Ruby 1.9 could take this idiom and add syntax
sugar to make the
> definition and invocation of local functions more like
the definition
> and invocation of regular methods.

My opinion is that the cure for this 'problem' is worse than
the disease.


-- 
Rick DeNatale

My blog on Ruby
http://talklikead
uck.denhaven2.com/

Rick DeNatale wrote:

<snip>

> My opinion is that the cure for this 'problem' is worse
than the disease.

I'd settle for a warning when run with -w. This code emits a
warning 
when run with -w:

# redef.rb
class Foo
    def bar
       puts "hello"
    end

    def bar
       puts "world"
    end
end

redef.rb:6: warning: method redefined; discarding old bar

But this does not:

# redef2.rb
class Foo
    def bar
       puts "hello"
    end
end

class Baz < Foo
    def bar
       puts "world"
    end
end

Is there any way to make the 2nd case emit a warning without
major 
changes to core Ruby?

Dan

On 6/14/07, Daniel Berger <Daniel.Bergerqwest.com> wrote:
> Rick DeNatale wrote:
>
> <snip>
>
> > My opinion is that the cure for this 'problem' is
worse than the disease.
>
> I'd settle for a warning when run with -w. This code
emits a warning
> when run with -w:
>
> # redef.rb
> class Foo
>     def bar
>        puts "hello"
>     end
>
>     def bar
>        puts "world"
>     end
> end
>
> redef.rb:6: warning: method redefined; discarding old
bar
>
> But this does not:
>
> # redef2.rb
> class Foo
>     def bar
>        puts "hello"
>     end
> end
>
> class Baz < Foo
>     def bar
>        puts "world"
>     end
> end
>
> Is there any way to make the 2nd case emit a warning
without major
> changes to core Ruby?

I'm not sure it should. Overriding a method is perfectly
natural,
sometimes you call super as part of that,when you are
augmenting the
inherited method, but sometimes you ARE replacing it and
that's fine.

 Also what should happen in a case like this:

class Foo
end

class Baz < Foo
  def bar
     puts "world"
  end
end

class Foo
   def bar
       puts "hello"
   end
end

Note that this is a temporal ordering, the code need not
appear in the
same file, for example the bar method might be added to the
base class
foo in a gem.

To make it more explicit, should Ruby examine all existing
superclasses AND subclasses AND including classes when
changing a
class/module?

What about singleton methods and classes?

There be dragons!

There's no magic bullet to warding off changes due to
changing
ancestors and descendants, the best one can do is to be
aware of what
CAN happen, and have a good set of regression tests to
detect
problems.


-- 
Rick DeNatale

My blog on Ruby
http://talklikead
uck.denhaven2.com/