List Info

Thread: Vmethods to add to 2.15




Vmethods to add to 2.15
user name
2006-01-31 23:19:10
Hi Andy / Everyone,

While we're looking at a new release, perhaps now is a
proper time to get some vmethods in that seem to be missing
for everyone. I have added both substr for scalars and
delete for hash ops, as I'm sure many others have done.
(Actually I just looked at CVS and it looks like you already
added delete.)

my substr vmethod:
-----------------------------------------------
$Template::Stash::SCALAR_OPS-> =
  sub {
    my ($scalar, $offset, $length) = _;
    return substr($scalar, $offset, $length);
  };
-----------------------------------------------



In addition, I just had another thought of something that
seems to be missing and a cause of problems on the mailing
list occasionally. That would be missing back references in
replace regular expressions. So here is a possible new
vmethod called 'replace_backref' that would allow backref
replacement without the security problems associated with
backrefs. It's obviously not perfect, but I think if we just
documented the feature correctly, then people would know
what to expect. I also added a parameter to check globally
or not, which also might be a good feature to add to the
current replace. 

------------------------------------------------------------
$Template::Stash::SCALAR_OPS->{'replace_backref'}  = sub
{
  my ($str, $search, $replace, $do_global_match) = _;
  if (!defined($do_global_match)){
    $do_global_match = 1; #match globally by default.
  }

  $replace = '' unless defined $replace;
  return $str unless defined $str and defined $search;

  if ($do_global_match){
    $str =~ s!$search!
      my vals = ($1, $2, $3, $4, $5, $6, $7, $8, $9);
      my $template = $replace;
      $template =~ s/#(d+)#/$vals[$1 - 1] || ''/eg;
      $template;
    !ge;
  }else{
    $str =~ s!$search!
      my vals = ($1, $2, $3, $4, $5, $6, $7, $8, $9);
      my $template = $replace;
      $template =~ s/#(d+)#/$vals[$1 - 1] || ''/eg;
      $template;
    !e;
  }

  return $str;
};
------------------------------------------------------------


Here is an example of how it would be used:
------------------------------------------------------------
[% string = "Josh is an ok guy" %]
New String: [% string.replace_backref('^(w+ is)', '#1#
not') %]
Should Output: Josh is not an ok guy

[% string2 = 'Josh is not an ok guy' %]
New String: [% string2.replace_backref('(not) (an)', '#1#
#2#') %]
Should output original string2.
------------------------------------------------------------

I used d+ in the regexp up there, thinking that we may
later on add more than the 9 backrefs, but for now we could
just use [1-9].

Some things that would need to be documented are:
Downfalls:
*) If you want to use #d+# literally in your replace string
your sol. However, the user could simply use the normal
replace instead.
*) Takes longer than the normal replace cause of the extra
templating involved.
*) Currently only goes from 1-9 backrefs.

Options:
*) You can specify to not globally replace now.

I can pretty much guarantee people won't care about the
extra replace time as long as they are able to use their
back references. (Those that are concerned can simply
override the vmethod.)

Any thoughts/changes on this idea/patch are welcome. (ie:
different names for it, or maybe a different format than
#d+#.)

-- Josh


_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 08:09:34
Josh Rosenbaum wrote:
> While we're looking at a new release, perhaps now is a
proper time to get 
> some vmethods in that seem to be missing for everyone.
I have added both 
> substr for scalars and delete for hash ops, as I'm sure
many others have 
> done. (Actually I just looked at CVS and it looks like
you already added 
> delete.)

Hi Josh,

Yes, I've added hash.delete(key) and scalar.remove(pattern)
which does
the same thing as scalar.replace(pattern, '')

I also added substr to the String plugin, but forget to add
it as a string
vmethod (one reason why this is all unified in TT3 - to
avoid all this 
duplication).  Thanks for the reminder.

> -----------------------------------------------
>  sub {
>    my ($scalar, $offset, $length) = _;
>    return substr($scalar, $offset, $length);
>  };
> -----------------------------------------------

Unfortunately it's not as simple as that.  Perl
distinguishes between
substr($scalar, $offset, undef) and substr($scalar, $offset)
so we have
to hard-code the alternatives (unless anyone knows another
way?).  I've
also added code the handle the 4th argument to provide a
replacement string,
thanks to http:/
/rt.cpan.org/Ticket/Display.html?id=2619

    'substr' => sub {
        my ($text, $offset, $length, $replacement) = _;
        $offset ||= 0;

        if(defined $length) {
            if (defined $replacement) {
                my $removed = substr( $text, $offset,
$length );
                substr( $text, $offset, $length ) =
$replacement;
                return $removed;
            }
            else {
                return substr( $text, $offset, $length );
            }
        }
        else {
            return substr( $text, $offset );
        }
    },

I think that covers all bases.  Damn shame when something
that should be
so simple turns out so messy.  Ho hum, that's Perl
sometimes.

> In addition, I just had another thought of something
that seems to be 
> missing and a cause of problems on the mailing list
occasionally. That 
> would be missing back references in replace regular
expressions. 

Yeah, this is a nuisance, but I haven't yet seen an
implementation that 
solves all the problems securely and robustly.  I'm not keen
on adding a
new syntax for back-references.  What was your reasoning for
that over
the regular $1 or 1 syntax?

I'll give it some more thought.  I'll also dig through the
archives and
see what other proposals we've had to solve this.  Any of
those people 
who have made suggestions are welcome to chip in again here,
to refresh
our memories if nothing else.

Cheers
A



_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 09:43:36
Andy Wardley wrote:
> I'll give it some more thought.  I'll also dig through
the archives and
> see what other proposals we've had to solve this.  Any
of those people 
> who have made suggestions are welcome to chip in again
here, to refresh
> our memories if nothing else.

This is the most promising candidate from Craig Barratt. 

sub replace { 
    my ($str, $search, $replace) = _;
    $replace = '' unless defined $replace;
    return $str unless defined $str and defined $search;

    $str =~ s{ $search } {
	my $r = $replace;
	my d = (0, $1, $2, $3, $4, $5, $6, $7, $8, $9);
	$r =~ s/$(d+)/$d[$1]/eg;
	$r;
    }egx;

    return $str;
}

It's limited to the first 9 captures only (no $10, $11, etc)
and it 
doesn't handle escaped '$' in the replacement string.  But
apart from
that, it seems to do the job without opening up any security
holes.

A


_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 02:40:11
Josh Rosenbaum wrote:
> While we're looking at a new release, perhaps now is a
proper time to 
> get some vmethods in that seem to be missing for
everyone.

In the same spirit -- not being able to use backrefs in
.replace() is why I 
wrote Template::Plugin::Subst.  I'm (obviously) biased, but
it would seem to 
be an appropriate addition to the core.

N

_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 14:57:20
Nik Clayton wrote:
> In the same spirit -- not being able to use backrefs in
.replace() is why I 
> wrote Template::Plugin::Subst.  I'm (obviously) biased,
but it would seem 
> to be an appropriate addition to the core.

Hi Nik,

Sorry I wasn't aware of your module, but it certainly looks
like the kind
of robust and properly thought-out solution that we need. 
Excellent job!

I noted one minor problem.  In the rather extreme case of
having more than 
9 backreferences, the $10, $11, etc, placeholders wouldn't
get replaced
because they would previously have been munched away by $1.

    foreach my $i (1..$#saved_match_start) {

Reversing the order of replacement fixes the problem.

    for (my $i = $#saved_match_start; $i; $i--) {

I was also able to re-write it using a loop (tail-recursion
style)
rather than recursively calling itself.  Although I
preferred the 
elegance of your recursive approach, this way makes it a
little easier
to patch directly into $Template::Stash::SCALAR_OPS without
having to 
create a separate named subroutine that we can recursively
call. 

So with your blessing, I'll include the following as the new
replace()
virtual method.

    'replace' => sub {
        my ($text, $pattern, $replace, $global) = _;
        my ($matched, $after, $backref, start,
end);
        my $result = '';

        $global = 1 unless defined $global;
        
        while ($text =~ m/$pattern/) {
            if($#- == 0) {  
                # no captured groups so do a simple search
and replace
                if($global) { $text =~ s/$pattern/$replace/g
}
                else        { $text =~ s/$pattern/$replace/ 
}
                last;
            }

            # extract the bit before the match, the match
itself, the 
            # bit after and the positions of all subgroups
            $result .= substr($text, 0, $-[0]) if $-[0];
            $matched = substr($text, $-[0], $+[0] - $-[0]);
            $after   = substr($text, $+[0]);
            start   = -;
            end     = +;

            # do the s/// leaving the placeholders
(literally '$1' etc) in place
            $matched =~ s/$pattern/$replace/;

            # then replace the $1, $2, etc., placeholders in
reverse order 
            # to ensure we do $10 before $1
            for (my $i = $#start; $i; $i--) {
                $backref = substr( $text, $start[$i],
$end[$i] - $start[$i] );
                $matched =~ s/$$i/$backref/g;
            }

            # add the modified $matched output to the result
and loop if global
            $result .= $matched;
            $text    = $after;
            last unless $global && length $text;
        }
        return $result . $text;
    },

Many thanks.
A


_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 18:48:17
>         while ($text =~ m/$pattern/) {

Most likely wanted a g on that - or else you could end up
with an infinite 
loop depending upon your swap.

Also - just wondering - how long has - been supported - and does
it incur any 
penalties.  (Guess I could look this up myself).

Paul Seamons

_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 19:00:44
mailseamons.com wrote:
> Most likely wanted a g on that - or else you could end
up with an infinite 
> loop depending upon your swap.

Nope, I don't think so.  $text is being munged each time
around until it
shrinks to zero length and drops out the bottom of the while
loop.

Or have I missed something?

Cheers
A


_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 19:18:12
> Nope, I don't think so.  $text is being munged each
time around until it
> shrinks to zero length and drops out the bottom of the
while loop.
>
> Or have I missed something?

You are absolutely right - forgot to view that it is
actually munging the 
$text string.

Here is another code snippet that you might consider.  It
has the unfortunate 
item of needing to be copied twice for global.  But it does
handle $10.  It 
also handles $10 more similarly to Perl.  The problem is
that if I have $10 
in my string - but I have less than 10 patterns, then I will
end up with 
"mymatch0" rather than "" should $1
contain mymatch.  Also, it should be 
decided about what to do with $1 variables in the
replacement portion should 
there not be any matches.  The version you had given leaves
them there - 
though I don't think that is desired behavior.

Any way - here is a quick code snippet - that I think I may
use in some of my 
applications. 

sub vmethod_replace {
    my ($str, $pat, $replace, $global) = _;
    $str     = '' if ! defined $str;
    $pat     = '' if ! defined $pat;
    $replace = '' if ! defined $replace;
    $global  = 1  if ! defined $global;
    if ($global) {
        $str =~ s{$pat}{
            my start = -;
            my end   = +;
            my $copy  = $replace;
            $copy =~ s{ (?<!\) $ (d+) }{
                ($1 > $#start || $1 == 0) ? '' :
substr($str, $start[$1],
$end[$1] - $start[$1]);
            }exg;
            $copy;
        }eg;
    } else {
        $str =~ s{$pat}{
            my start = -;
            my end   = +;
            my $copy  = $replace;
            $copy =~ s{ (?<!\) $ (d+) }{
                ($1 > $#start || $1 == 0) ? '' :
substr($str, $start[$1], 
$end[$1] - $start[$1]);
            }exg;
            $copy;
        }e;
    }
    return $str;
}

Paul

_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Vmethods to add to 2.15
user name
2006-02-01 19:48:30
Andy Wardley wrote:
>> -----------------------------------------------
>>  sub {
>>    my ($scalar, $offset, $length) = _;
>>    return substr($scalar, $offset, $length);
>>  };
>> -----------------------------------------------
> 
> Unfortunately it's not as simple as that.  Perl
distinguishes between
> substr($scalar, $offset, undef) and substr($scalar,
$offset) so we have
> to hard-code the alternatives (unless anyone knows
another way?).  I've
> also added code the handle the 4th argument to provide
a replacement string,
> thanks to http:/
/rt.cpan.org/Ticket/Display.html?id=2619


============================================================
Hmm, what a pain. I'd be interested in knowing another way
myself. Usually I'd just do something like ($length ?
$length : ()) which works with normal subs, but not this.
============================================================


>     'substr' => sub {
>         my ($text, $offset, $length, $replacement) =
_;
>         $offset ||= 0;
> 
>         if(defined $length) {
>             if (defined $replacement) {
>                 my $removed = substr( $text, $offset,
$length );
>                 substr( $text, $offset, $length ) =
$replacement;
>                 return $removed;
>             }
>             else {
>                 return substr( $text, $offset, $length
);
>             }
>         }
>         else {
>             return substr( $text, $offset );
>         }
>     },
> 
> I think that covers all bases.  Damn shame when
something that should be
> so simple turns out so messy.  Ho hum, that's Perl
sometimes.

============================================================
Hmm, it doesn't seem like the replacement would actually
even doing anything, since it's operating on a copy of the
text. Unless you pass by reference, I don't think you're
going to be able to do this. If you do manage to get
something worked out, then I'd suggest also adding
replacement to the second else, for the case that you want
to replace all ending text after a certain point with
something else.
============================================================



>> In addition, I just had another thought of
something that seems to be 
>> missing and a cause of problems on the mailing list
occasionally. That 
>> would be missing back references in replace regular
expressions. 
> 
> Yeah, this is a nuisance, but I haven't yet seen an
implementation that 
> solves all the problems securely and robustly.  I'm not
keen on adding a
> new syntax for back-references.  What was your
reasoning for that over
> the regular $1 or 1 syntax?


============================================================
I just didn't want people thinking they were using Perl
variables, and that things would parse the same. The $1
syntax would be fine, though.
============================================================


> This is the most promising candidate from Craig
Barratt. 
> 
> sub replace { 
>     my ($str, $search, $replace) = _;
>     $replace = '' unless defined $replace;
>     return $str unless defined $str and defined
$search;
> 
>     $str =~ s{ $search } {
> 	my $r = $replace;
> 	my d = (0, $1, $2, $3, $4, $5, $6, $7, $8, $9);
> 	$r =~ s/$(d+)/$d[$1]/eg;
> 	$r;
>     }egx;
> 
>     return $str;
> }
> 
> It's limited to the first 9 captures only (no $10, $11,
etc) and it 
> doesn't handle escaped '$' in the replacement string. 
But apart from
> that, it seems to do the job without opening up any
security holes.

============================================================
That's basically exactly what I did, but I allowed global
search and replace as well. I guess Craig should've patented
this one-click scheme! ;) j/k

I'm going to check out the replace based on what Nik did. I
had noticed him posting to this list before about that, and
actually started looking into what he was doing along with
some other ways to do it, but got lost in other stuff I
needed to get done.

Also I am also wondering about what Paul Seamons said:
> Also - just wondering - how long has - been
supported - and does it incur any 
> penalties.  (Guess I could look this up myself).

I had done a quick search earlier today, but didn't see much
related to what version - started in. (Makes me
think it was probably an earlier version.) I'll try to look
into this further if no one else responds.

-- Josh


_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
Correct replace vmethod
user name
2006-02-02 02:34:11
> So with your blessing, I'll include the following as
the new
replace()
> virtual method.

I have created a simple test, which compares results of
native
perl s/// (via eval) and proposed template 'replace'
vmethod. It
seems that we still have a lot of problems with
"last" version of
replace. The replacement is performed on source string, so
it may
easily break it:

$s = 'foo $1 bar';
$s =~ s/(foo)(.*)(bar)/$3$2$1/;
# we'll get 'bar $1 foo'
[% s.replace('(foo)(.*)(bar)','$3$2$1') %]
# we'll get 'bar foo foo'

There are other issues with backslashes - perl handles 1-9
as
$1-$9, but 10 and so on are handled as octal character code
(as
in double-quoed strings).

I've placed my test file (vmeth_replace.t) here:
http://martynoff.info/tt2/
 - please take a look at it. Maybe it
would be useful to include this test into distribution.


-- 
Sergey Martynoff


_______________________________________________
templates mailing list
templatestemplate-toolkit.org
http://lists.template-toolkit.org/mailman/listinfo/t
emplates
[1-10] [11-19]

about | contact  Other archives ( Real Estate discussion Medical topics )