Re: "memory size" Error

----- Original Message ----- 
> From: "Christopher Smith" <chrisjalakai.co.uk>
>
> Are you able to publish the page in question for
analysis?
> Or if not in public, send it to me in private so I can
take a look at
> where the memory usage is going.
>
> I did some tests on php4.4.7 & some very large DW
pages with lots of
> plugins installed, but only achieved memory usage in
the 10M-12M range.
>
> - Chris

The page is at http://music.organettes.com/instruments/celestina/home

I do use several plugins, both public and private, but this
particular page 
is pretty basic stuff. I also use a modified sidebar
template.

Todd Augsburger
toddrollerorgans.com
Roller Organs
www.rollerorgans.com

-- 
DokuWiki mailing list - more info at
http://wi
ki.splitbrain.org/wiki:mailinglist

On 20 Mar 2008, at 01:24, Todd Augsburger wrote:

>
> The page is at http://music.organettes.com/instruments/celestina/home

> I do use several plugins, both public and private, but
this  
> particular page
> is pretty basic stuff. I also use a modified sidebar
template.
>



Thanks.

For interest, that page is  97k, generates 1.2Mb of
instructions and  
requires 22MB in 18.5sec to process from raw wiki text to
rendered  
xhtml on my box.

I created a really big page, mixed content, no one piece of
content  
really large.

raw wiki file ... 750k
instructions cache ... 1.2M
memory required ... 26M
time required ... 95s

and I created a page that was pretty much all one large
table, 4 cells  
across, 3300 rows

raw wiki file ... 127k
instructions cache ... 3.2M
memory required ... 49.5M
time required ... 29s

My server has memory_limit set to 48M (that's just over 50
million  
hence the 49.5 above).  At 3500 rows it blows up and the
stack trace  
at the point shows ...

its still in the handler and its trying to finalize the
table
- at start of table finalisation:   27M
- at entering last part of finalisation: 49.5M

The table finalisation code uses two foreach on the initial
and final  
arrays of table instructions, by my calculations that should
double  
the memory requirements for finalisation.  However, I
swapped both to  
other iterators and although the page did process at 3500
rows, it  
still required 49.2M.

I'll do some more investigation later.

- Chris



-- 
DokuWiki mailing list - more info at
http://wi
ki.splitbrain.org/wiki:mailinglist

In addition to my earlier message, I've done a little more 

investigating.

- tried an iterator based on array_pop() on the basis that
as the  
input call stack is processed its memory requirements are
reduced in  
step with the increasing memory requirements of the output
call stack.

Virtually no change in memory use.  Significant increase in
processing  
time.

Tentative conclusion, PHP's shallow copy mechanism is
working in DW's  
favour, changes in processing of call rewriters seem to do
no more  
than achieve the same efficiencies already achieved by PHP
at a cost  
of greater processing time.

-----

Although this problem seems to apply to tables.  All of the
syntax  
modes which use call rewriters (e.g. lists, quotes,
footnotes) use a  
similar structure to process their call stacks so are
vulnerable to  
the same issue.  Its possible the problem doesn't seem to
occur with  
these other modes because pages aren't created with
extremely large  
instances of their syntax, ie. big tables are more common
and larger  
than big lists or big footnotes.  The instruction set
required to  
produce a table is more complex that the instruction set
required to  
produce the output of the other modes.  I think that is a
marginal  
issue, the main issue is that when the handler processes a
call  
rewriter's call stack it needs to double the memory
requirements of  
that stack.

It looks to me that significant improvements in memory usage
by the  
handler will be difficult to achieve without significant
changes to  
the way it operates.

For now, my best advice is:

- don't make really large wiki pages, especially ones with
really  
large tables.
- if you must have really large tables, try to break them
up.  
Processing several smaller tables requires less total memory
than  
processing a single large table.
- increase PHP's memory_limit

A couple of things to be looked at are:
- detailed look at where/how memory is being used in
parser/handler/ 
renderer
- reducing duplication of instruction strings, say by using
an index  
to a separate table containing the strings

Andi has opened an item for this in the bug tracker, http://bugs.splitbrain.org/index.php?do=details&
amp;task_id=1357

- Chris
-- 
DokuWiki mailing list - more info at
http://wi
ki.splitbrain.org/wiki:mailinglist