Suggest we cache the SDO model somehow

______________
INTRODUCTION

I have been having some recent correspondence with Ben
Barringer on
the performance of SCA. Some of us have been talking about
this
amongst ourselves,and in correspondence with Ben. I wanted
to move the
conversation onto here so that anyone can share and we
capture the
history.

_____________________
WHAT'S THE PROBLEM

The nub of it is that we spend a lot of time processing XML
schema
files in order to build an SDO model. Sometimes it is schema
that SCA
wants for itself - the schema for SOAP or the schema for
WSDL, for
example -  but just it is just as likely to be schema for
the
application's complex types, and these can be enormous: the
schema for
eBaY for example is enormous.

Ben is not the only one who has pointed out that we are
making
ourselves unusable for some applications with our
performance:
Adam Trachtenberg and Rob Richards have both commented
adversely on
the performance:
http://www.trachtenberg.com/blog/2006/10/12/php-soap
-vs-sdo/ ("Where
SDO really falls down for me is performance. ")

http://groups.google.co.uk/group/phpsoa/browse_thread/thread
/c827044441e74456/80b655115a3bca23?#80b655115a3bca23
(BTW this thread has some suggestions in it for things that
we might
do to help the performance)

It's not just one problem. We know that SCA for PHP
sometimes loads
the same schema more than once in a single request. We know
that SDO
runs any schema it does load through a SAX parser twice when
ideally
it would do it once. We know there are places in the parsing
that we
can get some improvement. I don't expect any truly dramatic
improvements if we just chip away at those though.

____________________
DISCUSSION TO DATE

Given how infrequently a given WSDL or schema file changes,
it makes
no sense to pound away on it building the SDO model from it
on every
request. We ought to cache the result of doing that: caching
either
the SDO model or the data factory that contains that model.

There are two approaches we could take:
   1. We could try to keep the interface unchanged, so all
PHP code
continues to use just SDO_DAS_XML::create() and addTypes()
...
   2. We could put in some explicit caching that is visible
at the PHP
level and is controlled by the SCA for PHP code or even the
application code somehow

There are, independently, a couple of possibilities for
where and what
we cache. Two options seem to be:
   A. we could serialise the SDO model out to a file and
read it back
in when needed ...
   B. we could hold on to the data factory within memory,
within the
sdo_php extension.

We examined option A, write the XML DAS to a file. What we
found is
that there is logic in the XML DAS to cache the model to a
file
already, but it caches as schema, so reading it back in just
gets us
back into loading schema again. So, we would need to come up
with a
format - binary or human-readable - that is quicker to
re-read. We
imagine by the way that anything cached in this way does not
have to
last very long. We would not want to get into the situation
of trying
to have file formats that were compatible across different
releases of
SDO, or between different platforms, or anything fancy.

So, we have concluded that the simplest thing to do is
probably to
cache in memory, option B.

Now look at the options 1. vs 2. i.e the interface. The
ideal is
probably to keep the interface unchanged, but in the
meantime we might
want to do something quicker to implement as a stop-gap,
even if it
puts a bit of responsibility into the SCA code.

The thing that worries me about option 1. comes about
because we have
addTypes(). If you do create(), followed by a string of
addTypes(), at
what point do you consider the data factory/model finished?
And then
they come back issuing the same string of create() and
addTypes()
(hence wanting the exact same model), how do you spot that
and use the
cached one? It seems to me that that needs a solution.
Perhaps allow
create() to take an array of types, and make that array the
the key to
the cached DAS?

You also need to consider what to do to catch when the files
changed
of course. Would you inspect the file modification times to
check they
had not changed? Would you want to do some quick hash of the
contents
as a backup check?

I now want to finish this posting and leave it up to others
to
comment. I intend to close with an extract form Ben's most
recent
note. We had got to the point where I was suggesting the
combination
of cache in memory, under PHP control, and I had suggested
explicit
caching calls like
$xmldas->saveModelunderKey
and
$xmldas->reloadModelFromKey

Ben's reply:

I have a couple of suggestions.  If you have a
saveModuleUnderKey
function it seems like you would need a clearModuleUnderKey
function
as well.  Not sure if it would be good or not but to reduce
the number
of function calls we could add another parameter to Create
such as
$cache='true.  Then if it wasn't cached it would cache it
with the
file name and if it was cached it would reload it from
cache.  In most
real world scenarios that I can think of most everyone will
want to
use the caching, so making it the default might be a good
idea.

So the code could be

$xmldas = SDO_DAS_XML::create('cmssys.xsd', true) ;

With this approach we will still need a
SDO_DAS_XML::ClearCache()
function.

I realize this gets more complicated because if we updated
our xsd
file we would need to clear the cache so that leaves me
checking the
modified date on the xsd file every page load in dev and QA.
 Not
trying to ask for too much here but it would be great if the
extension
could check for date modified on the xsd file and reload
automatically
if it has been updated.  So what I am proposing is

1. Add a new cache parameter to Create() that defaults to
true.
2. Use the file name as the key for caching.  This would
also avoid
accidentally caching the same file with different keys.
3. Check the xsd file modified time and refresh the cache if
changed.
This could be controlled via PHP.ini setting so it could be
turned off
for peak performance.  In our Dev/QA region we would have
it
automatically refreshed in production we would turn this off
for
better performance.

__________
EPILOGUE

I hope I have captured enough of the conversation to date
that we can
continue from here. Any comments, anyone?


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

On 27 Feb, 15:54, "Matthew Peters"
<matthew.f.pet...googlemail.com>
wrote:
> ______________
> INTRODUCTION
>
> I have been having some recent correspondence with Ben
Barringer on
> the performance of SCA. Some of us have been talking
about this
> amongst ourselves,and in correspondence with Ben. I
wanted to move the
> conversation onto here so that anyone can share and we
capture the
> history.
>
> _____________________
> WHAT'S THE PROBLEM
>
> The nub of it is that we spend a lot of time processing
XML schema
> files in order to build an SDO model. Sometimes it is
schema that SCA
> wants for itself - the schema for SOAP or the schema
for WSDL, for
> example -  but just it is just as likely to be schema
for the
> application's complex types, and these can be enormous:
the schema for
> eBaY for example is enormous.
>
> Ben is not the only one who has pointed out that we are
making
> ourselves unusable for some applications with our
performance:
> Adam Trachtenberg and Rob Richards have both commented
adversely on
> the performance:http://www.trachtenberg.com/blog/2006/10/12/php-soap
-vs-sdo/("Where
> SDO really falls down for me is performance. ")http://groups.google.co.uk/group/phpsoa/
browse_thread/thread/c8270444...
> (BTW this thread has some suggestions in it for things
that we might
> do to help the performance)
>
> It's not just one problem. We know that SCA for PHP
sometimes loads
> the same schema more than once in a single request. We
know that SDO
> runs any schema it does load through a SAX parser twice
when ideally
> it would do it once. We know there are places in the
parsing that we
> can get some improvement. I don't expect any truly
dramatic
> improvements if we just chip away at those though.
>
> ____________________
> DISCUSSION TO DATE
>
> Given how infrequently a given WSDL or schema file
changes, it makes
> no sense to pound away on it building the SDO model
from it on every
> request. We ought to cache the result of doing that:
caching either
> the SDO model or the data factory that contains that
model.
>
> There are two approaches we could take:
>    1. We could try to keep the interface unchanged, so
all PHP code
> continues to use just SDO_DAS_XML::create() and
addTypes() ...
>    2. We could put in some explicit caching that is
visible at the PHP
> level and is controlled by the SCA for PHP code or even
the
> application code somehow
>
> There are, independently, a couple of possibilities for
where and what
> we cache. Two options seem to be:
>    A. we could serialise the SDO model out to a file
and read it back
> in when needed ...
>    B. we could hold on to the data factory within
memory, within the
> sdo_php extension.
>
> We examined option A, write the XML DAS to a file. What
we found is
> that there is logic in the XML DAS to cache the model
to a file
> already, but it caches as schema, so reading it back in
just gets us
> back into loading schema again. So, we would need to
come up with a
> format - binary or human-readable - that is quicker to
re-read. We
> imagine by the way that anything cached in this way
does not have to
> last very long. We would not want to get into the
situation of trying
> to have file formats that were compatible across
different releases of
> SDO, or between different platforms, or anything
fancy.
>
> So, we have concluded that the simplest thing to do is
probably to
> cache in memory, option B.
>
> Now look at the options 1. vs 2. i.e the interface. The
ideal is
> probably to keep the interface unchanged, but in the
meantime we might
> want to do something quicker to implement as a
stop-gap, even if it
> puts a bit of responsibility into the SCA code.
>
> The thing that worries me about option 1. comes about
because we have
> addTypes(). If you do create(), followed by a string of
addTypes(), at
> what point do you consider the data factory/model
finished? And then
> they come back issuing the same string of create() and
addTypes()
> (hence wanting the exact same model), how do you spot
that and use the
> cached one? It seems to me that that needs a solution.
Perhaps allow
> create() to take an array of types, and make that array
the the key to
> the cached DAS?
>
> You also need to consider what to do to catch when the
files changed
> of course. Would you inspect the file modification
times to check they
> had not changed? Would you want to do some quick hash
of the contents
> as a backup check?
>
> I now want to finish this posting and leave it up to
others to
> comment. I intend to close with an extract form Ben's
most recent
> note. We had got to the point where I was suggesting
the combination
> of cache in memory, under PHP control, and I had
suggested explicit
> caching calls like
> $xmldas->saveModelunderKey
> and
> $xmldas->reloadModelFromKey
>
> Ben's reply:
>
> I have a couple of suggestions.  If you have a
saveModuleUnderKey
> function it seems like you would need a
clearModuleUnderKey function
> as well.  Not sure if it would be good or not but to
reduce the number
> of function calls we could add another parameter to
Create such as
> $cache='true.  Then if it wasn't cached it would cache
it with the
> file name and if it was cached it would reload it from
cache.  In most
> real world scenarios that I can think of most everyone
will want to
> use the caching, so making it the default might be a
good idea.
>
> So the code could be
>
> $xmldas = SDO_DAS_XML::create('cmssys.xsd', true) ;
>
> With this approach we will still need a
SDO_DAS_XML::ClearCache()
> function.
>
> I realize this gets more complicated because if we
updated our xsd
> file we would need to clear the cache so that leaves me
checking the
> modified date on the xsd file every page load in dev
and QA.  Not
> trying to ask for too much here but it would be great
if the extension
> could check for date modified on the xsd file and
reload automatically
> if it has been updated.  So what I am proposing is
>
> 1. Add a new cache parameter to Create() that defaults
to true.
> 2. Use the file name as the key for caching.  This
would also avoid
> accidentally caching the same file with different
keys.
> 3. Check the xsd file modified time and refresh the
cache if changed.
> This could be controlled via PHP.ini setting so it
could be turned off
> for peak performance.  In our Dev/QA region we would
have it
> automatically refreshed in production we would turn
this off for
> better performance.
>
> __________
> EPILOGUE
>
> I hope I have captured enough of the conversation to
date that we can
> continue from here. Any comments, anyone?

Hi Matthew, I aggree that the in memory cache is likely to
lead to the
desired result more quickly than going down the route of
defining a
new serialization format.

The API though is a little more tricky. You have raised a
couple of
potential issues.

- The type model may not be loaded in one go and hence the
name of the
originally loaded file may not uniquely identify the
resulting type
model.
   - Could optionally allow a key to be specified on create
and
default to the file name if a key is not provided.
- It may be the case that the initial schema file is
retrieved
remotely, via a URL, or that the local file includes remote
files.
   - This is where caching can give us the bigest payback.
   - This makes automatic change testing, i.e. based on
stating a
file, difficult because while the file itself may not have
changed
something that it includes may well have done.
   - typically live XML versioning is done via adopting
namespace
naming conventions. Uploading  changed xsd files
automatically to a
production system without due care, i.e. associated code
changes, will
likely lead to problems. We have this problem anyhow when we
restart
the server so maybe you would expect all schema files to be
copied to
the local file system but I don't know if you can guarantee
this.
   - On this basis I would avoid automatic cache refreshes
at the
moment. At least if you are going to provide it to make the
development environment more flexible make it switchable as
Ben
suggests. We could alternatively provide a cache clear
script that
does the job for us in a development environment without
taking the
service down.We would need the CacheClear interface Ben
suggests to
make this work.

Regards

Simon


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

On 27 Feb, 10:54, "Matthew Peters"
<matthew.f.pet...googlemail.com>
wrote:

> if it has been updated.  So what I am proposing is
>
> 1. Add a new cache parameter to Create() that defaults
to true.
> 2. Use the file name as the key for caching.  This
would also avoid
> accidentally caching the same file with different
keys.
> 3. Check the xsd file modified time and refresh the
cache if changed.
> This could be controlled via PHP.ini setting so it
could be turned off
> for peak performance.  In our Dev/QA region we would
have it
> automatically refreshed in production we would turn
this off for
> better performance.

This seems like a decent proposal imo. I do like the cache
in memory,
but eventually it would probably also need the special
binary format
caching mechanism.
This is pretty much the way ext/soap handles things. The
reason cache
in memory is not the default in ext/soap, IIRC, is for
shared host
installs. The number of schemas in memory could be quite big
depending
upon how many clients are using SDO/SCA on the machine.

As far as the cache refreshing goes, I completely agree with
that
statement. With a flush method, it should also be possible
then to
force a reload programatically in any event without having
to bounce
the web server, correct?

For the rest of the things mentioned, I am still digesting
some of the
finer points so will hold off commenting on them for now.

Rob



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

Matthew,

I've made a change to the signature of
SDO_DAS_XML::create(). Instead
of a single path to a schema file, this now takes an array
of paths to
schema files. This is checked into the BUZZARD branch.

The idea here is that people can in most cases avoid the
problem of
addTypes(), by instead making a single call to create() with
all the
required schema files at one go.

Your prototype caching code is still in place, but currently
only gets
involved when create() is called with the old-style
signature. I've
checked that it still works in that case. Did you have a
design in
mind for how to update the table key to represent a set of
schema
files?

I know this isn't a complete solution. Adding the schemas
all at once
will be useful, but I think there are still cases where
addTypes()
will be needed later. When that happens, I would like to be
able to
clone the DataFactory, that is, to make a copy without
having to re-
read the schema files, so that the additional types can be
added and
the original factory kept unchanged. But to do this would
require a
clone function that doesn't currently exist in the
underlying Tuscany
library.

On Feb 27, 3:54 pm, "Matthew Peters"
<matthew.f.pet...googlemail.com>
wrote:

>
> Given how infrequently a given WSDL or schema file
changes, it makes
> no sense to pound away on it building the SDO model
from it on every
> request. We ought to cache the result of doing that:
caching either
> the SDO model or the data factory that contains that
model.
>
> There are two approaches we could take:
>    1. We could try to keep the interface unchanged, so
all PHP code
> continues to use just SDO_DAS_XML::create() and
addTypes() ...
>    2. We could put in some explicit caching that is
visible at the PHP
> level and is controlled by the SCA for PHP code or even
the
> application code somehow
>
> There are, independently, a couple of possibilities for
where and what
> we cache. Two options seem to be:
>    A. we could serialise the SDO model out to a file
and read it back
> in when needed ...
>    B. we could hold on to the data factory within
memory, within the
> sdo_php extension.
>
> We examined option A, write the XML DAS to a file. What
we found is
> that there is logic in the XML DAS to cache the model
to a file
> already, but it caches as schema, so reading it back in
just gets us
> back into loading schema again. So, we would need to
come up with a
> format - binary or human-readable - that is quicker to
re-read. We
> imagine by the way that anything cached in this way
does not have to
> last very long. We would not want to get into the
situation of trying
> to have file formats that were compatible across
different releases of
> SDO, or between different platforms, or anything
fancy.
>
> So, we have concluded that the simplest thing to do is
probably to
> cache in memory, option B.
>
> Now look at the options 1. vs 2. i.e the interface. The
ideal is
> probably to keep the interface unchanged, but in the
meantime we might
> want to do something quicker to implement as a
stop-gap, even if it
> puts a bit of responsibility into the SCA code.
>
> The thing that worries me about option 1. comes about
because we have
> addTypes(). If you do create(), followed by a string of
addTypes(), at
> what point do you consider the data factory/model
finished? And then
> they come back issuing the same string of create() and
addTypes()
> (hence wanting the exact same model), how do you spot
that and use the
> cached one? It seems to me that that needs a solution.
Perhaps allow
> create() to take an array of types, and make that array
the the key to
> the cached DAS?
>


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

Caroline, thank you.

So, we are inching our way to a working solution. The steps
I can see
in front of me are these:

1. We were getting an SDO_UnsupportedOperationException with
the text
"Adding Properties after type completed" on a
fairly simple test case
that I thought ought to have worked. We need to get to the
bottom of
this. It is some interaction between the caching and
addTypes. It
might be that we can just sidestep it but I would like to
understand
it a little better first.

2. Next I will reorganise the places in the SCA PHP code
where we used
to do create() followed by addTypes(file) to just do the
single call
to create() with an array. I will need to generate some kind
of hash
of the set of filenames to use as a key to keep hold of the
cached
data factory. This, with the caching code that I have
already put in
should be enough so that a second and subsequent call to a
component
should find all the types that that component uses already
loaded. I
hope this will be enough to get Ben Barringer going.

3. Then I will have to tackle invalidating the cache when
the files
change. I suppose I will check the timestamps on the files.
I think
this is going to be harder than you might think. The reason
is that
the pathnames that I get passed in to the C code, if
relative, are
relative to the PHP working directory and not to the C
working
directory (I blogged about this on the developerWorks blog a
while
ago). This means that I cannot directly use within the C
code any
relative pathnames I have been given from PHP. I will have
to find a
way to call the PHP routines to check the file timestamps.

Also, I am never sure whether it is enough to just check the
file
timestamps. I wonder if I ought to open the file and do a
hash of the
contents, just to make sure that the contents have not
changed.

I am not sure how long this is going to take. I know Ben
wanted an
outlook for when we might complete. I am going to try to
complete the
first two steps i.e. running without invalidation by the end
of next
week.

Matthew

On Mar 21, 3:09 pm, "cem" <Caroline.Mayn...gmail.com> wrote:
> Matthew,
>
> I've made a change to the signature of
SDO_DAS_XML::create(). Instead
> of a single path to a schema file, this now takes an
array of paths to
> schema files. This is checked into the BUZZARD branch.
>
> The idea here is that people can in most cases avoid
the problem of
> addTypes(), by instead making a single call to create()
with all the
> required schema files at one go.
>
> Your prototype caching code is still in place, but
currently only gets
> involved when create() is called with the old-style
signature. I've
> checked that it still works in that case. Did you have
a design in
> mind for how to update the table key to represent a set
of schema
> files?
>
> I know this isn't a complete solution. Adding the
schemas all at once
> will be useful, but I think there are still cases where
addTypes()
> will be needed later. When that happens, I would like
to be able to
> clone the DataFactory, that is, to make a copy without
having to re-
> read the schema files, so that the additional types can
be added and
> the original factory kept unchanged. But to do this
would require a
> clone function that doesn't currently exist in the
underlying Tuscany
> library.
>
> On Feb 27, 3:54 pm, "Matthew Peters"
<matthew.f.pet...googlemail.com>
> wrote:
>
> > Given how infrequently a given WSDL or schema file
changes, it makes
> > no sense to pound away on it building the SDO
model from it on every
> > request. We ought to cache the result of doing
that: caching either
> > the SDO model or the data factory that contains
that model.
>
> > There are two approaches we could take:
> >    1. We could try to keep the interface
unchanged, so all PHP code
> > continues to use just SDO_DAS_XML::create() and
addTypes() ...
> >    2. We could put in some explicit caching that
is visible at the PHP
> > level and is controlled by the SCA for PHP code or
even the
> > application code somehow
>
> > There are, independently, a couple of
possibilities for where and what
> > we cache. Two options seem to be:
> >    A. we could serialise the SDO model out to a
file and read it back
> > in when needed ...
> >    B. we could hold on to the data factory within
memory, within the
> > sdo_php extension.
>
> > We examined option A, write the XML DAS to a file.
What we found is
> > that there is logic in the XML DAS to cache the
model to a file
> > already, but it caches as schema, so reading it
back in just gets us
> > back into loading schema again. So, we would need
to come up with a
> > format - binary or human-readable - that is
quicker to re-read. We
> > imagine by the way that anything cached in this
way does not have to
> > last very long. We would not want to get into the
situation of trying
> > to have file formats that were compatible across
different releases of
> > SDO, or between different platforms, or anything
fancy.
>
> > So, we have concluded that the simplest thing to
do is probably to
> > cache in memory, option B.
>
> > Now look at the options 1. vs 2. i.e the
interface. The ideal is
> > probably to keep the interface unchanged, but in
the meantime we might
> > want to do something quicker to implement as a
stop-gap, even if it
> > puts a bit of responsibility into the SCA code.
>
> > The thing that worries me about option 1. comes
about because we have
> > addTypes(). If you do create(), followed by a
string of addTypes(), at
> > what point do you consider the data factory/model
finished? And then
> > they come back issuing the same string of create()
and addTypes()
> > (hence wanting the exact same model), how do you
spot that and use the
> > cached one? It seems to me that that needs a
solution. Perhaps allow
> > create() to take an array of types, and make that
array the the key to
> > the cached DAS?


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

Sadly, I seem to have lost the post that I spent half an
hour
composing earlier today...I thought Google said it had been
successfully sent but it hasn't appeared yet...so, I will be
more
succinct this time. I looked through the posts so far from
Ben, Rob
and Simon, thought about them, and weighed up the pros and
cons of the
two approaches (1) cache and do the invalidation by
timestamp and (2)
require user involvement to tell us when to cache and when
to
invalidate.

The essential difficulty about caching and invalidating by
timestamp
is that xsds may use include to point to other files. I
cannot tell if
these other files have changed. Hence invalidation by
timestamp is
dodgy. Even if I open the file and look at the contents that
is not
enough - I would need to parse and follow the links to
included files.
That's horrible.

Add to that the bother caused by the fact that relative
pathnames the
extension gets are relative to the PHP cwd and not to the C
cwd, so
that C code cannot necessarily just stat what it is given,
and I would
have to work out how to call the PHP routines to check
timestamp, and
invalidating by timestamp is unattractive.

I prefer to put it somewhat under user control. Here is my
proposed
use case and interface:
// first time through...
$xmldas = SODO_DAS_XML::create(<array of
filenames>,<key>);
// first time through it creates and stores under the given
key
// use the das
// ...then in a later request, the call looks the same but
the
behaviour is different
$xmldas = SODO_DAS_XML::create(<array of
filenames>,<key>);
// ... the same call this time the das is retrieved based on
the key

I much prefer to generate the key in PHP space by the way.
If I have
to do it in C then I know I will write a lot of buggy code
to xor the
filenames together which creates a code, test and
maintenance burden.
If I do it in PHP space (for our case it will be in the SCA
code) I
can do it in a couple of lines of PHP.

The interface is a little odd, but it is to do with the fact
that in
our PHP code there is no equivalent to a first-time-through
flag, so
the call has to look the same each time. The code inside the
extension
will be something like:

if (<key> is not in table)
  generate new data factory from file names
  store in table under <key>
else
  retrieve stored data factory using <key>
put data factory into das and return

so, if they want the use on the second time through could
call
create(null, <key>). We will not do this in the SCA
code.

We will also need to introduce an explicit invalidate:
SDO_DAS_XML::invalidateSavedDataFactoryTableEntry(<key>
;); perhaps

OK, open for comments.

On Mar 21, 5:20 pm, "Matthew Peters"
<matthew.f.pet...googlemail.com>
wrote:
> Caroline, thank you.
>
> So, we are inching our way to a working solution. The
steps I can see
> in front of me are these:
>
> 1. We were getting an SDO_UnsupportedOperationException
with the text
> "Adding Properties after type completed" on a
fairly simple test case
> that I thought ought to have worked. We need to get to
the bottom of
> this. It is some interaction between the caching and
addTypes. It
> might be that we can just sidestep it but I would like
to understand
> it a little better first.
>
> 2. Next I will reorganise the places in the SCA PHP
code where we used
> to do create() followed by addTypes(file) to just do
the single call
> to create() with an array. I will need to generate some
kind of hash
> of the set of filenames to use as a key to keep hold of
the cached
> data factory. This, with the caching code that I have
already put in
> should be enough so that a second and subsequent call
to a component
> should find all the types that that component uses
already loaded. I
> hope this will be enough to get Ben Barringer going.
>
> 3. Then I will have to tackle invalidating the cache
when the files
> change. I suppose I will check the timestamps on the
files. I think
> this is going to be harder than you might think. The
reason is that
> the pathnames that I get passed in to the C code, if
relative, are
> relative to the PHP working directory and not to the C
working
> directory (I blogged about this on the developerWorks
blog a while
> ago). This means that I cannot directly use within the
C code any
> relative pathnames I have been given from PHP. I will
have to find a
> way to call the PHP routines to check the file
timestamps.
>
> Also, I am never sure whether it is enough to just
check the file
> timestamps. I wonder if I ought to open the file and do
a hash of the
> contents, just to make sure that the contents have not
changed.
>
> I am not sure how long this is going to take. I know
Ben wanted an
> outlook for when we might complete. I am going to try
to complete the
> first two steps i.e. running without invalidation by
the end of next
> week.
>
> Matthew
>
> On Mar 21, 3:09 pm, "cem"
<Caroline.Mayn...gmail.com> wrote:
>
> > Matthew,
>
> > I've made a change to the signature of
SDO_DAS_XML::create(). Instead
> > of a single path to a schema file, this now takes
an array of paths to
> > schema files. This is checked into the BUZZARD
branch.
>
> > The idea here is that people can in most cases
avoid the problem of
> > addTypes(), by instead making a single call to
create() with all the
> > required schema files at one go.
>
> > Your prototype caching code is still in place, but
currently only gets
> > involved when create() is called with the
old-style signature. I've
> > checked that it still works in that case. Did you
have a design in
> > mind for how to update the table key to represent
a set of schema
> > files?
>
> > I know this isn't a complete solution. Adding the
schemas all at once
> > will be useful, but I think there are still cases
where addTypes()
> > will be needed later. When that happens, I would
like to be able to
> > clone the DataFactory, that is, to make a copy
without having to re-
> > read the schema files, so that the additional
types can be added and
> > the original factory kept unchanged. But to do
this would require a
> > clone function that doesn't currently exist in the
underlying Tuscany
> > library.
>
> > On Feb 27, 3:54 pm, "Matthew Peters"
<matthew.f.pet...googlemail.com>
> > wrote:
>
> > > Given how infrequently a given WSDL or schema
file changes, it makes
> > > no sense to pound away on it building the SDO
model from it on every
> > > request. We ought to cache the result of
doing that: caching either
> > > the SDO model or the data factory that
contains that model.
>
> > > There are two approaches we could take:
> > >    1. We could try to keep the interface
unchanged, so all PHP code
> > > continues to use just SDO_DAS_XML::create()
and addTypes() ...
> > >    2. We could put in some explicit caching
that is visible at the PHP
> > > level and is controlled by the SCA for PHP
code or even the
> > > application code somehow
>
> > > There are, independently, a couple of
possibilities for where and what
> > > we cache. Two options seem to be:
> > >    A. we could serialise the SDO model out to
a file and read it back
> > > in when needed ...
> > >    B. we could hold on to the data factory
within memory, within the
> > > sdo_php extension.
>
> > > We examined option A, write the XML DAS to a
file. What we found is
> > > that there is logic in the XML DAS to cache
the model to a file
> > > already, but it caches as schema, so reading
it back in just gets us
> > > back into loading schema again. So, we would
need to come up with a
> > > format - binary or human-readable - that is
quicker to re-read. We
> > > imagine by the way that anything cached in
this way does not have to
> > > last very long. We would not want to get into
the situation of trying
> > > to have file formats that were compatible
across different releases of
> > > SDO, or between different platforms, or
anything fancy.
>
> > > So, we have concluded that the simplest thing
to do is probably to
> > > cache in memory, option B.
>
> > > Now look at the options 1. vs 2. i.e the
interface. The ideal is
> > > probably to keep the interface unchanged, but
in the meantime we might
> > > want to do something quicker to implement as
a stop-gap, even if it
> > > puts a bit of responsibility into the SCA
code.
>
> > > The thing that worries me about option 1.
comes about because we have
> > > addTypes(). If you do create(), followed by a
string of addTypes(), at
> > > what point do you consider the data
factory/model finished? And then
> > > they come back issuing the same string of
create() and addTypes()
> > > (hence wanting the exact same model), how do
you spot that and use the
> > > cached one? It seems to me that that needs a
solution. Perhaps allow
> > > create() to take an array of types, and make
that array the the key to
> > > the cached DAS?


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

This discussion thread hasn't explicitly spelled out the
scope of the
cache, but I've been assuming that we're intending it to
endure from
module init to module shutdown. In other words, depending on
the SAPI,
the cache entries may be available to multiple independent
SCA
programs, sequentially and simultaneously.

Did I get this wrong? I can see how the interface you
describe would
work for a request-scoped or possibly thread-scoped cache.
But it
doesn't feel right to me to hand out the task of key
management to
uncoordinated applications. Imagine if everyone used the
same key
string that's in the example code snippet  Please put
me right.



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

The Tuscany folk are looking into this. There's a relevant
discussion 
going on over at 
http://thread.gmane.org/gmane.comp.apac
he.webservices.tuscany.devel/16058


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

Yes, exactly right. The scope is intended to be module init
to
shutdown, and across all callers. I don't really know how I
could do
otherwise - a per-request cache would be throwing away all
the
benefit.

Of course you have worried me now. If the keys are chosen
poorly - for
example every call chooses key "my_first_key" or
suchlike - then that
is a recipe for disaster. The key needs to correspond to the
set of
types that are loaded into that DAS: the keys must be chosen
so that
if the model is different, so is the key. I can't enforce
that, of
course, all I can do is use it that way throughout the SCA
code.

On Mar 23, 1:52 pm, "cem" <Caroline.Mayn...gmail.com> wrote:
> This discussion thread hasn't explicitly spelled out
the scope of the
> cache, but I've been assuming that we're intending it
to endure from
> module init to module shutdown. In other words,
depending on the SAPI,
> the cache entries may be available to multiple
independent SCA
> programs, sequentially and simultaneously.
>
> Did I get this wrong? I can see how the interface you
describe would
> work for a request-scoped or possibly thread-scoped
cache. But it
> doesn't feel right to me to hand out the task of key
management to
> uncoordinated applications. Imagine if everyone used
the same key
> string that's in the example code snippet  Please put
me right.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---

Nice to know. If and when that Tuscany change gets into our
codebase I
can try that. It looks marginally safer than what we I am
proposing to
do right now, which is to create the data factory once then
use it
indefinitely.

On Mar 28, 5:58 pm, cem <c...php.net> wrote:
> The Tuscany folk are looking into this. There's a
relevant discussion
> going on over athttp://thread.gmane.org/gmane.comp.apach
e.webservices.tuscany.devel/1...


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "phpsoa" group.
To post to this group, send email to phpsoagooglegroups.com
To unsubscribe from this group, send email to
phpsoa-unsubscribegooglegroups.com
For more options, visit this group at http://
groups.google.co.uk/group/phpsoa?hl=en
-~----------~----~----~----~------~----~------~--~---