|
List Info
Thread: JDBC-cache sample
|
|
| JDBC-cache sample |
  United States |
2007-07-30 09:45:48 |
I have been trying to run the jdbc-cache example from the
samples
directory. The cache creation tool isn't working, and as far
as I can
ascertain it appears to be a config issue.
As far as my back track goes, I started working through
Penrose, from
the top down. I am using the xml files straight from the
samples. The
only changes have been to connections.xml to point to my
database. I
have loaded the tables as per the sql scripts in those
directories.
Now tracing down, looking into source and source sync. The
issue is,
that its not finding any source-sync info. The whole reason
that the
cache tool doesn't work, is it doesn't find anything to
sync, so it
just exits (Gets as far as create ( CacheUtil.java ln 88),
and finds
the default partition, but then tries to find sourceSyncs
(ln 96), and
nothing is returned).
I started poking around, and I find line Sources.java ln 38.
Here is
trying to pull a parameter called "sync" from the
sourceConfig, but
never finds it. I started poking back trying to figure out
where it
loads the config for sources. I have found the
sources-digester-
rules.xml in partition. I see that there is no source-sync
section
like is present in the sources.xml. So I'm confused as to
how its ever
supposed to pick up sync.
The only thing I have noticed, is, if I add another entry
to
sources.xml like this:
<source name="users">
<connection-name>MySQL</connection-name>
<field name="username"
primaryKey="true"/>
<field name="firstName"/>
<field name="lastName"/>
<field name="password"/>
<parameter>
<param-name>table</param-name>
<param-value>users</param-value>
</parameter>
</source>
BUT change param-name from table to sync, it then picks up a
sync, but
still fails as its obviously wanting some other params. So
I'm very
confused... I hope you can clear this up.
Thanks
P.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "safehaus" group.
To post to this group, send email to safehaus googlegroups.com
To unsubscribe from this group, send email to
safehaus-unsubscribe googlegroups.com
For more options, visit this group at http://
groups.google.com/group/safehaus?hl=en
-~----------~----~----~----~------~----~------~--~---
|
|
| Re: JDBC-cache sample |

|
2007-07-30 13:18:57 |
|
Paul,
Apparently the jdbc_cache example is outdated, sorry about that. But before proceeding further I need to tell you what this example does. It basicaly replicates the data from a master source, presumably from a remote or slow system, into a cache database which is local and faster. This would be useful if you don't want to burden the master source with traffic from Penrose. If your master source is already fast enough and can handle Penrose traffic you probably don't need this.
It is also useful if you want to join serveral sources that don't support join operation natively, for example joining 2 ldap directories. This feature will replicate the entire content of your master sources into the cache database and perform the join operation there. If you want to join tables from the same RDBMS database you don't need this. Unlike the previous versions, Penrose
1.2.x can perform the join operation on the database side.
If that is what you want to do, here's how to set up the example:
1. Create a MySQL database called "example". This is the master database.
2. Run the create.sql included in the example under the sql folder to create the tables and triggers. If you are using a different database system, you need to modify the script for your database system.
3. Create a MySQL database called "jdbc". This is the cache database.
4. Add the following parameter in the "users" source (keep the original "table" parameter).
<parameter> <param-name>sync</param-name> <param-value>users_cache</param-value> </parameter>
<parameter> <param-name>changelog</param-name> <param-value>users_changelog</param-value> </parameter>
These parameters tell Penrose to synchronize this source (i.e. users) with "users_cache" source using the change log information stored in "users_changelog" table. You can do the same thing with the other sources if you want. You can remove the <source-sync> entries from
sources.xml, they are no longer used.
5. Edit server.xml to install this partiiton. 6. Run this command to create the cache tables:
cache.bat -d create jdbc_cache
If it doesn't work with your database system, you can create the tables manually. See create-cache.sql.
7. To synchronize the cache, run:
cache.bat -d sync jdbc_cache
You can use your own scheduler (e.g. cron) to run this command periodically.
8. Try to update the data in the master source, the changes should be picked up when you run the sync command.
Let me know if you have any questions.
-- Endi S. Dewata
On 7/30/07, Nuzz <identitysolutions.co.nz" target="_blank">paul.edwards identitysolutions.co.nz
> wrote:
I have been trying to run the jdbc-cache example from the samples directory. The cache creation tool isn't working, and as far as I can
ascertain it appears to be a config issue.
As far as my back track goes, I started working through Penrose, from the top down. I am using the xml files straight from the samples. The only changes have been to
connections.xml to point to my database. I have loaded the tables as per the sql scripts in those directories.
Now tracing down, looking into source and source sync. The issue is, that its not finding any source-sync info. The whole reason that the
cache tool doesn't work, is it doesn't find anything to sync, so it just exits (Gets as far as create ( CacheUtil.java ln 88), and finds the default partition, but then tries to find sourceSyncs (ln 96), and
nothing is returned).
I started poking around, and I find line Sources.java ln 38. Here is trying to pull a parameter called "sync" from the sourceConfig, but never finds it. I started poking back trying to figure out where it
loads the config for sources. I have found the sources-digester- rules.xml in partition. I see that there is no source-sync section like is present in the sources.xml. So I'm confused as to how its ever supposed to pick up sync.
The only thing I have noticed, is, if I add another entry to sources.xml like this: <source name="users"> <connection-name>MySQL</connection-name> <field name="username" primaryKey="true"/>
<field name="firstName"/> <field name="lastName"/> <field name="password"/> <parameter> <param-name>table</param-name>
<param-value>users</param-value> </parameter> </source>
BUT change param-name from table to sync, it then picks up a sync, but still fails as its obviously wanting some other params. So I'm very
confused... I hope you can clear this up.
Thanks P.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "safehaus" group. To post to this group, send email to safehaus googlegroups.com To unsubscribe from this group, send email to safehaus-unsubscribe googlegroups.com For more options, visit this group at http://groups.google.com/group/safehaus?hl=en -~----------~----~----~----~------~----~------~--~---
|
| Re: JDBC-cache sample |
  United States |
2007-07-31 09:58:17 |
Endi,
That worked great. I have the system up.
However I'm trying to do a bit of load testing with slamd.
Now, when I
try to connect with slamd, penrose seems to not like the
default
filter that slamd is supplying (I cannot see a way at the mo
of
changing what slamd is supplying - without recompiling
slamd). As far
as I'm aware, what slamd is supplying is correct ldap
syntax. I
believe that the filter slamd is supplying on connect is:
(|
(objectclass=*)(objectclass=ldapsubentry)), which by my way
of
thinking seems a bit stupid, as it should just take the
objectclass=*
for all, but it appears that penrose is getting rid of the
objectclass=*, and then it all fails. Anyway heres what I'm
seeing:
[07/31/2007 10:46:29 DEBUG (PenroseSession.java:114)]
bind("uid=admin,ou=system", "secret")
[07/31/2007 10:46:29 WARN (Session.java:242)] Bind
"uid=admin,ou=system".
[07/31/2007 10:46:29 DEBUG (Session.java:248)]
------------------------------------------------------------
----------------------
[07/31/2007 10:46:29 DEBUG (Session.java:249)] BIND:
[07/31/2007 10:46:29 DEBUG (Session.java:250)] - Bind DN
:
uid=admin,ou=system
[07/31/2007 10:46:29 DEBUG (Session.java:251)] - Bind
Password :
secret
[07/31/2007 10:46:29 DEBUG (Session.java:252)]
[07/31/2007 10:46:29 DEBUG (Session.java:254)] Controls: []
[07/31/2007 10:46:29 DEBUG (EventManager.java:86)] Firing
BeforeBind
event.
[07/31/2007 10:46:29 DEBUG (PasswordUtil.java:209)]
Comparing
passwords:
[07/31/2007 10:46:29 DEBUG (PasswordUtil.java:242)]
Supplied
[secret]
[07/31/2007 10:46:29 DEBUG (PasswordUtil.java:243)] Stored
[secret]
[07/31/2007 10:46:29 DEBUG (PasswordUtil.java:247)] Result:
true
[07/31/2007 10:46:29 DEBUG (Session.java:283)] Bound as root
user.
[07/31/2007 10:46:29 DEBUG (EventManager.java:86)] Firing
AfterBind
event.
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:25)]
Converting
filter: OR
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:29)] Class:
org.apache.directory.shared.ldap.filter.BranchNode
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:25)]
Converting
filter: (objectclass=*)
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:29)] Class:
org.apache.directory.shared.ldap.filter.PresenceNode
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:25)]
Converting
filter: (objectclass=ldapsubentry)
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:29)] Class:
org.apache.directory.shared.ldap.filter.SimpleNode
[07/31/2007 10:46:29 DEBUG (FilterParser.java:223)] Parsing
filter
"objectclass=*"
[07/31/2007 10:46:29 DEBUG (FilterParser.java:237)] Filter:
[objectclass] [=] [*]
[07/31/2007 10:46:29 DEBUG (FilterParser.java:262)] Parsed
filter:
null
[07/31/2007 10:46:29 DEBUG (FilterParser.java:223)] Parsing
filter
"objectclass=ldapsubentry"
[07/31/2007 10:46:29 DEBUG (FilterParser.java:237)] Filter:
[objectclass] [=] [ldapsubentry]
[07/31/2007 10:46:29 DEBUG (FilterParser.java:262)] Parsed
filter:
(objectclass=ldapsubentry)
[07/31/2007 10:46:29 DEBUG (PenroseSession.java:269)]
search("ou=Users,dc=JDBC Cache,dc=Example,dc=com",
"(|
(objectclass=ldapsubentry))")
[07/31/2007 10:46:29 WARN (Session.java:757)] Search
"ou=Users,dc=JDBC Cache,dc=Example,dc=com" with
scope base and filter
"(|(objectclass=ldapsubentry))"
[07/31/2007 10:46:29 DEBUG (Session.java:763)]
------------------------------------------------------------
----------------------
[07/31/2007 10:46:29 DEBUG (Session.java:764)] SEARCH:
[07/31/2007 10:46:29 DEBUG (Session.java:765)] - Bind DN
:
uid=admin,ou=system
[07/31/2007 10:46:29 DEBUG (Session.java:766)] - Base DN
:
ou=Users,dc=JDBC Cache,dc=Example,dc=com
[07/31/2007 10:46:29 DEBUG (Session.java:767)] - Scope
: base
[07/31/2007 10:46:29 DEBUG (Session.java:768)] - Filter
: (|
(objectclass=ldapsubentry))
[07/31/2007 10:46:29 DEBUG (Session.java:769)] - Attributes
: [1.1]
[07/31/2007 10:46:30 DEBUG (Session.java:770)]
[07/31/2007 10:46:30 DEBUG (Session.java:772)] Controls: []
[07/31/2007 10:46:30 DEBUG (EventManager.java:236)] Firing
BeforeSearch event.
[07/31/2007 10:46:30 DEBUG (HandlerManager.java:419)]
Requested: [1.1]
[07/31/2007 10:46:30 DEBUG (HandlerManager.java:485)]
Searching
ou=Users,dc=JDBC Cache,dc=Example,dc=com in
ou=Users,dc=JDBC
Cache,dc=Example,dc=com
[07/31/2007 10:46:30 DEBUG (ACLManager.java:167)] Checking
object "s"
permission
[07/31/2007 10:46:30 DEBUG (ACLManager.java:176)] Root user
=> SUCCESS
[07/31/2007 10:46:30 DEBUG (DefaultHandler.java:234)]
Searching
ou=Users,dc=JDBC Cache,dc=Example,dc=com with scope base
[07/31/2007 10:46:30 DEBUG (DefaultHandler.java:240)] Entry
"ou=Users,dc=JDBC Cache,dc=Example,dc=com" doesn't
match search
filter.
[07/31/2007 10:46:30 DEBUG (HandlerSearchResponse.java:120)]
All
search threads have ended.
[07/31/2007 10:46:30 DEBUG (HandlerSearchResponse.java:146)]
Found
successes: true
[07/31/2007 10:46:30 DEBUG (EventManager.java:236)] Firing
AfterSearch
event.
[07/31/2007 10:46:30 DEBUG (PenroseSession.java:294)]
unbind()
On Jul 30, 2:18 pm, "Endi Sukma Dewata"
<end... gmail.com> wrote:
> Paul,
>
> Apparently the jdbc_cache example is outdated, sorry
about that. But before
> proceeding further I need to tell you what this example
does. It basicaly
> replicates the data from a master source, presumably
from a remote or slow
> system, into a cache database which is local and
faster. This would be
> useful if you don't want to burden the master source
with traffic from
> Penrose. If your master source is already fast enough
and can handle Penrose
> traffic you probably don't need this.
>
> It is also useful if you want to join serveral sources
that don't support
> join operation natively, for example joining 2 ldap
directories. This
> feature will replicate the entire content of your
master sources into the
> cache database and perform the join operation there. If
you want to join
> tables from the same RDBMS database you don't need
this. Unlike the previous
> versions, Penrose 1.2.x can perform the join operation
on the database side.
>
> If that is what you want to do, here's how to set up
the example:
>
> 1. Create a MySQL database called "example".
This is the master database.
> 2. Run the create.sql included in the example under the
sql folder to create
> the tables and triggers. If you are using a different
database system, you
> need to modify the script for your database system.
> 3. Create a MySQL database called "jdbc".
This is the cache database.
> 4. Add the following parameter in the "users"
source (keep the original
> "table" parameter).
>
> <parameter>
> <param-name>sync</param-name>
>
<param-value>users_cache</param-value>
> </parameter>
> <parameter>
> <param-name>changelog</param-name>
>
<param-value>users_changelog</param-value>
> </parameter>
>
> These parameters tell Penrose to synchronize this
source (i.e. users) with
> "users_cache" source using the change log
information stored in
> "users_changelog" table. You can do the same
thing with the other sources if
> you want. You can remove the <source-sync>
entries from sources.xml, they
> are no longer used.
> 5. Edit server.xml to install this partiiton.
> 6. Run this command to create the cache tables:
>
> cache.bat -d create jdbc_cache
>
> If it doesn't work with your database system, you can
create the tables
> manually. See create-cache.sql.
> 7. To synchronize the cache, run:
>
> cache.bat -d sync jdbc_cache
>
> You can use your own scheduler (e.g. cron) to run this
command periodically.
> 8. Try to update the data in the master source, the
changes should be picked
> up when you run the sync command.
>
> Let me know if you have any questions.
>
> --
> Endi S. Dewata
>
> On 7/30/07, Nuzz <paul.edwa... identitysolutions.co.nz>
wrote:
>
>
>
> > I have been trying to run the jdbc-cache example
from the samples
> > directory. The cache creation tool isn't working,
and as far as I can
> > ascertain it appears to be a config issue.
>
> > As far as my back track goes, I started working
through Penrose, from
> > the top down. I am using the xml files straight
from the samples. The
> > only changes have been to connections.xml to point
to my database. I
> > have loaded the tables as per the sql scripts in
those directories.
>
> > Now tracing down, looking into source and source
sync. The issue is,
> > that its not finding any source-sync info. The
whole reason that the
> > cache tool doesn't work, is it doesn't find
anything to sync, so it
> > just exits (Gets as far as create ( CacheUtil.java
ln 88), and finds
> > the default partition, but then tries to find
sourceSyncs (ln 96), and
> > nothing is returned).
>
> > I started poking around, and I find line
Sources.java ln 38. Here is
> > trying to pull a parameter called "sync"
from the sourceConfig, but
> > never finds it. I started poking back trying to
figure out where it
> > loads the config for sources. I have found the
sources-digester-
> > rules.xml in partition. I see that there is no
source-sync section
> > like is present in the sources.xml. So I'm
confused as to how its ever
> > supposed to pick up sync.
>
> > The only thing I have noticed, is, if I add
another entry to
> > sources.xml like this:
> > <source name="users">
> >
<connection-name>MySQL</connection-name>
> > <field name="username"
primaryKey="true"/>
> > <field name="firstName"/>
> > <field name="lastName"/>
> > <field name="password"/>
> > <parameter>
> > <param-name>table</param-name>
> > <param-value>users</param-value>
> > </parameter>
> > </source>
>
> > BUT change param-name from table to sync, it then
picks up a sync, but
> > still fails as its obviously wanting some other
params. So I'm very
> > confused... I hope you can clear this up.
>
> > Thanks
> > P.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "safehaus" group.
To post to this group, send email to safehaus googlegroups.com
To unsubscribe from this group, send email to
safehaus-unsubscribe googlegroups.com
For more options, visit this group at http://
groups.google.com/group/safehaus?hl=en
-~----------~----~----~----~------~----~------~--~---
|
|
| Re: JDBC-cache sample |

|
2007-07-31 11:57:04 |
|
Paul,
Please try replacing lib/penrose-server-1.2.4.jar with this:
Btw, in modules.xml there's already a module configured to synchronize the sources every 30 seconds. You can just use this instead of cron.
-- Endi S. Dewata
On 7/31/07, Nuzz < paul.edwards identitysolutions.co.nz">paul.edwards identitysolutions.co.nz> wrote:
Endi,
That worked great. I have the system up.
However I'm trying to do a bit of load testing with slamd. Now, when I
try to connect with slamd, penrose seems to not like the default filter that slamd is supplying (I cannot see a way at the mo of changing what slamd is supplying - without recompiling slamd). As far as I'm aware, what slamd is supplying is correct ldap syntax. I
believe that the filter slamd is supplying on connect is: (| (objectclass=*)(objectclass=ldapsubentry)), which by my way of thinking seems a bit stupid, as it should just take the objectclass=* for all, but it appears that penrose is getting rid of the
objectclass=*, and then it all fails. Anyway heres what I'm seeing:
[07/31/2007 10:46:29 DEBUG (PenroseSession.java:114)] bind("uid=admin,ou=system", "secret") [07/31/2007 10:46:29 WARN (
Session.java:242)] Bind "uid=admin,ou=system". [07/31/2007 10:46:29 DEBUG (Session.java:248)] ---------------------------------------------------------------------------------- [07/31/2007 10:46:29 DEBUG (
Session.java:249)] BIND: [07/31/2007 10:46:29 DEBUG (Session.java:250)] - Bind DN : uid=admin,ou=system [07/31/2007 10:46:29 DEBUG (Session.java:251)] - Bind Password : secret [07/31/2007 10:46:29 DEBUG (
Session.java:252)] [07/31/2007 10:46:29 DEBUG (Session.java:254)] Controls: [] [07/31/2007 10:46:29 DEBUG (EventManager.java:86)] Firing BeforeBind event. [07/31/2007 10:46:29 DEBUG (PasswordUtil.java:209)] Comparing
passwords: [07/31/2007 10:46:29 DEBUG (PasswordUtil.java:242)] Supplied [secret] [07/31/2007 10:46:29 DEBUG (PasswordUtil.java:243)] Stored [secret] [07/31/2007 10:46:29 DEBUG (PasswordUtil.java:247)] Result: true
[07/31/2007 10:46:29 DEBUG (Session.java:283)] Bound as root user. [07/31/2007 10:46:29 DEBUG (EventManager.java:86)] Firing AfterBind event. [07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:25)] Converting
filter: OR [07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:29)] Class: org.apache.directory.shared.ldap.filter.BranchNode [07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:25)] Converting filter: (objectclass=*)
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:29)] Class: org.apache.directory.shared.ldap.filter.PresenceNode [07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:25)] Converting filter: (objectclass=ldapsubentry)
[07/31/2007 10:46:29 DEBUG (MinaFilterTool.java:29)] Class: org.apache.directory.shared.ldap.filter.SimpleNode [07/31/2007 10:46:29 DEBUG (FilterParser.java:223)] Parsing filter "objectclass=*" [07/31/2007 10:46:29 DEBUG (
FilterParser.java:237)] Filter: [objectclass] [=] [*] [07/31/2007 10:46:29 DEBUG (FilterParser.java:262)] Parsed filter: null [07/31/2007 10:46:29 DEBUG (FilterParser.java:223)] Parsing filter "objectclass=ldapsubentry"
[07/31/2007 10:46:29 DEBUG (FilterParser.java:237)] Filter: [objectclass] [=] [ldapsubentry] [07/31/2007 10:46:29 DEBUG (FilterParser.java:262)] Parsed filter: (objectclass=ldapsubentry) [07/31/2007 10:46:29 DEBUG (
PenroseSession.java:269)] search("ou=Users,dc=JDBC Cache,dc=Example,dc=com", "(| (objectclass=ldapsubentry))") [07/31/2007 10:46:29 WARN (Session.java:757)] Search "ou=Users,dc=JDBC Cache,dc=Example,dc=com" with scope base and filter
"(|(objectclass=ldapsubentry))" [07/31/2007 10:46:29 DEBUG (Session.java:763)] ---------------------------------------------------------------------------------- [07/31/2007 10:46:29 DEBUG (Session.java
:764)] SEARCH: [07/31/2007 10:46:29 DEBUG (Session.java:765)] - Bind DN : uid=admin,ou=system [07/31/2007 10:46:29 DEBUG (Session.java:766)] - Base DN : ou=Users,dc=JDBC Cache,dc=Example,dc=com [07/31/2007 10:46:29 DEBUG (
Session.java:767)] - Scope : base [07/31/2007 10:46:29 DEBUG (Session.java:768)] - Filter : (| (objectclass=ldapsubentry)) [07/31/2007 10:46:29 DEBUG (Session.java:769)] - Attributes : [1.1] [07/31/2007 10:46:30 DEBUG (
Session.java:770)] [07/31/2007 10:46:30 DEBUG (Session.java:772)] Controls: [] [07/31/2007 10:46:30 DEBUG (EventManager.java:236)] Firing BeforeSearch event.
| |