Ok,
I did some investigating and found some surprising results.
When I did a ps -ef | grep java on qacosmo I saw 30 Sun JRE
processes
running for one Tomcat instance.
This was highly unusual as only one process should be
running for Tomcat.
The next obvious conclusion was that Cosmo was not getting
shutdown
properly and that orphaned processes were still running.
However, when I shutdown Cosmo via a 'osafsrvctl stop'
command all 30
processes shutdown.
Weird!
So at that point it could either be an OS, Sun JVM, a Tomcat
configuration, or a Cosmo issue.
I uploaded a Jrockit JRE to see whether running Cosmo with
it would also
spawn 30 processes.
If it did not then the problem would most likely be with the
Sun JVM.
The Jrockit installer is a .bin file which is executed on
the command
line with a -mode=console flag.
Running the installer caused a stack overflow. Jrockit
unpacks itself
using its own JVM in a single process.
However, when I did a ps -ef | grep java I saw that there
were 5 jrockit
JVM processes running orphaned.
That is certainly not the expected outcome!
After killing the orphaned processes I uploaded to qacosmo
an already
unpacked jrockit JRE from cosmo-demo.
I then created a test program which looped long enough for
me to view
the number of java processes running.
I wanted to see if multiple JVM's were created.
The program is:
public class Test {
public static void main(String[] args) {
for(int i = 0; i < 1000000000; i++) {
continue;
}
for(int i = 0; i < 1000000000; i++) {
continue;
}
System.out.println("This is a test");
}
}
I ran the program on qacosmo using the Sun JVM and in
another terminal
did a ps -ef | grep java.
Here is what I got:
osafuser qacosmo:~/cosmo/tomcat$ ps -ef | grep java
osafuser 12120 11623 99 17:29 pts/0 00:00:01
./jre1.5.0_06/bin/java Test
osafuser 12121 12120 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12122 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12123 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12124 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12125 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12126 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12127 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java Test
osafuser 12128 12121 0 17:29 pts/0 00:00:00
./jre1.5.0_06/bin/java
Test
9 Sun JVM processes were created. The expected result of
course is that
only one process would be running.
I then tried the program on qacosmo using the Jrockit JVM
and did a ps
-ef | grep java.
Here is what I got:
osafuserqacosmo:~/cosmo/tomcat$ ps -ef | grep java
osafuser 12108 11623 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12109 12108 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12110 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12111 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12112 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12113 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12114 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12115 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12116 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
osafuser 12117 12109 0 17:28 pts/0 00:00:00
./jrockit-jre-R26/bin/java Test
The same result 9 JVM processes spawned for a simple
"Hello World" type
program.
It seems there is something very strange about the way the
qacosmo box
itself is functioning.
Possibly Heikki, Jared, or Bear can shed light on to why
this may be
happening.
I personally have never seen anything like this and the
results are
puzzling to say the least.
Thoughts?
--Brian
Brian Kirsch - Cosmo Developer / Chandler
Internationalization Engineer
Open Source Applications Foundation
543 Howard St. 5th Floor
San Francisco, CA 94105
http://www.osafoundation
.org
bug-commentosafoundation.org wrote:
>http://bugzilla.osafoundation.org/show_bug.cgi?id=5643
>
> Summary: Cosmo crashes after multiple large
event PUTs
> Product: Cosmo
> Version: 0.3
> Platform: All
> OS/Version: All
> Status: NEW
> Severity: normal
> Priority: P3
> Component: Configuration
> AssignedTo: bkirschosafoundation.org
> ReportedBy: mikealosafoundation.org
>
>
>Repro:
>
>You can run the chandler functional test TestSharing to
recreate the bug. It
>loads in a large ICS file and then shares it with cosmo.
There are over 500
>events in the ics file.
>
>You can look at all the cosmo logs here;
>http://qacosmo
.osafoundation.org/logs/ #/cosmo/logs
>http:/
/qacosmo.osafoundation.org/logs/slashdir #/cosmo/
>http
://qacosmo.osafoundation.org/logs/tomcatlogs
#/cosmo/tomcat/logs
>
>The timestamp when the problem occurred first is;
>2006-04-12 15:01:37,138
>
>The error in catalina.out is;
>#
># An unexpected error has been detected by HotSpot
Virtual Machine:
>#
># SIGSEGV (0xb) at pc=0x40576cd1, pid=10454, tid=114696
>#
># Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05
mixed mode)
># Problematic frame:
># V [libjvm.so+0x3c9cd1]
>#
># An error report file with more information is saved as
hs_err_pid10454.log
>#
># If you would like to submit a bug report, please
visit:
># http:
//java.sun.com/webapps/bugreport/crash.jsp
>
>Error in osafsrvlog;
>2006-04-12 15:01:37,138 DEBUG [JcrDavException] handling
exception with
>errorCode 403
>javax.jcr.ItemNotFoundException:
>cc757545-e94b-485b-b549-6344a6b4cea5/{http://www.jcp.org/jcr/1.0
}mimeType
> at
>org.apache.jackrabbit.core.ItemManager.createItemInstanc
e(ItemManager.java:467)
> at
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.j
ava:319)
> at
>org.apache.jackrabbit.core.ItemImpl.restoreTransientItem
s(ItemImpl.java:691)
> at
org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1192)
> at
>org.apache.jackrabbit.webdav.simple.DavResourceImpl.addM
ember(DavResourceImpl.java:541)
> at
>org.apache.jackrabbit.server.AbstractWebdavServlet.doPut
(AbstractWebdavServlet.java:531)
> at
org.osaf.cosmo.dav.CosmoDavServlet.doPut(Unknown Source)
> at
>org.apache.jackrabbit.server.AbstractWebdavServlet.execu
te(AbstractWebdavServlet.java:260)
> at
org.osaf.cosmo.dav.CosmoDavServlet.execute(Unknown Source)
> at
>org.apache.jackrabbit.server.AbstractWebdavServlet.servi
ce(AbstractWebdavServlet.java:212)
> at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at
>org.apache.catalina.core.ApplicationFilterChain.internal
DoFilter(ApplicationFilterChain.java:252)
> at
>org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:173)
> at
>org.acegisecurity.util.FilterChainProxy$VirtualFilterCha
in.doFilter(FilterChainProxy.java:292)
> at
>org.acegisecurity.intercept.web.FilterSecurityIntercepto
r.invoke(FilterSecurityInterceptor.java:116)
> at
>org.acegisecurity.intercept.web.FilterSecurityIntercepto
r.doFilter(FilterSecurityInterceptor.java:79)
> at
>org.acegisecurity.util.FilterChainProxy$VirtualFilterCha
in.doFilter(FilterChainProxy.java:303)
> at
>org.acegisecurity.ui.ExceptionTranslationFilter.doFilter
(ExceptionTranslationFilter.java:143)
> at
>org.acegisecurity.util.FilterChainProxy$VirtualFilterCha
in.doFilter(FilterChainProxy.java:303)
> at
>org.osaf.cosmo.acegisecurity.ticket.TicketProcessingFilt
er.doFilter(Unknown
>Source)
> at
>org.acegisecurity.util.FilterChainProxy$VirtualFilterCha
in.doFilter(FilterChainProxy.java:303)
> at
>org.acegisecurity.ui.basicauth.BasicProcessingFilter.doF
ilter(BasicProcessingFilter.java:207)
> at
>org.acegisecurity.util.FilterChainProxy$VirtualFilterCha
in.doFilter(FilterChainProxy.java:303)
> at
>org.osaf.cosmo.acegisecurity.HttpRequestContextIntegrati
onFilter.doFilter(Unknown
>Source)
> at
>org.acegisecurity.util.FilterChainProxy$VirtualFilterCha
in.doFilter(FilterChainProxy.java:303)
> at
>org.acegisecurity.util.FilterChainProxy.doFilter(FilterC
hainProxy.java:173)
> at
>org.acegisecurity.util.FilterToBeanProxy.doFilter(Filter
ToBeanProxy.java:120)
> at
>org.apache.catalina.core.ApplicationFilterChain.internal
DoFilter(ApplicationFilterChain.java:202)
> at
>org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:173)
> at
>org.springmodules.jcr.support.OpenSessionInViewFilter.do
FilterInternal(OpenSessionInViewFilter.java:119)
> at
>org.springframework.web.filter.OncePerRequestFilter.doFi
lter(OncePerRequestFilter.java:76)
> at
>org.apache.catalina.core.ApplicationFilterChain.internal
DoFilter(ApplicationFilterChain.java:202)
> at
>org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:173)
> at
org.osaf.cosmo.ui.VersionHeaderFilter.doFilter(Unknown
Source)
> at
>org.apache.catalina.core.ApplicationFilterChain.internal
DoFilter(ApplicationFilterChain.java:202)
> at
>org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:173)
> at
>org.apache.catalina.core.StandardWrapperValve.invoke(Sta
ndardWrapperValve.java:213)
> at
>org.apache.catalina.core.StandardContextValve.invoke(Sta
ndardContextValve.java:178)
> at
>org.apache.catalina.core.StandardHostValve.invoke(Standa
rdHostValve.java:126)
> at
>org.apache.catalina.valves.ErrorReportValve.invoke(Error
ReportValve.java:105)
> at
>org.apache.catalina.valves.AccessLogValve.invoke(AccessL
ogValve.java:526)
> at
>org.apache.catalina.core.StandardEngineValve.invoke(Stan
dardEngineValve.java:107)
> at
>org.apache.catalina.connector.CoyoteAdapter.service(Coyo
teAdapter.java:148)
> at
>org.apache.coyote.http11.Http11Processor.process(Http11P
rocessor.java:856)
> at
>org.apache.coyote.http11.Http11Protocol$Http11Connection
Handler.processConnection(Http11Protocol.java:744)
> at
>org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket
(PoolTcpEndpoint.java:527)
> at
>org.apache.tomcat.util.net.LeaderFollowerWorkerThread.ru
nIt(LeaderFollowerWorkerThread.java:80)
> at
>org.apache.tomcat.util.threads.ThreadPool$ControlRunnabl
e.run(ThreadPool.java:684)
> at java.lang.Thread.run(Unknown Source)
>2006-04-12 15:07:11,293 ERROR [[webdav]]
Servlet.service() for servlet webdav
>threw exception
>java.lang.OutOfMemoryError: Java heap space
>2006-04-12 15:13:42,842 ERROR [[webdav]]
Servlet.service() for servlet webdav
>threw exception
>java.lang.OutOfMemoryError: Java heap space
>2006-04-12 15:39:22,931 ERROR [[webdav]]
Servlet.service() for servlet webdav
>threw exception
>java.lang.OutOfMemoryError: Java heap space
>2006-04-12 15:40:36,670 ERROR [[cmp]] Servlet.service()
for servlet cmp threw
>exception
>java.lang.OutOfMemoryError: Java heap space
>2006-04-12 15:40:53,593 ERROR [[cmp]] Servlet.service()
for servlet cmp threw
>exception
>java.lang.OutOfMemoryError: Java heap space
>
>
>
>
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev
|