Kenneth Lakin wrote:
> All,
>
> I'm running vss2svn and ssphys (from SVN rev 309) on a
Windows XP Pro machine w/ 2GB of RAM. I compiled perl from
the ActiveState 5.8.8 sources. I've enabled perl's native
memory management.
> I have two separate issues.
> The first:
> *Summary:
> SanityChecker needs to start writing out its data to
disk; perhaps in an SQLite DB or something.
>
> *Exposition:
> I'm processing a SSafe database that has well over 3000
commits.
> Everything's fine until the IMPORTSVN phase. As we work
through the IMPORTSVN phase, perl keeps using more and more
memory. 'Round about the time that SVN revision 2947 is
being processed, perl's using 1.5GB of ram. vss2svn goes to
load a ~250MB file, and cannot allocate the memory to do so.
perl halts, on line 719 of Dumpfile.pm ( $node-> =
do { local( $/ ) ; <EXP> } ; ) and I'm left with a 9GB
SVN dump file that still has two years of commits to go!
> I've looked all through Vss2Svn: umpFile
and associated classes, and made a short run thru the
debugger. Unless I've messed up my perl compilation, I can't
find anything other than SanityChecker that might be eating
so much memory.
>
> I'm going to get to work on making SanityChecker write
to an SQLite database today, unless someone has a better
idea.
>
Hi Kenneth, thanks for writing. You're losing me a bit here
though. The
issue in Dumpfile.pm in which it pulls the entire file into
memory is
certainly an issue (as documented in ticket 25 which you
mention) and
one that I would like to fix. Doing a buffered read isn't
terribly
difficult, but rewriting it to write the data directly from
source to
target file will take some more coding.
However, I don't understand where you're getting to
SanityChecker, or
what else you're planning on writing to SQLite. It certainly
is odd that
you are seeing steadily increasing memory though; I've never
seen it get
to that high of usage although others might have. The number
of commits
(~3000) you're dealing with is certainly not huge but if you
have
numerous multi-hundred-megabyte files then that could
definitely be a
problem. But again, that would be solved by fixing
Dumpfile.pm.
Is there a reason you compiled Perl from source, instead of
using
ActiveState's binary version?
> The second issue:
> I have a patch for DumpFile::get_export_contents() that
(for me) works better than the one here:
>
> ht
tp://www.pumacode.org/projects/vss2svn/ticket/25
> (I'll submit the patch to the list within 24 hrs.)
>
> All in all, this is a really fine tool! Thanks for
making it available.
>
Glad it is working well for you; please feel free to send
the patches if
you get it working with less memory. My time to work on this
project is
very rare these days but I would definitely be interested in
reducing
the overall memory footprint.
toby
_______________________________________________
vss2svn-users mailing list
Project homepage:
http://www.
pumacode.org/projects/vss2svn/
Subscribe/Unsubscribe/Admin:
http://lists.pumacode.org/mailman/lis
tinfo/vss2svn-users-lists.pumacode.org
Mailing list web interface (with searchable archives):
http://dir.gmane.org/gmane.comp.version-c
ontrol.subversion.vss2svn.user
|
