pass6 (out of) Memory Error

Hi,

Here's the summary of the issue I mentioned in IRC.
I'm trying to convert a huge repo (11G, 406k files, ~1500
symbols).
I tried 1.5 but ran into issue #106, so I reran the
conversion with
2.0rc1 and trunk.
The first 5 passes took some time (24h pass1, 14h pass3, in
total about
2 days). The rc1 conversion was run without internal-co on a
AMD64 X2 3800 with
2G RAM (Fedora 7) and died with 4.9G in cvs-items-sorted.
Pass6 created a 756M
cvs-item-to-changeset after a couple of minutes, which
remained at that size
until it crashed after running for a whole week.
The output was:

Creating preliminary commit sets...
Traceback (most recent call last):
    File "./cvs2svn", line 31, in <module>
      main(sys.argv[0], sys.argv[1:])
    File
"/home/marks/work/scm/cvs2svn/cvs2svn-2.0.0-rc1/cvs2svn
_lib/main.py", line 92, in main
      pass_manager.run(run_options.start_pass,
run_options.end_pass)
    File
"/home/marks/work/scm/cvs2svn/cvs2svn-2.0.0-rc1/cvs2svn
_lib/pass_manager.py", line 171, in run
      the_pass.run(stats_keeper)
    File
"/home/marks/work/scm/cvs2svn/cvs2svn-2.0.0-rc1/cvs2svn
_lib/passes.py", line 502, in run
      for cvs_item in changeset.get_cvs_items():
    File
"/home/marks/work/scm/cvs2svn/cvs2svn-2.0.0-rc1/cvs2svn
_lib/changeset.py", line 41, in get_cvs_items
      return
set(Ctx()._cvs_items_db.get_many(self.cvs_item_ids))
    File
"/home/marks/work/scm/cvs2svn/cvs2svn-2.0.0-rc1/cvs2svn
_lib/database.py", line 277, in get_many
      yield self._fetch(offset)
    File
"/home/marks/work/scm/cvs2svn/cvs2svn-2.0.0-rc1/cvs2svn
_lib/database.py", line 251, in _fetch
      return self.serializer.loadf(self.f)
MemoryError

I have currently the same conversion at pass6 running on an
Opteron285 with
4Gig RAM (current FC6) and internal-co for about 2 days now.
It shows the same
behaviour, uses 1.8G RSS, but only half of one CPU core. The
conversion is as
slow as on the other machine, spending most of its time
flushing the tables.

227132.897805: Flushing cache for
RecordTable('cvs2svn-tmp/cvs-item-to-changeset.dat')
227179.482381: Flushing cache for
RecordTable('cvs2svn-tmp/cvs-item-to-changeset.dat')
227234.237808: Flushing cache for
RecordTable('cvs2svn-tmp/cvs-item-to-changeset.dat')
227294.243359: Flushing cache for
RecordTable('cvs2svn-tmp/cvs-items-filtered-index.dat')
227335.088815: Flushing cache for
RecordTable('cvs2svn-tmp/cvs-items-filtered-index.dat')
227370.870331: Flushing cache for
RecordTable('cvs2svn-tmp/cvs-items-filtered-index.dat')

Best regards,
Markus

------------------------------------------------------------
---------
To unsubscribe, e-mail: users-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: users-helpcvs2svn.tigris.org