CVSItem dependencies and symbols

If you have three revisions like this, the dependencies are
obvious.

3.2
3.1
3.0

3.0 -> 3.1 -> 3.2

Now introduce a label X

3.2
3.1(X)
3.0

3.0 -> 3.1(X) -> 3.2

In changeset_graph.py/add_changeset() the predecessor and
successor
change set links are built. For CVSRevsions this is
happening
correctly. But for a Symbol it is not correct.

CVSRevision 3.1 lies between 3.0 and 3.2. Label X doesn't
lie between
3.0 and 3.2, it is between 3.1 and 3.2.

I believe a dependency chain like this is needed:
3.0 -> 3.1->X -> 3.2

This will allow change set to be located between 3.1 and X.
It is not
possible to place a change set in the location with a single
CVS file,
but it is possible when multiple files are involved.

Does this make sense? If so, there needs to be a CVSItem
corresponding
to symbols.

-- 
Jon Smirl
jonsmirlgmail.com

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

Jon Smirl wrote:
> If you have three revisions like this, the dependencies
are obvious.
> 
> 3.2
> 3.1
> 3.0
> 
> 3.0 -> 3.1 -> 3.2
> 
> Now introduce a label X
> 
> 3.2
> 3.1(X)
> 3.0
> 
> 3.0 -> 3.1(X) -> 3.2
> 
> In changeset_graph.py/add_changeset() the predecessor
and successor
> change set links are built. For CVSRevsions this is
happening
> correctly. But for a Symbol it is not correct.
> 
> CVSRevision 3.1 lies between 3.0 and 3.2. Label X
doesn't lie between
> 3.0 and 3.2, it is between 3.1 and 3.2.

Label X *should* be between 3.1 and 3.2 (unless a
non-current revision
was tagged).  You can't tag revision 3.1 until after 3.1
exists.

> I believe a dependency chain like this is needed:
> 3.0 -> 3.1->X -> 3.2
> 
> This will allow change set to be located between 3.1
and X. It is not
> possible to place a change set in the location with a
single CVS file,
> but it is possible when multiple files are involved.
> 
> Does this make sense? If so, there needs to be a
CVSItem corresponding
> to symbols.

It does more than make sense--it is pretty much blindingly
obvious.

Please look in cvs_item.py.  There, along with CVSRevision,
you will see
CVSBranch, and CVSTag.  I believe that they already know
their
dependencies correctly, aside from a few fine points.  In
fact there is
already a SymbolChangeset, which is already populated with
the symbol
creation events in the obvious way.  But the use of the
symbol changeset
graph simply hasn't been implemented yet.  I know pretty
much what to
do; I simply haven't had time to do it yet.

The subtlety involved in processing symbols is not in
setting up the
dependency graph.  It is in deciding where to base a branch.
 If you
have a revision 1.5 and branches 1.5.2 and 1.5.4, then there
is no way
to determine from a single *,v file whether the user created
1.5.4 off
of 1.5 or off of 1.5.2 (before branch 1.5.2 had any commits
on it).  And
it is not trivial to find a cheap way to decide what moment
in time is
most fortuitous to create the tag such that all of the
needed revisions
are in place.  These are the main problems addressed by
Daniel
Jacobowitz's patch, if I understand correctly.  Thus there
is even a
known solution to these problems.

Michael

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

I'm looking for a solution to the problem illustrated in
this test
case. In the test the change sets got built in a manner such
that it
was impossible to achieve a single branch base. I poked
around in the
data manually and convinced myself that it was possible to
achieve a
single base if things were rearranged.

I read though Daniel's original patch and it is definitely
much
smarter about choosing the symbol base. But the only way I
see to fix
the problem illustrated below is by altering the dependency
trees and
Daniel's code doesn't do that.
http://cvs2svn.tigris.org/servlets/ReadMsg?lis
t=dev&msgNo=1441

I admit to not understanding everything completely yet but
it looks to
me like there are two problems.

1) Getting the change sets arranged in the correct order and
broken up
in the right places.
2) Picking an optimal change set(s) to base the branch on.

The test case illustrate a problem in #1. Fixing issues
around #1 make
things much easier for #2.

A test case for the symbol base problem.

 ../branches/graph-based/cvs2svn --no-cross-branch-commits
--skip-cleanup -v --dry-run --force-tag=CVS --force-tag=N3
--force-branch=RDF_122898_BRANCH
--force-branch=FASTLOAD_20010529_BRANCH
--force-branch=MozillaSourceClassic_19981026_BRANCH
/opt/mozcvs/mozilla/xpcom >foo

diff --git a/branches/graph-based/cvs2svn_lib/passes.py
b/branches/graph-based/cvs2svn_lib/passes.py
index bb9165b..b2fb16e 100644
--- a/branches/graph-based/cvs2svn_lib/passes.py
+++ b/branches/graph-based/cvs2svn_lib/passes.py
 -558,11
+558,13  class InitializeChangesetsPass(Pass):
       
artifact_manager.get_temp_file(config.CHANGESETS_DB),
DB_OPEN_NEW)
    self.changeset_key_generator = KeyGenerator(1)

+    Log().verbose("Creating preliminary revision
commit sets...")
    for changeset in self.get_revision_changesets():
      for split_changeset in
self.break_internal_dependencies(changeset):
        Log().verbose(repr(changeset))
        self.store_changeset(split_changeset)

+    Log().verbose("Creating preliminary symbol commit
sets...")
    for changeset in self.get_symbol_changesets():
      Log().verbose(repr(changeset))
      self.store_changeset(changeset)
 -662,8
+664,8  class BreakCVSRevisionChangesetLoopsPass
      changeset = old_changesets_db[changeset_id]
      print repr(changeset) # 
      self.changesets_db.store(changeset)
-      if isinstance(changeset, RevisionChangeset):
-        self.changeset_graph.add_changeset(changeset)
+      #if isinstance(changeset, RevisionChangeset):
+      self.changeset_graph.add_changeset(changeset)

    self.changeset_key_generator =
KeyGenerator(changeset_ids[-1] + 1)
    del changeset_ids
 -726,8
+728,8  class TopologicalSortPass(Pass):

    for changeset_id in changeset_ids:
      changeset = changesets_db[changeset_id]
-      if isinstance(changeset, RevisionChangeset):
-        changeset_graph.add_changeset(changeset)
+      #if isinstance(changeset, RevisionChangeset):
+      changeset_graph.add_changeset(changeset)

    del changeset_ids

 -867,7
+869,8  class CreateRevsPass(Pass):

    creator = CVSRevisionCreator()
    for (changeset, timestamp) in self.get_changesets():
-      creator.process_changeset(changeset, timestamp)
+      if isinstance(changeset, RevisionChangeset):
+        creator.process_changeset(changeset, timestamp)

    if not Ctx().trunk_only:
      Ctx()._symbolings_logger.close()


If you look in the output:

CVS Revision grouping:
 Time: Fri May 22 17:48:59 1998
Creating Subversion r81 (commit)
...
  src/nsID.cpp 3.3
...
Creating Subversion r86 (closing tag/branch 'MANIFEST')

Check the CVS file:

jonsmirljonsmirl:/opt/mozcvs/mozilla/xpcom$ grep -r
MANIFEST: * | grep nsID
src/Attic/nsID.cpp,v:   MANIFEST:3.2
jonsmirljonsmirl:/opt/mozcvs/mozilla/xpcom$

---------------------------

'src/nsID.cpp 3.3' clearly got pulled into r81 which is
before the
symbol closing in r86. Is should have had a dependency on
the MANIFEST
symbol closing which would have stoped it from being pulled
forward.

Once this happened it became impossible to find a single
base for the symbol.

-- 
Jon Smirl
jonsmirlgmail.com

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

Jon Smirl wrote:
> I'm looking for a solution to the problem illustrated
in this test
> case. In the test the change sets got built in a manner
such that it
> was impossible to achieve a single branch base. I poked
around in the
> data manually and convinced myself that it was possible
to achieve a
> single base if things were rearranged.
> 
> I read though Daniel's original patch and it is
definitely much
> smarter about choosing the symbol base. But the only
way I see to fix
> the problem illustrated below is by altering the
dependency trees and
> Daniel's code doesn't do that.
> http://cvs2svn.tigris.org/servlets/ReadMsg?lis
t=dev&msgNo=1441

It would be great to have concrete test cases that we could
add to our
test suite (even though the tests still fail!)  It would be
very helpful
if you would:

- add a test to run-tests.py

- boil your test case down small enough to add to our test
suite (this
is mostly automated; you can use the tools described in the
following
FAQ entries:

    http
://cvs2svn.tigris.org/faq.html#reportingbugs
    http://cv
s2svn.tigris.org/faq.html#testcase

- package both up together as a patch

Please read HACKING in the cvs2svn project and the
subversion hacking
document referenced therein for guidelines on how to format
a patch.


If you have more energy after that (and it seems like you
have lots of
energy  ), it
would be very helpful if you would concoct some other
perverse repositories to use in testing graph-based.  For
example, our
test suite should have test repositories with simple cycles
like

+-> A -> B --+
|            |
+------------+

A: a.txt<1.1> b.txt<1.2>
B: a.txt<1.2> b.txt<1.1>

and

+-> A -> B -> C --+
|                 |
+-----------------+

A: a.txt<1.1>            c.txt<1.2>
B: a.txt<1.2> b.txt<1.1>
C:            b.txt<1.2> c.txt<1.1>

And also some more perverse ones with passthru dependencies
like the
following

A: a.txt<1.1> b.txt<1.3> c.txt<1.2>
B: a.txt<1.2> b.txt<1.1> c.txt<1.3>
C: a.txt<1.3> b.txt<1.2> c.txt<1.1>

and

A: a.txt<1.1>            c.txt<1.3>
d.txt<1.2>
B: a.txt<1.2> b.txt<1.1>           
d.txt<1.3>
C: a.txt<1.3> b.txt<1.2> c.txt<1.1>
D:            b.txt<1.3> c.txt<1.2>
d.txt<1.1>

and

A: a.txt<1.1> c.txt<1.3>
B: a.txt<1.2> c.txt<1.4>
C: a.txt<1.3> c.txt<1.1>
D: a.txt<1.4> c.txt<1.2>

(this last one has the nasty property that each changeset
has either one
LINK_PREV or LINK_SUCC and also one LINK_PASSTHRU).

I think it would be enough to ensure_conversion() of these
repositories
and to make sure that the total number of resulting svn
revisions is
minimal, but if you can think of other things to test that
would be even
better.

Thanks,
Michael

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

I'm chopping down the test case for this one. I am at 4.4MB.
Is that
small enough? If not I need a tool that will remove all of
the CVS
activity after a certain date. The problem occurs in 1998 so
I don't
need the next eight years worth of activity.

I already ran destroy_repository.py on the test data. I am
down to
about 340 files. There are 3546 svn commits in the test data
but the
problem is illustrated in the first 100 commits.

-- 
Jon Smirl
jonsmirlgmail.com

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

Jon Smirl wrote:
> I'm chopping down the test case for this one. I am at
4.4MB. Is that
> small enough? If not I need a tool that will remove all
of the CVS
> activity after a certain date. The problem occurs in
1998 so I don't
> need the next eight years worth of activity.
> 
> I already ran destroy_repository.py on the test data. I
am down to
> about 340 files. There are 3546 svn commits in the test
data but the
> problem is illustrated in the first 100 commits.

Did you try shrink_test_case.py?  I bet that can condense it
down to
just a few files.

Michael

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

Here is a test case involving only 12 small files. I picked
another
case of the same problem and was able to achieve a much
smaller file
set.

To run it:
cvs2svn -v --dry-run --force-tag=CVS --force-tag=N3
--force-branch=RDF_122898_BRANCH
--force-branch=FASTLOAD_20010529_BRANCH
--force-branch=MozillaSourceClassic_19981026_BRANCH  xpcom

In the verbose output look for this SVN change set which
creates the
symbol MANIFEST.

============================================================
Starting Subversion r61 / 231
Filling symbolic name: MANIFEST
  Copying revision 52 of trunk
                to tags/MANIFEST
  Deleting tags/MANIFEST/ds
  Copying revision 60 of trunk/src/export.mac
                to tags/MANIFEST/src/export.mac
  Copying revision 59 of trunk/src/nsISupportsArray.h
                to tags/MANIFEST/src/nsISupportsArray.h
  Copying revision 59 of trunk/src/nsSupportsArray.cpp
                to tags/MANIFEST/src/nsSupportsArray.cpp

It comes after this change set which involes nsID.cpp

Starting Subversion r59 / 231
Committing 5 CVSRevisions
  Changing trunk/src/nsID.cpp
  Changing trunk/src/nsISupports.h
  Changing trunk/src/makefile.win
  Adding trunk/src/nsSupportsArray.cpp
  Adding trunk/src/nsISupportsArray.h

Check the CVS versions for r59

CVS Revision grouping:
  Time: Fri May 22 17:48:52 1998
Creating Subversion r59 (commit)
  src/nsID.cpp 3.3
  src/nsISupports.h 3.3
  src/makefile.win 3.2
  src/nsSupportsArray.cpp 3.1
  src/nsISupportsArray.h 3.1

Note that src/nsID.cpp 3.3 is used in r59
Now look in nsID.cpp,v for the definition of MANIFEST.
        MANIFEST:3.2

src/nsID.cpp 3.3 which is used in r59 depends on the
definition of
MANIFEST which occured in r61. The dependencies are out of
order.

-- 
Jon Smirl
jonsmirlgmail.com
------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

Jon Smirl wrote:
> Here is a test case involving only 12 small files. I
picked another
> case of the same problem and was able to achieve a much
smaller file
> set.
> 
> To run it:
> cvs2svn -v --dry-run --force-tag=CVS --force-tag=N3
> --force-branch=RDF_122898_BRANCH
> --force-branch=FASTLOAD_20010529_BRANCH
> --force-branch=MozillaSourceClassic_19981026_BRANCH 
xpcom
> 
> In the verbose output look for this SVN change set
which creates the
> symbol MANIFEST.
> [...]

Thanks for the test case, but it would be *much* more useful
if it were
automated as part of run-tests.py so that we could integrate
it into the
test suite.

There are utilities in the test framework that make it
possible to
inquire about specific revisions and see which files were
affected/copied/whatever.  It's not documented very well,
but you can
probably figure it out from existing test cases.

Thanks,
Michael

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

I have the two output lines now as Python variables.

A: Creating Subversion r59 (commit)
B: Starting Subversion r61 / 231

How do i parse out the 59 and 61 in Python?

A > B Pass
A < B Fail
-- 
Jon Smirl
jonsmirlgmail.com

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org

Jon Smirl wrote:
> I have the two output lines now as Python variables.
> 
> A: Creating Subversion r59 (commit)
> B: Starting Subversion r61 / 231
> 
> How do i parse out the 59 and 61 in Python?
> 
> A > B Pass
> A < B Fail

You mean you want to extract the number from the string? 
Use something like

import re

pattern = re.compile(r'Subversion r(d+)')
m = pattern.search(A)
print int(m.group(1))
m = pattern.search(B)
print int(m.group(1))

The regular expression can be more specific if necessary.

Michael

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribecvs2svn.tigris.org
For additional commands, e-mail: dev-helpcvs2svn.tigris.org