Slowdown

I just noticed that the version of offlineimap I've been
using
(3.99.17) is well over four years old.  How time flies.  I
haven't had
any problems with it, but out of curiosity I decided to pull
in 5.99.2
from the fedora repository.  It turns out to take
consistently over
twice as long as the old version to sync the same account. 
Is this
expected?

-Aaron

On Mon January 7 2008 5:58:35 pm Aaron Kaplan wrote:
> I just noticed that the version of offlineimap I've
been using
> (3.99.17) is well over four years old.  How time flies.
 I haven't had
> any problems with it, but out of curiosity I decided to
pull in 5.99.2
> from the fedora repository.  It turns out to take
consistently over
> twice as long as the old version to sync the same
account.  Is this
> expected?

No, it's not, and I don't recall hearing anyone else having
this problem.  
But there are a lot of variables there.  You probably have
had to upgrade 
your Python version as well.  Are you using multithreaded
sync?

What are the characteristics of your account?  Lots of
folders?  Few or many 
messages per folder?  What UI are you using?

Incidentally, 5.99.4 is the latest release.

-- John

>
> -Aaron

On Tue January 8 2008, Aaron Kaplan wrote:
> I just noticed that the version of offlineimap I've
been using
> (3.99.17) is well over four years old.  How time flies.
 I haven't had
> any problems with it, but out of curiosity I decided to
pull in 5.99.2
> from the fedora repository.  It turns out to take
consistently over
> twice as long as the old version to sync the same
account.  Is this
> expected?
Which distribution are you using? The oldes offlineimap
version in a supported 
Fedora Release is offlineimap-4.0.16-3.fc7. 5.99.4 will be
in Fedora 8 
updates-testing repository soon.

Regards,
Till

-- Attached file included as plaintext by Ecartis --
-- File: signature.asc
-- Desc: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iQIVAwUAR4O7kGo6ELMcEJUXAQpNLg//ct2/GGs91uG/NihbUV1Ck+U/KN3/
UX5a
oLgohrBxgOhpSLVJlfkX60AtEIanpWZ2ZjYGwVwHOyD7hIvGzeoogSetXfxP
wON7
OF+Eh8WYvxjaRZmeAFlddlPCj0rVZv10KB3CzTXXy46/Ny23M1LJgHSF3pJ5
ul0a
kaVxA6cjne4oI7VB6CiY10ebw+tdaJBuE6GkrV4T2UuSZVuRun4mzuZT/xHA
QKlA
cOJHfYIMt2qKiLITxnrYYbvRVc5kPNQpAbhkihwbHjbt2Sju/F7oqeQxffN1
+4sh
YkxmzmovuAZGdVGu0AwnoQY4MI/Sj9Jhg4NCtk2ypNzLioAOGOJ8MUcfbduZ
AldK
5gihyxDnP3ib6IQ9mRQan9i9sJ9VWDsmY5V6HVSjR3v+kX4DdDNs25smJm3e
3IXf
l33+0ppG/1tDEn7UiNGC4WO4JUQOZiW+NLWLC3yfLU7AoMizlYXU65cvES7I
6pyl
efyu29j8BobNbTlDOLWdLaUoBP8gAqytG8wEnwutiw5iWpPSEXb8oiv8TadW
6S/5
dnRddaEB+RXz9wE23wZGr5kYePy1uM9YVKa/z27ayuq+ohyP7D/BHqUCItYO
oCRk
84fEEVjBJzVYYD9Rotu/0DJ4dXf6YepLUd5trTNvMHA/KJu970lrrCzJuXRl
7u9F
bKcywDPEBqA=
=UNr0
-----END PGP SIGNATURE-----

On Tue, 8 Jan 2008 08:27:04 -0600
John Goerzen <jgoerzencomplete.org> wrote:

> But there are a lot of variables there.  You probably
have had to upgrade 
> your Python version as well.

No, I'm comparing 3.99.17 and 5.99.2 (and now 5.99.4, same
results)
installed side-by-side on the same machine, with the same
python.

> Are you using multithreaded sync?

I tried with maxconnections = 1, 2, and 3.  The old version
is
significantly faster in each case.

> What are the characteristics of your account?  Lots of
folders?  Few or many 
> messages per folder?

35 folders; about 17k messages total, with the four biggest
folders
accounting for 11k of that.

>  What UI are you using?

TTY.TTYUI

-Aaron

On Wed January 9 2008 4:32:59 pm Aaron Kaplan wrote:
> > What are the characteristics of your account? 
Lots of folders?  Few or
> > many messages per folder?
>
> 35 folders; about 17k messages total, with the four
biggest folders
> accounting for 11k of that.

Well, this is pretty puzzling.  There have been some patches
that have been 
applied that are reported to significantly *increase*
performance in this 
situation.

I might suggest you grab the darcs tree and try unapplying a
few patches, and 
see what happens.  The biggest ones that are on your path
are:

Mon Oct  1 11:20:37 CDT 2007  John Goerzen <jgoerzencomplete.org>
  * Daniel Jacobowitz patches

Fri Jul  6 06:46:29 CDT 2007  John Goerzen <jgoerzencomplete.org>
  * Re-scan list of remote folders on each sync

Wed Mar 28 10:23:18 CDT 2007  John Goerzen <jgoerzencomplete.org>
  * Improve filesystem flushing semantics

Mon Aug 22 21:15:09 CDT 2005  John Goerzen <jgoerzencomplete.org>
  * [319117] Unroll loop to speed performance on large
folders

Tue Nov 16 10:41:09 CST 2004  John Goerzen <jgoerzencomplete.org>
  * Added code to limit command line length for very large
mailboxes

There may be others that I didn't see.

Try removing each of those in turn and see if any of them
speed it up.  I'd 
be interested in your results.

-- John

On Wed, 9 Jan 2008 16:52:38 -0600, "John Goerzen"
<jgoerzencomplete.org> said:
> I might suggest you grab the darcs tree and try
unapplying a few patches,
> and 
> see what happens.

The slowdown appears to be related to the removal of the
local version
of imaplib.py.  This threw me for a loop for a while: I
unpulled big
bunches of patches until I got to a version that was fast,
then I
started pulling patches back in, to see where it would get
slow again,
but I pulled all the way back up to the current version and
the slowdown
didn't reappear!  This turned out to be because after I
pulled the patch
"Remove local imaplib.py", the local imaplib.py
was gone, but the local
imaplib.pyc that had been compiled from it was still there. 
Once I
deleted that, it got slow again.

I did my testing on a single folder with 1500 messages, and
it takes 12
seconds with the current version vs. 7 seconds for the old
version. 
/usr/lib/python2.5/imaplib.py on my system says it's version
2.58; the
last version included with offlineimap says version 2.52.

-Aaron

On Tuesday 12 February 2008 5:33:31 pm Aaron Kaplan wrote:
> On Wed, 9 Jan 2008 16:52:38 -0600, "John
Goerzen"
>
> <jgoerzencomplete.org> said:
> > I might suggest you grab the darcs tree and try
unapplying a few
> > patches, and
> > see what happens.
>
> The slowdown appears to be related to the removal of
the local version
> of imaplib.py.  This threw me for a loop for a while: I
unpulled big

Ugh.  Unfortunately, that's almost the worst possible news,
because 
imaplib.py is *huge*.  I'm going to grasp at straws here to
suggest where 
you might start on this.  I guess the first thing to do is
figure out if an 
OfflineIMAP patch made the change, or an upstream change
made the 
difference.

You could look at the darcs history for imaplib.py from the
time I imported 
2.52 until the time we got rid of it. You might try the
unmodified 2.52 
imaplib.py with the current code and see if it is any
better.  If it is, 
that would point the bloody finger squarely at the Python
maintainers.  If 
it's not, that means one of the OfflineIMAP patches boosted
performance.  
Then at that point, it's a matter of figuring out which one,
and seeing if 
we can get it sent upstream.  I do not want to reintroduce
imaplib.py to the 
source tree.

-- John

> bunches of patches until I got to a version that was
fast, then I
> started pulling patches back in, to see where it would
get slow again,
> but I pulled all the way back up to the current version
and the slowdown
> didn't reappear!  This turned out to be because after I
pulled the patch
> "Remove local imaplib.py", the local
imaplib.py was gone, but the local
> imaplib.pyc that had been compiled from it was still
there.  Once I
> deleted that, it got slow again.
>
> I did my testing on a single folder with 1500 messages,
and it takes 12
> seconds with the current version vs. 7 seconds for the
old version.
> /usr/lib/python2.5/imaplib.py on my system says it's
version 2.58; the
> last version included with offlineimap says version
2.52.
>
> -Aaron

The following changeset is the one responsible for the
difference in
speed I was noticing between the imaplib.py that was
packaged with older
versions of offlineimap and the one that comes with python:

* /offlineimap/head: changeset 169
  More optimizations -- this time fix readline() to not work
character-
  by-character!

-Aaron

On Thu February 14 2008 3:03:57 pm Aaron Kaplan wrote:
> The following changeset is the one responsible for the
difference in
> speed I was noticing between the imaplib.py that was
packaged with older
> versions of offlineimap and the one that comes with
python:
>
> * /offlineimap/head: changeset 169
>   More optimizations -- this time fix readline() to not
work character-
>   by-character!
>
> -Aaron

Ah.  I remember that one.  Huge difference indeed.  Do you
feel up to working 
with upstream Python to adopt that patch?

On Thu, Feb 14, 2008 at 03:58:52PM -0600, John Goerzen
wrote:
> On Thu February 14 2008 3:03:57 pm Aaron Kaplan wrote:
> > The following changeset is the one responsible for
the difference in
> > speed I was noticing between the imaplib.py that
was packaged with older
> > versions of offlineimap and the one that comes
with python:
> >
> > * /offlineimap/head: changeset 169
> >   More optimizations -- this time fix readline()
to not work character-
> >   by-character!
> >
> > -Aaron
> 
> Ah.  I remember that one.  Huge difference indeed.  Do
you feel up to working 
> with upstream Python to adopt that patch?

Aaron,

I have checked in a fix (err, workaround) for this here:

  http://git.complete.org/o
fflineimap?a=commit;h=1ebc45d96354cdc5b76e57d2ded17502e882b8
aa

You can get the raw diff here:

  http://git.comp
lete.org/offlineimap?a=commitdiff_plain;h=1ebc45d96354cdc5b7
6e57d2ded17502e882b8aa

Can you let me know if that fixes this for you?

On Mon, 3 Mar 2008 08:26:42 -0600
John Goerzen <jgoerzencomplete.org> wrote:

> I have checked in a fix (err, workaround) for this
here:
> 
>   http://git.complete.org/o
fflineimap?a=commit;h=1ebc45d96354cdc5b76e57d2ded17502e882b8
aa
> 
> Can you let me know if that fixes this for you?

It does indeed.

To answer your previous question, yes I can submit the
original patch
upstream.  Just haven't gotten around to it yet.

-Aaron

On Mon March 3 2008 2:42:09 pm Aaron Kaplan wrote:
> On Mon, 3 Mar 2008 08:26:42 -0600
>
> John Goerzen <jgoerzencomplete.org> wrote:
> > I have checked in a fix (err, workaround) for this
here:
> >
> >  
> > http://git.complete.org/offlineimap?a=co
mmit;h=1ebc45d96354cdc5b76e57d2d
> >ed17502e882b8aa
> >
> > Can you let me know if that fixes this for you?
>
> It does indeed.
>
> To answer your previous question, yes I can submit the
original patch
> upstream.  Just haven't gotten around to it yet.

Excellent on both fronts.

That patch made it into 5.99.5, so that should work out well
for folks.

-- John

I filed the imaplib patch here:
http://bugs.python.o
rg/issue2228

-Aaron