Performance Update: Why is Chandler slow on Mac?

More numbers. On my dual 2GHz G5 I got 31,446. I then ran it
again in  
python without quitting and got 28,736. Again: 29069.

On my 1.6GHz PowerBook: 27,778. Then I realized I was using
/opt/ 
local/bin/python (v2.4.2, built with GCC 4), so I tried
again with / 
usr/bin/python (v2.3.5, built with GCC 3.3) and got 24,510.
Huh. The  
G5 was using 2.4.2/GCC4.

So these numbers have a fuzz factor of about 3% or so, and
are highly  
impacted by build environment and/or Python version. I am
assuming  
John's numbers are from using GCC 4 and Python 2.4.2.

I threw these numbers into Excel and added a
"stones/GHz" field:

                      PyStone Benchmarks
        System            GHz   CPU results stones/GHz
        1.2Ghz iBook      1.2   G4   18939   15782.5
        3GHz P4 XP        3.0   P4   39463   13154.3
        3GHz P4 FC4       3.0   P4   27932    9310.7
        1.67GHz PowerBook 1.67  G4   27778   16633.5
        2x2GHz G5         2.0   G5   31446   15723.0

When you factor in the actual speed of the CPU, it looks
like the G4  
CPUs in the PowerBook and iBook fare the best, with the
Fedora Core  
3GHz P4 the worst. I would expect an AMD CPU to outshine the
G4's.

These numbers don't really affect the final conclusion all
that much  
- that Python is CPU limited. And with Apple's switch to
Intel, the  
differences amongst OS X, XP and Linux should fade.

Reid


On Mar 8, 2006, at 18:50, John Townsend wrote:
> Hi John,
>
> I actually had access to an Intel iMac today so I ran
your python  
> test on the machine.
>
> Intel-based iMac (running 1.83 GHz Intel Dual Core)
with 1GB of RAM.
>
> The results: 31250 pystones/sec.
>
> By comparison, G4 based PowerBook (1.67GHz PowerPC G4)
with 1GB of  
> RAM:
>
> The results: 22831 pystones/sec.
>
> Just thought you might be interested.
>
> --> towns
>
> On Mar 8, 2006, at 3:09 PM, John Anderson wrote:
>
>> During the sprints at PyCon I worked on
Performance, focusing on  
>> trying to understand why Chandler is slower on Mac
than other  
>> platforms. Preliminary results seem to indicate
that OSX Macs are  
>> just overall slower than Windows and Linux PCs.
Read on for the  
>> nitty-gritty details.
>>
>> Launching Chandler with a preexisting repository --
the normal  
>> user experience -- on the 1.2Ghz OSX iBook G4 I had
at PyCon took  
>> 14 seconds compared to about 4 seconds on my 3Ghz
P3 Shuttle  
>> running XP or Fedora Core3. Alec's XP Pentium M
laptop was not far  
>> behind the 4 second mark.
>>
>> The 3.5 times slowdown in launch translated into a
significant  
>> loss of productivity for me. When developing
Chandler I usually  
>> run in WingIDE. On my 3Ghz P4 shuttle, launching
Chandler in Wing  
>> with an existing repository takes 12 seconds and
launching while  
>> creating a new repository takes 36 seconds.
Multiplying those  
>> times by 3.5 made the iBook too slow for
development. This  
>> probably explains why Macintosh developers avoid
using Wing or  
>> even running Chandler whenever possible.
>>
>> Next I decided to run some benchmarks to see how C
and Python  
>> performance compared on the different machines.
Anyone who has run  
>> benchmarks on modern multitasking operating systems
knows that  
>> running the same benchmark can sometimes vary as
much as a factor  
>> of two on the same machine for a variety of
reasons. Sometimes  
>> it's because some other task is using the CPU.
Sometimes the  
>> machine saves power by slowing down the processor.
I ran all the  
>> tests several times to try to get representative
numbers.
>>
>> The first test I ran was Python's pystone, which I
ran as follows:
>>
>> from test import pystone
>> pystone.pystones (loops=pystone.LOOPS)
>>
>> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did
39,463. 3Ghz P4  
>> Fedora Core 4 did 27,932. XP Pentium M was
comparable to the 3Ghz  
>> P4 XP result. So on first glance, Python
performance seems twice  
>> as slow on the iBook compared to the XP machines.
>>
>> Running the Python "make test" on the
iBook vs. XP Pentium M  
>> produced a similar XP speed advantage of roughly a
factor of 2.
>>
>> Since Fedora and OSX use the same gcc compiler, yet
XP is 1.4  
>> times faster than Fedora on the same hardware, it
looks like a big  
>> part of the difference in performance is explained
by the  
>> Microsoft compiler. Not only does the Microsoft
compiler produce  
>> much better code, it runs significantly faster --
Bear mentioned  
>> that builds on the iBook are about 5 times slower
than Windows.
>>
>> I have a 10,000 line C program I wrote years ago
that I use for  
>> performance measures on different processors. It's
an interpreted  
>> language, not unlike Python, which I suspect
compares to the kind  
>> of C code that makes up Chandler. The iBook did
141K instructions/ 
>> sec. 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did 
800K/sec.  
>> Pentium M XP did about 1000K/sec. A 1Ghz PowerBook
did about 300K/ 
>> sec. The first NeXT machine did about 30K/sec. So
in this test the  
>> iBook was almost 9 times slower than Intel/XP. The
Microsoft  
>> compiler advantage, 1.5 times speed up, was
comparable to the  
>> pystone tests -- even using -O3, the highest level
of optimization  
>> for the gcc builds.
>>
>> Alec pointed out a C benchmark, GeekBench (http:// 
>> www.geekpatrol.ca/geekbench/) which has lots of
tests that measure  
>> the speed of different tasks on a variety of
processors. Although  
>> their tests didn't compare the same processors I
tested, the most  
>> varied CPU test gave the Microsoft/Intel vs OSX/G4
a 2 to 4 speed  
>> advantage. The exciting news, however, is that the
iMac Core Duo  
>> 2.0 was roughly equal to the best Microsoft/Intel
machines.
>>
>> Here are my conclusions: Although these results
don't explain the  
>> whole story, the XP/Intel Chandler launch advantage
of 3.5 seems  
>> like it could be entirely be explained by the
difference in C  
>> performance. Furthermore, a significant part of
this advantage is  
>> due to the Microsoft compiler. There are more tests
that would be  
>> interesting to do: profile the C code in Chandler
to see where  
>> time is spent during launch and other time
consuming tasks. Run  
>> tests on the Intel Macs. Measure how much IO is
involved running  
>> Chandler. Construct some Berkeley DB benchmarks and
measure them  
>> on the different platforms. Finally, Given Moore's
law, the  
>> current state of Chandler, and its probable
lifetime (assuming  
>> it's successful), performance probably won't be a
significant  
>> obstacle to adoption.
>>
>> John

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

Reid Ellis wrote:
> More numbers. On my dual 2GHz G5 I got 31,446. I then
ran it again in 
> python without quitting and got 28,736. Again: 29069.
>
> On my 1.6GHz PowerBook: 27,778. Then I realized I was
using 
> /opt/local/bin/python (v2.4.2, built with GCC 4), so I
tried again 
> with /usr/bin/python (v2.3.5, built with GCC 3.3) and
got 24,510. Huh. 
> The G5 was using 2.4.2/GCC4.
>
> So these numbers have a fuzz factor of about 3% or so,
and are highly 
> impacted by build environment and/or Python version. I
am assuming 
> John's numbers are from using GCC 4 and Python 2.4.2.
I used Chandler's latest release build of Python for all my
tests.
>
> I threw these numbers into Excel and added a
"stones/GHz" field:
>
>                      PyStone Benchmarks
>        System            GHz   CPU results stones/GHz
>        1.2Ghz iBook      1.2   G4   18939   15782.5
>        3GHz P4 XP        3.0   P4   39463   13154.3
>        3GHz P4 FC4       3.0   P4   27932    9310.7
>        1.67GHz PowerBook 1.67  G4   27778   16633.5
>        2x2GHz G5         2.0   G5   31446   15723.0
>
> When you factor in the actual speed of the CPU, it
looks like the G4 
> CPUs in the PowerBook and iBook fare the best, with the
Fedora Core 
> 3GHz P4 the worst. I would expect an AMD CPU to
outshine the G4's.
>
> These numbers don't really affect the final conclusion
all that much - 
> that Python is CPU limited. And with Apple's switch to
Intel, the 
> differences amongst OS X, XP and Linux should fade.
>
> Reid
>
>
> On Mar 8, 2006, at 18:50, John Townsend wrote:
>> Hi John,
>>
>> I actually had access to an Intel iMac today so I
ran your python 
>> test on the machine.
>>
>> Intel-based iMac (running 1.83 GHz Intel Dual Core)
with 1GB of RAM.
>>
>> The results: 31250 pystones/sec.
>>
>> By comparison, G4 based PowerBook (1.67GHz PowerPC
G4) with 1GB of RAM:
>>
>> The results: 22831 pystones/sec.
>>
>> Just thought you might be interested.
>>
>> --> towns
>>
>> On Mar 8, 2006, at 3:09 PM, John Anderson wrote:
>>
>>> During the sprints at PyCon I worked on
Performance, focusing on 
>>> trying to understand why Chandler is slower on
Mac than other 
>>> platforms. Preliminary results seem to indicate
that OSX Macs are 
>>> just overall slower than Windows and Linux PCs.
Read on for the 
>>> nitty-gritty details.
>>>
>>> Launching Chandler with a preexisting
repository -- the normal user 
>>> experience -- on the 1.2Ghz OSX iBook G4 I had
at PyCon took 14 
>>> seconds compared to about 4 seconds on my 3Ghz
P3 Shuttle running XP 
>>> or Fedora Core3. Alec's XP Pentium M laptop
was not far behind the 4 
>>> second mark.
>>>
>>> The 3.5 times slowdown in launch translated
into a significant loss 
>>> of productivity for me. When developing
Chandler I usually run in 
>>> WingIDE. On my 3Ghz P4 shuttle, launching
Chandler in Wing with an 
>>> existing repository takes 12 seconds and
launching while creating a 
>>> new repository takes 36 seconds. Multiplying
those times by 3.5 made 
>>> the iBook too slow for development. This
probably explains why 
>>> Macintosh developers avoid using Wing or even
running Chandler 
>>> whenever possible.
>>>
>>> Next I decided to run some benchmarks to see
how C and Python 
>>> performance compared on the different machines.
Anyone who has run 
>>> benchmarks on modern multitasking operating
systems knows that 
>>> running the same benchmark can sometimes vary
as much as a factor of 
>>> two on the same machine for a variety of
reasons. Sometimes it's 
>>> because some other task is using the CPU.
Sometimes the machine 
>>> saves power by slowing down the processor. I
ran all the tests 
>>> several times to try to get representative
numbers.
>>>
>>> The first test I ran was Python's pystone,
which I ran as follows:
>>>
>>> from test import pystone
>>> pystone.pystones (loops=pystone.LOOPS)
>>>
>>> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP
did 39,463. 3Ghz P4 
>>> Fedora Core 4 did 27,932. XP Pentium M was
comparable to the 3Ghz P4 
>>> XP result. So on first glance, Python
performance seems twice as 
>>> slow on the iBook compared to the XP machines.
>>>
>>> Running the Python "make test" on
the iBook vs. XP Pentium M 
>>> produced a similar XP speed advantage of
roughly a factor of 2.
>>>
>>> Since Fedora and OSX use the same gcc compiler,
yet XP is 1.4 times 
>>> faster than Fedora on the same hardware, it
looks like a big part of 
>>> the difference in performance is explained by
the Microsoft 
>>> compiler. Not only does the Microsoft compiler
produce much better 
>>> code, it runs significantly faster -- Bear
mentioned that builds on 
>>> the iBook are about 5 times slower than
Windows.
>>>
>>> I have a 10,000 line C program I wrote years
ago that I use for 
>>> performance measures on different processors.
It's an interpreted 
>>> language, not unlike Python, which I suspect
compares to the kind of 
>>> C code that makes up Chandler. The iBook did
141K instructions/sec. 
>>> 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did 
800K/sec. Pentium M XP 
>>> did about 1000K/sec. A 1Ghz PowerBook did about
300K/sec. The first 
>>> NeXT machine did about 30K/sec. So in this test
the iBook was almost 
>>> 9 times slower than Intel/XP. The Microsoft
compiler advantage, 1.5 
>>> times speed up, was comparable to the pystone
tests -- even using 
>>> -O3, the highest level of optimization for the
gcc builds.
>>>
>>> Alec pointed out a C benchmark, GeekBench 
>>> (http://www.geekpa
trol.ca/geekbench/) which has lots of tests that 
>>> measure the speed of different tasks on a
variety of processors. 
>>> Although their tests didn't compare the same
processors I tested, 
>>> the most varied CPU test gave the
Microsoft/Intel vs OSX/G4 a 2 to 4 
>>> speed advantage. The exciting news, however, is
that the iMac Core 
>>> Duo 2.0 was roughly equal to the best
Microsoft/Intel machines.
>>>
>>> Here are my conclusions: Although these results
don't explain the 
>>> whole story, the XP/Intel Chandler launch
advantage of 3.5 seems 
>>> like it could be entirely be explained by the
difference in C 
>>> performance. Furthermore, a significant part of
this advantage is 
>>> due to the Microsoft compiler. There are more
tests that would be 
>>> interesting to do: profile the C code in
Chandler to see where time 
>>> is spent during launch and other time consuming
tasks. Run tests on 
>>> the Intel Macs. Measure how much IO is involved
running Chandler. 
>>> Construct some Berkeley DB benchmarks and
measure them on the 
>>> different platforms. Finally, Given Moore's
law, the current state 
>>> of Chandler, and its probable lifetime
(assuming it's successful), 
>>> performance probably won't be a significant
obstacle to adoption.
>>>
>>> John
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev"
mailing list
> h
ttp://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

Reid Ellis wrote:
>
> On my 1.6GHz PowerBook: 27,778. Then I realized I was
using 
> /opt/local/bin/python (v2.4.2, built with GCC 4), so I
tried again 
> with /usr/bin/python (v2.3.5, built with GCC 3.3) and
got 24,510. Huh. 
> The G5 was using 2.4.2/GCC4.
>
> So these numbers have a fuzz factor of about 3% or so,
and are highly 
> impacted by build environment and/or Python version. I
am assuming 
> John's numbers are from using GCC 4 and Python 2.4.2.
>
It is important to note that there were some specific
performance 
improvements in Python going from 2.3 -> 2.4, and that
Chandler uses 
some of the new 2.4 syntax and thus requires Python 2.4. The
only real 
differences in 2.4.1/2.4.2 have been security fixes, so
there shouldn't 
be any noticeable performance change between those versions.
Since 
Python is open-source, there isn't really a notion of the 
factory-installed Python being any "better", so
in general its best to 
go off of the python that is built in Chandler.

Alec
> I threw these numbers into Excel and added a
"stones/GHz" field:
>
>                      PyStone Benchmarks
>        System            GHz   CPU results stones/GHz
>        1.2Ghz iBook      1.2   G4   18939   15782.5
>        3GHz P4 XP        3.0   P4   39463   13154.3
>        3GHz P4 FC4       3.0   P4   27932    9310.7
>        1.67GHz PowerBook 1.67  G4   27778   16633.5
>        2x2GHz G5         2.0   G5   31446   15723.0
>
> When you factor in the actual speed of the CPU, it
looks like the G4 
> CPUs in the PowerBook and iBook fare the best, with the
Fedora Core 
> 3GHz P4 the worst. I would expect an AMD CPU to
outshine the G4's.
>
> These numbers don't really affect the final conclusion
all that much - 
> that Python is CPU limited. And with Apple's switch to
Intel, the 
> differences amongst OS X, XP and Linux should fade.
>
> Reid
>
>
> On Mar 8, 2006, at 18:50, John Townsend wrote:
>> Hi John,
>>
>> I actually had access to an Intel iMac today so I
ran your python 
>> test on the machine.
>>
>> Intel-based iMac (running 1.83 GHz Intel Dual Core)
with 1GB of RAM.
>>
>> The results: 31250 pystones/sec.
>>
>> By comparison, G4 based PowerBook (1.67GHz PowerPC
G4) with 1GB of RAM:
>>
>> The results: 22831 pystones/sec.
>>
>> Just thought you might be interested.
>>
>> --> towns
>>
>> On Mar 8, 2006, at 3:09 PM, John Anderson wrote:
>>
>>> During the sprints at PyCon I worked on
Performance, focusing on 
>>> trying to understand why Chandler is slower on
Mac than other 
>>> platforms. Preliminary results seem to indicate
that OSX Macs are 
>>> just overall slower than Windows and Linux PCs.
Read on for the 
>>> nitty-gritty details.
>>>
>>> Launching Chandler with a preexisting
repository -- the normal user 
>>> experience -- on the 1.2Ghz OSX iBook G4 I had
at PyCon took 14 
>>> seconds compared to about 4 seconds on my 3Ghz
P3 Shuttle running XP 
>>> or Fedora Core3. Alec's XP Pentium M laptop
was not far behind the 4 
>>> second mark.
>>>
>>> The 3.5 times slowdown in launch translated
into a significant loss 
>>> of productivity for me. When developing
Chandler I usually run in 
>>> WingIDE. On my 3Ghz P4 shuttle, launching
Chandler in Wing with an 
>>> existing repository takes 12 seconds and
launching while creating a 
>>> new repository takes 36 seconds. Multiplying
those times by 3.5 made 
>>> the iBook too slow for development. This
probably explains why 
>>> Macintosh developers avoid using Wing or even
running Chandler 
>>> whenever possible.
>>>
>>> Next I decided to run some benchmarks to see
how C and Python 
>>> performance compared on the different machines.
Anyone who has run 
>>> benchmarks on modern multitasking operating
systems knows that 
>>> running the same benchmark can sometimes vary
as much as a factor of 
>>> two on the same machine for a variety of
reasons. Sometimes it's 
>>> because some other task is using the CPU.
Sometimes the machine 
>>> saves power by slowing down the processor. I
ran all the tests 
>>> several times to try to get representative
numbers.
>>>
>>> The first test I ran was Python's pystone,
which I ran as follows:
>>>
>>> from test import pystone
>>> pystone.pystones (loops=pystone.LOOPS)
>>>
>>> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP
did 39,463. 3Ghz P4 
>>> Fedora Core 4 did 27,932. XP Pentium M was
comparable to the 3Ghz P4 
>>> XP result. So on first glance, Python
performance seems twice as 
>>> slow on the iBook compared to the XP machines.
>>>
>>> Running the Python "make test" on
the iBook vs. XP Pentium M 
>>> produced a similar XP speed advantage of
roughly a factor of 2.
>>>
>>> Since Fedora and OSX use the same gcc compiler,
yet XP is 1.4 times 
>>> faster than Fedora on the same hardware, it
looks like a big part of 
>>> the difference in performance is explained by
the Microsoft 
>>> compiler. Not only does the Microsoft compiler
produce much better 
>>> code, it runs significantly faster -- Bear
mentioned that builds on 
>>> the iBook are about 5 times slower than
Windows.
>>>
>>> I have a 10,000 line C program I wrote years
ago that I use for 
>>> performance measures on different processors.
It's an interpreted 
>>> language, not unlike Python, which I suspect
compares to the kind of 
>>> C code that makes up Chandler. The iBook did
141K instructions/sec. 
>>> 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did 
800K/sec. Pentium M XP 
>>> did about 1000K/sec. A 1Ghz PowerBook did about
300K/sec. The first 
>>> NeXT machine did about 30K/sec. So in this test
the iBook was almost 
>>> 9 times slower than Intel/XP. The Microsoft
compiler advantage, 1.5 
>>> times speed up, was comparable to the pystone
tests -- even using 
>>> -O3, the highest level of optimization for the
gcc builds.
>>>
>>> Alec pointed out a C benchmark, GeekBench 
>>> (http://www.geekpa
trol.ca/geekbench/) which has lots of tests that 
>>> measure the speed of different tasks on a
variety of processors. 
>>> Although their tests didn't compare the same
processors I tested, 
>>> the most varied CPU test gave the
Microsoft/Intel vs OSX/G4 a 2 to 4 
>>> speed advantage. The exciting news, however, is
that the iMac Core 
>>> Duo 2.0 was roughly equal to the best
Microsoft/Intel machines.
>>>
>>> Here are my conclusions: Although these results
don't explain the 
>>> whole story, the XP/Intel Chandler launch
advantage of 3.5 seems 
>>> like it could be entirely be explained by the
difference in C 
>>> performance. Furthermore, a significant part of
this advantage is 
>>> due to the Microsoft compiler. There are more
tests that would be 
>>> interesting to do: profile the C code in
Chandler to see where time 
>>> is spent during launch and other time consuming
tasks. Run tests on 
>>> the Intel Macs. Measure how much IO is involved
running Chandler. 
>>> Construct some Berkeley DB benchmarks and
measure them on the 
>>> different platforms. Finally, Given Moore's
law, the current state 
>>> of Chandler, and its probable lifetime
(assuming it's successful), 
>>> performance probably won't be a significant
obstacle to adoption.
>>>
>>> John
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev"
mailing list
> h
ttp://lists.osafoundation.org/mailman/listinfo/dev

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

During the sprints at PyCon I worked on Performance,
focusing on trying 
to understand why Chandler is slower on Mac than other
platforms. 
Preliminary results seem to indicate that OSX Macs are just
overall 
slower than Windows and Linux PCs. Read on for the
nitty-gritty details.

Launching Chandler with a preexisting repository -- the
normal user 
experience -- on the 1.2Ghz OSX iBook G4 I had at PyCon took
14 seconds 
compared to about 4 seconds on my 3Ghz P3 Shuttle running XP
or Fedora 
Core3. Alec's XP Pentium M laptop was not far behind the 4
second mark.

The 3.5 times slowdown in launch translated into a
significant loss of 
productivity for me. When developing Chandler I usually run
in WingIDE. 
On my 3Ghz P4 shuttle, launching Chandler in Wing with an
existing 
repository takes 12 seconds and launching while creating a
new 
repository takes 36 seconds. Multiplying those times by 3.5
made the 
iBook too slow for development. This probably explains why
Macintosh 
developers avoid using Wing or even running Chandler
whenever possible.

Next I decided to run some benchmarks to see how C and
Python 
performance compared on the different machines. Anyone who
has run 
benchmarks on modern multitasking operating systems knows
that running 
the same benchmark can sometimes vary as much as a factor of
two on the 
same machine for a variety of reasons. Sometimes it's
because some other 
task is using the CPU. Sometimes the machine saves power by
slowing down 
the processor. I ran all the tests several times to try to
get 
representative numbers.

The first test I ran was Python's pystone, which I ran as
follows:

from test import pystone
pystone.pystones (loops=pystone.LOOPS)

The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did 39,463.
3Ghz P4 Fedora 
Core 4 did 27,932. XP Pentium M was comparable to the 3Ghz
P4 XP result. 
So on first glance, Python performance seems twice as slow
on the iBook 
compared to the XP machines.

Running the Python "make test" on the iBook vs.
XP Pentium M produced a 
similar XP speed advantage of roughly a factor of 2.

Since Fedora and OSX use the same gcc compiler, yet XP is
1.4 times 
faster than Fedora on the same hardware, it looks like a big
part of the 
difference in performance is explained by the Microsoft
compiler. Not 
only does the Microsoft compiler produce much better code,
it runs 
significantly faster -- Bear mentioned that builds on the
iBook are 
about 5 times slower than Windows.

I have a 10,000 line C program I wrote years ago that I use
for 
performance measures on different processors. It's an
interpreted 
language, not unlike Python, which I suspect compares to the
kind of C 
code that makes up Chandler. The iBook did 141K
instructions/sec. 3Ghz 
P4 XP did 1226K/sec. 3Ghz P4 Fedora did  800K/sec. Pentium M
XP did 
about 1000K/sec. A 1Ghz PowerBook did about 300K/sec. The
first NeXT 
machine did about 30K/sec. So in this test the iBook was
almost 9 times 
slower than Intel/XP. The Microsoft compiler advantage, 1.5
times speed 
up, was comparable to the pystone tests -- even using -O3,
the highest 
level of optimization for the gcc builds.

Alec pointed out a C benchmark, GeekBench 
(http://www.geekpa
trol.ca/geekbench/) which has lots of tests that 
measure the speed of different tasks on a variety of
processors. 
Although their tests didn't compare the same processors I
tested, the 
most varied CPU test gave the Microsoft/Intel vs OSX/G4 a 2
to 4 speed 
advantage. The exciting news, however, is that the iMac Core
Duo 2.0 was 
roughly equal to the best Microsoft/Intel machines.

Here are my conclusions: Although these results don't
explain the whole 
story, the XP/Intel Chandler launch advantage of 3.5 seems
like it could 
be entirely be explained by the difference in C performance.

Furthermore, a significant part of this advantage is due to
the 
Microsoft compiler. There are more tests that would be
interesting to 
do: profile the C code in Chandler to see where time is
spent during 
launch and other time consuming tasks. Run tests on the
Intel Macs. 
Measure how much IO is involved running Chandler. Construct
some 
Berkeley DB benchmarks and measure them on the different
platforms. 
Finally, Given Moore's law, the current state of Chandler,
and its 
probable lifetime (assuming it's successful), performance
probably won't 
be a significant obstacle to adoption.

John

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

Hi John,

I actually had access to an Intel iMac today so I ran your
python  
test on the machine.

Intel-based iMac (running 1.83 GHz Intel Dual Core) with 1GB
of RAM.

The results: 31250 pystones/sec.

By comparison, G4 based PowerBook (1.67GHz PowerPC G4) with
1GB of RAM:

The results: 22831 pystones/sec.

Just thought you might be interested.

--> towns

On Mar 8, 2006, at 3:09 PM, John Anderson wrote:

> During the sprints at PyCon I worked on Performance,
focusing on  
> trying to understand why Chandler is slower on Mac than
other  
> platforms. Preliminary results seem to indicate that
OSX Macs are  
> just overall slower than Windows and Linux PCs. Read on
for the  
> nitty-gritty details.
>
> Launching Chandler with a preexisting repository -- the
normal user  
> experience -- on the 1.2Ghz OSX iBook G4 I had at PyCon
took 14  
> seconds compared to about 4 seconds on my 3Ghz P3
Shuttle running  
> XP or Fedora Core3. Alec's XP Pentium M laptop was not
far behind  
> the 4 second mark.
>
> The 3.5 times slowdown in launch translated into a
significant loss  
> of productivity for me. When developing Chandler I
usually run in  
> WingIDE. On my 3Ghz P4 shuttle, launching Chandler in
Wing with an  
> existing repository takes 12 seconds and launching
while creating a  
> new repository takes 36 seconds. Multiplying those
times by 3.5  
> made the iBook too slow for development. This probably
explains why  
> Macintosh developers avoid using Wing or even running
Chandler  
> whenever possible.
>
> Next I decided to run some benchmarks to see how C and
Python  
> performance compared on the different machines. Anyone
who has run  
> benchmarks on modern multitasking operating systems
knows that  
> running the same benchmark can sometimes vary as much
as a factor  
> of two on the same machine for a variety of reasons.
Sometimes it's  
> because some other task is using the CPU. Sometimes the
machine  
> saves power by slowing down the processor. I ran all
the tests  
> several times to try to get representative numbers.
>
> The first test I ran was Python's pystone, which I ran
as follows:
>
> from test import pystone
> pystone.pystones (loops=pystone.LOOPS)
>
> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did
39,463. 3Ghz P4  
> Fedora Core 4 did 27,932. XP Pentium M was comparable
to the 3Ghz  
> P4 XP result. So on first glance, Python performance
seems twice as  
> slow on the iBook compared to the XP machines.
>
> Running the Python "make test" on the iBook
vs. XP Pentium M  
> produced a similar XP speed advantage of roughly a
factor of 2.
>
> Since Fedora and OSX use the same gcc compiler, yet XP
is 1.4 times  
> faster than Fedora on the same hardware, it looks like
a big part  
> of the difference in performance is explained by the
Microsoft  
> compiler. Not only does the Microsoft compiler produce
much better  
> code, it runs significantly faster -- Bear mentioned
that builds on  
> the iBook are about 5 times slower than Windows.
>
> I have a 10,000 line C program I wrote years ago that I
use for  
> performance measures on different processors. It's an
interpreted  
> language, not unlike Python, which I suspect compares
to the kind  
> of C code that makes up Chandler. The iBook did 141K
instructions/ 
> sec. 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did 
800K/sec.  
> Pentium M XP did about 1000K/sec. A 1Ghz PowerBook did
about 300K/ 
> sec. The first NeXT machine did about 30K/sec. So in
this test the  
> iBook was almost 9 times slower than Intel/XP. The
Microsoft  
> compiler advantage, 1.5 times speed up, was comparable
to the  
> pystone tests -- even using -O3, the highest level of
optimization  
> for the gcc builds.
>
> Alec pointed out a C benchmark, GeekBench (http://www.geekpatrol.ca/ 
> geekbench/) which has lots of tests that measure the
speed of  
> different tasks on a variety of processors. Although
their tests  
> didn't compare the same processors I tested, the most
varied CPU  
> test gave the Microsoft/Intel vs OSX/G4 a 2 to 4 speed
advantage.  
> The exciting news, however, is that the iMac Core Duo
2.0 was  
> roughly equal to the best Microsoft/Intel machines.
>
> Here are my conclusions: Although these results don't
explain the  
> whole story, the XP/Intel Chandler launch advantage of
3.5 seems  
> like it could be entirely be explained by the
difference in C  
> performance. Furthermore, a significant part of this
advantage is  
> due to the Microsoft compiler. There are more tests
that would be  
> interesting to do: profile the C code in Chandler to
see where time  
> is spent during launch and other time consuming tasks.
Run tests on  
> the Intel Macs. Measure how much IO is involved running
Chandler.  
> Construct some Berkeley DB benchmarks and measure them
on the  
> different platforms. Finally, Given Moore's law, the
current state  
> of Chandler, and its probable lifetime (assuming it's
successful),  
> performance probably won't be a significant obstacle
to adoption.
>
> John
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev"
mailing list
> h
ttp://lists.osafoundation.org/mailman/listinfo/dev

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

John Anderson wrote:
> Here are my conclusions: Although these results don't
explain the 
> whole story, the XP/Intel Chandler launch advantage of
3.5 seems like 
> it could be entirely be explained by the difference in
C performance. 
> Furthermore, a significant part of this advantage is
due to the 
> Microsoft compiler. There are more tests that would be
interesting to 
> do: profile the C code in Chandler to see where time is
spent during 
> launch and other time consuming tasks. Run tests on the
Intel Macs. 
> Measure how much IO is involved running Chandler.
Construct some 
> Berkeley DB benchmarks and measure them on the
different platforms. 
> Finally, Given Moore's law, the current state of
Chandler, and its 
> probable lifetime (assuming it's successful),
performance probably 
> won't be a significant obstacle to adoption.
Is it worthwhile trying to compile some of the smaller
benchmarks on 
Windows using gcc to get some idea of how bad off we are
using GCC?    
We could do release builds on the mac using either the IBM
XLC compiler 
or on Mac OS intel using intel's compiler

Ted
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

Ted Leung wrote:
> Is it worthwhile trying to compile some of the smaller
benchmarks on 
> Windows using gcc to get some idea of how bad off we
are using GCC?    
> We could do release builds on the mac using either the
IBM XLC 
> compiler or on Mac OS intel using intel's compiler
>
I'm guessing that trying to get chandler/wx building on a
different 
compiler is going to be a nightmare - not because of
compiler 
incompatibilities, but because getting all the Makefiles to
do what we 
want is going to be extremely tedious.. and its not clear to
me if there 
is any benefit beyond confirming that in fact VC++ is a
better compiler 
than gcc.

It seems like what we've determined is "the python
benchmarks are slower 
on mac than windows because of gcc" - and that this is
probably true for 
all python apps... but Chandler on Linux isn't that bad. 
While I think 
this was a useful exercise so that we can have a baseline to
compare 
windows to mac, it doesn't really explain why the mac is
also much 
faster than Linux. It sounds like we need to start focusing
on 
chandler-specific stuff, specifically wx and berkeleydb.

How about making some benchmarks:
1) Resizing the chandler window (tests wx & wxPython,
shouldn't have 
much repository interference)
2) in headless, loading 3000 items and accessing some values
on them

This might at least give us a clue if the problems are
mostly in the UI, 
or mostly in the database.

[As an aside: when I was doing the calendar performance work
in 0.6, we 
got benefits tweaking both the DB and wx - Andi did the db
work of 
course and I was focused on wx. One thing I found was that
the 
Python-to-C++ layer (SWIG) was a definite bottleneck for
drawing. It 
wouldn't surprise me if the slower gcc is making wx that
much slower on 
the mac because the SWIG'ed code is, of course, C++ and
thus slow. I 
will try to write up a wiki page next week with
"lessons learned" from 
wxPython performance tweaking.]

Alec
> Ted
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev"
mailing list
> h
ttp://lists.osafoundation.org/mailman/listinfo/dev

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

Alec Flett wrote:
> Ted Leung wrote:
>> Is it worthwhile trying to compile some of the
smaller benchmarks on 
>> Windows using gcc to get some idea of how bad off
we are using 
>> GCC?    We could do release builds on the mac using
either the IBM 
>> XLC compiler or on Mac OS intel using intel's
compiler
>>
> I'm guessing that trying to get chandler/wx building
on a different 
> compiler is going to be a nightmare - not because of
compiler 
> incompatibilities, but because getting all the
Makefiles to do what we 
> want is going to be extremely tedious.. and its not
clear to me if 
> there is any benefit beyond confirming that in fact
VC++ is a better 
> compiler than gcc.
>
> It seems like what we've determined is "the
python benchmarks are 
> slower on mac than windows because of gcc" - and
that this is probably 
> true for all python apps... but Chandler on Linux
isn't that bad.  
> While I think this was a useful exercise so that we can
have a 
> baseline to compare windows to mac, it doesn't really
explain why the 
> mac is also much faster than Linux. It sounds like we
need to start 
> focusing on chandler-specific stuff, specifically wx
and berkeleydb.
>
I didn't mean compiling all of chandler using gcc, just the
micro 
benchmarks so that we could see if the perf ratios change
with the 
compiler on the same platform.

Also my experience is the reverse, namely Linux is much
faster than the 
Mac.  Or is that what you meant?   I did some benchmarks on
berkeleydb a 
while back, and perf on mac was definitely not the same as
on the other 
platforms.

Ted
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev

On Fri, 10 Mar 2006, Alec Flett wrote:

> Ted Leung wrote:
>> Is it worthwhile trying to compile some of the
smaller benchmarks on 
>> Windows using gcc to get some idea of how bad off
we are using GCC?    We 
>> could do release builds on the mac using either the
IBM XLC compiler or on 
>> Mac OS intel using intel's compiler
>> 
> I'm guessing that trying to get chandler/wx building
on a different compiler 
> is going to be a nightmare - not because of compiler
incompatibilities, but 
> because getting all the Makefiles to do what we want is
going to be extremely 
> tedious.. and its not clear to me if there is any
benefit beyond confirming 
> that in fact VC++ is a better compiler than gcc.

Yeah, good luck getting the ./configure scripts to produce
working Makefile 
unless these compilers are option-compatible with gcc...
Also, since these compilers are not free (free as in libre)
and probably not 
free (free as in beer), this move wouldn't be well received
in the open source 
world, no ?

> It seems like what we've determined is "the
python benchmarks are slower on 
> mac than windows because of gcc" - and that this
is probably true for all

What we've mostly determined is that the mac, the power-pc
mac, is slower than 
anything else. We've also determined that gcc produces
slower code than vc++.
In either case, no news here. I'd be curious to see how
fast Chandler runs, 
natively, on the new intel macs once we can produce a
functional build there.

> python apps... but Chandler on Linux isn't that bad. 
While I think this was 
> a useful exercise so that we can have a baseline to
compare windows to mac, 
> it doesn't really explain why the mac is also much
faster than Linux. It

Mac faster than linux ?? Are you saying OS X is faster than
Linux on the same 
hardware ? are you saying Mac faster than Intel PC ? My
Linux PC sure is a lot 
faster than my Mac Powerbook...

> sounds like we need to start focusing on
chandler-specific stuff, 
> specifically wx and berkeleydb.

That is not chandler-specific. How about focusing on actual
chandler code such 
as the repository or the chandler app code, like we did
towards the end of the 
0.6 release cycle ?

> How about making some benchmarks:
> 1) Resizing the chandler window (tests wx &
wxPython, shouldn't have much 
> repository interference)

Sure !

> 2) in headless, loading 3000 items and accessing some
values on them

About Berkeley DB on Mac OS X: I had found that it was
slower there than could 
be, in particular with caching added (!) and Sleepycat said
it was a problem 
with the OS. Then versions 4.3.29 and 4.4.20 came out and
suddenly one has to 
use cache or it is 6 times slower, and a little faster
compared to without 
cache in earlier releases. The plot has thickened a bit...

I suspect that the "loading of 3000 items and
accessing some values on them" 
benchmark is going to exercise a lot more repository code
than Berkeley DB 
code. I'm hoping to move more of that code to C for some
significant speed 
improvements during the 0.7 release cycle.

> This might at least give us a clue if the problems are
mostly in the UI, or 
> mostly in the database.

I suspect neither. They're most likely all over the place,
in the 'death by a 
thousand' cuts style. Nothing we can't overcome but no
magic bullet to 
resolve them either.

> [As an aside: when I was doing the calendar performance
work in 0.6, we got 
> benefits tweaking both the DB and wx - Andi did the db
work of course and I 
> was focused on wx.

The biggest speed improvements during 0.6 came from tweaking
the app 
proper, not the db or wx. For example: using indexes to find
relevant calendar 
events within a time interval instead of iterating all
calendar events 
linearly in the entire repository. I suspect there are more
such things to 
come. For instance, we might make some significant
improvements by avoiding 
the use of unbounded collections that have little benefit
except in an empty 
repository such as the 'all' collection.

> One thing I found was that the Python-to-C++ layer
(SWIG) was a definite 
> bottleneck for drawing. It wouldn't surprise me if the
slower

Yes, definitely. The code SWIG generates for wrapping python
is thick, sticky, 
needlessly slow and generally useless. For example, in
PyLucene, the 5000 line 
swig interface file translates into almost 60,000 lines of
C++ and almost 
3,000 lines of Python code. It turns out that *NONE* of the
generated python 
code is needed. All of that can be done in C++, using
Python's C API and C 
types API for the wrappers, thereby cutting out an entire
layer of python and 
python calls.

For PyLucene, I've just embarked on rewriting these
wrappers by hand, using 
only C++, in other words, not using SWIG for PyLucene
anymore, with some very 
encouraging results. Yes, that's a lot of boilerplate but
it's not as bad as I 
had first feared. 

For example, instantiating a simple Java Object instance
from python on my 
1ghz powerbook g4 takes:

> python -m timeit -s "from PyLucene import
Object" "Object()"

with swig 1.3.28:  10000 loops, best of 3: 20.6 usec per
loop
without swig    : 100000 loops, best of 3: 4.19 usec per
loop

Simple instantiation is about 5 times faster without swig...

> gcc is making wx that much slower on the mac because
the SWIG'ed code is, of 
> course, C++ and thus slow. I will try to write up a
wiki page next week with 
> "lessons learned" from wxPython performance
tweaking.]

Andi..
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Open Source Applications Foundation "Dev"
mailing list
h
ttp://lists.osafoundation.org/mailman/listinfo/dev