Ignoring -jN when linking executable targets?

Hi,

Does anyone know why bjam ignores the -jN option if the only
targets  
to build are executables to link?  In this case you can see
that  
there are 38 targets to build, all of them executables to
link, and,  
you'll have to take my word on this, it serializes all the
links even  
though I asked for a parallel (-j2) build.

-bash-3.00$ bjam --v2 -j2
--infinibanddir=/apps/x86_64/mpi/mvapich/ 
gcc-3.4.6/mvapich-0.9.8-ofed --builddir=/tmp/kbelco/nightly
--g77dir=/ 
usr/bin/g77 --mkldir=/projects/global/x86_64/libraries/ 
l_mkl_p_8.0.1.006/lib/em64t --address-model=64 gcc-3.4.6
link=static  
debug
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...patience...
...found 40708 targets...
...updating 38 targets...
gcc.link
/tmp/kbelco/nightly/adagio/votd/gcc-3.4.6/debug/address- 
model-64/blas-mkl/link-static/mpi-infiniband/adagio

Has anyone seen this before or is this a known feature?

Thanks,

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

Hi Noel !

On Friday 22 December 2006 19:31, K. Noel Belcourt wrote:
> Hi,
>
> Does anyone know why bjam ignores the -jN option if the
only targets
> to build are executables to link?  

Yes.

>  it serializes all the links even  
> though I asked for a parallel (-j2) build.

Well, -j2 is just mild parallel )

> Has anyone seen this before or is this a known feature?

It is a feature, definetely. 

A look at gcc.jam at line 535 reveals:
---
    # Serialize execution of the 'link' action, since
    # running N links in parallel is just slower.
    # For now, serialize only gcc links, it might be a good
    # idea to serialize all links.
    JAM_SEMAPHORE on $(targets) =
<s>gcc-link-semaphore ;
---

which should make things clearer. 
This statement is especially true when doing -j10 and higher
)
Using our compile farm, I can use -j17 and believe me,
having 17 linker 
instances  trying to link 17 different executable is no fun
:-((. Been 
there the semaphore was added.  But you can try and remove
it and 
measure if things go faster for you.
I think that running #number of processor linker instances
on modern 
multi -core systems might be better, though. But not more,
definetely.
But I think this is for 1.35. 

Yours,

Jürgen 
-- 
* Dipl.-Math. Jürgen Hunold  ! Ingenieurgesellschaft für 
* voice: ++49 511 262926 57  ! Verkehrs- und Eisenbahnwesen
mbH  
* fax  : ++49 511 262926 99  ! Lister Straße 15
* juergen.hunoldivembh.de   ! www.ivembh.de

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

On Dec 22, 2006, at 12:00 PM, Juergen Hunold wrote:

>>  it serializes all the links even
>> though I asked for a parallel (-j2) build.
>
> Well, -j2 is just mild parallel )

True, but that's just on our old dual-core linux desktop
machines.

>> Has anyone seen this before or is this a known
feature?
>
> It is a feature, definetely.
>
> A look at gcc.jam at line 535 reveals:
> ---
>     # Serialize execution of the 'link' action, since
>     # running N links in parallel is just slower.
>     # For now, serialize only gcc links, it might be a
good
>     # idea to serialize all links.

Based on our experience with our Sun, Ibm and Sgi clusters, 

serializing all links would be a mistake.  We routinely link
-j8 to - 
j16 on these systems without noticeable performance
degradation.

>     JAM_SEMAPHORE on $(targets) =
<s>gcc-link-semaphore ;
> ---
>
> which should make things clearer.
> This statement is especially true when doing -j10 and
higher )
> Using our compile farm, I can use -j17 and believe me,
having 17  
> linker
> instances  trying to link 17 different executable is no
fun :-((. Been
> there the semaphore was added.  But you can try and
remove it and
> measure if things go faster for you.
> I think that running #number of processor linker
instances on modern
> multi -core systems might be better, though. But not
more, definetely.

Awesome, thanks Jürgen.

We need 2 Gb per application being linked so our 8Gb dual
duo-cores  
can definitely link 4 at a time.  We'll play around with
this and see  
how things behave.

Thanks for the help.

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build