multi-threading with gcc43 openmp on dual core

I am messing around with gcc43's openmp capabilities on an
IBM Thinkcentre, which has a dual core Intel vPro.  I am
running FreeBSD 6.2 with SMP.

At this stage in the game I am trying to push around the
limits of the systems capabilities, and I was wondering if
FreeBSD attempts to offload some of the threads to the 2nd
core.  I admittedly don't know the chip architecure or how
FreeBSD handles man threads in SMP, but I'd like too, which
is why I am playing with this.

As the number of threads increases, one core seems to be
oversubscribed according to top (i.e., wcpu > 100%) yet
the system is said to be 50% idle (+/- a few points). The
load average also seems to not get much higher than 1.0.
While I am happy with the load handling, I am wondering:

1. why are threads not evenly distributed b/w CPU0 and CPU1
2. is there a way to explicitly facilitate this thread
balancing (don't know if this is related to CPU affinity)
3. what is a good way to push the system? 

The program I am running spawns an increasing number of
threads from 1 to OMP_NUM_THREADS, where each thread has its
own loop that simply assigns the sum of j and i to k. My
OpenMP-foo is not advanced, but it's steadily getting there.


Here is the code I am using right now:

#include <stdio.h>
int main (int argc, char *argv[]) {
 int i,j,k,m,n;
 int MAX=500;
 n=omp_get_max_threads();
 for (m=1;m<=n;m++) {
   k=0;
   omp_set_num_threads(m);
   #pragma omp parallel private(i,j) shared(MAX)
reduction(+:k)
   { for (i=0;i<MAX;i++) {
       for (j=0;j<MAX;j++) {
         k = j+i;
       }
       #pragma omp barrier
     }
   }
   printf("%8d threads: %dn",m,k);
 }
 return 0;
}

It is compiled and executed with the following command:

% gcc43 -fopenmp test.c && export
OMP_NUM_THREADS=1500 && ./a.out

Any thoughts?

Once I get familiar with what 6.x is doing, I plan on
putting 7.0 on this system to see how it compares.
Ultimately, I'd like to a straightforward code I can use to
evaluate how a particular SMP system running FreeBSD scales.
 I know it is not a "real" workload, but it would
give /me some information that I'd find useful.

TIA && Cheers,
Brett

-- 
HPC Enablement Group
Louisiana Optical Network Initiative
http://www.loni.org
225-578-1920

_______________________________________________
freebsd-smpfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-smp
To unsubscribe, send any mail to
"freebsd-smp-unsubscribefreebsd.org"

B. Estrade wrote:
> I am messing around with gcc43's openmp capabilities on
an IBM Thinkcentre, which has a dual core Intel vPro.  I am
running FreeBSD 6.2 with SMP.
> 
> At this stage in the game I am trying to push around
the limits of the systems capabilities, and I was wondering
if FreeBSD attempts to offload some of the threads to the
2nd core.  I admittedly don't know the chip architecure or
how FreeBSD handles man threads in SMP, but I'd like too,
which is why I am playing with this.
> 
> As the number of threads increases, one core seems to
be oversubscribed according to top (i.e., wcpu > 100%)
yet the system is said to be 50% idle (+/- a few points).
The load average also seems to not get much higher than 1.0.
While I am happy with the load handling, I am wondering:
> 
> 1. why are threads not evenly distributed b/w CPU0 and
CPU1
> 2. is there a way to explicitly facilitate this thread
balancing (don't know if this is related to CPU affinity)
> 3. what is a good way to push the system? 
> 
> The program I am running spawns an increasing number of
threads from 1 to OMP_NUM_THREADS, where each thread has its
own loop that simply assigns the sum of j and i to k. My
OpenMP-foo is not advanced, but it's steadily getting there.

> 
> Here is the code I am using right now:
> 
> #include <stdio.h>
> int main (int argc, char *argv[]) {
>  int i,j,k,m,n;
>  int MAX=500;
>  n=omp_get_max_threads();
>  for (m=1;m<=n;m++) {
>    k=0;
>    omp_set_num_threads(m);
>    #pragma omp parallel private(i,j) shared(MAX)
reduction(+:k)
>    { for (i=0;i<MAX;i++) {
>        for (j=0;j<MAX;j++) {
>          k = j+i;
>        }
>        #pragma omp barrier
>      }
>    }
>    printf("%8d threads: %dn",m,k);
>  }
>  return 0;
> }
> 
> It is compiled and executed with the following
command:
> 
> % gcc43 -fopenmp test.c && export
OMP_NUM_THREADS=1500 && ./a.out
> 
> Any thoughts?
> 
> Once I get familiar with what 6.x is doing, I plan on
putting 7.0 on this system to see how it compares.
Ultimately, I'd like to a straightforward code I can use to
evaluate how a particular SMP system running FreeBSD scales.
 I know it is not a "real" workload, but it would
give /me some information that I'd find useful.
> 
> TIA && Cheers,
> Brett
> 

FreeBSD uses all CPUs for running processes and threads. 
This relies on 
your program being sufficiently parallel though.  I don't
know enough 
about openmp but maybe it is not able to parallelize your
workload very 
well.

Kris
_______________________________________________
freebsd-smpfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-smp
To unsubscribe, send any mail to
"freebsd-smp-unsubscribefreebsd.org"

On Wed, Oct 17, 2007 at 07:31:05PM +0200, Kris Kennaway
wrote:
> B. Estrade wrote:
snip
> >1. why are threads not evenly distributed b/w CPU0
and CPU1
> >2. is there a way to explicitly facilitate this
thread balancing (don't 
> >know if this is related to CPU affinity)
> >3. what is a good way to push the system? 
snip
> >The program I am running spawns an increasing
number of threads from 1 to 
> >OMP_NUM_THREADS, where each thread has its own loop
that simply assigns 
> >the sum of j and i to k. My OpenMP-foo is not
advanced, but it's steadily 
> >getting there. 
snip
> >Brett
> >
> 
> FreeBSD uses all CPUs for running processes and
threads.  This relies on 
> your program being sufficiently parallel though.  I
don't know enough 
> about openmp but maybe it is not able to parallelize
your workload very 
> well.

Thanks, Kris.  This is what I am trying to determine. After
some more observations, it seems as if there is some
swapping of duties between CPU0 and CPU1 - but only one of
them has the entire set of threads at any one point in time,
while the other is idle.  I am execeuting a code that uses
only explicitly declared private variables, and this
includes taking out the reduction shown in the code above
from my original email. It should be obvious to the compiler
and OS that these threads are not dependent on each other in
any way. 

I am going to look deeper into this - but I am still very
interested in anything that anyone wants to say.  I think my
question can be best summed up as: how does one make it
obvious to FreeBSD that all threads should be shared among
all CPUs equally? I don't think this is an OpenMP specific
question - I am just using it as a way to generate the
threading...is it unreasonable of me to expect that all
threads should be shared by all CPUs?

Thanks,
Brett

> 
> Kris

-- 
HPC Enablement Group
Louisiana Optical Network Initiative
http://www.loni.org
225-578-1920

_______________________________________________
freebsd-smpfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-smp
To unsubscribe, send any mail to
"freebsd-smp-unsubscribefreebsd.org"

B. Estrade wrote:
> On Wed, Oct 17, 2007 at 07:31:05PM +0200, Kris Kennaway
wrote:
>> B. Estrade wrote:
> snip
>>> 1. why are threads not evenly distributed b/w
CPU0 and CPU1
>>> 2. is there a way to explicitly facilitate this
thread balancing (don't 
>>> know if this is related to CPU affinity)
>>> 3. what is a good way to push the system? 
> snip
>>> The program I am running spawns an increasing
number of threads from 1 to 
>>> OMP_NUM_THREADS, where each thread has its own
loop that simply assigns 
>>> the sum of j and i to k. My OpenMP-foo is not
advanced, but it's steadily 
>>> getting there. 
> snip
>>> Brett
>>>
>> FreeBSD uses all CPUs for running processes and
threads.  This relies on 
>> your program being sufficiently parallel though.  I
don't know enough 
>> about openmp but maybe it is not able to
parallelize your workload very 
>> well.
> 
> Thanks, Kris.  This is what I am trying to determine.
After some more observations, it seems as if there is some
swapping of duties between CPU0 and CPU1 - but only one of
them has the entire set of threads at any one point in time,
while the other is idle.  I am execeuting a code that uses
only explicitly declared private variables, and this
includes taking out the reduction shown in the code above
from my original email. It should be obvious to the compiler
and OS that these threads are not dependent on each other in
any way. 
> 
> I am going to look deeper into this - but I am still
very interested in anything that anyone wants to say.  I
think my question can be best summed up as: how does one
make it obvious to FreeBSD that all threads should be shared
among all CPUs equally? I don't think this is an OpenMP
specific question - I am just using it as a way to generate
the threading...is it unreasonable of me to expect that all
threads should be shared by all CPUs?

I dunno if there is something special you have to do with
OpenMP, with 
pthreads there is nothing special one needs to do, apart
from making 
sure your code is such that multiple threads can actually
run at once 
(i.e. they are not blocking each other for shared
resources).

Kris

P.S. Please wrap your lines, it makes your emails hard to
read.

_______________________________________________
freebsd-smpfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-smp
To unsubscribe, send any mail to
"freebsd-smp-unsubscribefreebsd.org"

On Wed, Oct 17, 2007 at 08:34:07PM +0200, Kris Kennaway
wrote:
> I dunno if there is something special you have to do
with OpenMP, with 
> pthreads there is nothing special one needs to do,
apart from making 
> sure your code is such that multiple threads can
actually run at once 
> (i.e. they are not blocking each other for shared
resources).

I am using OpenMP because it is easy and coincides with some
stuff
I am already doing.  I will take a look at Pthreads since my
purpose
is also to see how things work.  Thanks for the suggestion.

I'm going back my cave for a while to take a look at a few
things. If 
I have any FreeBSD specific questions or find something that
I think is
interestig, I'll stick my head out again.

Hope the lines are better now 

Cheers,
Brett

> 
> 
> Kris
> 
> P.S. Please wrap your lines, it makes your emails hard
to read.
> 

-- 
HPC Enablement Group
Louisiana Optical Network Initiative
http://www.loni.org
225-578-1920

_______________________________________________
freebsd-smpfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-smp
To unsubscribe, send any mail to
"freebsd-smp-unsubscribefreebsd.org"