Warning popping up during -j builds

On Fri, Aug 18, 2006 at 04:31:54PM +0200, Martin Husemann
wrote:
> On Fri, Aug 18, 2006 at 04:07:01PM +0200, Quentin
Garnier wrote:
> > It's a bug in distrib/sets/sets.subr:  it calls
$ without
> > clearing MAKEFLAGS, so it interprets the -J
although the parent of
> > sets.subr isn't aware of that instance of
make(1).
> 
> I don't claim to understand the inner workings of
make, but is it realy
> necessary that -J shows up in MAKEFLAGS at all? It is
kind of an internal
> argument, so keeping it out of the variable sounds like
a good thing.

But how is it supposed to go to a child instance of make,
then?
I don't claim to understand anything about make, let alone
its inner
workings 

Note:  jumping to tech-userlevel.

-- 
Quentin Garnier - cubecubidou.net - cubeNetBSD.org
"When I find the controls, I'll go where I like,
I'll know where I want
to be, but maybe for now I'll stay right here on a silent
sea."
KT Tunstall, Silent Sea, Eye to the Telescope, 2004.

On Fri, Aug 18, 2006 at 04:37:54PM +0200, Quentin Garnier
wrote:
> But how is it supposed to go to a child instance of
make, then?
> I don't claim to understand anything about make, let
alone its inner
> workings 

By make adding it (silently) to the args (but not publishing
that through
MAKEFLAGS) when creating a child make process, if it has
been invoked with -j
or -J itself.

If make invokes a shell script, and that invokes another
make, it is IMHO
not part of the recursive/parallel make tree - so the token
queue should not
be passed on to it.

Martin

On Fri, Aug 18, 2006 at 05:19:02PM +0200, Martin Husemann
wrote:
> On Fri, Aug 18, 2006 at 04:37:54PM +0200, Quentin
Garnier wrote:
> > But how is it supposed to go to a child instance
of make, then?
> > I don't claim to understand anything about make,
let alone its inner
> > workings 
> 
> By make adding it (silently) to the args (but not
publishing that through
> MAKEFLAGS) when creating a child make process, if it
has been invoked with -j
> or -J itself.
> 
> If make invokes a shell script, and that invokes
another make, it is IMHO
> not part of the recursive/parallel make tree - so the
token queue should not
> be passed on to it.

Somehow it makes me think a lot of magic will be required
for make to
know it's invoking make--and moreover, the correct one. 
Also, I think
that the current scheme allow other make programs to
co-operate,
although it might not be an intended feature.

-- 
Quentin Garnier - cubecubidou.net - cubeNetBSD.org
"When I find the controls, I'll go where I like,
I'll know where I want
to be, but maybe for now I'll stay right here on a silent
sea."
KT Tunstall, Silent Sea, Eye to the Telescope, 2004.

Quentin Garnier wrote:

>On Fri, Aug 18, 2006 at 05:19:02PM +0200, Martin
Husemann wrote:
>  
>
>>On Fri, Aug 18, 2006 at 04:37:54PM +0200, Quentin
Garnier wrote:
>>    
>>
>>>But how is it supposed to go to a child instance
of make, then?
>>>I don't claim to understand anything about
make, let alone its inner
>>>workings 
>>>      
>>>
>>By make adding it (silently) to the args (but not
publishing that through
>>MAKEFLAGS) when creating a child make process, if it
has been invoked with -j
>>or -J itself.
>>
>>If make invokes a shell script, and that invokes
another make, it is IMHO
>>not part of the recursive/parallel make tree - so
the token queue should not
>>be passed on to it.
>>    
>>
>
>Somehow it makes me think a lot of magic will be
required for make to
>know it's invoking make--and moreover, the correct one.
 Also, I think
>that the current scheme allow other make programs to
co-operate,
>although it might not be an intended feature.
>
>  
>
I don't think we are on the right track here.
One thing that puzzles me is that make apparently complains
about childs it does not know about. My understanding of
Unix
principles is that SIGCLD and/or the wait-results is either
delivered to the immediate parent or init in case a process
is orphaned.
That means make must have forked the childs it claims to
know nothing
about. I cannot see how a sub-make(/process) could fork
processes where the
grand parent gets to see the exit codes via wait(pid)().
Where did I miss
something here ? I still have the impression that the job
list handling may
be incomplete (unless, of course, make has pid 1 ). It may
be that the
forks happen at a(nother) place (e.g. library) where they
are not 
entered in the job
list. If that is the case (and can be proven), then we
should get rid of 
the message
as it is of limited (and confusing) value if we are not able
by design 
to catch all
forked pids in the job list. If the joblist is definitely
complete, we 
have a coordination
issue.

Frank

On Fri, Aug 18, 2006 at 06:18:27PM +0200, Frank Kardel
wrote:
[...]
> I don't think we are on the right track here.
> One thing that puzzles me is that make apparently
complains
> about childs it does not know about. My understanding
of Unix
> principles is that SIGCLD and/or the wait-results is
either
> delivered to the immediate parent or init in case a
process is orphaned.
> That means make must have forked the childs it claims
to know nothing
> about. I cannot see how a sub-make(/process) could fork
processes where the

In that case, no.  make forks sh to run sets.subr, which in
turns forks a
make process.  File-descriptors are kept open all along
(it's probably part
of the problem, although the child make would complain in
turn if the fds
were closed and still it got -J).

> grand parent gets to see the exit codes via
wait(pid)(). Where did I miss
> something here ? I still have the impression that the
job list handling may
> be incomplete (unless, of course, make has pid 1 ). It may
be that the
> forks happen at a(nother) place (e.g. library) where
they are not 
> entered in the job
> list. If that is the case (and can be proven), then we
should get rid of 
> the message
> as it is of limited (and confusing) value if we are not
able by design 
> to catch all
> forked pids in the job list. If the joblist is
definitely complete, we 
> have a coordination
> issue.

There's an issue somewhere, clearly, but otoh, no-one
really bothered to
debug it in about three years.

-- 
Quentin Garnier - cubecubidou.net - cubeNetBSD.org
"When I find the controls, I'll go where I like,
I'll know where I want
to be, but maybe for now I'll stay right here on a silent
sea."
KT Tunstall, Silent Sea, Eye to the Telescope, 2004.

On Fri, Aug 18, 2006 at 06:35:58PM +0200, Quentin Garnier
wrote:
> On Fri, Aug 18, 2006 at 06:18:27PM +0200, Frank Kardel
wrote:
> [...]
> > I don't think we are on the right track here.
> > One thing that puzzles me is that make apparently
complains
> > about childs it does not know about. My
understanding of Unix
> > principles is that SIGCLD and/or the wait-results
is either
> > delivered to the immediate parent or init in case
a process is orphaned.
> > That means make must have forked the childs it
claims to know nothing
> > about. I cannot see how a sub-make(/process) could
fork processes where the
> 
> In that case, no.  make forks sh to run sets.subr,
which in turns forks a
> make process.  File-descriptors are kept open all along
(it's probably part
> of the problem, although the child make would complain
in turn if the fds
> were closed and still it got -J).

The complaint is silent - it is under '#if 0'.
If the fds are closed, the child make uses the -j n argument
to start another
'n' programs, with a different token pipe.  You don't
want this to happen.

I also (think) I've run that part of the build with make
erroring out
if the token pipe was closed - and it didn't fail building
the scripts.

make's job list processing is horrid, it fakes up calls to
its own
signal handler for 'job start' events.  I've a plan to
rewrite it using
an array with one slot per job (sized from the -j n
parameter) so that
almost all the signal interlocks can be removed.  If nothing
else that
will save 1000s of system calls.

But I need another round tuit.

	David

-- 
David Laight: davidl8s.co.uk