jackd shortcomings in logging and direction for solution

Hello JACK folk,

I am a bit frustrated with jackd and it's logging
abilities.  But, I am
prepared to put some effort to improve the situation.

In my ideal world, jackd would log via syslog with multiple
levels of
verbosity.  This would allow for centralized logging and
also allow for
differnt level messages to be looked at or ignored (ex:
debug vs error
messages).  This logging (at least errors) would happen all
the time, so when
there is a problem, it can be inspected and corrected.


As I see things now, jack has either "no
logging" (default behaviour) or a
plethora of debug logging (with the "-v
--verbose" option passed at the
command line).  While this logging solution may work for a
developer doing
debugging, it is not an acceptable solution for logging in
production
environments with users that are likely not even aware of
jackd.  Even worse,
the current logging does not work at all if jackd is started
with the
"JACK_START_SERVER" environment variable.  Even
adding adding stdout or stderr
redirections in /etc/jackdrc, the output gets thrown away.


While I am not yet very familiar with the jack codebase, I
am willing to
submit a patch that works within the planned jack framework.
 I seek feedback
from other veteran jackd developers on how to best proceed
with getting better
logging with jackd.

thank you for your feedback,
donfede


-- 
Federico Grau
Free Software Developer and Sysadmin
Radio Free Asia
2025 M Street, NW
Suite 300
Washington, DC  20036
202-587-2046  Telephone
202-721-7468  Facsimile
CONFIDENTIAL COMMUNICATION
This e-mail message is intended only for the use of the
addressee and may
contain information that is privileged and confidential. 
Any unauthorized
dissemination, distribution, or copying is strictly
prohibited.  If you
receive this transmission in error, please contact
networkrfa.org.


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Fri, 2006-06-09 at 14:49 -0400, Federico Grau wrote:
> Hello JACK folk,
> 
> I am a bit frustrated with jackd and it's logging
abilities.  But, I am
> prepared to put some effort to improve the situation.

i am not saying that i don't think it could be improved.
but i think you
misunderstand the overally goal behind jack's design. in my
view, if you
need logging to be improved, then jack isn't working as
well as it
should and logging won't help fix it.

> As I see things now, jack has either "no
logging" (default behaviour) or a
> plethora of debug logging (with the "-v
--verbose" option passed at the
> command line).  While this logging solution may work
for a developer doing
> debugging, it is not an acceptable solution for logging
in production

this isn't true, actually. if jack is built with
--enable-debug at
configure time, there is a *lot* more output from -v than in
a non-debug
build.

there is already an abstraction built into jackd for
"logging", which
writes to a buffer rather than deliver directly to some kind
of output.
changing where the output goes is not that hard. 

however, i must repeat what i said earlier - if there are
problems with 
using jack, i don't believe that more or redirected or more
controlled
logging is part of the solution.

--p






_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Fri, Jun 09, 2006 at 03:09:01PM -0400, Paul Davis wrote:
> On Fri, 2006-06-09 at 14:49 -0400, Federico Grau wrote:
> > Hello JACK folk,
> > 
> > I am a bit frustrated with jackd and it's logging
abilities.  But, I am
> > prepared to put some effort to improve the
situation.
> 
> i am not saying that i don't think it could be
improved. but i think you
> misunderstand the overally goal behind jack's design.
in my view, if you
> need logging to be improved, then jack isn't working
as well as it
> should and logging won't help fix it.
> 
> > As I see things now, jack has either "no
logging" (default behaviour) or a
> > plethora of debug logging (with the "-v
--verbose" option passed at the
> > command line).  While this logging solution may
work for a developer doing
> > debugging, it is not an acceptable solution for
logging in production
> 
> this isn't true, actually. if jack is built with
--enable-debug at
> configure time, there is a *lot* more output from -v
than in a non-debug
> build.
> 
> there is already an abstraction built into jackd for
"logging", which
> writes to a buffer rather than deliver directly to some
kind of output.
> changing where the output goes is not that hard. 
> 
> however, i must repeat what i said earlier - if there
are problems with 
> using jack, i don't believe that more or redirected or
more controlled
> logging is part of the solution.
> 


Hi Paul and others,

I know jack should be "rock solid", but in our
experience "jack is not working
as well as it should be".  While in our environment
jack works well 80%+ of
the time, we experience a variety of problems.  Regardless
to the source of
the problems, without central logging getting a grasp on
what the problems may
be is next to impossible, a bit like flying blind.

There is no doubt that jack is a very useful piece of code
(I look forward to
help "jackyfy various Free Software apps over time),
but it is definitely
still a work in progress (there are regular cvs commits and
new releases
comming out with API changes still happening).  As jack
continues to mature, I
cannot imagine a "production" daemon without
logging.  This brings back
nightmares of windows 9x and not having any decent logs to
analyze when there
is a problem.  Imagine and MTA or scheduler daemon with no
logging ...  sure
they are 'supposed' to work correctly all the time ... but
there are always
exceptions or times that things need further inspection.  


Thank you for the pointer to the logging abstraction layer,
I'll take a look
at that and see if I have any worthwhile contributions.

respectfully,
donfede

-- 
Federico Grau
Free Software Developer and Sysadmin
Radio Free Asia
2025 M Street, NW
Suite 300
Washington, DC  20036
202-587-2046  Telephone
202-721-7468  Facsimile
CONFIDENTIAL COMMUNICATION
This e-mail message is intended only for the use of the
addressee and may
contain information that is privileged and confidential. 
Any unauthorized
dissemination, distribution, or copying is strictly
prohibited.  If you
receive this transmission in error, please contact
networkrfa.org.


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Fri, 2006-06-09 at 15:31 -0400, Federico Grau wrote:
> I know jack should be "rock solid", but in
our experience "jack is not working
> as well as it should be".  While in our
environment jack works well 80%+ of
> the time, we experience a variety of problems. 
Regardless to the source of
> the problems, without central logging getting a grasp
on what the problems may
> be is next to impossible, a bit like flying blind.
> 
> There is no doubt that jack is a very useful piece of
code (I look forward to
> help "jackyfy various Free Software apps over
time), but it is definitely
> still a work in progress (there are regular cvs commits
and new releases
> comming out with API changes still happening).  As jack
continues to mature, I
> cannot imagine a "production" daemon
without logging.  This brings back
> nightmares of windows 9x and not having any decent logs
to analyze when there
> is a problem.  Imagine and MTA or scheduler daemon with
no logging ...  sure
> they are 'supposed' to work correctly all the time
... but there are always
> exceptions or times that things need further
inspection.  

in its current incarnation, jackd is not intended (nor is it
possible)
to be used by multiple users. therefore, it does not
represent a "system
daemon". in fact, i would argue that its not actually
a daemon at all.

what kinds of problems are you experiencing and what kind of
logging do
you think would help?




_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Fri, Jun 09, 2006 at 03:36:50PM -0400, Paul Davis wrote:
> On Fri, 2006-06-09 at 15:31 -0400, Federico Grau wrote:
> > I know jack should be "rock solid",
but in our experience "jack is not working
> > as well as it should be".  While in our
environment jack works well 80%+ of
> > the time, we experience a variety of problems. 
Regardless to the source of
> > the problems, without central logging getting a
grasp on what the problems may
> > be is next to impossible, a bit like flying blind.
> > 
> > There is no doubt that jack is a very useful piece
of code (I look forward to
> > help "jackyfy various Free Software apps
over time), but it is definitely
> > still a work in progress (there are regular cvs
commits and new releases
> > comming out with API changes still happening).  As
jack continues to mature, I
> > cannot imagine a "production" daemon
without logging.  This brings back
> > nightmares of windows 9x and not having any decent
logs to analyze when there
> > is a problem.  Imagine and MTA or scheduler daemon
with no logging ...  sure
> > they are 'supposed' to work correctly all the
time ... but there are always
> > exceptions or times that things need further
inspection.  
> 
> in its current incarnation, jackd is not intended (nor
is it possible)
> to be used by multiple users. therefore, it does not
represent a "system
> daemon". in fact, i would argue that its not
actually a daemon at all.
> 
> what kinds of problems are you experiencing and what
kind of logging do
> you think would help?
> 

I am most interested in seeing "why" jackd stops
when it does, and to have
that recorded in a central place (syslog is ideal for this,
dozens of
workstations can log to a central logging server and allow
for easy analysis
if there is a problem).

Other logs that would be interesting are when jack kicks a
client (why, when,
how often).  This will allow me to identify a problem,
possibly with jack or
possibly with the client.


The most common example of jack failures we are seeing
recently is when
starting an app that will start jack (via the
JACK_START_SERVER env); the app
dies and jack is no-where to be seen.   Starting the app a
second time works
most of the time (but there is obviously a problem present).
 The app is
Rivendell's "cae" and jack is 0.100.0 on a
debian stable system.

We have had other other times when jack has failed, but I do
not have
worthwhile records of that now (thus the logging would be
ideal).  

I am also now trying to study how jackd exits (and cleans
up), especially when
started within an X session and dying on exit of the X
session.  I am unable
to get those logs out of jack when it's started with teh
JACK_START_SERVER
env.


As I am working on deploying this project, jack is a key
piece of code that
HAS to work.  Before I deploy it to dozens of workstations,
the sysadmin in me
says "I have to have eyes" for WHEN there is a
problem (there will be one
sooner or later) so I can give a proper answer to my users
and take actions to
correct it.

regards,
donfede



-- 
Federico Grau
Free Software Developer and Sysadmin
Radio Free Asia
2025 M Street, NW
Suite 300
Washington, DC  20036
202-587-2046  Telephone
202-721-7468  Facsimile
CONFIDENTIAL COMMUNICATION
This e-mail message is intended only for the use of the
addressee and may
contain information that is privileged and confidential. 
Any unauthorized
dissemination, distribution, or copying is strictly
prohibited.  If you
receive this transmission in error, please contact
networkrfa.org.


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

Thanks for the explanation.  I personally have no objection
to adding
a syslog option to jackd.  The main difficulty would
probably be doing
a good job of classifying the levels of all those messages. 
That would
be worthwhile, but a lot of work.

I believe that quite a few --verbose messages from jackd are
actually
errror messages that should appear even without that option.
 Fixing
some of those would probably help a lot of new users figure
out what
has gone wrong, even without syslog.
-- 
 joq


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Fri, Jun 09, 2006 at 03:58:31PM -0400, Federico Grau
wrote:
> The most common example of jack failures we are seeing
recently is when
> starting an app that will start jack (via the
JACK_START_SERVER env); the app
> dies and jack is no-where to be seen.   Starting the
app a second time works
> most of the time (but there is obviously a problem
present).  The app is
> Rivendell's "cae" and jack is 0.100.0 on a
debian stable system.

You are aware that Debian stable uses libpthread-0.60 in
libc6? This is
broken so threaded apps won't work properly. You need to
tell jackd to
use an earlier thread model by using LD_ASSUME_KERNEL, eg.:
LD_ASSUME_KERNEL=2.4.22 jackd -R -d alsa ...
Clients that are supposed to start jackd will have to be
configured to
do this.

If you're using AMD64 you won't have this option as no
earlier thread model
is provided in the kernel. Debian 3.1r2 has some thread
related changes
in libc6 for AMD64 which might fix it, but I don't think
it's out yet.

John


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Sat, Jun 10, 2006 at 09:44:29AM +0100, John Rigg wrote:
> If you're using AMD64 you won't have this option as
no earlier thread model
> is provided in the kernel. Debian 3.1r2 has some thread
related changes
> in libc6 for AMD64 which might fix it, but I don't
think it's out yet.

Hmm. I could be wrong about earlier thread models not being
in the AMD64
kernel. I do remember seeing somewhere that only NPTL works
on AMD64 but
I can't remember where. Perhaps LD_ASSUME_KERNEL works on
AMD64 after
all. I switched to Debian testing/unstable some time ago to
avoid all
this.

John


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el

On Sat, Jun 10, 2006 at 09:44:29AM +0100, John Rigg wrote:
> On Fri, Jun 09, 2006 at 03:58:31PM -0400, Federico Grau
wrote:
> > The most common example of jack failures we are
seeing recently is when
> > starting an app that will start jack (via the
JACK_START_SERVER env); the app
> > dies and jack is no-where to be seen.   Starting
the app a second time works
> > most of the time (but there is obviously a problem
present).  The app is
> > Rivendell's "cae" and jack is 0.100.0
on a debian stable system.
> 
> You are aware that Debian stable uses libpthread-0.60
in libc6? This is
> broken so threaded apps won't work properly. You need
to tell jackd to
> use an earlier thread model by using LD_ASSUME_KERNEL,
eg.:
> LD_ASSUME_KERNEL=2.4.22 jackd -R -d alsa ...
> Clients that are supposed to start jackd will have to
be configured to
> do this.

Looking into this further I see that jackd 0.99 (the version
in Debian
stable) was patched to work around the pthreads/NPTL
problem. You
might still need to use LD_ASSUME_KERNEL with jack clients,
though.

John


_______________________________________________
Jackit-devel mailing list
Jackit-devellists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jackit-dev
el