Individual logging output of multiple objects to multiple files

Hallo,

I'm writing a simulation application, that creates and
destroys several objects
of the same class during the simulation. I want to perform
logging for each of
these objects to a single file (i.e. I want only the output
of one object in the
corresponding object's logging file). The application is
single threaded.

Can I achieve this behaviour with log4cxx - if so - how? I
tried to implement
this behaviour by initializing a new logger for each new
object. But since there
are up to 10000 objects this seems to be inefficient.
Furthermore I could not
figure out how to delete such a logger in case the
associated object gets
destroyed (how todo this???) and hence I experienced a lot
of memory leaks. To
my best knowledge MDC is also no alternative, since it does
not allow for the
restricted object->file logging.

Is there any chance to achieve this logging behaviour with
log4cxx?

Thank you!

Cheers Michael

You can certainly do it, but I wouldn't recommend it. You
would create
creating a new Logger instance for each instance and
attaching to it a
file appender unique to each instance. The overhead of
object creation
will be nothing compared to the overhead of creating 10000
files!

The question is why do you think you want to create 10000
files as
output. It would be a much better idea to log it all into
one file and
then process the log file to see the log for individual
instances by
differentiating the log events per instance.

The easiest way to differentiate the logging events is
creating a
logger per instance. Diagnostic context (MDC or NDC) can do
this as
well, but is really thread centric so probably does not
apply here
unless your simulation is such that object lifetimes do not
overlap.

If you just need to view them one at a time consider saving
a log with
XML Layout and using Chainsaw on the log to focus on
individual
loggers. If you really want 10000 files then it would be
fairly simple
to postprocess the log and create them.


On Thu, May 8, 2008 at 8:12 AM, Michael Duerr <duerrmgmail.com> wrote:
> Hallo,
>
> I'm writing a simulation application, that creates and
destroys several
> objects
> of the same class during the simulation. I want to
perform logging for each
> of
> these objects to a single file (i.e. I want only the
output of one object in
> the
> corresponding object's logging file). The application
is single threaded.
>
> Can I achieve this behaviour with log4cxx - if so -
how? I tried to
> implement
> this behaviour by initializing a new logger for each
new object. But since
> there
> are up to 10000 objects this seems to be inefficient.
Furthermore I could
> not
> figure out how to delete such a logger in case the
associated object gets
> destroyed (how todo this???) and hence I experienced a
lot of memory leaks.
> To
> my best knowledge MDC is also no alternative, since it
does not allow for
> the
> restricted object->file logging.
>
> Is there any chance to achieve this logging behaviour
with log4cxx?
>
> Thank you!
>
> Cheers Michael
>



-- 
Dale King

On May 8, 2008, at 7:12 AM, Michael Duerr wrote:

> Hallo,
>
> I'm writing a simulation application, that creates and
destroys  
> several objects
> of the same class during the simulation. I want to
perform logging  
> for each of
> these objects to a single file (i.e. I want only the
output of one  
> object in the
> corresponding object's logging file). The application
is single  
> threaded.
>
> Can I achieve this behaviour with log4cxx - if so -
how? I tried to  
> implement
> this behaviour by initializing a new logger for each
new object. But  
> since there
> are up to 10000 objects this seems to be inefficient.
Furthermore I  
> could not
> figure out how to delete such a logger in case the
associated object  
> gets
> destroyed (how todo this???) and hence I experienced a
lot of memory  
> leaks. To
> my best knowledge MDC is also no alternative, since it
does not  
> allow for the
> restricted object->file logging.
>
> Is there any chance to achieve this logging behaviour
with log4cxx?
>
> Thank you!
>
> Cheers Michael

I agree with Dale's comments, but want to add some more.

The motivation of having a logger hierarchy is so to allow
the user to  
control the processing of logging requests by discarding
some requests  
and selectively routing others to specific appenders. 
Having 10000  
loggers with dynamic names has little value to someone who
is trying  
to shape logging in the configuration file.

If you really wanted 10000 output files, the best approach
would be to  
use something like the mythical MultiFileAppender that lies
unfinished  
in the log4j sandbox.  One appender, but can manage multiple
open  
files.  Search log4j-dev for MultiFileAppender for the back
story if  
you are interested.

However, I agree with Dale that you'd be better off writing
to a  
single file and then appropriately sorting or filtering.

If you were using the mythical MFA, you'd need to have
something in  
the logging event to determine what file the event should be
routed,  
some sort of identification of the object.  In the single
file  
approach, you could write this identifier out into the log
file and  
then use grep to pull out only the relevant log entries.

The question is then how do you get this object identifier
into the  
logging event.  Your options are:

1. Put the object identifier into the message.  Either
something like:

LOG4CXX_INFO(logger, this << "Hello,
World");

You'd need to provide an insertion operator for your object
that works  
on a log4cxx::helpers::MessageBuffer to insert the id.  If
advantage  
of this is that it costs nothing if logging is disabled.

A cleaner solution would be to define your own macro that
handled  
inserting the object ID into the message.

2. Put the object identifier in the NDC on public method
entry:

void foo() {
	NDC entry(getID());

}

This will add an entry to the nested diagnostic context on
method  
entry and remove it on method exit.  However, this adds a
fixed cost  
to the method call even if logging is disabled.

3. Put the object identifier in the MDC on public method
entry:

void foo() {
      MDC entry("objID", getID());

Same disadvantage and likely slower.

Hi Dale and Arnold,

thanks a lot for your support. The idea to log from the
beginning into different 
files was motivated by the size a single log file would have
(up to 8GB) after a 
single run). But actually I have not thought at all about
post processing the 
data which probably could take quite long. I'll give it a
try.
Thanks also for the advice to use the MultiFileAppender.
I'll definitely have a 
look at it!

Cheers,
Michael

Dale King wrote:
> You can certainly do it, but I wouldn't recommend it.
You would create
> creating a new Logger instance for each instance and
attaching to it a
> file appender unique to each instance. The overhead of
object creation
> will be nothing compared to the overhead of creating
10000 files!
> 
> The question is why do you think you want to create
10000 files as
> output. It would be a much better idea to log it all
into one file and
> then process the log file to see the log for individual
instances by
> differentiating the log events per instance.
> 
> The easiest way to differentiate the logging events is
creating a
> logger per instance. Diagnostic context (MDC or NDC)
can do this as
> well, but is really thread centric so probably does not
apply here
> unless your simulation is such that object lifetimes do
not overlap.
> 
> If you just need to view them one at a time consider
saving a log with
> XML Layout and using Chainsaw on the log to focus on
individual
> loggers. If you really want 10000 files then it would
be fairly simple
> to postprocess the log and create them.
> 
> 
> On Thu, May 8, 2008 at 8:12 AM, Michael Duerr
<duerrmgmail.com> wrote:
>> Hallo,
>>
>> I'm writing a simulation application, that creates
and destroys several
>> objects
>> of the same class during the simulation. I want to
perform logging for each
>> of
>> these objects to a single file (i.e. I want only
the output of one object in
>> the
>> corresponding object's logging file). The
application is single threaded.
>>
>> Can I achieve this behaviour with log4cxx - if so -
how? I tried to
>> implement
>> this behaviour by initializing a new logger for
each new object. But since
>> there
>> are up to 10000 objects this seems to be
inefficient. Furthermore I could
>> not
>> figure out how to delete such a logger in case the
associated object gets
>> destroyed (how todo this???) and hence I
experienced a lot of memory leaks.
>> To
>> my best knowledge MDC is also no alternative, since
it does not allow for
>> the
>> restricted object->file logging.
>>
>> Is there any chance to achieve this logging
behaviour with log4cxx?
>>
>> Thank you!
>>
>> Cheers Michael
>>
> 
> 
> 

On May 8, 2008, at 12:22 PM, Michael Dürr wrote:

> Hi Dale and Arnold,
>
> thanks a lot for your support. The idea to log from the
beginning  
> into different files was motivated by the size a single
log file  
> would have (up to 8GB) after a single run). But
actually I have not  
> thought at all about post processing the data which
probably could  
> take quite long. I'll give it a try.
> Thanks also for the advice to use the
MultiFileAppender. I'll  
> definitely have a look at it!
>
> Cheers,
> Michael
>


MultiFileAppender is not a viable choice at the moment since
it is  
just a skeleton implementation in the log4j sandbox.  Let
alone  
something that has been tested in log4j and ported to
log4cxx.   
However, it is the right design for an appender that needs
to write to  
distinct files per thread or some other criteria.  The
reason I  
brought it into the conversation was the need for you to get
that  
"criteria" (in this case an object ID) into the
logging event which  
would be needed regardless of whether you were using an MFA
or a  
FileAppender with post-processing.

If file size is your problem, then you could use a
RollingFileAppender  
with a sized based triggering policy.

Logging 8 GB per run seems like you are collecting more
information  
that you could practically analyze.