I have an ETL process set up in perl to process a number of files, and load them to a database.
Recently, for performance reasons I set up the code to be multi-threaded, through use of a fork() call and a call to system("perl someOtherPerlProcess.pl $arg1 $arg2").
I end up with about 12 instances of someOtherPerlProcess.pl running with different arguments, and these processes each work through one directories worth of files (corresponding to a single table in our database).
The applications main functions work, but I am having issues with figuring out how to configure my logging.
Ideally, I would like to have all the someOtherPerlProcess.pl share the same $log_config value to initialize their loggers, but have each of those create a log file in the directory they are working on.
I haven't been able to figure out how to do that. I also noticed that in the directory I am calling these perl scripts from I see several files (ARRAY(0x260eec), ARRAY(0x313f8), etc) that contain all my logging messages!
Is there a simple way to change the log4perl.appender.A1.filename value from running code? Or to otherwise dynamically configure the file name we use, but use all other values from a config file?
I came up with a less than ideal solution for this, which is to configure my logger from someOtherPerlProcess.pl directly.
my $FORKED_LOG_CONF = "log4perl.appender.A1.filename=$directory_to_load/log.txt
log4perl.rootLogger=WARN, A1
log4perl.appender.A1=Log::Log4perl::Appender::File
log4perl.appender.A1.mode=append
log4perl.appender.A1.autoflush=1
log4perl.appender.A1.layout=PatternLayout
log4perl.appender.A1.layout.ConversionPattern=[%p] %d{yyyy-MM-dd HH:mm:ss}: %m%n";
#Logger start up
Log::Log4perl::init( \$FORKED_LOG_CONF);
my $logger = get_logger();
The $directory_to_load is the process specific portion of the logger, which works in the context of the perl process that is running and has a (local) value for that variable, but that method will fail if used in an external config file.
I would be happy to hear of any alternative solutions.