Log4j format (via -Dlog4j.configurationFile) not working in some env

Question

Our app uses log4j to control logging behavior. It specifies the format of the log line via Field elements in the Configuration.Appenders.RollingFile.StructuredLayout section of the log4 config xml file (full config below).

Problem: The log format is only being used in some of our environments, but not others.

We pass the log4j xml config to the app like so, the same for each environment

# command in stage
java -Dlog4j.configurationFile=/opt/ais/config/log4j2.xml -jar myApp.jar

# command in production
java -Dlog4j.configurationFile=/opt/ais/config/log4j2.xml -jar myApp.jar

The logs in stage have no formatting. The logs in production have the formatting we expect (specified in the xml).

# in stage (no format, not working)
> cat /opt/ais/logs/splunk/myApp/host-0-443.log
test log message

# in prod (format, working as expected)
> cat /opt/ais/logs/splunk/myApp/host-0-443.log
timestamp="2023-05-03 07:13:15,461",level="INFO",thread="pool-2-thread-1",message="test log message",logger="MessageProcessor",appName="myApp",instanceNum="300",dataCenter="MR",partition="MR",buildVersion="1.0.0"

The xml file is present on each host, and the file path matches. So what could be causing one environment to honor the log format, and the other does not?

Full log4j config xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.apple.jvm.commons.logging.structured">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <StructuredLayout/>
    </Console>
    <RollingFile name="RollingFile"
                 fileName="/opt/ais/logs/splunk/${sys:ISApplicationName}/${sys:ISHostId}-${sys:ISInstanceId}-${sys:WOPort}.log"
                 filePattern="/opt/ais/logs/${sys:ISApplicationName}/${sys:ISHostId}-${sys:ISInstanceId}-${sys:WOPort}.%i.log">
      <StructuredLayout serializer="com.mycompany.jvm.commons.logging.structured.kv.KeyValueSerializer" fieldSeparator="," timeZoneId="America/Los_Angeles">
        <Field name="timestamp" pattern="%date{yyyy-MM-dd HH:mm:ss,SSS}{America/Los_Angeles}"/>
        <Basic exclude="timestamp"/>
        <Throwable name="exception"/>
        <Field name="appName" value="${sys:ISApplicationName}"/>
        <Field name="instanceNum" value="${sys:ISInstanceId}"/>
        <Field name="dataCenter" value="${sys:ISDataCenter}"/>
        <Field name="partition" value="${sys:partition}"/>
        <Field name="buildVersion" value="${sys:build.version}"/>
      </StructuredLayout>
      <Policies>
        <SizeBasedTriggeringPolicy size="500 MB"/>
      </Policies>
    </RollingFile>
  </Appenders>
  <Loggers>
    <Root level="info">
      <AppenderRef ref="RollingFile"/>
      <AppenderRef ref="Console"/>
    </Root>
  </Loggers>
</Configuration>

Answer 1

The issue was that serializer in the StructuredLayout com.mycompany.jvm.commons.logging.structured.kv.KeyValueSerializer was not available in the dependency graph. Our production build is working but some time in between the last production build and this current build in staging, that dependency got dropped. Who knows why, but it could have been due to the fact we got "lucky" and had the dependency available via a transitive dependency, but that is no longer true.

The fix was to add a runtime dependency to gradle, to include that missing KeyValueSerializer type.

runtimeOnly("com.mycompany.jvm.commons.commons-logging:structured-layout-log4j2:1.11.1")

Note that this dependency is our own internal artifact, YMMV