Enabling JSON logging in ververica flink

Question

Preface - I was using flink before without ververica and was able to set up log4j JSON logging by leveraging the pre entrypoint script to download the needed jackson dependencies

Question is - for ververica, without building a custom image, is it possible to enable json logging?

I tried using JSONLayout type of logs but jackson is not in the class path, and for some reason marking them as artifacts is not enough for the job/task manager logs

Next, i tried JsonTemplateLayout which does not require jackson or any other dependencies but it does look like the the layouts such as EcsLayout.json or LogstashJsonEventLayoutV1.json also does not exist

my logging config is attached below, it is the same as the default but with the rollingFile appender layout changed

Am I missing something or does this require custom image creation?

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<Configuration xmlns="http://logging.apache.org/log4j/2.0/config" strict="true">
    <Appenders>
        <Appender name="StdOut" type="Console">
            <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/>
        </Appender>
        <Appender name="RollingFile" type="RollingFile" fileName="${sys:log.file}" filePattern="${sys:log.file}.%i">
            <JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json"/>
            <Policies>
                <SizeBasedTriggeringPolicy size="50 MB"/>
            </Policies>
            <DefaultRolloverStrategy max="5"/>
        </Appender>
    </Appenders>
    <Loggers>
        <Logger level="INFO" name="org.apache.hadoop"/>
        <Logger level="INFO" name="org.apache.kafka"/>
        <Logger level="INFO" name="org.apache.zookeeper"/>
        <Logger level="INFO" name="akka"/>
        <Logger level="ERROR" name="org.jboss.netty.channel.DefaultChannelPipeline"/>
        <Logger level="OFF" name="org.apache.flink.runtime.rest.handler.job.JobDetailsHandler"/>
        {%- for name, level in userConfiguredLoggers -%}
        <Logger level="{{ level }}" name="{{ name }}"/>
        {%- endfor -%}
        <Root level="{{ rootLoggerLogLevel }}">
            <AppenderRef ref="StdOut"/>
            <AppenderRef ref="RollingFile"/>
        </Root>
    </Loggers>
</Configuration>

Answer 1

Yes, you can use JSON logging without a custom image in Ververica Platform. Let me take JsonTemplateLayout as an example.

Step-1:
Prepare the log4j-layout-template-json-2.14.1.jar (or whatever version you are using) and the LogstashJsonEventLayoutV1.json (from https://logging.apache.org/log4j/2.x/manual/json-template-layout.html)

Step-2:
Create a k8s volume from configmap using the log4j-layout-template-json-2.14.1.jar in the k8s namespace where you will run this job:

kubectl create configmap jsonlogging --from-file=log4j-layout-template-json-2.14.1.jar -n vvp-ops-jobs

Step-3:
In the Deployment configuration page (YAML), add the volume mount for json template jar (under spec.template.spec.kubernetes):

pods:
  volumeMounts:
    - name: jsonlogging
      volume:
        configMap:
          name: jsonlogging
        name: jsonlogging
      volumeMount:
        mountPath: /flink/lib/log4j-layout-template-json-2.14.1.jar
        name: jsonlogging
        subPath: log4j-layout-template-json-2.14.1.jar

Step-4:
Add the LogstashJsonEventLayoutV1.json to the Deployment Artifacts and refer to it under the Deployment configuration page (YAML):

spec:
  template:
    spec:
      artifact:
        additionalDependencies:
          - >-
            s3://vvp-lab/artifacts/namespaces/default/LogstashJsonEventLayoutV1.json

Step-5:
Configure the logging template for the Deployment:

<Appender name="RollingFile" type="RollingFile" fileName="${sys:log.file}" filePattern="${sys:log.file}.%i">
  <Layout type="JsonTemplateLayout" eventTemplateUri="file:///flink/usrlib/LogstashJsonEventLayoutV1.json" />
  <Policies>
    <SizeBasedTriggeringPolicy size="50 MB"/>
  </Policies>
  <DefaultRolloverStrategy max="1"/>
</Appender>

Notes:

• For Step-2 and Step-3, you cannot just add this jar as an additional dependency for a deployment, because the jar will be available only in the user class loader. Flink needs this jar to be available in the parent class loader in order to format logs. You can also bake this jar into the /flink/lib directory with a custom image.
• For Step-4, the json file can be placed either in the 'additional dependencies' or in the same place as the jar file. (but the latter one means you will need to refer to that json using 'classpath:LogstashJsonEventLayoutV1.json' rather than absolute file path in Step-5)