spring hibernate log4j databricks logback

Azure Databricks Logging Configuration problems

I'm running Spring Boot 2.7.4 with logback 1.2.11 and also slf4j 1.7.36, and I have a console appender in my logback-spring.xml:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.7.4</version>
    <relativePath/>
</parent>

<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
</dependency>

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
</dependency>

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>log4j-over-slf4j</artifactId>
</dependency>

<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>databricks-jdbc</artifactId>
    <version>2.6.29</version>
</dependency>

<pattern>%mask(${CONSOLE_LOG_PATTERN})</pattern>
<pattern>%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){magenta} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %mask(%m%n)</pattern>

My entire application is logging just fine, but I needed to add the Databricks JDBC drivers, and I get this in my log when I execute a SQL from both the Spring JPA framework and with the JDBC Template:

ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.databricks.client.jdbc42.internal.io.netty.util.internal.ReflectionUtil (file:/C:/Users/[REDACTED]/com/databricks/databricks-jdbc/2.6.29/databricks-jdbc-2.6.29.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of com.databricks.client.jdbc42.internal.io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I found this issue that other people had, and it seems like some dependency is bringing in lo4j configurations from somewhere, like some .dat file or a log4j properties file, but I cannot find these file s anywhere in the Databricks jars. There are some log4j files in the jar, but not sure if these are what is causing the issue or not.

Log4j-charsets.properties
Log4j-config.xsd
Log4j-events.dtd
Log4j-events.xsd
Log4j-levels.xsd

It's clearly conflicting with my logging pattern, because it cannot understand the format pattern for "msg" or "thread" etc.

When I debug the my code and get my datasource, and getParentLogger(), it's using java.util.logging.Logger, just like the rest of my datasources (using Spring JPA/Hibernate).

I've tried to exclude any logging from the dependency and cannot figure out how to force it to use the logging for my project:

<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>databricks-jdbc</artifactId>
    <version>2.6.29</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
        </exclusion>
        <exclusion>
            <groupId>io.netty</groupId>
            <artifactId>netty-buffer</artifactId>
        </exclusion>
        <exclusion>
            <groupId>io.netty</groupId>
            <artifactId>netty-common</artifactId>
        </exclusion>
        <exclusion>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
        </exclusion>
    </exclusions>
</dependency>

I tried adding some of the other dependencies, like these:

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>jul-to-slf4j</artifactId>
</dependency>

And this doesn't fix the issue either.

I have also tried to turn off the logging for the StatusLogger in my logback-spring.xml, but I still see the errors in red text in the log (ANSI is turned on).

<logger name="org.apache.logging.log4j.status.StatusLogger" level="OFF" additivity="false">
   <appender-ref ref="CONSOLE" />
</logger>

<logger name="com.databricks.*" level="OFF" additivity="false">
   <appender-ref ref="CONSOLE" />
</logger>

It only happens once when I use the datasource, and hibernate is printing the SQL and everything just fine, so I don't know if I can suppress this somehow, but ideally, I would like it to use the logging framework that I'm using for everything else and not try to override my logging patterns.

Solution

This is apparently related to Arrow Serialization, and can be fixed by disabling it via your JDBC URL.

Add ;EnableArrow=0 to the end of your URL.

Warning: I do not know the overall implications of disabling this serialization, but I know this will also allow you to use the driver in the latest versions of Java as well.