I'm using Spark and there's one of my Spring Boot-based application beans:
@Component
@RequiredArgsConstructor
public class SomeService implements FlatMapFunction<T, K> {
private final ObjectMapper mapper;
}
ObjectMapper
here is the standard one taken from application context. The problem is that the app fails with org.apache.spark.SparkException: Task not serializable
. Here's serialization stack:
Caused by: java.io.NotSerializableException: org.springframework.http.converter.json.SpringHandlerInstantiator
Serialization stack:
- object not serializable (class: org.springframework.http.converter.json.SpringHandlerInstantiator, value: org.springframework.http.converter.json.SpringHandlerInstantiator@6e4912db)
- field (class: com.fasterxml.jackson.databind.cfg.BaseSettings, name: _handlerInstantiator, type: class com.fasterxml.jackson.databind.cfg.HandlerInstantiator)
- object (class com.fasterxml.jackson.databind.cfg.BaseSettings, com.fasterxml.jackson.databind.cfg.BaseSettings@155616d8)
- field (class: com.fasterxml.jackson.databind.cfg.MapperConfig, name: _base, type: class com.fasterxml.jackson.databind.cfg.BaseSettings)
- object (class com.fasterxml.jackson.databind.DeserializationConfig, com.fasterxml.jackson.databind.DeserializationConfig@66e72ca2)
- field (class: com.fasterxml.jackson.databind.ObjectMapper, name: _deserializationConfig, type: class com.fasterxml.jackson.databind.DeserializationConfig)
- object (class com.fasterxml.jackson.databind.ObjectMapper, com.fasterxml.jackson.databind.ObjectMapper@433ef204)
- field (class: com.smth.SomeService, name: mapper, type: class com.fasterxml.jackson.databind.ObjectMapper)
So the problem is about non-serializable SpringHandlerInstantiator
.
This far I work this around by assigning mapper
field in constructor manually:
public SomeService() {
this.mapper = new ObjectMapper();
}
Is there a way to somehow solve this properly, i. e. relying on Spring's DI?
I use Spring Boot 2.6.7 and Spark 2.11.
Try to configure ObjectMapper bean to use a serializable HandlerInstantiator. In the following example MyHandlerInstantiator
is a custom implementation of HandlerInstantiator
which is serializable (e.g. it can be started with minimal patching the code of SpringHandlerInstantiator
). This should allow SomeService class to be serializable as well and compatible for usage in a distributed environment like Apache Spark.
@Configuration
public class MyConfiguration {
@Bean
public ObjectMapper objectMapper() {
ObjectMapper mapper = new ObjectMapper();
mapper.setHandlerInstantiator(new MyHandlerInstantiator());
return mapper;
}
}