Search code examples
javaapache-kafkaapache-zookeeperapache-storm

I am using StormConfigs startOffsetTime as LatestTime but if storm is stopped then it is Losing data produced by the producer

I am using KafkaProducer to send data to storm and when configuring spout I have written below code. 

spoutConfig.useStartOffsetTimeIfOffsetOutOfRange=true;
spoutConfig.startOffsetTime=kafka.api.OffsetRequest.LatestTime();

but the problem is when i stop Storm Application(Dev environment) storm loses the data produced by the producer. If i use:

spoutConfig.startOffsetTime=kafka.api.OffsetRequest.ErliestTime()

the data insertion starts from beginning. Here I don't want any data loss if storm goes down still storm should take all data produced by the Producer.

Solution

  • LocalClusters clear their state every time you restart them (it essentially sets up a new Zookeeper cluster every time). They are for testing, not production workloads. This is why your spout is starting over.