I am using Morphline Solr Sink to store information in Solr. The problem that I am facing is that flume agent never stops retrying the failed requests, which sometimes can increase over time. This results in the flume warning of MaxIO Workers being used and the system suffers with performance issues. Is there any way other than writing my own sink, that can make flume stop retrying or backoff exponentially to have a better system performance? My source is an avroSource.
Thanks.
You should fix the reason for the failed requests.
Flume is doing exactly what it's designed to do. It's transactionally trying to store the batch of events in your store. If it can't store those events then, yes, it keeps on trying.
You haven't explained what the problem is causing these failures. I would recommend thinking about an interceptor to fix whatever is wrong in the data or to drop events you don't want to store.