I'm developing a Spring MVC app where users can search for products. The products are stored in a local database. The database gets filled or updated everyday by a Spring Batch job. The product data comes from multiple csv files. So far so good.
My problem:
It may happen that some products will expire and won't appear in the csv files anymore. Well I don't want to check the whole database table against all incoming csv files everyday to check which database records I have to remove.
Otherwise I could truncate all records from my database before the csv import. But in that case users in the web app won't find any products during the batch job execution. Since the csvs come from different merchants I could only delete all products from this merchant before inserting the new ones. In that case only these products won't be available in the web app during batch job execution.
Does anyone have an idea how to deal with this problem?
You can use a flag indicates 'product to remove' and clear this flag when a product is read from CSV.
Next step for your job is to delete records who have the flag still set.