How to avoid duplicates from saving in database parsed from external JSON file with sidekiq in Rails

Question

I have a small To Do list in a .json file that I´m reading, parsing, and saving to a rails app with Sidekiq. Everytime I refresh the browser, the worker executes and duplicates the entries on the database. How do I maintain a unique database that is synchronized with the .json file and avoid duplicate entries to save on my database and show on my browser?

Here's the worker:

class TodoWorker
  include Sidekiq::Worker

  def perform
    json_text = File.read('todo_json.json')
    json = JSON.parse(json_text, :headers => true)
    json.each do |todo|
      t = TodoList.create(name: todo["name"], done: todo["done"])
      t.save
    end
  end
end

And the controller:

class TodoListsController < ApplicationController
  def index
    @todo_lists = TodoList.all
    TodoWorker.perform_async
  end
end

Thanks

Answer 1

This is a terrible solution btw, you have a huge race condition in your read/store code, and you're not going to be able to use a large part of what Rails is good at. If you want a simple DB why not just use sqlite?

That being said, you need some way of recognizing duplicates, in most DBs this is done with a primary key that is sent to the browser along with the rest of the data, and then back again with any changes. That primary key is used to ensure that existing data is updated, rather than duplicated.

You will need the same thing in your JSON file, and then you can change your create method to be something more like ActiveRecord's find_or_create_by