I am still very new to Ruby so I hope you can help. I have a Ruby on Rails app that needs to watch a specific directory "Dir A" to which I keep adding txt files. Once a new file appears it needs to be processed into csv file, which then appears in a tmp directory before being attached to a user, and disappears from tmp after the file goes into ActiveStorage, while keeping the original txt file in "Dir A" for a limited amount of time.
Now, I am using filewatcher gem to watch "Dir A" , however I need it to run on server startup and continue to run in the background. I understand I need to daemonize the process, but how can I do it from *.rb files rather than terminal? Atm I am using Threads, but I am not sure if that's the best solution...
I also have the following issues: - how to process files which have already appeared in the folder before the server start up? - filewatcher does not seem to pick up another new file while it's still processing the previous one, and threads don't seem to help with that - what would you recommend to be the best way to keep track of processed files - a database, or renaming/copying files into a different folder, or some global variables or maybe there's something else? I have to know which files are processed so I don't repeat the process espacially in cases when I need to schedule filewatcher restarts due to its declining performance (filewatcher documentation states it is best to restart the process if it's been long-running)
I'm sorry to bombard with questions, but I need some guidance, maybe there's a better gem I've missed, I looked at guard gem but I am not entirely sure how it works and filewatcher seemed simpler.
This question should probably be split into two, one about running filewatcher as a background process, and another about managing processed files, but as far as filewatcher goes, a simple solution would be to use the foreman gem with a Procfile.
You can start your Rails app in one process and filewatcher in another with a Procfile in the root of your app, like this:
# Procfile
web: bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development}
filewatcher: filewatcher "**/*.txt" "bundle exec rake process_txt_files"
and move whatever processing needs to be done into a rake task. With this you could just run foreman start locally to start both processes, and if your production server supports Procfiles (like Heroku, for example), this makes it easy to do the same in production.