Say I have a multithreaded application made up of two separate threads and a Queue. Thread 1 finishes it's computation and puts the result in the Queue. Meanwhile thread 2 is constantly looping and checking if there is any data in the Queue for it to process.
How can I save the values in the queue to disk temporarily in case the for some reason the program faults or the computer is restarted? Would it be reasonable to save the value in a SQLite DB after the first thread is finished and then delete it after thread 2 is done?
If SQLite is in fact the best solution is it quicker to insert a flag on a row saying that it's been processed and have another thread come through every half hour and delete those rows that are flagged or just immediately delete that row?
Don't roll your own here. Asynchronous processing routines, restartability, program faults, etc. are best managed with message queues.
You could use the built-in one to Windows, go open source with ActiveMQ from the Apache Software Foundation, or leverage web services in the cloud with Amazon's Simple Queue Service.
There are a lot of different ways you can go here. Investigate these options before repeating the functionality yourself. Reliability in these areas is more challenging than you might expect.