Django: Can I build a model upon local files instead of a real database?

Question

I'll provide more background information first. The question is asked again in the last bullet of "My Thoughts & Questions" section.

Background

• I am working on a legacy system which looks kind of like a batch processing system: A job is passed along a series of worker programs that each works on part of the whole job until it is finished. The job status/management information is generated by each worker program along the way and written into local text files with names like "status_info.txt" and "mgmt_info.txt".
• This processing system does NOT use any databases, just pure text files on the local file system. Changing their code to use a databases is also kind of expensive which I want to avoid.
• I am trying to add a GUI to this system for primarily two purposes. First, let the users view (a read operation) the job status and management information so they can get a big picture of what's going on and whether there is any error in any steps. Second, allow the users to redo one or more steps by changing (a write operation) the job status and management information.

My Current Solution

• I am thinking of using Django to develop the GUI because:
  • Django is fast on development; Django is web-based so almost no installation is required;
  • I want to enable remote monitoring of the system so a web-based, in-browser GUI makes more sense;
  • I used some Django before so had some experience.
• However, I see Django mostly works with a real database: SQLite, MySQL, PostgreSQL, etc.. The user-defined model will be matched into the tables in these databases by Django automatically. However, the legacy system only produces text files.

My Thoughts & Questions

• Fortunately, I noticed that the text files are all in one of the two formats:
  • Multiple lines of strings;
  • Multiple lines of key-value pairs.
• Both formats look easy to match to a database table design. For example, a "multiple lines of strings" can be considered as a DB table of a single column of text, while a "multiple lines of key-value pairs" as a two-column table.
• Therefore, my question is: Can I build my model upon local text files instead of a real database, and somehow override some code somewhere in Django that acts as the interface between the core framework and the external database, so these text files will play the role of a "database" to Django and the reading/writing operations will happen to these files?? I've searched on internet and stackoverflow but wasn't lucky enough. Will appreciate if you can give me some helpful links.

Answer 1

What Not to Do.

If you are going to reproduce an RDBMS using files you are in for a lot and I mean a lot of grief and hard work. Even the simplest RDBMS like sqlite has thousands of man hours of work invested on it. If you were to bring your files into django or any other framework you would need to write a custom backend for it.

What To Do

Create django models backed by an RDBMS and import the files into it. Alternatively since this data appears to be mostly in Key Value pairs, you might be able to use Mongodb or redis.

You can use inotify to monitor the file system to detect when a new file has been created by the batch processing system. When that happens you can invoke a django CLI script to process that file and import it's data into the database.

The rest of it is a straight forward django app.