Search code examples
loggingdatabase-designmongodb-queryviewmodelview-model-pattern

should we Log in same Database or separate database

So far I was using a file system for the logging, now planning to move for the Database system.

I am planning to the same database and different collection ( NoSQL term) /table for the logs. Is it safe to use the same database or better to use a different database for loggers?

Does it require indexing, the user can query to get all the logs between this day to this day and will show only X logs in the current screen, on click on next button it will fetch another X logs. Is it good to load all the data asked from the user for a range of day and keep in memory so for next and back call frontend can ask respective data?

If anyone of you already have database connection for logger could you please suggest the best practice and most failure scenario.

Solution

  • I have seen quite a few loggers running against ElasticSearch (whole ELK stack, actually). There are things to consider if you are going to implement a new solution:

    1. Logger has to be fast. It may not slow down the application. If you are implementing it yourself, circular buffer and multi-thread processing is a good start.
    2. There will be A LOT of data. Separate DB is a must. This way you can move it later to a server with cheaper storage or scale it to your needs or whatever.
    3. Log data is time-based. Designing a storage you may want to do hourly/daily partitions for easier manipulation.
    4. Same goes for reading. You would expect data to be ordered by time, latest data is the most needed. Your indexes should accommodate that.
    5. Technical things like pagination and caching depends on the database vendor. I don't know much about MongoDB, but I guess pagination should be in place. So, clicking Next you should just request some extra rows from previous query.

    One more point: personally, I would not spend my time implementing something new. I would install an ELK instance, configure Log4J (or whatever your application logger is) and spend extra time teaching service desk guys to use the new environment.