Search code examples
mysqlmongodbloggingevent-logdatabase

Is it good to store mysql event logs in MongoDB?

Earlier in our database design, we use to create mandate fields for each of the table and few important fields were:

created_by
created_time
created_by_ip
updated_by
updated_time
updated_by_ip

Now, its an era of no-schema design. We prefer mongodb or some other just writing databases.

My question here is:

  1. Is it a good practise to maintain logs in a separate database?

  2. Do we need to create separate log table for each mysql tables considering mongodb or is it okay to have single mongodb audit table for all mysql tables?

  3. What things need to be considered in querying the results from mongodb?

  4. What should be the structure for mongodb table structure?

  5. Any other alternatives to store logs?

Considering situation where if we want to delete registered user if not authenticated in specified time(max of 48hrs).

If all the time logs are handled in mongodb. How can we query the same from mysql?

Solution

  • You usually want this (audit?) data next to the real data and definitely not in a different DB engine as the number of partial errors to support becomes quite a nightmare (e.g. someone registered, but you fail to insert audit data - is this ok? should the account become orphan? What happens if the app goes down half way?).

    Systems that have this separation usually use messaging and 2 different listeners are responsible for storing the data and storing the audit (e.g. one in a relational DB and the other in an event store). In this way you have a higher chance of achieving eventual consistency.

    Edit

    There are a few options around using messaging and the assumption here is that both sources of data must be in sync (or as close as possible). Please bear in mind that I still think that storing data+audit together is by far the simplest and more sensible approach.

    Using messaging, your app can emit a message on certain events (e.g. user created). Then 2 different listeners react to this message. One listener stores the data in one DB engine; Another listener stores the audit data. The problem with this approach is that you might need to ensure ordering on the messages, which makes it really slow.

    Another (scary) approach is to use distributed (XA) transactions between MySQL and a messaging system (as mongo doesn't support transactions). Then the data to MySQL and the message would be committed together, and a listener can receive the audit data and store it in mongo.

    I need to emphasize that the 2 approaches above are horrible and should never be implemented.

    There are more sensible approaches but might require a different tech stack. For example using an EventSourcing+CQRS you can store the events (with the audit data) and store the final read models without the audit data.