Search code examples
mysqlfile-ioserver-load

Can massive writing & deleting files hurt our server performance?


We run a system that for cache purposes, currently writes and deletes about 1,000 small files (10k) every hour. In the near future this number will raise to about 10,000 - 20,000 files being written and deleted every hour.

For every files that is being written a new row on our mysql DB is added and deleted when the file is deleted an hour later.

My question:

  1. Can this excessive write & delete operation hurt our server performance eventually somehow? (btw we currently run this on a VPS and soon on a dedicated server.)
  2. Can writing and deleting so many rows eventually slow our DB?

Solution

  • This greatly depends on operating system, file system and configuration of file system caching. Also this depends on whether your database is stored on the same disk as files that are written/deleted.

    Usually, operation that affect file system structure such as file creations and file deletions require some synchronous disk IO, so operating system will not loose these changes after power failure. Though, some operating systems and file systems may support more relaxed policy for this. For example, UFS file system on FreeBSD has nice "soft updates" option that does this. Probably etx3/Linus should have similar feature.

    Once you will move to dedicated server I think it would be reasonable to attach several HDDs to it and to make sure that database is stored on once disk while massive file operations are performed on another disk. In this cases DB performance should not be affected.