I'm creating an Archiving System that let me store scanned files in a directory. I put my images in one directory called upload.
I made the system using Laravel framework in windows base XAMPP server.
The files will be corrupted or will reach file limit.
The searching of images could be very slow
As of this moment I have 12 thousand images stored in that directory and find searching not a problem. But I worry it could be troublesome in the future.
So any recommendation or storing million images in one directory is fine.?
Hopefully that helps in removing your concerns:
One thing that worries me is that images that will be stored in that directory can go about millions.
Hardware right now is capable of handling a huge number of files for a cheap price, and it heavily depends on the platform you are building and the ratio those files are growing. But if you are not into managing this yourself, a solution like Amazon S3 can help.
In the case of directories, preferably you use a mechanism that helps you splitting those files in multiple ones, an example would be to use the first 2-3 characters of the name of the file to build a directory you can store the file in :
test.doc will turn into t/e/s/test.doc
The files will be corrupted or will reach file limit.
If the upload process is done successfully, the case where the file is corrupt needs more logic to be handled; depending on the file type, like this
The searching of images could be very slow
An index is your best solution , to have a database of all the files and its paths, this will easily give you a direct access to file without going through searching for them; keeping the name, path, and other info for example.
if you are talking about searching their content, if it is not only images, I recommend taking a look at Lucene based products like Solr or Elastic Search