Search code examples
databaseimagecassandrastorelarge-data

Store large amount of images on multiple servers

I would like to know what is the best solution for storing large amount of images on multiple servers like google, facebook.

It seems that storing in filesystem is better then inside a database but what about using a noSQL DB like cassandra.

Do Google/Facebooke store the same image in multiple servers for the load balancing. How does it work? What is the best solution?

Thx a lot

Solution

  • There's nothing wrong with the approach you're taking. As mentioned, there are caveats, however, the possibilities do exist, and a lot of people and companies are successfully storing files in Apache Cassandra.

    • zjffdu/cassandra-fs is the first solution i'd look into. Now, this was last developed 2 years ago, so I'd be a bit cautious on it working the first time, out of the box. Apache Cassandra is now at version 1.0.x, with 1.1.x on the way. 2 years ago, that was version 0.6.x maybe? A lot has changed & improved in 24 months.
    • semantico/cassandra-fs a fork ... last touched 7 months ago
    • favoritas37/cassandra-fs another fork ... last touched 3 months ago and indicates compatibility with the 1.0.5 branch of Cassandra

    The principal behind this is to take a file, break it into a set of chunks and store those chunks as columns in a row. When retrieving, pull each column, reassemble the file and voila.

    Cassandra FAQ: large file and blog storage

    ...files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks...

    Lucene indexes in Cassandra

    ...its files are broken down into blocks (whose sizes are capped), where each block (see FileBlock) is stored as the value of a column in the corresponding row...

    You'll get more positive feedback on the Cassandra mailing list and on the IRC channel.

    Finally, this is from 2009, and written by folks at Facebook, which should go some way to help answer more of the fundamental questions you have: Cassandra - A Decentralized Structured Storage System.