I have already tried out keeping in text file(Obviosuly it takes more space as it is not customized to hold tables as an excel is). I also tried keeping serialized data(c#, just an fyi). There wasn't any benefit in either cases.
Since most 'tabular' data will be text, you'll get large gains in compressing this before saving (using GZip and BsonWriter as an example):
public static byte[] Compress(object entity)
{
using (var stream = new MemoryStream())
{
using (var zipStream = new GZipStream(stream, CompressionLevel.Optimal))
{
using (var writer = new BsonWriter(zipStream))
{
var serializer = new JsonSerializer();
serializer.Serialize(writer, entity);
}
}
return stream.ToArray();
}
}
** Update **
Protobuf is far superior to both XmlSerializer and BinaryFormatter in terms of serialized entity size and speed. Try using this to serialize your entities before compressing:
public static byte[] Compress(object entity)
{
using (var stream = new MemoryStream())
{
using (var zipStream = new GZipStream(stream, CompressionLevel.Optimal))
{
Serializer.Serialize(stream, entity);
}
return stream.ToArray();
}
}