Search code examples
hadoopintegrationkdbin-memory-database

Is it possible to integrate Kdb+ and hadoop?


Does KDB+ can be integrated with Hadoop like OpensTSDB/MongoDB/Cassandra?

KDB+ works in distributed architecture and supports mapreduce. Is it an alternate option to Hadoop?

Can anyone explain in detail about this?


Solution

  • kdb is a column oriented database plus and array based language q.

    I wrote a comparison of the top column oriented databases, which should give you an idea of the common competitors.

    I consider kdb and hadoop very different, as for the use cases I have seen the overlap is small. What is the problem you are trying to solve?

    Sayantan Ghosh wrote this comparison https://www.quora.com/Whats-the-difference-between-Hadoop-and-KDB:

    The major differences are between KDB and Hadoop are :

    KDB uses a terse programming language Q. In Q the expressions are evaluated from right to left instead of left to right for other programming languages like Java and C++. Hadoop is based on Java though you can write map reduce jobs in a lot of languages like Java and Python. So, for a programmer, getting the hang of Q may take some time, though it depends from person to person, while I would expect a developer to be up to speed with Hadoop really fast. q-SQL is a SQL like language to query the KDB database. In the hadoop world, the corresponding language is Hive. Hive and q-SQL were mainly built so as to make it easy for business or analyst folks to query the database without prior knowledge of KDB+ or hadoop. KDB+ has an inmemory component called the RDB(real time database) which holds the current days data. The historical data is stored in the historical database(HDB). Hadoop does not have any real time component. It uses the HDFS(Hadoop File System) to store the data.