Search code examples
modulequery-optimizationxquerymarklogic

How to implment auto increment numberic primary key for MarkLogic with Xquery


How to impmlement Auto-Increment-Primary-Key RMDB feature in MarkLogic?

Any side effect if that key is stored as an element in a configuration XML file and use "Singleton" pattern of XQuery libary to generate it. (Multiple difffernet Xquery code will use that same libary to gerenate the unique key.) I guess it won't work for a ML cluster? It may only work on a single node ML. Will that be the bottomeneck of the multi threaded Corbs rountine?


Solution

  • Incrementing numeric keys are generally considered an anti-pattern, as it will require locking and shared state, which can become a performance bottleneck. It will work, but it just limits your performance and throughput.

    You should really consider whether you need sequential numeric values, or just unique numeric values.

    You could leverage xdmp:random() to generate a random 64-bit numeric value. It just won't be sequential.

    Or, for non-numeric unique values, sem:uuid-string()

    If you really, really, need auto-incrementing numbers then you would need to have a document to track that number, and then obtaining the next value in the sequence would involve reading that document and updating it's value (which would acquire an exclusive write lock). You could get more elaborate and have shards responsible for ranges of numbers in an attempt to get some additional throughput and a parallel activity at the expense of not 100% sequential with regards to insert time. You could draw inspiration from Twitter Snowflake: https://github.com/twitter-archive/snowflake/tree/snowflake-2010