Parallel Table Creation Performance Issue in Ignite 2.16

Question

I am working in a Node.js 20 environment and using Ignite 2.16 to communicate with Ignite via the REST API. My application primarily performs SQL operations on tables.

While Ignite provides excellent performance when creating and inserting into one table at a time (around 0.2 seconds), I am encountering performance issues when attempting to create a large number of tables in parallel. It takes approximately 15 seconds to create 100 tables simultaneously.

I suspect this delay is due to the synchronous nature of table creation in Ignite. My application requires the ability to create and insert into multiple tables concurrently, and we are not using persistence for storage. Increasing the number of server nodes does not seem to improve the parallel table creation time.

Is there a configuration setting or approach that can enable more efficient parallel table creation in Ignite?

I have tried scaling Ignite both horizontally and vertically, along with increasing queryThreadPoolSize and queryParallelism in cache configuration to 1000 and 100 respectively in Ignite XML configuration. However, that didn't help. I also tried adding parallelJobsNumber in the collisionSpi property.

Here are the Collision and Cache configurations I have used:

<property name="collisionSpi">
    <bean class="org.apache.ignite.spi.collision.fifoqueue.FifoQueueCollisionSpi">
        <!-- Execute 1000 jobs at a time. -->
        <property name="parallelJobsNumber" value="1000"/>
    </bean>
</property>
              
<bean class="org.apache.ignite.configuration.CacheConfiguration">
    <property name="name" value="myTableCache"/>
    <property name="backups" value="0"/>
    <property name="queryParallelism" value="10"/>
    <property name="sqlOnheapCacheEnabled" value="true"/>
    <property name="onheapCacheEnabled" value="true"/>
</bean>

Thanks!

Answer 1

Creating and dropping tables is a relatively heavy operation in Ignite and not something you want to do very often. It's a cluster-wide operation, so it cannot be performed in parallel.

You can create multiple tables simultaneously using the Java API (Ignite#createCaches()).

You can insert/update/delete records across multiple tables in parallel. Using the node.js client rather than the REST API will give you better control and performance.