Search code examples
azureazure-cosmosdb

Azure Cosmos DB not working as excepted -get throughput control to work with Cosmos DB Spark connector

I am trying to get throughput control to work with Cosmos DB Spark connector, but I don't see the new containers created.

Here is the code:

val changeFeedCfg = Map("spark.cosmos.accountEndpoint" -> cosmosEndpoint,"spark.cosmos.accountKey" -> cosmosMasterKey,"spark.cosmos.database" -> cosmosDatabaseName,"spark.cosmos.container" -> cosmosContainerName,"spark.cosmos.read.inferSchema.enabled" -> "false","spark.cosmos.changeFeed.startFrom" -> "Beginning","spark.cosmos.changeFeed.mode" -> "Incremental","spark.cosmos.changeFeed.itemCountPerTriggerHint" -> "100000"//optional configuration for throughput control// "spark.cosmos.throughputControl.enabled" -> "true",// "spark.cosmos.throughputControl.name" -> "SourceContainerThroughputControl",// "spark.cosmos.throughputControl.targetThroughputThreshold" -> "0.30",// "spark.cosmos.throughputControl.globalControl.database" -> "database-v4",// "spark.cosmos.throughputControl.globalControl.container" -> "ThroughputControl"
"spark.cosmos.throughputControl.useGatewayMode" -> "true")`

I had tried the code and it is not working as excepted.

Solution

  • I see you are using Gateway mode.

    "spark.cosmos.throughputControl.useGatewayMode" -> "true"

    Currently we do not have support for throughput control in gateway mode

    https://github.com/Azure/azure-sdk-for-java/issues/34229