[Question posted by a user on YugabyteDB Community Slack]
I just setup a Yugabyte cluster. Before I migrate my database (6TB) over to yugabytedb cluster, I wanted to get my configs right. What is the recommended way of configuring automatic tablet splitting for my database with just 20 tables (out of which only 5 tables are 0.5-1.5TBs large, the rest are small). We plan to keep the number of tables low (<30) always, but each table will grow upto multiple TBs. [ YB Cluster Spec: 10 nodes x 32 core x 512 GB Memory x 5 TB SSD ]
Sadly, without having a full understanding of a situation, the most correct answer is "it depends", which doesn't help you. With just < 30 tables, you can just go with the default settings (presplit tables -- where the number of tablets is automatically determined, rather than automatic tablet splitting).
Let me give an example here about "tablets is automatically determined", doing the math from your spec ([ YB Cluster Spec: 10 nodes x 32 core x 512 GB Memory x 5 TB SSD ]) and based on 30 DocDB tables (which is what you will have with 20 tables 10 secondary indexes)
On > 4 vCPUs servers like here, this "automatically determined" will be 8 tablets per server which means that each table or index create will be split to 80 tablets.
For 30 tables, that means 30*80 = 2400 tablets in total. If Replication Factor RF=3 (the minimum for HA) this means 7200 distributed on 10 servers = 720 tablets peers per server.
RAM: If they are all active this will be 90 GB of regulardb memtable, more with intents. Plus the DocDB cache, plus the filesystem cache. That may fit in the 512GB servers. CPU: 720 tablet peers should be ok 32 cores if they are not active all the time. You say 32 cores, does it mean 64 threads?
Recommendation, with auto-split disabled (enable_automatic_tablet_splitting=false) and with all other defaults you will have 80 tablets per table.
However, if you have small tables you can reduce their number of tablets to one per server with a SPLIT INTO 10 TABLETS at CREATE TABLE time Note that this supposes HASH sharding (the default). If you have range queries or pagination, you will create range sharded indexes (mentioning ASC or DESC) or maybe even for the primary key. Those will be created with only one tablet, whatever the settings. You can define the range split when creating them, or this is where auto-splitting may be useful.
Another point: 10 servers is not a multiple of the replication factor. do you plan fault tolerance per server and not per zones?
1st you should identify the range queries so that the corresponding primary key or secondary indexes are defined as range (ASC or DESC).
Then suggest that you identify the small reference tables for which you better have 1 table replicated with duplicate indexes. And medium tables where one tablet per server is probably far enough. And the large ones that you pre-split. And then enable automatic splitting which may split more, especially range ones where you may not know at which value to split.