Can't check key distribution with FNV-1 algorithm

Question

As recommended by a lot of places, I am using the FNV-1 method to generate my partition keys for a Service Fabric cluster with 5 partitions as below:

[
  {
    "lowKey": -5534023222112865000,
    "highKey": -1844674407370955300,
    "id": "92b4c32e-cb3c-432f-aea0-4579fc72850c",
    "kind": 2
  },
  {
    "lowKey": 5534023222112865000,
    "highKey": 9223372036854776000,
    "id": "dd9b084d-88fa-4773-94d9-f634d22a2c03",
    "kind": 2
  },
  {
    "lowKey": -9223372036854776000,
    "highKey": -5534023222112865000,
    "id": "d7322e52-cf1e-47c2-b713-e983f443b533",
    "kind": 2
  },
  {
    "lowKey": -1844674407370955300,
    "highKey": 1844674407370955300,
    "id": "dfe2f994-bdc7-4b32-b54f-78a0245e4c4c",
    "kind": 2
  },
  {
    "lowKey": 1844674407370955300,
    "highKey": 5534023222112865000,
    "id": "6ad81ffb-5f3c-48ba-91d9-e65b23c528a4",
    "kind": 2
  }
]

I want to check what the distribution of my data is like.

However, I can't see how to do this because the numbers are so large.

I am assuming that I have to use LowKey="-9223372036854775808" HighKey="9223372036854775807" for this?

If so, these numbers are too big/small for me to process in either SQL or C# (C# because the constants are too big).

Is there a way around this? I get into overflow issues if I try to save these values in my database.

I am using SQL Server 2017.

DECLARE @tbl TABLE(LowKey BIGINT, HighKey BIGINT, TotalITems INT)

INSERT INTO @tbl(LowKey, HighKey, TotalITems) VALUES (5534023222112865000, 9223372036854776000, 0)
INSERT INTO @tbl(LowKey, HighKey, TotalITems) VALUES (-9223372036854776000, -5534023222112865000, 0)

INSERT INTO @tbl(LowKey, HighKey, TotalITems) VALUES (-5534023222112865000, -1844674407370955300, 0)
INSERT INTO @tbl(LowKey, HighKey, TotalITems) VALUES (-1844674407370955300, 1844674407370955300, 0)
INSERT INTO @tbl(LowKey, HighKey, TotalITems) VALUES (1844674407370955300, 5534023222112865000, 0)

The first 2 lines fail with the error

Msg 8115, Level 16, State 2, Line 168
Arithmetic overflow error converting expression to data type bigint.

Here is the logic I use to see what the partitions are:

[ActionName("GetPartitions")]
public async Task<ActionResult> GetPartitionsAsync()
{
   var result = new List<ServicePartitionInformation>();
   ServicePartitionList partitions = await FabricClient.QueryManager.GetPartitionListAsync(FullServiceName).ConfigureAwait(false);

   foreach (var partition in partitions)
   {
       result.Add(partition.PartitionInformation);
   }

   return new JsonResult(result);
}

Paul

Answer 1

Service Fabric can use a UniformInt64Partition. That means a minimum low of -9223372036854775808 and a maximum high of 9223372036854775807.

You can check that using

Console.WriteLine(long.MinValue);
Console.WriteLine(long.MaxValue);

So your json is incorrect. For example, -9223372036854776000 is not a valid long/Int64 value.

I am assuming that I have to use LowKey="-9223372036854775808" HighKey="9223372036854775807" for this?

If so, these numbers are to big/small for me to process in either SQL or C# (C# because the constants are too big)

Can't be, these are the defaults for a stateful service which is Int64 based.

<Service Name="Stateful1" ServicePackageActivationMode="ExclusiveProcess">
  <StatefulService ServiceTypeName="Stateful1Type" TargetReplicaSetSize="[Stateful1_TargetReplicaSetSize]" MinReplicaSetSize="[Stateful1_MinReplicaSetSize]">
    <UniformInt64Partition PartitionCount="[Stateful1_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
  </StatefulService>
</Service>

When I run your code against a new default stateful service I get

[
    {
        "LowKey": 5534023222112865485,
        "HighKey": 9223372036854775807,
        "Id": "ef982303-dd13-49f0-ad3c-5de7428eccad",
        "Kind": 2
    },
    {
        "LowKey": -1844674407370955161,
        "HighKey": 1844674407370955161,
        "Id": "e2e5f51b-d3f5-4381-b7ff-6267124d0adb",
        "Kind": 2
    },
    {
        "LowKey": -5534023222112865484,
        "HighKey": -1844674407370955162,
        "Id": "4715db41-124a-4eee-b5b9-2587256ed6e1",
        "Kind": 2
    },
    {
        "LowKey": -9223372036854775808,
        "HighKey": -5534023222112865485,
        "Id": "10e9d445-53e0-4f27-9434-0ea5621ac188",
        "Kind": 2
    },
    {
        "LowKey": 1844674407370955162,
        "HighKey": 5534023222112865484,
        "Id": "687c1c4d-9fc1-4574-a916-66ac5ebfed25",
        "Kind": 2
    }
]

As you can see your highkey 9223372036854776000 and lowkey -9223372036854776000 are invalid and are not in my generated json. Not sure where that comes from then.

My code to reproduce:

    protected override async Task RunAsync(CancellationToken cancellationToken)
    {
        var result = new List<ServicePartitionInformation>();
        ServicePartitionList partitions = await new FabricClient().QueryManager.GetPartitionListAsync(Context.ServiceName).ConfigureAwait(false);

        foreach (var partition in partitions)
        {
            result.Add(partition.PartitionInformation);
        }

        var info = JsonConvert.SerializeObject(result);
        Console.WriteLine(info);
    }