Document AI Custom Doc Splitter - Internal error encountered

I created a training set for custom document splitter with a total 3803 docs and 158 labels. I checked with documents quota and limits and all my docs and pages are within the limits. When I run the training I got this following error message

{
  "name": "projects/945951207848/locations/us/operations/4890234249887707766",
  "done": true,
  "result": "error",
  "response": {},
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.documentai.uiv1beta3.TrainProcessorVersionMetadata",
    "commonMetadata": {
      "state": "FAILED",
      "createTime": "2023-10-11T10:41:01.258619Z",
      "updateTime": "2023-10-11T16:41:45.453832Z",
      "resource": "projects/945951207848/locations/us/processors/fd97f52566718e15/processorVersions/43dd9a2405017f3c"
    },
    "trainingDatasetValidation": {},
    "testDatasetValidation": {}
  },
  "error": {
    "code": 13,
    "message": "Internal error encountered.",
    "details": []
  }
}

I retrained it again, but I still get the same error.

Solution

I checked with the product development team, and this is due to a product infrastructure issue that is in the process of being resolved.

For this particular instance, reducing the number of docs and labels will prevent the issue from occurring. However, after the issue is resolved, you shouldn't need to reduce the number of docs/labels.