GCP Dataproc - Error: Unknown name "optionalComponents" at 'cluster.config': Cannot find field

Question

I am trying to create dataproc cluster using configurations mentioned in YAML file (using import):

The command I have been using successfully:

$ gcloud beta dataproc clusters import $CLUSTER_NAME --region=$REGION 
--source=cluster_conf_file.yaml

Later on I tried adding HABSE component which is a part of available optional components using attribute --optional-components:

$ gcloud beta dataproc clusters import $CLUSTER_NAME --optional-components=HBASE --region=$REGION 
--source=cluster_conf_file.yaml

(Documentation referred: https://cloud.google.com/dataproc/docs/concepts/components/hbase#installing_the_component)

Which caused below error:

ERROR: (gcloud.beta.dataproc.clusters.import) unrecognized arguments: --optional-components=HBASE 

Then I tried adding the attribute --optional-components as optionalComponents in the YAML file (instead of passing through command line) by referring this documentation.

Sample YAML:

config:
  endpointConfig:
    enableHttpPortAccess: BOOLEAN_VALUE
  configBucket: BUCKET_NAME
  gceClusterConfig:
    serviceAccount: SERVICE_ACCOUNT
    subnetworkUri: SUBNETWORK_URI
    tags:
      - Tag1
      - TAG2
  optionalComponents:            <---- Attribute causing error
    - HBASE
  softwareConfig:
    imageVersion: IMAGE_VERSION
    properties:
      PROPERTY: VALUE
      .
      .
      .
  masterConfig:
    diskConfig:
      bootDiskSizeGb: SIZE
      bootDiskType: TYPE
    machineTypeUri: TYPE_URI
    numInstances: COUNT

Which caused below error:

ERROR: (gcloud.dataproc.clusters.import) INVALID_ARGUMENT: Invalid JSON payload received. Unknown name "optionalComponents" at 'cluster.config': Cannot find field.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: "Invalid JSON payload received. Unknown name \"optionalComponents\"\
      \ at 'cluster.config': Cannot find field."
    field: cluster.config

Is there a way to fix this?

Answer 1

optionalComponents should be under config.softwareConfig:

config:
  ...
  softwareConfig:
    imageVersion: IMAGE_VERSION
    optionalComponents:
    - ZOOKEEPER
    - HBASE

You can prove it by first creating a cluster with optional components, then export it to a YAML file.