Kinesis Data Analytics (Flink) - how do I configure environment variables?

we are currently running a flink cluster in a standalone mode on Kubernetes. We have wanted to explore whether we could migrate over to managed flink on AWS (KDA).

But I don't seem to find any documentation or indication that it is possible to inject environment variables? Do these need to be provided as runtime arguments?

Related, is it possible to override default flink configurations that we currently specify in our flink-conf.yml in managed Flink?

Thanks in advance!

Solution

I'll answer my own question, I seems there is no way to provide the environment variables in the same fashion you would with configmaps in Kubernetes. Instead, we need to use the Runtime properties that can be defined in KDA. These can then be retrieved using the KinesisAnalyticsRuntime.getApplicationProperties()

As an example:

      val params: ParameterTool = ParameterTool.fromArgs(args)
      val config = params.get("env", "") match {
        case "local" => AppConfiguration.initialize(sys.env)
        case _ => // KDA
          val kdaProperties = KinesisAnalyticsRuntime.getApplicationProperties()
          logger.error(
            s"kdaProperties $kdaProperties",
            Some(Map("kda" -> kdaProperties))
          )
          Option(kdaProperties.get("DevProperties")) match {
            case Some(kdaProperties) =>
              val kdaPropsToMap = kdaProperties.asScala.toMap
              AppConfiguration.initialize(kdaPropsToMap)
            case None =>
              logger.error(s"could not read KDA runtime properties", Some(Map("kda" -> kdaProperties)))
              throw new Error(
                "unable to read KDA runtime properties"
              ) // scalafix:ok
          }
      }

And where the grouping key defined in KDA for the Runtime properties is used to fetch these.

This also means configuring flink-conf.yml will be possible as Runtime properties which then need to be set during runtime (is my understanding)