Search code examples
apache-flinkflink-streaming

Apache flink Kafka Connector not found


I'm fairly new to flink/Java/Scala so this might be a non-question but any help is appreciated. I haven't been able to find an example that uses the Flink Kafka connector with Flink 1.13 (and works for me).

My project is here: https://github.com/sysarcher/flink-scala-tests

I want to I'm unable to use FlinkKafkaConsumer (link) which I want to try out.

I'm using IntelliJ Idea. The project was generated from the tutorial on Flink's website

  • The First problem seemed to be the provided scope as suggested here: https://stackoverflow.com/a/63667067/3760442 ... Now, DataStream API (and the example) seem to work.
  • The current problem is that I'm not able to use the Kafka connector which I'm looking to try out.

The following link was used to generate the project: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/try-flink/datastream/#how-to-follow-along

Error Encountered

$ mvn clean compile
[INFO] ...
[WARNING] Multiple versions of scala libraries detected
[ERROR] /hide/abs/path/github/flink-stuff/frauddetection/src/main/scala/spendreport/FraudDetectionJob.scala:42: error: not found: type FlinkKafkaConsumer
[ERROR]     val kafkaConsumer = new FlinkKafkaConsumer[String]("car.create", new SimpleStringSchema(), properties)
[ERROR]                             ^
[ERROR] one error found

UPDATE:

One problem was the missing import statement:

import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

I can build the artifacts using mvn clean compile but IntelliJ still complains with the following error:

/long/path/flink-stuff/frauddetection/src/main/scala/spendreport/FraudDetectionJob.scala:30:35
object connectors is not a member of package org.apache.flink.streaming
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

Solution

  • It turns out the problem was with IntelliJ's indexing (or something, I'm not sure). I pressed CTRL+SHIFT+A and typed Reload All Maven Projects. Thereafter, the build was successful.

    This comment helped: https://lists.apache.org/x/thread.html/rc332d1054886e35af65de4a7c38553e02a2304a2ebdab8197badd94d@%3Cuser.flink.apache.org%3E

    It seems sometime IntelliJ does not works well for index, perhaps you could choose mvn -> reimport project from the context menu, if it still not work, perhaps you might try remove the .idea and .iml file and re-open the project again.