Can't read data from Nifi output port using Flink-nifi connector

Question

I have an instance of Nifi in docker on virtual machine with exposed ports: 8080 and 10000. On thin instance i created a simple pipeline with output port named "flink" and i want to read this data using flink-nifi connector:

 SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
                .url("http://vm-address:8080/nifi")
                .portName("flink")
                .requestBatchCount(100)
                .buildConfig();
        DataStream<NiFiDataPacket> nifi = environment.addSource(new NiFiSource(clientConfig));

        nifi.map(new MapFunction<NiFiDataPacket, JsonNode>() {
            @Override
            public JsonNode map(NiFiDataPacket value) throws Exception {
                return DataConverter.byte2Json(value.getContent());
            }
        }).print();

In this case i got error: Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: java.net.UnknownHostException If i add localAddress in config:

 SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()
                .url("http://vm-address:8080/nifi")
                .localAddress(InetAddress.getByName("vm-address"))
                .portName("flink")
                .requestBatchCount(100)
                .buildConfig();

I got this error: Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: java.net.BindException: Cannot assign requested address: JVM_Bind

I run this code from local pc on windows and flink is started in standalone mode. Also, i tried to run this directly on virtual machine but i got the same error.

---

In logs there is a lot of retries:

execchain.RetryExec: I/O exception (java.net.BindException) caught when processing request to /vm-address->{}->http://vm-address:8080: Cannot assign requested address: JVM_Bind

Answer 1

Finally, solved it! The problem was in my docker configuration. Firstly, i run nifi like this: docker run --name nifi -p 8008:8080 -p 10000:10000 -d apache/nifi:1.7.1 The network, by default, was bridge. In this case my container has some random hostname and i don't communicate with container directly, but through the docker. When i choose network=host: docker run --name nifi --network host -d apache/nifi:1.7.1 everything goes well. Probably, i could solve it another way (maybe, explicitly resolve container hostname), but this was the easiest way