We are running a flink cluster 3 nodes, and S3 as state backend. What we have noticed is after a couple of weeks of running, one of the jobs starts failing with checkpoint failures. At this point, we also loose the "Web UI" console for cluster so we can cancel ths failing job and restart it via UI console. As the Web UI Rest call are throwing 500 error for /jars endpoint. At this point we need to restart the cluster and resubmit jobs to recover from the error. Any insights into this will be helpful. Flink version is 1.3.1
Following is list of errors from
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
java.lang.Exception: An async function call terminated with an exception. Failing the AsyncWaitOperator.
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Async function call has timed out.
Caused by: java.util.concurrent.TimeoutException: Async function call has timed out.
java.lang.Exception: An async function call terminated with an exception. Failing the AsyncWaitOperator.
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Async function call has timed out.
Caused by: java.util.concurrent.TimeoutException: Async function call has timed out.
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/361dc299a5660bf46f335c98cc55a000dab0b6b23eb050cd42791f54127ec3c5.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/c235988e4448b0649b609261fff311be685205cc7394eb9a0cb66c958198ca6c.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/361dc299a5660bf46f335c98cc55a000dab0b6b23eb050cd42791f54127ec3c5.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/8fd69bc697889096d9d7f5595144dcb655c3d0f72f729870fe32d21c8f4c59b6.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/8fd69bc697889096d9d7f5595144dcb655c3d0f72f729870fe32d21c8f4c59b6.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/6851aa013bdf007587e023a880d799ab11a0feee5a2cb12cc411f94132c7114b.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/dc36c57d88c098e78733a2d376c6599b18ef47d12575bdfd3ece47908f6bb8d3.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/6851aa013bdf007587e023a880d799ab11a0feee5a2cb12cc411f94132c7114b.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/a92bffa6cc199b7ba29ebfcac5db914cc83291da3edf53ff3d59cdd0abf66e4b.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/a92bffa6cc199b7ba29ebfcac5db914cc83291da3edf53ff3d59cdd0abf66e4b.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/53327713f638a37dd59cc0a4e72e763229e91e050d1767af8608fb0418f2fb39.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/53327713f638a37dd59cc0a4e72e763229e91e050d1767af8608fb0418f2fb39.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/0e9ca7e5a82fce473fb6ffb698a59e05fea25e97bc0b67c226b893f25c56a176.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/6c7cbdd10df09fc93793d94d6b1aeb7c3cd21f2839b92c380796aaf27d56f02f.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/6c7cbdd10df09fc93793d94d6b1aeb7c3cd21f2839b92c380796aaf27d56f02f.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/cb4062b597c2b31aa1e57b51e1ce993fca80aff6dc41ed4164a61f9f86171c96.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/cb4062b597c2b31aa1e57b51e1ce993fca80aff6dc41ed4164a61f9f86171c96.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/704e543afb61a45f1e5d0fb919422514ee6634feef72224803b6213d4bad30e1.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/704e543afb61a45f1e5d0fb919422514ee6634feef72224803b6213d4bad30e1.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/5de242870ca414f84c0a172ceae717ca3a018ec707e9ea36b510c6bcacfa05b6.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/5de242870ca414f84c0a172ceae717ca3a018ec707e9ea36b510c6bcacfa05b6.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/f8e8def8cf4fe3486136672eb2c06aff2e1f666522b031d80d09e7c944aa5f3c.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/f8e8def8cf4fe3486136672eb2c06aff2e1f666522b031d80d09e7c944aa5f3c.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/2d1b032e661e18675524f72255a65a6da769d3357250083b90e3481bfcc028d9.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/2d1b032e661e18675524f72255a65a6da769d3357250083b90e3481bfcc028d9.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/8e469dc73aa082f60889edb39bd4fc964606a99941e0c26b63d0b795adf9c59c.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/8e469dc73aa082f60889edb39bd4fc964606a99941e0c26b63d0b795adf9c59c.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/486a86de1d3739614fb7ab35589fb2c5f0d7ee963a4fd60011246966005e5863.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/486a86de1d3739614fb7ab35589fb2c5f0d7ee963a4fd60011246966005e5863.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/2be77bff7d041d57cf50a744f3d9393559f70488f99d6c248cc13b7e3ddea35a.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/2be77bff7d041d57cf50a744f3d9393559f70488f99d6c248cc13b7e3ddea35a.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/fcccafac16636433ee3cd487c9479361885219ca72194da913478a47cfbb8278.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/fcccafac16636433ee3cd487c9479361885219ca72194da913478a47cfbb8278.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/dc163596a1f387e3689c4d9784d1a9eb613a7d88c01bd07611ad91bebc2bb1ea.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/dc163596a1f387e3689c4d9784d1a9eb613a7d88c01bd07611ad91bebc2bb1ea.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/c0692b18f402d398b4011a05a6f1c0d603e1b29bf0dca0de4779a4bdd6e55c1f.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/c0692b18f402d398b4011a05a6f1c0d603e1b29bf0dca0de4779a4bdd6e55c1f.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/6942b876e25f74be283e03d6c80976d8d02ffb4c00ee42f12c412bcfcb21220e.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/6942b876e25f74be283e03d6c80976d8d02ffb4c00ee42f12c412bcfcb21220e.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/9e1c86c32f24641d771ddf586e1caf08293bbef64c2481cda6f65b003c9cc2c5.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/9e1c86c32f24641d771ddf586e1caf08293bbef64c2481cda6f65b003c9cc2c5.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/de71895b951861ddc8c216993f3d7a728df14e2d8c6e238711deff44febc8c26.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/de71895b951861ddc8c216993f3d7a728df14e2d8c6e238711deff44febc8c26.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/c66ce72d469adcd4df71c23e2273386e596ba2df6a31816e8f14dab27be15b5f.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/c66ce72d469adcd4df71c23e2273386e596ba2df6a31816e8f14dab27be15b5f.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/ad7c949287ce1afc1922b42eca1abfc21719af6f1801d8a5bab9f61a8acc0615.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/ad7c949287ce1afc1922b42eca1abfc21719af6f1801d8a5bab9f61a8acc0615.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/5f17800457a16c1252ee60480893033fbee063919b68406b8c384ad8f15f48ad.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/5f17800457a16c1252ee60480893033fbee063919b68406b8c384ad8f15f48ad.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/b181484de45dfb1f840080abe91be2978eebe09e8946652a8f10274492269600.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/b181484de45dfb1f840080abe91be2978eebe09e8946652a8f10274492269600.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/72e9e3deda978a0e2e8458bfab4b50f4d8ad947cda58b54e2321b8feacbfa746.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/72e9e3deda978a0e2e8458bfab4b50f4d8ad947cda58b54e2321b8feacbfa746.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/8bee9581987f2819f56589c712fcb8f049ff1892b0d990c8db5a811ceae1de2a.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/8bee9581987f2819f56589c712fcb8f049ff1892b0d990c8db5a811ceae1de2a.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/dd5eb1ca51a83a3468936e3751d418d4cef67fd34591085f12c6cc8382f6333a.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/dd5eb1ca51a83a3468936e3751d418d4cef67fd34591085f12c6cc8382f6333a.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/46992c7278b45d570bf4f4b003eaf45b68d9e478e891c26f1ae6977114ad0526.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/46992c7278b45d570bf4f4b003eaf45b68d9e478e891c26f1ae6977114ad0526.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/0abb492a1db0e23fe769c689113e7af0e8090b166fa3ecf70457340942f205f1.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/0abb492a1db0e23fe769c689113e7af0e8090b166fa3ecf70457340942f205f1.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/281dca34d7a33bbc05ed4a9c4e69aa604b7a86655a8ed52b652b16a9062a6da7.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/281dca34d7a33bbc05ed4a9c4e69aa604b7a86655a8ed52b652b16a9062a6da7.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/0f0161f59b4f1ff868da2db396fef8f35e08f16bdc876e9cac3002f3af4f8310.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/0f0161f59b4f1ff868da2db396fef8f35e08f16bdc876e9cac3002f3af4f8310.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/5e88d7e85750f7b6bb371640f882b47c9dba9f2973d3b3d46ebf2dd9eeff688b.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/5e88d7e85750f7b6bb371640f882b47c9dba9f2973d3b3d46ebf2dd9eeff688b.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/4429cc77825848c87761b11e56799c6b6d73106db708d5339820e431341b13bb.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/4429cc77825848c87761b11e56799c6b6d73106db708d5339820e431341b13bb.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/d4b98fc37626dce729b1839e2c4fabb684d87f028bf39c2d8be261d757141dc0.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/d4b98fc37626dce729b1839e2c4fabb684d87f028bf39c2d8be261d757141dc0.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/9348477c4e6189f009355ef8bbdd0a426c165ec3590e914dce2edb6d004db8d0.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/9348477c4e6189f009355ef8bbdd0a426c165ec3590e914dce2edb6d004db8d0.0.buffer (No such file or directory)
org.apache.http.NoHttpResponseException: mybuckets3.amazonaws.com:443 failed to respond
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/b6554e128110283bd5277545e5cd19f0a97714f87b2792dbaba4310637e49db3.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/b6554e128110283bd5277545e5cd19f0a97714f87b2792dbaba4310637e49db3.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/bdde44428ffa65b68d443fb1240f3eb927aac952a2c66ee97dd1e8effd990519.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/bdde44428ffa65b68d443fb1240f3eb927aac952a2c66ee97dd1e8effd990519.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/d096ea3ca445ceadcdcea9cf87d7a584d657a4d81f2f872d97d778dd8472e4c9.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/d096ea3ca445ceadcdcea9cf87d7a584d657a4d81f2f872d97d778dd8472e4c9.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/79b56533897cbc38a161f355f626804f5818ad6211ea13fa9eb9db3db1213eb5.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/126fa227415b4146d8df1c243027b249fa75ec0cb05f0721052b953f04655d76.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/126fa227415b4146d8df1c243027b249fa75ec0cb05f0721052b953f04655d76.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/ebc7b353e8e620d4c91386d635abbcaff1bfacb64d67ed8c619d04285f65bb41.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/ebc7b353e8e620d4c91386d635abbcaff1bfacb64d67ed8c619d04285f65bb41.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/9240a687a6534c1c84d0e909e84642aae71fe3fc098f563ec127d5dc31453be2.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-9d166a0f-82d7-4fd0-b0de-55c06efbe9e2/9240a687a6534c1c84d0e909e84642aae71fe3fc098f563ec127d5dc31453be2.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/437b9623c2a0a01d08180dc21fc627508eb911ac97676fafff621f7bbbd8ab33.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-e85870d4-9214-4d52-bdfd-0a00fc08d459/437b9623c2a0a01d08180dc21fc627508eb911ac97676fafff621f7bbbd8ab33.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/7f48f14bf8d86dfb1f693472b1b025aec05c4e0189d8b15e2cc8781c60068bd3.0.buffer (No such file or directory)
java.io.FileNotFoundException: /tmp/flink-io-7c72ca50-3b57-4a11-9940-f88a52b25407/7f48f14bf8d86dfb1f693472b1b025aec05c4e0189d8b15e2cc8781c60068bd3.0.buffer (No such file or directory)
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
org.apache.http.NoHttpResponseException: mybucket.s3.amazonaws.com:443 failed to respond
2018-06-05 15:51:54,505 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x3637530d74e0005 type:create cxid:0x139 zxid:0x100111017 txntype:-1 reqpath:n/a Error Path:/flink/default/checkpoints/c3bfe1335ad19483b758a09c7044b0a2/0000000000000152470/0d6c7449-6a3e-4b46-a4c2-9e29a93841c5 Error:KeeperErrorCode = NodeExists for /flink/default/checkpoints/c3bfe1335ad19483b758a09c7044b0a2/0000000000000152470/0d6c7449-6a3e-4b46-a4c2-9e29a93841c5 EndOfStreamException: Unable to read additional data from client sessionid 0x3637530d74e0005, likely client has closed socket 2018-06-05 15:55:40,024 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16375313e4d0000 type:create cxid:0xb5 zxid:0x100111060 txntype:-1 reqpath:n/a Error Path:/flink/default/checkpoint-counter/68139a9b7a561ed5099402f5579edd64 Error:KeeperErrorCode = NodeExists for /flink/default/checkpoint-counter/68139a9b7a561ed5099402f5579edd64 2018-06-05 15:55:40,120 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16375313e4d0000 type:create cxid:0xc4 zxid:0x100111062 txntype:-1 reqpath:n/a Error Path:/flink/default/checkpoint-counter/1d84f8bbb75ab8325cf46c1f12b9a1a2 Error:KeeperErrorCode = NodeExists for /flink/default/checkpoint-counter/1d84f8bbb75ab8325cf46c1f12b9a1a2 2018-06-05 15:55:40,189 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16375313e4d0000 type:create cxid:0xd3 zxid:0x100111064 txntype:-1 reqpath:n/a Error Path:/flink/default/checkpoint-counter/164209b48b68d3f9e07f205ffe23706a Error:KeeperErrorCode = NodeExists for /flink/default/checkpoint-counter/164209b48b68d3f9e07f205ffe23706a 2018-06-05 15:55:40,226 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16375313e4d0000 type:create cxid:0xe2 zxid:0x100111066 txntype:-1 reqpath:n/a Error Path:/flink/default/checkpoint-
Flink 1.3.1 is known to have issues with checkpointing. I'm not sure if one or more of the bugs fixed in 1.3.2 and in 1.3.3 explains the problem you're having, but it seems reasonably likely.