Search code examples
apache-storm

Storm 1.0.1 Worker error on submitting topology

I am getting the error below (worker.log) when submitting a topology to Storm 1.0.1 (despite the error, the topology does get submitted and appears in the Storm UI):

o.a.s.d.worker [ERROR] Error on initialization of server mk-worker java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[?:1.8.0_91]
    at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_91]
    at org.apache.storm.timer$mk_timer.doInvoke(timer.clj:77) ~[storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:457) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$mk_halting_timer.invoke(worker.clj:244) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.worker$worker_data$fn__8190.invoke(worker.clj:293) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.util$assoc_apply_self.invoke(util.clj:930) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.worker$worker_data.invoke(worker.clj:268) ~[storm-core-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.worker$fn__8450$exec_fn__2461__auto__$reify__8452.run(worker.clj:611) ~[storm-core-1.0.1.jar:1.0.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_91]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_91]
    at org.apache.storm.daemon.worker$fn__8450$exec_fn__2461__auto____8451.invoke(worker.clj:609) ~[storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
    at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
    at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$fn__8450$mk_worker__8545.doInvoke(worker.clj:583) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$_main.invoke(worker.clj:771) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
    at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.1.jar:1.0.1]
2016-07-14 11:48:29.568 o.a.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
    at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$fn__8450$mk_worker__8545.doInvoke(worker.clj:583) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$_main.invoke(worker.clj:771) [storm-core-1.0.1.jar:1.0.1]
    at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
    at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.0.1.jar:1.0.1]

Also found this entry in the supervisor.log:

o.a.s.util [WARN] Worker Process ea92e1b7-c870-4b4c-b0a6-00272f27f521:# There is insufficient memory for the Java Runtime Environment to continue.

However, "free -m" command suggests enough free memory.

Further messages seem to suggest that the worker fails to start and supervisor.log then starts logging INFO message "{id of worker process} still hasn't started" multiple times.

Solution

  • I separated kafka & storm and now they are running on separate VMs. The topology is running OK now without any errors.