Search code examples
apache-storm

storm supervisor and worker not starting


I have search this forum and lot other over google but, unable to find solution of my problem. please help.

Issue is:

I am using apache-storm-0.9.3, zookeeper-3.4.6 and kafka_2.8.0-0.8.1.1

I am setting everything on my local machine.

My supervisor throw error as soon as it starts and throw msg --->>kill: No such process...

Below is what I found on screen ...

> storm supervisor

Running: /usr/local/jdk1.8.0_05/bin/java -server -Dstorm.options=  -Dstorm.home=/usr/local/apache-storm-0.9.3 -Dstorm.log.dir=/usr/local/apache-storm-0.9.3/logs -Djava.library.path=/usr/local/jdk1.8.0_05 -Dstorm.conf.file= -cp /usr/local/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/usr/local/apache-storm-0.9.3/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.3/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.3/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.3/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.3/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.3/lib/slf4j-api-1.7.5.jar:/usr/local/apache-storm-0.9.3/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.3/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.3/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/usr/local/apache-storm-0.9.3/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.3/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.3/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/usr/local/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.3/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.3/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.3/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.3/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.3/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.3/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.3/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/usr/local/apache-storm-0.9.3/lib/json-simple-1.1.jar:/usr/local/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.3/conf -Xmx256m -Dlogfile.name=supervisor.log -Dlogback.configurationFile=/usr/local/apache-storm-0.9.3/logback/cluster.xml backtype.storm.daemon.supervisor


kill: No such process

my supervisor log say,( multiple times), ---

2015-02-17T20:11:55.942+0530 b.s.d.supervisor [INFO] 1867e54f-f296-4279-8af7-50ca0674951f still hasn't started

and error on worker-6701 is ....

2015-02-17T20:23:42.679+0530 b.s.d.worker [ERROR] Error on initialization of server mk-worker
java.io.FileNotFoundException: File '/usr/local/apache-storm-0.9.3/data/supervisor/stormdist/cdr-event-processor-2-1424179110/stormconf.ser' does not exist
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) ~[commons-io-2.4.jar:2.4]
at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) ~[commons-io-2.4.jar:2.4]
at backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) ~[storm-core-0.9.3.jar:0.9.3]
at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:182) ~[storm-core-0.9.3.jar:0.9.3]
at backtype.storm.daemon.worker$fn__3743$exec_fn__1108__auto____3744.invoke(worker.clj:363) ~[storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) [storm-core-0.9.3.jar:0.9.3]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) [storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.3.jar:0.9.3]
2015-02-17T20:23:42.684+0530 b.s.util [ERROR] Halting process: ("Error on initialization")



java.lang.RuntimeException: ("Error on initialization")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.3.jar:0.9.3]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) [storm-core-0.9.3.jar:0.9.3]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) [storm-core-0.9.3.jar:0.9.3]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.3.jar:0.9.3]

and my storm.yaml is....

storm.zookeeper.servers: 
- "localhost"
storm.zookeeper.port: 2181
nimbus.host: "localhost"
#nimbus.thrift.port: 6627
ui.port: 8080
storm.local.dir: "/usr/local/apache-storm-0.9.3/data" 
java.library.path: "/usr/local/jdk1.8.0_05"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
storm.messaging.netty.max_retries: 30
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100

JAVA_HOME is /usr/local/jdk1.8.0_05

Solution

  • I believe you are running into bug storm-130:

    https://issues.apache.org/jira/browse/STORM-130

    The storm team seems quite close to shipping a fix. In the meantime, you can work around this by stopping supervisor and all workers, delete all storm data from zookeeper, then restart the storm cluster.