I have a few custom modules which I use in Spring XD. I can use 'module upload' in singlenode deployment of XD without a problem, but when running in distributed mode I have the following exception:
org.springframework.xd.dirt.module.NoSuchModuleException: Could not find module with name 'grobid' and type 'processor'
I am creating the XD cluster with docker using Spring XD 1.1.1.RELEASE. Here is the command sequence that leads to this exception:
server-unknown:>admin config server http://docker:9393
Successfully targeted http://docker:9393
xd:>module upload --type processor --name grobid --file /opt/.../grobid-1.0-SNAPSHOT.jar
Successfully uploaded module 'processor:grobid'
xd:>stream create --name grobidTest --definition "http --port=11001 | grobid --grobidHost=http://${GROBID_PORT_8080_TCP_ADDR}:${GROBID_PORT_8080_TCP_PORT} | processedXml:log"
Created new stream 'grobidTest'
xd:>stream deploy --name grobidTest --properties "module.grobid.count=1"
Deployed stream 'grobidTest'
As per looking at the documentation I've setup a standalone redis instance and I have no logs while booting XD indicating any connection problem with it.
What could cause my module not being available to the nodes while in the shell I have access to its information:
xd:>module info --name processor:grobid
Information about processor module 'grobid':
Option Name Description Default Type
----------- ----------------------------------------------------- ------- --------
grobidHost Grobid Host <none> String
outputType how this module should emit messages it produces <none> MimeType
inputType how this module should interpret messages it consumes <none> MimeType
The only problem (I do not know if it is a blocker) I can see so far in the logs are when I create the stream I have a few ZK errors that gets logged:
zk_1 | 2015-04-13 19:55:37,257 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14cb4593f940001 type:create cxid:0x59 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/xd/modules/sink/log/dependencies Error:KeeperErrorCode = NoNode for /xd/modules/sink/log/dependencies
zk_1 | 2015-04-13 19:55:37,273 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14cb4593f940001 type:create cxid:0x64 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/xd/modules/processor/grobid/dependencies Error:KeeperErrorCode = NoNode for /xd/modules/processor/grobid/dependencies
zk_1 | 2015-04-13 19:55:37,289 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x14cb4593f940001 type:create cxid:0x6e zxid:0x41 txntype:-1 reqpath:n/a Error Path:/xd/modules/source/http/dependencies Error:KeeperErrorCode = NoNode for /xd/modules/source/http/dependencies
Spring XD added the concept of "shared remote module registry" in https://github.com/spring-projects/spring-xd/commit/208181c9f4693d2ef2e4e0d4065d9d8727fa1601 (which is 1.2, unreleased at the time of writing). This relies on a shared location, presumably HDFS.
Before that, if you're going to use a cluster with custom uploaded modules, then you need to make the location where modules are written available to all containers somehow. The traditional way of doing that is using something like NFS.
If you're using docker though (and it seems you are, not sure if this is on one host only though), you could use a Docker data volume for this.
And to clarify, those two last paragraphs apply even with Spring XD 1.1.x