How should I embed a Jetty server in an Apache Ignite service?

Question

I'm trying to embed a Jetty server in an Apache Ignite service (as per this thread) so I can make the HTTP endpoint the entry point to my data pipeline. Here's my basic test:

Main.scala

object Main {
    def main(args: Array[String]): Unit = {
        val ignite = Ignition.start()
        val group = ignite.cluster.forLocal
        val services = ignite.services(group)
        services.deployNodeSingleton("myTestService", new TestServiceImpl)
    }
}

TestService.scala

trait TestService {
    def test()
}

class TestServiceImpl extends Service with TestService {
    val server = new Server(8090)

    def cancel(ctx: ServiceContext) = {
        server.stop()
        server.join()
    }

    def init(ctx: ServiceContext) = {
        println("TestServiceImpl#init")
    }

    def execute(ctx: ServiceContext) = {
        println("TestServiceImpl#execute")
        server.start()
    }

    def test() = {
        println("Tested")
    }
}

When I run it, I get the following error:

[01:52:57] Ignite node started OK (id=626c1302)
[01:52:57] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=2.0GB]
TestServiceImpl#init
TestServiceImpl#execute
Oct 17, 2017 1:52:57 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Service execution stopped with error [name=myTestService, execId=565f4fb4-5726-4c37-857d-0c74f3b334ce]
java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.util.thread.NonBlockingThread@56a1831d
  at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:362)
  at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:160)
  at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:258)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
  at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:256)
  at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
  at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
  at org.eclipse.jetty.server.Server.doStart(Server.java:366)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
  at me.danellis.ignite.TestServiceImpl.execute(TestService.scala:23)
  at org.apache.ignite.internal.processors.service.GridServiceProcessor$2.run(GridServiceProcessor.java:1160)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:748)

Does something need to be configured differently in either Ignite or Jetty for this to work?

Answer 1

You should instantiate jetty Server class in init method, because it will be invoked on the target machine right after service deployment. Instantiating Server class in the constructor is useless - right after creating a Service instance(which could be performed on other nodes in most cases), this instance will be serialized and added to the internal cache and only after it will be started on the target machine.

I think it's clear that jetty Server object can't be serialized properly. For example, serializing of ThreadPool can't be done, because Thread implementation contains blocks of native code.