Search code examples
jsonweb-servicestalend

Talend - URI based on values obtained from a list

I need to feed a collection in MongoDB based on values obtained from webservices that provide JSON, but I'm having trouble mounting the job because the URI of some of the web services rely on values that can be obtained from other webservices. For example, the URI http://172.31.15.180:80/ws/getAgenciasUF/52 provides a JSON with a collection, in this format:

{ "COD_AGENCIA", "521800300", "NAME", "PORANGATU", "UORG": "902", "INTRA_MUNICIPAL": "0"},
{ "COD_AGENCIA", "521830000", "NAME", "HOLD", "UORG": "904", "INTRA_MUNICIPAL": "0"}
...
(20 other values)
...

Through this webservice, I could have insert in a MongoDB collection using the tREST and tExtractJSONFields components. However, there is another webservice whose URI is http://172.31.15.180:80/ws/getCidadesPorAgencia/521800300, where the latter value is one of COD_AGENCIA available in JSON above. That is, if I read the COD_AGENCIA up and put in some component that iterate on these values and shoot x times the second URI, varying only the code, I could get all the values needed to feed another collection MongoDB.

Using the ESB TOS 6.2.1, I tried to interconnect a Trest a tExtractJSONField and this to a tRESTRequest, like this:

my job

but I received the following error:

[statistics] connecting to socket on port 3587
[statistics] connected

[WARN ]: org.eclipse.jetty.util.component.AbstractLifeCycle - FAILED [email protected]:80: java.net.BindException: Cannot assign requested address: bind
java.net.BindException: Cannot assign requested address: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:293)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine.addServant(JettyHTTPServerEngine.java:472)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.activate(JettyHTTPDestination.java:175)
at org.apache.cxf.transport.AbstractObservable.setMessageObserver(AbstractObservable.java:53)
at org.apache.cxf.binding.AbstractBindingFactory.addListener(AbstractBindingFactory.java:95)
at org.apache.cxf.jaxrs.JAXRSBindingFactory.addListener(JAXRSBindingFactory.java:88)
at org.apache.cxf.endpoint.ServerImpl.start(ServerImpl.java:123)
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:206)
at bdogo.teste_0_1.teste$Thread4RestServiceProviderEndpoint.run(teste.java:791)

[WARN ]: org.eclipse.jetty.util.component.AbstractLifeCycle - FAILED org.eclipse.jetty.server.Server@13df084: java.net.BindException: Cannot assign requested address: bind
java.net.BindException: Cannot assign requested address: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:293)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine.addServant(JettyHTTPServerEngine.java:472)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.activate(JettyHTTPDestination.java:175)
at org.apache.cxf.transport.AbstractObservable.setMessageObserver(AbstractObservable.java:53)
at org.apache.cxf.binding.AbstractBindingFactory.addListener(AbstractBindingFactory.java:95)
at org.apache.cxf.jaxrs.JAXRSBindingFactory.addListener(JAXRSBindingFactory.java:88)
at org.apache.cxf.endpoint.ServerImpl.start(ServerImpl.java:123)
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:206)
at bdogo.teste_0_1.teste$Thread4RestServiceProviderEndpoint.run(teste.java:791)

[ERROR]: org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine - Could not start Jetty server on port 80: Cannot assign requested address: bind
org.apache.cxf.service.factory.ServiceConstructionException
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:219)
at bdogo.teste_0_1.teste$Thread4RestServiceProviderEndpoint.run(teste.java:791)

Caused by: org.apache.cxf.interceptor.Fault: Could not start Jetty server on port 80: Cannot assign requested address: bind
at org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine.addServant(JettyHTTPServerEngine.java:483)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.activate(JettyHTTPDestination.java:175)
at org.apache.cxf.transport.AbstractObservable.setMessageObserver(AbstractObservable.java:53)
at org.apache.cxf.binding.AbstractBindingFactory.addListener(AbstractBindingFactory.java:95)
at org.apache.cxf.jaxrs.JAXRSBindingFactory.addListener(JAXRSBindingFactory.java:88)
at org.apache.cxf.endpoint.ServerImpl.start(ServerImpl.java:123)
at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:206)
... 1 more

Caused by: java.net.BindException: Cannot assign requested address: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:293)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine.addServant(JettyHTTPServerEngine.java:472)
... 7 more

If I eliminate tRESTRequest and link the tExtractJSONField directly to tLog this error doesn't happen, and the output of tExtract is listed on the console. The use of this tRESTRequest component (which is new to me) is that I seem to be something wrong. It is so even that it should be used (see figure below)?

tRESTRequest

Note that there is a warning in tExtractJSONFields (the text says: "This component has not enough Row type outputs"). The following figures shows as the configuration of the components was made.

Could help me on how to configure it from values received from tExtractJSONFields? What am I doing wrong? There is another way to get the desired result ?

Solution

  • Your approach is fine, you only seem to lack a few components.

    Job design

    1. Get list of COD_AGENCIA

    The components tREST and tExtractJSONFields are well suited for this.

    At this point you should have a list of all the values you got with the first REST call.

    Now for the connection between the first and the second call.

    1. Get list of getCidadesPorAgencia per COD_AGENCIA

    Here, you could again use a tREST component and maybe a tExtractJSONFields. Also, here would be the spot where to use tMongoDBOutput.

    To connect both of those requests, use a tFlowToIterate component after the first tExtractJSONFields.

    Add a key value for COD_AGENCIA and connect the second tREST with Iterate. Now in the second tREST you will have access to a global variable which you just named as the key value. Use this in the call, e.g.: "http://172.31.15.180:80/ws/getCidadesPorAgencia/" + globalMap.get("var_agencia")"

    Now you should be able to loop through every agency and get all the cities connected to it.

    Reason

    What you did was a great idea but the wrong connector. OnComponentOk waits until the connected component runs through without an error. Then the next component will be started. No data will be transmitted. And no row iteration will happen, which seems to be key here.