Search code examples
etltalend

How to combine multiple Talend contexts? (e.g. different database connections)

product: Talend Open Studio for Big Data (5.6.0M3_20140806_1834)

One can use contexts and/or context groups (you can think of it e.g. as environment parameter sets, e.g. source and target database settings for development, test and production).

See here for an example: http://www.talendforge.org/tutorials/tutorial.php?idTuto=14#step183 (5mn Demo)

So one could do this with some db connection objects in a job and configure them based on the context (e.g. the server name with context.srcDbHostName).

If you create such a context group you can nicely use it e.g. to create a job to copy data from some source to some destination database (e.g. in a data warehouse environment). And then test it in all of those environments with just one click selecting the specific context of our context group.

Now if you also like to debug the databases via customized SQL queries and the wizards related to them and so on this is nicely integrated based on the Metadata->DB connections area.

Now it would be nice, if we could reuse our context group with these connections (instead of manually creating a db connection for all our different source and target databases), meaning letting them be based on our single context group. Thus in our example we would require only two connections, one for the source db connection and another for the destination db connection where a dialog will appear on access, letting us choose the proper environment.

But these connections are first to be configured by discrete values. There is the option to export such a connection as a context in a new context group. But if you have two such connections like in the example above you would only end up with two different such context groups although you need only one where both contexts are contained. :-(

I found no way to accomplish this kind of context group merge or export merge functionality.

But I found another rather easy but not wizard supported way to do this anyways. (See my answer below)

Solution

  • Just one straight solution that works, but as you get the picture you can adjust the steps to a scenario that is optimal for your problem...

    1. create a db connection under Metadata->db connections (e.g. for test env src db)
    2. export this as your desired context group as a starting point
    3. add all other environment contexts to this group (e.g. all values for prod and demo src dbs)
    4. create the other db connection (e.g. for test env dst db)
    5. now locate the following files and their content in your project on the disk:
      • context/<myCtxGroup>.properties
        • xmi:XMI/TalendProperties:Property/@id = "myCtxGrpId"
      • metadata/connections/<srcdb>.item
        • xmi:XMI/TalendMetadata:DatabaseConnection ContextId="myCtxGrpId" contextName="test" ... ServerName="context.srcDbHost" ...
      • metadata/connections/<dstdb>.item
        • xmi:XMI/TalendMetadata:DatabaseConnection ...
    6. add all the ContextId, contextName, ... attributes found in the properly working srcdb connection to your dstdb connection there
      • of course there some things should look little different like e.g. ServerName="context.dstDbHost" (not srcDbHost)
    7. test it and it should work like a charm for all your environments :-)