Search code examples
datastage

DataStage: run low level socket connection

I am Unix java developer trying to help a datastage developer, so out of my aquarium.

The datastage process connects to a database hosting financial transactions on a unix server. there is a datastage process for migrating financial transactions to the ACCOUNTING system. The ETL developers for one reason or another have specified they cannot run one or more specific ETL while in-taking new financial transactions and have specified the process that inserts transactions into the DB be stopped.

me java geek thinks have some process checking a service running at port 55555 would be perfect. But we cannot find a way for datastage to create a socket connection to a port to check. I don't do datastage so I don't know how to work around it's limitations.

The ETL developer thinks a cron script running every minute that inserts an up/down status for the process into a special table would be perfect. I think it is a waste of cpu.

I cannot be the only company that cannot run an ETL when some process is running on a remote system.

How did you solve this issue? Is there a way to connect to a remote servers socket and run the service from datastage???

thanks

Solution

  • after a bunch of discussion. options we found

    1. Add a step to the start stop server scripts that writes process status to a table. pro: easy to implement. con: not turely accurate (some geek like me is likely to bypass the start/stop script and run only build/run the executable bypassing the start stop script and bypassing the step that inserts the status.) No network and InfoSec paperwork

    2. Cron based script that updates the table with the status on a minute by minute basis. what a pain!!!! No network and InfoSec paperwork.

    3. A script made available to the network through inet or xinet. Problem is datastage ETL developer does not know how to connect to a socket via C or java program. Creates Infosec and network paperwork issues.

    4. New web service (there is a tomcat server serving up a number of web services) Problem is datastage ETL developer does not know how to connect to a socket via C or java program. Creates Infosec and network paperwork issues.

    options 3 and 4 are accurate and realtime. options 1 and 2 opens up the possibility of inaccuracies by bypassing process, but that opens up a different can of worms.

    We are probably going to implement option 1