Search code examples
datastage

Insert job's information to statistic table after job finished DataStage


Currently, I have multiple jobs to load data from source to target (Oracle Connector -> Transformer Stage -> Oracle Connector). I want to get those job's information to a statistic table to track the progress every day.

My thought is after the job has done, it will automatically insert 1 row for each job to my statistic table. For example, after Job_1 (load to Target_1 table) and Job_2 (load to Target_2 table) finished, each job will insert 1 row to my statistic table and it will look like below:

TABLE_NAME   DATE_1   DATE_2    TIME_STAMP            TOTAL_RECORD
----------   ------   -------   -------------------   ------------
Target_1     041120   2020309   2020-11-04 11:09:00   500
Target_2     041120   2020309   2020-11-04 11:10:00   1000

Is it possible to do with a routine or something else?


Solution

  • Such load stats are really useful and there are several way to achieve this. The main question is how to get the total record information amd how those are defined. The is a number of rows getting read from the source and a number written into the target - only if you do not have any filtering in your Transformer (and no rejects) those shoule be the same.

    You can get this from

    • the link information via DSGetLinkInfo
    • by running a count on your target table
    • or - my recommendation - by using the DSODB

    Check out the Section Monitoring job and job runs by using the Operations Console in the Knowledge Center or if you want to create your own table Extracting Monitor data from the operations database