How to do a Custom Build of Spring Cloud Data Flow server with Oracle driver dependency?

Question

I've been trying out the SCDF for sometime with intention to use Oracle Database as datasource. Due to licensing issues Oracle driver has to be added to the classpath of SCDF server or we have to do a custom build of SCDF server with Oracle Driver dependency(Which I have). When I download the custom build project dataflow-server-22x (only this project) from github and try to execute I get a missing artifact issue in pom.xml as below.

Missing artifact io.pivotal:pivotal-cloudfoundry-client-reactor:jar:1.1.0.RELEASE 
at <xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> of pom.xml

So how exactly I have to perform the custom build of this SCDF jar. Am I missing something here?

Also my intention is just to build a jar containing set of batch jobs which can be deployed in SCDF and orchestrated from SCDF. But I'm not using Docker or Kubernetes/CloudFoundry here.

Note: I already asked one question to get clarification regarding this issue, which lead my to this issue. There they said I should use custom build, But couldn't exactly tell how, or how to solve the issues arise from the custom build. Hence I posted this question. SCDF+Oracle

Thanks in advance.

Update 1:

The above issue got resolved after Ilayaperumals suggestion. However I got stuck up with another issue.

org.springframework.context.ApplicationContextException: Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgumentException: Invalid TaskExecution, ID 3 not found

But I see Id=3, in task_execution table after the I execute the task from SCDF. The custom SCDF project has the same database config property values as my Spring batch job properties. Few things to note here are,

• Spring-boot-starter-parent : 2.2.5.RELEASE,
• Spring-cloud-dataflow : 2.2.0.RELEASE
• I load all of my Batchjobs from main class of Boot using the instance of batch job class and only the main class (which kickstarts all jobs)contains @EnableTask annotation. Below is my class structure.

    @SpringBootApplication
    @EnableScheduling
    @EnableTask
    public class SpringBootMainApplication{
        @Autowired
        Job1Loader job1Loader;

        public static void main(String[] args) {
            SpringApplication.run(SpringBootMainApplication.class, args);
        }

        @Scheduled(cron = "0 */1 * * * ?")
        public void executeJob1Loader() throws Exception
        {
            JobParameters param = new JobParametersBuilder()
                                        .addString("JobID",         
                                     String.valueOf(System.currentTimeMillis()))
                                        .toJobParameters();
            jobLauncher.run(job1Loader.loadJob1(), param);
        }
    }

    //Job Config
    @Configuration
    @EnableBatchProcessing
    public class Job1Loader {
    @Bean
        public Job loadJob1()
        {
            return jobBuilderFactory().get("JOb1Loader")
                .incrementer(new RunIdIncrementer())
                .flow(step01())
                .end()
                .build();;//return job
    }

I use two different datasources in my Spring job project, both are oracle datasource(Different servers).I marked one of them as primary and used that Datasource in my custom implementation of "DefaultTaskConfigurer" as below.

    @Configuration
    public class TaskConfig extends DefaultTaskConfigurer {
        @Autowired
        DatabaseConfig databaseConfig; 
        @Override
        public DataSource getTaskDataSource() {
            return databaseConfig.dataSource();//dataSource() returns the 
    primary ds
        }
    }

• Below are the properties I use in both SCDF custom serer and Spring Batch project.

    **Spring batch Job :**
    spring.datasource.jdbc-url=jdbc:oracle:thin:@mydb
    spring.datasource.username=db_user
    spring.datasource.password=db_pwd
    spring.datasource.driver-class-name=oracle.jdbc.OracleDriver

    **SCDF customer Server:**
    spring.datasource.url=jdbc:oracle:thin:@mydb
    spring.datasource.username=db_user
    spring.datasource.password=db_pwd
    spring.datasource.driver-class-name=oracle.jdbc.OracleDriver

I tried supplying the db config as args while Starting the server and few other options like adding @Enabletask to all job config classes but none of them seems to work.

What am I missing here?

Answer 1

Since you mentioned you don't run this on CloudFoundry and the specific dependency io.pivotal:pivotal-cloudfoundry-client-reactor:jar comes from the spring-cloud-dataflow-platform-cloudfoundry, you need to remove this dependency from the custom build configuration as below:

                 <dependency>
                        <groupId>org.springframework.cloud</groupId>
                        <artifactId>spring-cloud-starter-dataflow-server</artifactId>
                        <exclusions>
                                        <exclusion>
                                                <groupId>org.springframework.cloud</groupId>
                                                <artifactId>spring-cloud-dataflow-platform-cloudfoundry</artifactId>
                                        </exclusion>
                                </exclusions>
                </dependency>

Also, doing ./mvnw dependency:tree will help you figure where does the dependency come from:

\- org.springframework.cloud:spring-cloud-dataflow-platform-cloudfoundry:jar:2.5.0.BUILD-SNAPSHOT:compile
[INFO] |  |        +- org.springframework.cloud:spring-cloud-deployer-cloudfoundry:jar:2.3.0.BUILD-SNAPSHOT:compile
[INFO] |  |        |  +- org.cloudfoundry:cloudfoundry-client-reactor:jar:4.1.0.RELEASE:compile
[INFO] |  |        |  | 
[INFO] |  |        |  +- io.projectreactor.addons:reactor-extra:jar:3.3.2.RELEASE:compile
[INFO] |  |        |  \- io.pivotal:pivotal-cloudfoundry-client-reactor:jar:2.0.0.RELEASE:compile
[INFO] |  |        |     \- io.pivotal:pivotal-cloudfoundry-client:jar:2.0.0.RELEASE:compile