springspring-batchspring-cloud-dataflowspring-cloud-task
Set SCDF CTR to execute next task immediately when step is complete
I'm using SCDF 2.9.6, which is mariadb 10.3.x(used in my company, cannot update version) compatible with flyway. Compiling SCDF 2.10.x project with lower flyway dependency resulted whole compile process failure... I guess this is the highest version I could use.
Now question arises, when I have to set Composed Task Runner property that 2.9.x version does not support - setting app.composed-task-runner.interval-time-between-checks property in 2.9.x seems not working, so I have to find walkaround. I know this Runner is Spring Batch project, so added Spring Batch tag.
There is a simple Composed Task asdf1, which runs a,b,c,d,e task consequently.
Each of child tasks are not that heavy - they complete their job in 5 seconds. However, CTR executes next task(step) after 10 seconds.
It seems, in some property file, there is a "minimum runtime" for each of task(step) in Composed Task Runner.
Is there any way to set this minimum runtime to 0, so the task(step) is always executed when the predecessor finishes, in lower SCDF version?
Thank for any Information provided for this question.
Solution
The property intervalTimeBetweenChecks is present in 2.9.x as well.
The documentation show how to set properties for Composed Task Runner.
app.composed-task-runner.interval-time-between-checks=1000 will set it to 1 second.
I did a quick test and use kail -ns=default | tee pods.log to combine all the logs. I use 5s and the logs show that there was 7s between the last log entry from a and the first log entry from b. This will include the time for launching the container and the application being active.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.755 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ctr]] launched with the following parameters: [{run.id=3}]
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.762 INFO 1 --- [ main] o.s.c.t.b.l.TaskBatchExecutionListener : The job execution id 11 was run within the task execution 22
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.773 INFO 1 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [ctr-a_0]
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:34.893 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Starting TaskApplication v2.0.2 using Java 1.8.0_352 on ctr-a-q30xnqr0r4 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:34.895 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : No active profile set, falling back to 1 default profile: "default"
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.363 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.533 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.548 DEBUG 1 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.548 DEBUG 1 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : No EntityManager was found, using DataSourceTransactionManager
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.641 DEBUG 1 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for postgresql database
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.714 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Starting: TaskExecution{executionId=23, parentExecutionId=22, exitCode=null, taskName='ctr-a', startTime=Fri Jan 20 12:17:35 UTC 2023, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--management.metrics.tags.service=task-application, --spring.datasource.username=root, --spring.datasource.url=jdbc:postgresql://postgresql:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --management.metrics.tags.application=${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}, --spring.cloud.task.name=ctr-a, --spring.datasource.password=postgres-root, --spring.cloud.task.parent-execution-id=22, --spring.cloud.task.executionid=23]}
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.752 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Started TaskApplication in 1.068 seconds (JVM running for 1.231)
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.755 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : 2023-01-20 12:17:35.754
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.764 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=23 with the following {exitCode=0, endTime=Fri Jan 20 12:17:35 UTC 2023, exitMessage='null', errorMessage='null'}
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.772 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.783 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.648 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-a_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.650 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-a_0 with taskExecution 23
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.654 INFO 1 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ctr-a_0] executed in 6s881ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.664 INFO 1 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [ctr-b_0]
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:42.755 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Starting TaskApplication v2.0.2 using Java 1.8.0_352 on ctr-b-kg57k4plg0 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:42.757 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : No active profile set, falling back to 1 default profile: "default"
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.274 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.443 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.455 DEBUG 1 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.456 DEBUG 1 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : No EntityManager was found, using DataSourceTransactionManager
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.558 DEBUG 1 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for postgresql database
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.630 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Starting: TaskExecution{executionId=24, parentExecutionId=22, exitCode=null, taskName='ctr-b', startTime=Fri Jan 20 12:17:43 UTC 2023, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--management.metrics.tags.service=task-application, --spring.datasource.username=root, --spring.datasource.url=jdbc:postgresql://postgresql:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --management.metrics.tags.application=${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}, --spring.cloud.task.name=ctr-b, --spring.datasource.password=postgres-root, --spring.cloud.task.parent-execution-id=22, --spring.cloud.task.executionid=24]}
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.649 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Started TaskApplication in 1.114 seconds (JVM running for 1.311)
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.651 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : 2023-01-20 12:17:43.650
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.657 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=24 with the following {exitCode=0, endTime=Fri Jan 20 12:17:43 UTC 2023, exitMessage='null', errorMessage='null'}
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.664 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.677 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.421 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-b_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.422 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-b_0 with taskExecution 24
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.425 INFO 1 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ctr-b_0] executed in 7s761ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.433 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ctr]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 14s675ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.441 INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.443 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.445 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.421 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-b_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.422 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-b_0 with taskExecution 24
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.425 INFO 1 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ctr-b_0] executed in 7s761ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.433 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ctr]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 14s675ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.441 INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
- Spring @Sql Annotations, possible to run once before all tests?
- Springboot mySQL Failed to determine a suitable driver class
- Add JSF Message From Spring Bean
- Spring Boot - no log file written (logging.file is not respected)
- Handling Internal Server Error When OAuth Issuer Is Not Available in WebFluxSecurity
- Return file from Spring @Controller having OutputStream
- With deprecation of the `SecurityContextRepository.loadContext(HttpRequestResponseHolder)` method, how do we add `Set-Cookie` to the response?
- How to correctly read Flux<DataBuffer> and convert it to a single inputStream
- Implementing Custom Expression Handling with Spring Security 6
- Can I set a TTL for @Cacheable
- Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect)
- How to fix: Error creating bean with name : Unsatisfied dependency expressed through field
- Error in Java Import statement "The import javax.validation.constraints.NotNull cannot be resolved"
- How to conditionally enable or disable scheduled jobs in Spring?
- How do I get the connection inside of a Spring transaction?
- Kafka No Acknowledgment available as an argument
- How to use fixed values in Spring CrudRepository?
- WARNING: Exception encountered during context initialization - cancelling refresh attempt
- How to increase the maximum length of SpEL expressions in spring boot 3.3.0?
- Start a spring batch job when already within a transaction
- How to persist data in H2 database
- is using DTOs in every case always a good practice?
- How to add custom ApplicationContextInitializer to a spring boot application?
- How do I make the update query executed by JdbcPollingChannelAdapter transactional?
- How to configure a TransactionInterceptor equivalent to the @Transactional annotation for use with PollerMetadata advice
- Spring Webflux - WebClient over TLSv1.2
- Spring 5 WebClient using ssl
- Application version does not show up in Spring Boot banner.txt
- package org.springframework.boot does not exist
- How to specify prefix for all controllers in Spring Boot?