springspring-batchspring-cloud-dataflowspring-cloud-task
Set SCDF CTR to execute next task immediately when step is complete
I'm using SCDF 2.9.6, which is mariadb 10.3.x(used in my company, cannot update version) compatible with flyway. Compiling SCDF 2.10.x project with lower flyway dependency resulted whole compile process failure... I guess this is the highest version I could use.
Now question arises, when I have to set Composed Task Runner property that 2.9.x version does not support - setting app.composed-task-runner.interval-time-between-checks property in 2.9.x seems not working, so I have to find walkaround. I know this Runner is Spring Batch project, so added Spring Batch tag.
There is a simple Composed Task asdf1, which runs a,b,c,d,e task consequently.
Each of child tasks are not that heavy - they complete their job in 5 seconds. However, CTR executes next task(step) after 10 seconds.
It seems, in some property file, there is a "minimum runtime" for each of task(step) in Composed Task Runner.
Is there any way to set this minimum runtime to 0, so the task(step) is always executed when the predecessor finishes, in lower SCDF version?
Thank for any Information provided for this question.
Solution
The property intervalTimeBetweenChecks is present in 2.9.x as well.
The documentation show how to set properties for Composed Task Runner.
app.composed-task-runner.interval-time-between-checks=1000 will set it to 1 second.
I did a quick test and use kail -ns=default | tee pods.log to combine all the logs. I use 5s and the logs show that there was 7s between the last log entry from a and the first log entry from b. This will include the time for launching the container and the application being active.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.755 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ctr]] launched with the following parameters: [{run.id=3}]
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.762 INFO 1 --- [ main] o.s.c.t.b.l.TaskBatchExecutionListener : The job execution id 11 was run within the task execution 22
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.773 INFO 1 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [ctr-a_0]
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:34.893 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Starting TaskApplication v2.0.2 using Java 1.8.0_352 on ctr-a-q30xnqr0r4 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:34.895 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : No active profile set, falling back to 1 default profile: "default"
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.363 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.533 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.548 DEBUG 1 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.548 DEBUG 1 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : No EntityManager was found, using DataSourceTransactionManager
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.641 DEBUG 1 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for postgresql database
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.714 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Starting: TaskExecution{executionId=23, parentExecutionId=22, exitCode=null, taskName='ctr-a', startTime=Fri Jan 20 12:17:35 UTC 2023, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--management.metrics.tags.service=task-application, --spring.datasource.username=root, --spring.datasource.url=jdbc:postgresql://postgresql:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --management.metrics.tags.application=${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}, --spring.cloud.task.name=ctr-a, --spring.datasource.password=postgres-root, --spring.cloud.task.parent-execution-id=22, --spring.cloud.task.executionid=23]}
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.752 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Started TaskApplication in 1.068 seconds (JVM running for 1.231)
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.755 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : 2023-01-20 12:17:35.754
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.764 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=23 with the following {exitCode=0, endTime=Fri Jan 20 12:17:35 UTC 2023, exitMessage='null', errorMessage='null'}
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.772 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.783 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.648 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-a_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.650 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-a_0 with taskExecution 23
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.654 INFO 1 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ctr-a_0] executed in 6s881ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.664 INFO 1 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [ctr-b_0]
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:42.755 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Starting TaskApplication v2.0.2 using Java 1.8.0_352 on ctr-b-kg57k4plg0 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:42.757 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : No active profile set, falling back to 1 default profile: "default"
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.274 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.443 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.455 DEBUG 1 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.456 DEBUG 1 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : No EntityManager was found, using DataSourceTransactionManager
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.558 DEBUG 1 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for postgresql database
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.630 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Starting: TaskExecution{executionId=24, parentExecutionId=22, exitCode=null, taskName='ctr-b', startTime=Fri Jan 20 12:17:43 UTC 2023, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--management.metrics.tags.service=task-application, --spring.datasource.username=root, --spring.datasource.url=jdbc:postgresql://postgresql:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --management.metrics.tags.application=${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}, --spring.cloud.task.name=ctr-b, --spring.datasource.password=postgres-root, --spring.cloud.task.parent-execution-id=22, --spring.cloud.task.executionid=24]}
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.649 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : Started TaskApplication in 1.114 seconds (JVM running for 1.311)
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.651 INFO 1 --- [ main] io.spring.timestamp.TaskApplication : 2023-01-20 12:17:43.650
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.657 DEBUG 1 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=24 with the following {exitCode=0, endTime=Fri Jan 20 12:17:43 UTC 2023, exitMessage='null', errorMessage='null'}
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.664 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.677 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.421 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-b_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.422 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-b_0 with taskExecution 24
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.425 INFO 1 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ctr-b_0] executed in 7s761ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.433 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ctr]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 14s675ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.441 INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.443 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.445 INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.421 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-b_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.422 INFO 1 --- [ main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-b_0 with taskExecution 24
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.425 INFO 1 --- [ main] o.s.batch.core.step.AbstractStep : Step: [ctr-b_0] executed in 7s761ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.433 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=ctr]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 14s675ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.441 INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
- ApplicationContextException: Unable to start ServletWebServerApplicationContext due to missing ServletWebServerFactory bean
- Autowired gives Null value in Custom Constraint validator
- Spring 6 TransactionManager Isolation Level Support in IBM Liberty
- how to debug spring application with gradle
- Can I set the property spring.cloud.function.definition programmatically?
- Spring Boot 3.4.0 Broken HTTP interface
- How to register spring 5 application to eureka client
- spring kafka No type information in headers and no default type provided
- Spring Integration and TCP server socket - how can I send a message to a client?
- Spring Boot doesn't use datasource properties
- Error after updating logback to version ^1.5.13
- Recurring AsyncRequestTimeoutException in Spring Boot Admin log
- No Dialect mapping for JDBC type: 1111
- spring boot An error happened during template parsing
- Trying to Dockerize Maven Application but got "release version 21 not supported"
- Jasypt out of maintenance? What to use for encryption with Spring Boot
- @Value not found from application.properties in main @SpringBootApplication class
- Hibernate doesn't creating Join Table on his own
- migrate Spring app from XML to annotations
- Session not persisted between Spring Boot (Spring Session + MySQL) and Angular frontend
- Spring Camel integration tests slow with Timeout message
- Where to place events in a Spring Modulith application?
- Springdoc: ClassNotFoundException: org.springframework.web.servlet.resource.LiteWebJarsResourceResolver
- Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured
- Java Date Time discrepancy
- Keycloak Account API returns Unauthorized 401 with token from user logged in via my spring client
- How exactly does the Spring BeanPostProcessor work?
- Spring Integration: Aggregator pattern - a way for downstream component to know if group is complete
- Upgrading to Spring Boot 3/ JDK 17/ Spring 6 - JAXB conflict issue IllegalAnnotationsException
- What's the difference between Hibernate and Spring Data JPA