Quartz Spring Batch Example
Through this article, we are going to show you how to run Spring Batch using Quartz. Spring Batch and Quartz have different goals. Spring Batch provides functionality for processing large volumes of data and Quartz provides functionality for scheduling tasks. So Quartz could complement Spring Batch, a common combination would be to use Quartz as a trigger for a Spring Batch job using a Cron expression and the Spring Core convenience SchedulerFactoryBean
.
Quartz has three main components: a scheduler
, a job
, and a trigger
. A scheduler, which is obtained from a SchedulerFactory
, serves as a registry of JobDetails
(a reference to a Quartz job) and triggers
and it is responsible for executing a job when its associated trigger fires. A job is a unit of work that can be executed. A trigger defines when a job is to be run. When a trigger fires, telling Quartz to execute a job, a JobDetails
object is created to define the individual execution of the job.
In order to integrate Quartz with your Spring Batch process, you need to do the following:
- Add the required dependencies to your
pom.xml
file. - Write your own Quartz job to launch your job using Spring’s
QuartzJobBean
. - Configure a
JobDetailBean
provided by Spring to create a Quartz JobDetail. - Configure a
trigger
to define when your job should run.
To show how Quartz can be used to periodically execute a Spring Batch job, let’s use our job of the previous example Spring Batch ETL Job which calculates the financial stock market OHLC data, By adding Quartz scheduling feature to this job, it will automatically be executed every day when the daily trading session is finished.
1. Project Environment
- Spring Boot 1.3.3.RELEASE
- Apache Maven 3.0.5
- Quartz 2.2.3
- JDK 1.8
- Eclipse 4.4 (Luna)
2. Project Structure
3. Dependencies
We added additional required dependencies to our POM file. In this example, there are three new dependencies. The first is the Quartz framework itself. The second dependency we add, is for the spring-context-support
artifact. This package from Spring, provides the classes required to integrate Quartz easily with Spring.
pom.xml:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.quantvalley.examples</groupId> <artifactId>quartz-spring-batch-example</artifactId> <version>0.1.0</version> <name>Quartz Spring Batch Example</name> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.3.3.RELEASE</version> </parent> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-batch</artifactId> </dependency> <!-- Includes spring's support classes for quartz --> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-context-support</artifactId> </dependency> <dependency> <groupId>org.quartz-scheduler</groupId> <artifactId>quartz</artifactId> <version>2.2.3</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
4. Quartz Batch Java Configuration
We use the same Spring Batch ETL Job configuration while we add more Quartz specific configuration through the QuartzConfiguration.java
class where we defined our Quartz configuration, then we will import this Quartz configuration into BatchConfiguration.java
using @import
annotation.
QuartzConfiguration.java:
package com.quantvalley.batch.quartz; import java.util.HashMap; import java.util.Map; import org.springframework.batch.core.configuration.JobLocator; import org.springframework.batch.core.configuration.JobRegistry; import org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor; import org.springframework.batch.core.launch.JobLauncher; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.scheduling.quartz.CronTriggerFactoryBean; import org.springframework.scheduling.quartz.JobDetailFactoryBean; import org.springframework.scheduling.quartz.SchedulerFactoryBean; /** * The Class QuartzConfiguration. * * @author ashraf */ @Configuration public class QuartzConfiguration { @Autowired private JobLauncher jobLauncher; @Autowired private JobLocator jobLocator; @Bean public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) { JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor(); jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry); return jobRegistryBeanPostProcessor; } @Bean public JobDetailFactoryBean jobDetailFactoryBean() { JobDetailFactoryBean factory = new JobDetailFactoryBean(); factory.setJobClass(QuartzJobLauncher.class); Map map = new HashMap(); map.put("jobName", "fxmarket_prices_etl_job"); map.put("jobLauncher", jobLauncher); map.put("jobLocator", jobLocator); factory.setJobDataAsMap(map); factory.setGroup("etl_group"); factory.setName("etl_job"); return factory; } // Job is scheduled after every 2 minute @Bean public CronTriggerFactoryBean cronTriggerFactoryBean() { CronTriggerFactoryBean stFactory = new CronTriggerFactoryBean(); stFactory.setJobDetail(jobDetailFactoryBean().getObject()); stFactory.setStartDelay(3000); stFactory.setName("cron_trigger"); stFactory.setGroup("cron_group"); stFactory.setCronExpression("0 0/2 * 1/1 * ? *"); return stFactory; } @Bean public SchedulerFactoryBean schedulerFactoryBean() { SchedulerFactoryBean scheduler = new SchedulerFactoryBean(); scheduler.setTriggers(cronTriggerFactoryBean().getObject()); return scheduler; } }
4.1. JobRegistryBeanPostProcessor
A BeanPostProcessor
that registers Job
beans with a JobRegistry
. Include a bean of this type along with your job configuration, and use the same JobRegistry
as a JobLocator
when you need to locate a Job
to launch.
@Bean public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) { JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor(); jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry); return jobRegistryBeanPostProcessor; }
4.2. JobDetailFactoryBean
Spring provides JobDetailFactoryBean
that uses Quartz JobDetail
. We use it to configure complex job such as job scheduling using CRON expression. Create job implementing QuartzJobBean
interface and configure to JobDetailFactoryBean
. We also configure job name and group name. To pass the parameter to job, it provides setJobDataAsMap(Map<String,?> jobDataAsMap)
method.
@Bean public JobDetailFactoryBean jobDetailFactoryBean() { JobDetailFactoryBean jobfactory = new JobDetailFactoryBean(); jobfactory.setJobClass(QuartzJobLauncher.class); Map<String, Object> map = new HashMap<String, Object>(); map.put("jobName", "fxmarket_prices_etl_job"); map.put("jobLauncher", jobLauncher); map.put("jobLocator", jobLocator); jobfactory.setJobDataAsMap(map); jobfactory.setGroup("etl_group"); jobfactory.setName("etl_job"); return jobfactory; }
4.3. CronTriggerFactoryBean
Spring provides CronTriggerFactoryBean
that uses Quartz CronTrigger
. CronTriggerFactoryBean
configures JobDetailFactoryBean
. We also configure start delay, trigger name, group name and CRON expression to schedule the job.
@Bean public CronTriggerFactoryBean cronTriggerFactoryBean() { CronTriggerFactoryBean ctFactory = new CronTriggerFactoryBean(); ctFactory.setJobDetail(jobDetailFactoryBean().getObject()); ctFactory.setStartDelay(3000); ctFactory.setName("cron_trigger"); ctFactory.setGroup("cron_group"); ctFactory.setCronExpression("0 0/2 * 1/1 * ? *"); return ctFactory; }
4.4. SchedulerFactoryBean
Spring provides SchedulerFactoryBean
that uses Quartz Scheduler
. Using SchedulerFactoryBean
we register all the triggers. In our case we have CronTriggerFactoryBeantrigger
that is being registered.
@Bean public SchedulerFactoryBean schedulerFactoryBean() { SchedulerFactoryBean scheduler = new SchedulerFactoryBean(); scheduler.setTriggers(cronTriggerFactoryBean().getObject()); return scheduler; }
Tip
- By default, Spring boot’s autoconfiguration service will run all configured job beans after application start. Setting
spring.batch.job.enabled
to false in theapplication.properties
prevents the launching of all jobs - After the job first run, Spring Batch will throws a
JobInstanceAlreadyCompleteException
which says job instance already exists, To avoid that, set stepallowStartIfComplete(boolean allowStartIfComplete)
totrue
.
4.5. QuartzJobBean
QuartzJobLauncher.java
is a single class that extends Spring’s QuartzJobBean
. This implementation of Quartz’s Job
interface is a helpful class that allows you to implement only the pieces of logic that pertain to your work, leaving the manipulation of the scheduler and so on to Spring. In this case, we override the executeInternal(org.quartz.JobExecutionContext context)
method from which to execute the job. In this case, we want to reference one parameter: the name of the job. With the name of the job obtained, you use the JobLocator
to retrieve the Spring Batch job from the JobRegistry
. Once that is complete, we can execute the job using the JobLauncher
.
QuartzJobLauncher.java:
package com.quantvalley.batch.quartz; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.batch.core.Job; import org.springframework.batch.core.JobExecution; import org.springframework.batch.core.JobParameters; import org.springframework.batch.core.configuration.JobLocator; import org.springframework.batch.core.launch.JobLauncher; import org.springframework.scheduling.quartz.QuartzJobBean; /** * The Class QuartzJobLauncher. * * @author ashraf */ public class QuartzJobLauncher extends QuartzJobBean { private static final Logger log = LoggerFactory.getLogger(QuartzJobLauncher.class); private String jobName; private JobLauncher jobLauncher; private JobLocator jobLocator; public String getJobName() { return jobName; } public void setJobName(String jobName) { this.jobName = jobName; } public JobLauncher getJobLauncher() { return jobLauncher; } public void setJobLauncher(JobLauncher jobLauncher) { this.jobLauncher = jobLauncher; } public JobLocator getJobLocator() { return jobLocator; } public void setJobLocator(JobLocator jobLocator) { this.jobLocator = jobLocator; } @Override protected void executeInternal(JobExecutionContext context) throws JobExecutionException { try { Job job = jobLocator.getJob(jobName); JobExecution jobExecution = jobLauncher.run(job, new JobParameters()); log.info("{}_{} was completed successfully", job.getName(), jobExecution.getId()); } catch (Exception e) { log.error("Encountered job execution exception!"); } } }
5. Running Quartz Batch Job
Application.java
is our main class to our Quartz Batch Job.
Application.java:
package com.quantvalley.batch; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; /** * The Class Application. * * @author ashraf */ @SpringBootApplication public class Application { public static void main(String[] args) throws Exception { SpringApplication.run(Application.class, args); } }
Output:
. ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v1.3.3.RELEASE) 2016-07-23 14:53:05.671 INFO 4347 --- [ main] com.quantvalley.batch.Application : Starting Application on HP-ProBook with PID 4347 (started by ashraf in /home/ashraf/me/jcg/examples/Quartz Spring Batch Example/quartz-spring-batch-example) 2016-07-23 14:53:05.673 INFO 4347 --- [ main] com.quantvalley.batch.Application : No active profile set, falling back to default profiles: default 2016-07-23 14:53:05.757 INFO 4347 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@4e41089d: startup date [Sat Jul 23 14:53:05 EET 2016]; root of context hierarchy 2016-07-23 14:53:07.012 WARN 4347 --- [ main] o.s.c.a.ConfigurationClassEnhancer : @Bean method ScopeConfiguration.stepScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean javadoc for complete details. 2016-07-23 14:53:07.023 WARN 4347 --- [ main] o.s.c.a.ConfigurationClassEnhancer : @Bean method ScopeConfiguration.jobScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean javadoc for complete details. 2016-07-23 14:53:07.135 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.datasource.CONFIGURATION_PROPERTIES' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.141 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$EnhancerBySpringCGLIB$a23a1ff] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.164 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.EmbeddedDataSourceConfiguration' of type [class org.springframework.boot.autoconfigure.jdbc.EmbeddedDataSourceConfiguration$EnhancerBySpringCGLIB$bdf9d4e4] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.213 INFO 4347 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:hsqldb:mem:testdb', username='sa' 2016-07-23 14:53:07.727 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSource' of type [class org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory$EmbeddedDataSourceProxy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.730 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration$DataSourceInitializerConfiguration' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration$DataSourceInitializerConfiguration$EnhancerBySpringCGLIB$64e7b346] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.738 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceInitializer' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.742 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration' of type [class org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$EnhancerBySpringCGLIB$cc9327a5] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.783 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'jobLauncher' of type [class com.sun.proxy.$Proxy33] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.789 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'jobRegistry' of type [class com.sun.proxy.$Proxy35] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:07.790 INFO 4347 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'quartzConfiguration' of type [class com.quantvalley.batch.quartz.QuartzConfiguration$EnhancerBySpringCGLIB$2f4a7e79] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2016-07-23 14:53:08.141 INFO 4347 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor 2016-07-23 14:53:08.223 INFO 4347 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2016-07-23 14:53:08.223 INFO 4347 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.2.3 created. 2016-07-23 14:53:08.224 INFO 4347 --- [ main] org.quartz.simpl.RAMJobStore : RAMJobStore initialized. 2016-07-23 14:53:08.225 INFO 4347 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.2.3) 'schedulerFactoryBean' with instanceId 'NON_CLUSTERED' Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads. Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered. 2016-07-23 14:53:08.225 INFO 4347 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'schedulerFactoryBean' initialized from an externally provided properties instance. 2016-07-23 14:53:08.226 INFO 4347 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler version: 2.2.3 2016-07-23 14:53:08.227 INFO 4347 --- [ main] org.quartz.core.QuartzScheduler : JobFactory set to: org.springframework.scheduling.quartz.AdaptableJobFactory@478ee483 2016-07-23 14:53:08.427 INFO 4347 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: HSQL 2016-07-23 14:53:09.120 INFO 4347 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor. 2016-07-23 14:53:09.278 INFO 4347 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] 2016-07-23 14:53:09.286 INFO 4347 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 8 ms. 2016-07-23 14:53:09.338 INFO 4347 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup 2016-07-23 14:53:09.345 INFO 4347 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647 2016-07-23 14:53:09.345 INFO 4347 --- [ main] o.s.s.quartz.SchedulerFactoryBean : Starting Quartz Scheduler now 2016-07-23 14:53:09.346 INFO 4347 --- [ main] org.quartz.core.QuartzScheduler : Scheduler schedulerFactoryBean_$_NON_CLUSTERED started. 2016-07-23 14:53:09.356 INFO 4347 --- [ main] com.quantvalley.batch.Application : Started Application in 4.455 seconds (JVM running for 6.628) 2016-07-23 14:54:00.081 INFO 4347 --- [ryBean_Worker-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=fxmarket_prices_etl_job]] launched with the following parameters: [{}] 2016-07-23 14:54:00.098 INFO 4347 --- [ryBean_Worker-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [Extract -> Transform -> Aggregate -> Load] 2016-07-23 14:54:08.603 INFO 4347 --- [ryBean_Worker-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=fxmarket_prices_etl_job]] completed with the following parameters: [{}] and the following status: [COMPLETED] 2016-07-23 14:54:08.603 INFO 4347 --- [ryBean_Worker-1] c.q.batch.quartz.QuartzJobLauncher : fxmarket_prices_etl_job_0 was completed successfully 2016-07-23 14:56:00.028 INFO 4347 --- [ryBean_Worker-2] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=fxmarket_prices_etl_job]] launched with the following parameters: [{}] 2016-07-23 14:56:00.039 INFO 4347 --- [ryBean_Worker-2] o.s.batch.core.job.SimpleStepHandler : Executing step: [Extract -> Transform -> Aggregate -> Load] 2016-07-23 14:56:07.436 INFO 4347 --- [ryBean_Worker-2] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=fxmarket_prices_etl_job]] completed with the following parameters: [{}] and the following status: [COMPLETED] 2016-07-23 14:56:07.436 INFO 4347 --- [ryBean_Worker-2] c.q.batch.quartz.QuartzJobLauncher : fxmarket_prices_etl_job_1 was completed successfully 2016-07-23 14:58:00.007 INFO 4347 --- [ryBean_Worker-3] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=fxmarket_prices_etl_job]] launched with the following parameters: [{}] 2016-07-23 14:58:00.020 INFO 4347 --- [ryBean_Worker-3] o.s.batch.core.job.SimpleStepHandler : Executing step: [Extract -> Transform -> Aggregate -> Load] 2016-07-23 14:58:07.516 INFO 4347 --- [ryBean_Worker-3] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=fxmarket_prices_etl_job]] completed with the following parameters: [{}] and the following status: [COMPLETED] 2016-07-23 14:58:07.516 INFO 4347 --- [ryBean_Worker-3] c.q.batch.quartz.QuartzJobLauncher : fxmarket_prices_etl_job_2 was completed successfully
6. Download the Source Code
This was an example to show how to integrate Quartz with Spring Batch.
You can download the full source code of this example here: QuartzSpringBatchExampleCode.zip
I have tried this out. The only problem I see is that, after I gave a cron expression 0 09 22 ? * *, the job starts at 22:09, but the launcher executes every 5 minutes after that. I wanted this to run every day at 22:09. What am I doing wrong?
Itemreader postconstuct metod is called only once in execution. I want to call it and generate the list every quartz iterate. How can I achive this. Thank you
doesn’t work, keep getting this error:
Description:
Field jobLauncher in org.springbatch.examples.quartz.QuartzConfiguration required a bean of type ‘org.springframework.batch.core.launch.JobLauncher’ that could not be found.
The injection point has the following annotations:
– @org.springframework.beans.factory.annotation.Autowired(required=true)