This issue recommends ElasticJob, a distributed scheduling solution for Internet ecology and massive tasks, which is composed of two independent subprojects, ElasticJob-Lite and ElasticJob-Cloud.

ElasticJob uses the functions of elastic scheduling, resource management, and job governance to create a distributed scheduling solution applicable to the Internet scenario, and provides a diversified job ecology through an open architecture design. Its products use a unified job API, developers only need to develop once, can be deployed at will.

ElasticJob became a subproject of Apache ShardingSphere on May 28, 2020.

Official website:
https://shardingsphere.apache.org/elasticjob/

project introduction

Using ElasticJob frees development engineers from worrying about non-functional requirements such as linear throughput increases for tasks, allowing them to focus more on business-oriented coding design. At the same time, it can also free up operations engineers, so that they no longer have to worry about the availability of tasks and related management requirements, only by easily adding service nodes can achieve the purpose of automated operations.

ElasticJob-Lite

Positioned as a lightweight, decentralized solution that provides coordination services for distributed tasks in the form of jars.

function list

Elastic scheduling

Task fragmentation and high availability in distributed scenarios The task throughput and execution efficiency can be expanded horizontally. The task processing capability varies flexibly with resource allocation

Resource allocation

Allocate the right resources to the task at the right time and make it effective
The same tasks are aggregated into the same actuator for unified processing
Dynamically allocate additional resources to newly assigned tasks

Operational governance

failover
The missed job was re-executed
Self-diagnostic repair

Job Dependence (TODO)

Interjob dependency based on directed acyclic graph (DAG)
Job-shard dependencies based on directed acyclic graph (DAG)

Operational open ecology

Extensible unified interface for job types
Rich library of job types such as data streams, scripts, HTTP, files, big data, etc
Easy to interface with business operations and seamless integration with Spring dependency injection

Visual control terminal

Job control terminal
Job execution history data tracking
Registry management

environmental requirement

Java

Use Java 8 or later.

Maven

Use Maven 3.5.0 or later.

ZooKeeper

Use ZooKeeper 3.6.0 or later.

Mesos (only used by ElasticJob-Cloud)

Use Mesos 1.1.0 and its compatible versions.

Quick start

ElasticJob-Lite

Introducing Maven dependencies

<dependency>    <groupId>org.apache.shardingsphere.elasticjob</groupId>    <artifactId>elasticjob-lite-core</artifactId>    <version>${latest.release.version}</version></dependency>

Job development

public class MyJob implements SimpleJob {        @Override    public void execute(ShardingContext context) {        switch (context.getShardingItem()) {            case 0:                 // do something by sharding item 0                break;            case 1:                 // do something by sharding item 1                break;            case 2:                 // do something by sharding item 2                break;            // case n: ...        }    }}

Job configuration

    JobConfiguration jobConfig = JobConfiguration.newBuilder("MyJob", 3).cron("0/5 * * * * ?").build();

job scheduling

public class MyJobDemo {        public static void main(String[] args) {        new ScheduleJobBootstrap(createRegistryCenter(), new MyJob(), createJobConfiguration()).schedule();    }        private static CoordinatorRegistryCenter createRegistryCenter() {        CoordinatorRegistryCenter regCenter = new ZookeeperRegistryCenter(new ZookeeperConfiguration("zk_host:2181", "my-job"));        regCenter.init();        return regCenter;    }        private static JobConfiguration createJobConfiguration() {        // 创建作业配置        // ...    }}

ElasticJob-Cloud

Introducing Maven dependencies

<dependency>    <groupId>org.apache.shardingsphere.elasticjob</groupId>    <artifactId>elasticjob-cloud-executor</artifactId>    <version>${latest.release.version}</version></dependency>

Job development

public class MyJob implements SimpleJob {        @Override    public void execute(ShardingContext context) {        switch (context.getShardingItem()) {            case 0:                 // do something by sharding item 0                break;            case 1:                 // do something by sharding item 1                break;            case 2:                 // do something by sharding item 2                break;            // case n: ...        }    }}

Job initiation

public class MyJobDemo {        public static void main(final String[] args) {        JobBootstrap.execute(new MyJob());    }}

Job packaging

tar -cvf my-job.tar.gz my-job