Taier is a distributed visual DAG task scheduling system. In order to reduce the development cost of ETL and improve the stability of big data platform, big data developers can directly develop business logic in Taier, without worrying about the complex dependency of tasks and the implementation of the underlying big data platform architecture, and focus more on the business.

Function feature

Stability

Single point of failure: decentralized distributed mode
High availability mode: Zookeeper
Overload handling: distributed node + two-level storage strategy + queuing mechanism. Each node can handle task scheduling and submission; When a large number of tasks are performed, they are preferentially cached in the memory queue. When the number of tasks exceeds the configured maximum number of queues, all tasks are stored in the database. Task processing is consumed in a queue, which asynchronously fetches an executable instance from the database
Actual test: hundreds of enterprise customers production environment actual test

Ease of use

Support big data job scheduling Spark, Flink,
Supports many types of tasks, currently supports Spark SQL, data synchronization

Later open source: SparkMR, PySpark, FlinkMR, Python, Shell, Jupyter, Tersorflow, Pytorch, HadoopMR, Kylin, Odps,

SQL tasks (MySQL, PostgreSQL, Hive, Impala, Oracle, SQLServer, TiDB, greenplum, inceptor, kingbase, presto)
Visual workflow configuration: supports encapsulated workflows, supports single-task running, does not need to encapsulate workflows, supports drag-and-drop mode to draw DAG
DAG monitoring interface: operation and maintenance center, support to view cluster resources, understand the remaining situation of current cluster resources, support to batch stop tasks in the scheduling queue, task status, task type, retry times, task running machine, visual variables and other key information ata glance
Scheduling time configuration: visual configuration
Multi-cluster connection: Support a scheduling system to connect multiple Hadoop clusters

Multi-version engine

Support for multiple versions of Spark, Flink and other engines

Kerberos support

Spark
Flink

System parameters

Rich, supports 3 time benchmarks, and can flexibly set the output format

Extensibility

The design considers the distributed mode, and currently supports the overall Taier horizontal expansion mode
Scheduling capability increases linearly with cluster

Architecture Design

DatasourceX is a data source plug-in, responsible for metadata and data operations of various types of data sources, such as obtaining table structure, previewing table data and other functions are implemented by DatasourceX
Chunjun is a batch flow unified data synchronization tool based on Flink, which can collect both static data, such as MySQL, HDFS, etc., and real-time changing data, such as MySQL binlog, Kafka, etc.

A distributed visual DAG task scheduling system插图1

Main interface

A distributed visual DAG task scheduling system插图2

A distributed visual DAG task scheduling system插图3

A distributed visual DAG task scheduling system插图4

A distributed visual DAG task scheduling system插图5

A distributed visual DAG task scheduling system插图6

—END—

Open Source protocol: Apache2.0

资源下载此资源为免费资源立即下载

Telegram:@John_Software

collect(0) Like (0)

Disclaimer: This article is published by a third party and represents the views of the author only and has nothing to do with this website. This site does not make any guarantee or commitment to the authenticity, completeness and timeliness of this article and all or part of its content, please readers for reference only, and please verify the relevant content. The publication or republication of articles by this website for the purpose of conveying more information does not mean that it endorses its views or confirms its description, nor does it mean that this website is responsible for its authenticity.

Ictcoder Free source code A distributed visual DAG task scheduling system https://ictcoder.com/kyym/a-distributed-visual-dag-task-scheduling-system.html

lllll

Share free open-source source code

Previous article： Lapce: Lightning-fast and powerful code editor

Next article： Ghost is an open source professional publishing platform based on the modern Node.js technology stack

Q&A

What is the delivery method?

1, automatic: after taking the photo, click the (download) link to download; 2. Manual: After taking the photo, contact the seller to issue it or contact the official to find the developer to ship.

View details

How long is the trading cycle?

1, the default transaction cycle of the source code: manual delivery of goods for 1-3 days, and the user payment amount will enter the platform guarantee until the completion of the transaction or 3-7 days can be issued, in case of disputes indefinitely extend the collection amount until the dispute is resolved or refunded!

View details

Matters needing attention

1. Heptalon will permanently archive the process of trading between the two parties and the snapshots of the traded goods to ensure that the transaction is true, effective and safe! 2, Seven PAWS can not guarantee such as "permanent package update", "permanent technical support" and other similar transactions after the merchant commitment, please identify the buyer; 3, in the source code at the same time there is a website demonstration and picture demonstration, and the site is inconsistent with the diagram, the default according to the diagram as the dispute evaluation basis (except for special statements or agreement); 4, in the absence of "no legitimate basis for refund", the commodity written "once sold, no support for refund" and other similar statements, shall be deemed invalid; 5, before the shooting, the transaction content agreed by the two parties on QQ can also be the basis for dispute judgment (agreement and description of the conflict, the agreement shall prevail); 6, because the chat record can be used as the basis for dispute judgment, so when the two sides contact, only communicate with the other party on the QQ and mobile phone number left on the systemhere, in case the other party does not recognize self-commitment. 7, although the probability of disputes is very small, but be sure to retain such important information as chat records, mobile phone messages, etc., in case of disputes, it is convenient for seven PAWS to intervene in rapid processing.

View details

Systemhere declaration

1. As a third-party intermediary platform, Qichou protects the security of the transaction and the rights and interests of both buyers and sellers according to the transaction contract (commodity description, content agreed before the transaction); 2, non-platform online trading projects, any consequences have nothing to do with mutual site; No matter the seller for any reason to require offline transactions, please contact the management report.

View details