Taier is a distributed visual DAG task scheduling system. In order to reduce the development cost of ETL and improve the stability of big data platform, big data developers can directly develop business logic in Taier, without worrying about the complex dependency of tasks and the implementation of the underlying big data platform architecture, and focus more on the business.

Function feature

Stability

Single point of failure: decentralized distributed mode
High availability mode: Zookeeper
Overload handling: distributed node + two-level storage strategy + queuing mechanism. Each node can handle task scheduling and submission; When a large number of tasks are performed, they are preferentially cached in the memory queue. When the number of tasks exceeds the configured maximum number of queues, all tasks are stored in the database. Task processing is consumed in a queue, which asynchronously fetches an executable instance from the database
Actual test: hundreds of enterprise customers production environment actual test

Ease of use

Support big data job scheduling Spark, Flink,
Supports many types of tasks, currently supports Spark SQL, data synchronization

Later open source: SparkMR, PySpark, FlinkMR, Python, Shell, Jupyter, Tersorflow, Pytorch, HadoopMR, Kylin, Odps,

SQL tasks (MySQL, PostgreSQL, Hive, Impala, Oracle, SQLServer, TiDB, greenplum, inceptor, kingbase, presto)
Visual workflow configuration: supports encapsulated workflows, supports single-task running, does not need to encapsulate workflows, supports drag-and-drop mode to draw DAG
DAG monitoring interface: operation and maintenance center, support to view cluster resources, understand the remaining situation of current cluster resources, support to batch stop tasks in the scheduling queue, task status, task type, retry times, task running machine, visual variables and other key information ata glance
Scheduling time configuration: visual configuration
Multi-cluster connection: Support a scheduling system to connect multiple Hadoop clusters

Multi-version engine

Support for multiple versions of Spark, Flink and other engines

Kerberos support

Spark
Flink

System parameters

Rich, supports 3 time benchmarks, and can flexibly set the output format

Extensibility

The design considers the distributed mode, and currently supports the overall Taier horizontal expansion mode
Scheduling capability increases linearly with cluster

Architecture Design

DatasourceX is a data source plug-in, responsible for metadata and data operations of various types of data sources, such as obtaining table structure, previewing table data and other functions are implemented by DatasourceX
Chunjun is a batch flow unified data synchronization tool based on Flink, which can collect both static data, such as MySQL, HDFS, etc., and real-time changing data, such as MySQL binlog, Kafka, etc.

A distributed visual DAG task scheduling system插图1

Main interface

A distributed visual DAG task scheduling system插图2

A distributed visual DAG task scheduling system插图3

A distributed visual DAG task scheduling system插图4

A distributed visual DAG task scheduling system插图5

A distributed visual DAG task scheduling system插图6

—END—

Open Source protocol: Apache2.0

资源下载此资源为免费资源立即下载

Telegram:@John_Software

collect(0) Like (0)

Disclaimer: This article is published by a third party and represents the views of the author only and has nothing to do with this website. This site does not make any guarantee or commitment to the authenticity, completeness and timeliness of this article and all or part of its content, please readers for reference only, and please verify the relevant content. The publication or republication of articles by this website for the purpose of conveying more information does not mean that it endorses its views or confirms its description, nor does it mean that this website is responsible for its authenticity.

Ictcoder Free Source Code A distributed visual DAG task scheduling system https://ictcoder.com/a-distributed-visual-dag-task-scheduling-system/

lllll

Share free open-source source code

Previous article： Lapce: Lightning-fast and powerful code editor

Next article： Ghost is an open source professional publishing platform based on the modern Node.js technology stack

Q&A

Ordering Process

1. Automatic: After making an online payment, click the (Download) link to download the source code; 2. Manual: Contact the seller or the official to check if the template is consistent. Then, place an order and make payment online. The seller ships the goods, and both parties inspect and confirm that there are no issues. ICTcoder will then settle the payment for the seller. Note: Please ensure to place your order and make payment through ICTcoder. If you do not place your order and make payment through ICTcoder, and the seller sends fake source code or encounters any issues, ICTcoder will not assist in resolving them, nor can we guarantee your funds!

View details

Transactions and Disputes

1. Default transaction cycle for source code: The seller manually ships the goods within 1-3 days. The amount paid by the user will be held in escrow by ICTcoder until 7 days after the transaction is completed and both parties confirm that there are no issues. ICTcoder will then settle with the seller. In case of any disputes, ICTcoder will have staff to assist in handling until the dispute is resolved or a refund is made! If the buyer places an order and makes payment not through ICTcoder, any issues and disputes have nothing to do with ICTcoder, and ICTcoder will not be responsible for any liabilities!

View details

Matters needing attention

1. ICTcoder will permanently archive the transaction process between both parties and snapshots of the traded goods to ensure the authenticity, validity, and security of the transaction! 2. ICTcoder cannot guarantee services such as "permanent package updates" and "permanent technical support" after the merchant's commitment. Buyers are advised to identify these services on their own. If necessary, they can contact ICTcoder for assistance; 3. When both website demonstration and image demonstration exist in the source code, and the text descriptions of the website and images are inconsistent, the text description of the image shall prevail as the basis for dispute resolution (excluding special statements or agreements); 4. If there is no statement such as "no legal basis for refund" or similar content, any indication on the product that "once sold, no refunds will be supported" or other similar declarations shall be deemed invalid; 5. Before the buyer places an order and makes payment, the transaction details agreed upon by both parties via WhatsApp or email can also serve as the basis for dispute resolution (in case of any inconsistency between the agreement and the description of the conflict, the agreement shall prevail); 6. Since chat records and email records can serve as the basis for dispute resolution, both parties should only communicate with each other through the contact information left on the system when contacting each other, in order to prevent the other party from denying their own commitments. 7. Although the probability of disputes is low, it is essential to retain important information such as chat records, text messages, and email records, in case a dispute arises, so that ICTcoder can intervene quickly.

View details

ICTcoder reminds and declares

1. As a third-party intermediary platform, ICTcoder solely protects transaction security and the rights and interests of both buyers and sellers based on the transaction contract (product description, agreed content before the transaction); 2. For online trading projects not on the ICTcoder platform, any consequences are unrelated to this platform; regardless of the reason why the seller requests an offline transaction, please contact the administrator to report.

View details