ETL/ELT pipelines
With the help of this add-on module large amounts of data can be moved (and transformed) with high performance, e.g. from database to database, CSV to database or database to CSV. The processing of extremely large data sets can also be outsourced to another server to conserve the resources of the main system.
Examples
Once you have read through this page here, it is best to start directly with the examples. This is the easiest way to get started without getting lost in the details of this powerful module. There we start with the simplest use cases and step by step work our way through the more sophisticated functionalities. You can then read up on detailed settings later.
Overview
In the overview you will find all created ETL/ELT pipelines.
Before you can create an ETL/ELT pipeline, you must create a new group using the context menu. Within a group, you can then create, duplicate, view, view in the "Data Flow" module, edit, delete, import/export, and run pipelines or start a test run.
Settings
(1) Plus icon: Creates a new pipeline.
(2) Context menu of an ETL/ELT pipeline (self explanatory options are not listed):
|
Compare (Diff) with |
Compares the ETL/ELT pipeline configuration with another ETL/ELT pipeline configuration (you can also compare with an imported ETL/ELT pipeline) . As a result you get a diff file with the differences. |
|
Quick view |
Provides the most important information about the ETL/ELT pipeline (source data, destination data), a diagram view and an XML debug file (provides information about the execution of the actions of the ETL/ELT tasks). |
|
Open in new window |
The pipeline will be opened in another browser tab. Attention: Your browser may block this as an unauthorised popup. Check your browser settings if necessary. |
|
In use by |
Shows where the ETL/ELT pipeline is used, e.g. in a workflow. |
|
Export/Import entry |
The format is a signed XML, so it is no longer possible to import the file if it is changed. Note: Old " .obj" files can still be imported. Note: We also provides a Transport Manager . |
(3) Expert search: See section "Expert search".
(4) Display: Two views organised in groups and a tile view (without groups). The tile view displays images that have been assigned in the respective entries.
(5) Selection on the right top: The selected client.
ETL/ELT pipeline
The following tabs are available for configuring an ETL/ELT pipeline.
Main settings
Variables
ETL/ELT tasks
Cron settings
Logs and running jobs
Logs of completed ETL/ELT pipelines can be found on page ETL/ELT logs.
Running ETL/ELT pipelines are displayed on page Active ETL/ELT.
External and automatic access to ETL/ELT pipelines
ETL/ELT pipelines can, in addition to being started manually, also be started via cron job, via ETL/ELT functions in a profile, via REST or from the Workflow module.