ETL/ELT Pipelines
With the help of this add-on module large amounts of data can be moved (and transformed) with high performance, e.g. from database to database, CSV to database or database to CSV. The processing of extremely large data sets can also be outsourced to another server to conserve the resources of the main system.
Examples
Once you have read through this page here, it is best to start directly with the examples. This is the easiest way to get started without getting lost in the details of this powerful module. There we start with the simplest use cases and step by step work our way through the more sophisticated functionalities. You can then read up on detailed settings later.
User Rights
To use the module, the Lobster_data user must have the user rights ETL/ELT and ETL/ELT administration.
Overview
In the overview you will find all created ETL/ELT pipelines.
Before you can create an ETL/ELT pipeline, you must create a new group using the context menu. Within a group, you can then create, duplicate, view, view in the Data Flow module, edit, delete, import/export, and run pipelines or start a test run.
(1) Creates a new pipeline.
(2) Context menu of an ETL/ELT pipeline (self explanatory options are not listed).
|
Compare (Diff) with |
Compares the ETL/ELT pipeline with another ETL/ELT pipeline (you can also compare with an imported ETL/ELT pipeline) . As a result you get a diff file with the differences. |
|
Quick view |
Provides the most important information about the ETL/ELT pipeline (source data, destination data), a diagram view and an XML debug file (provides information about the execution of the actions of the ETL/ELT tasks). |
|
Open in new window |
The pipeline will be opened in another browser tab. Attention: Your browser may block this as an unauthorised popup. Check your browser settings if necessary. |
|
In use by |
Shows where the ETL/ELT pipeline is used, e.g. in a workflow. |
|
Export/Import entry |
The format is a signed XML, so it is no longer possible to import the file if it is changed. Note: Old .obj files can still be imported. Note: Lobster_data also provides the Transport Manager , which allows an online transport of profiles without the intermediate step via files. However, the versions of both systems must be within the same major release, e.g. 4.0.0 and 4.1.0. Versions 3.5.4 and 4.0.0 would not work. |
(3) Expert search.
(4) Two views organized in groups and a tile view (without groups). The tile view displays images that have been assigned in the respective entries.
ETL/ELT Pipeline
The following tabs are available for configuring an ETL/ELT pipeline.
Logs and Running Jobs
Logs of completed ETL/ELT pipelines can be found on page ETL/ELT Logs.
Running ETL/ELT pipelines are displayed on page Active ETL/ELT.
External and Automatic Access to ETL/ELT Pipelines
ETL/ELT pipelines can, in addition to being started manually, also be started via cron job, via ETL/ELT functions in a profile, via REST or from the Workflow module.