ETL/ELT
ETL stands for Extract (of data from different sources), Transform (of data into the format of the target system) and Load (loading of data into the target system) and ELT accordingly for Extract, Load and Transform.
With the help of this add-on module large amounts of data can be moved (and transformed), e.g. from database to database, CSV to database or database to CSV. A great advantage is, for example, the reading and processing of extremely large CSV files, which otherwise cannot be completely read in the memory. The processing of extremely large data sets can also be outsourced to another server to conserve the resources of the main system.
This module is currently only available as a beta version. The following documentation is only a rough overview of the module. At a later date we will provide a book on the subject and also expand this documentation here.
User Rights
To use the ETL/ELT module, the Lobster_data user must have the user rights ETL/ELT and ETL/ELT administration.
Overview
In the overview you will find all created ETL/ELT entries.
Before you can create an ETL/ELT entry, you must create a new group using the context menu. Within a group, you can then create, duplicate, view, view in the Data Flow module, edit, delete, import/export, and run entries or start a test run.
(1) Creates a new entry.
(2) Compares the ETL/ELT entry with another ETL/ELT entry. As a result you get a diff file with the differences.
(3) Expert search.
(4) Provides the most important information about the ETL/ELT entry (source data, target data), a diagram view and an XML debug file (provides information about the execution of the actions of the ETL/ELT tasks).
(5) The entry will be opened in another browser tab. Attention: Your browser may block this as an unauthorised popup. Check your browser settings if necessary.
ETL/ELT Entry
The following tabs are available for configuring an ETL/ELT entry.
External and Automatic Access to ETL/ELT Entries
ETL/ELT entries can also be started via cron job, via ETL/ELT functions in a profile, via REST or from the Workflow module.